| David Tseng | dbb358b | 2021-11-24 00:01:11 | [diff] [blame] | 1 | # How accessibility works |
| 2 | |
| 3 | ## Overview |
| 4 | In modern operating systems, accessibility can be thought of as the set of |
| 5 | features which allow for alternative interactions with the system. In practice, |
| 6 | it is all about letting users work in the best way they can, regardless of |
| 7 | ability or disability. Since users have almost an unbounded set of needs, |
| 8 | accessibility has to constantly keep up and adapt to include increasing numbers |
| 9 | of alternate interactions. It is this very challenge which gives accessibility |
| 10 | at the level of an operating system such complexity, depth, and interest. |
| 11 | |
| 12 | For Chrome OS, this document will provide the reader with a technical survey of |
| 13 | the major system components to give a good foundation for additional |
| 14 | exploration. |
| 15 | |
| David Tseng | 8057e1b | 2021-12-21 16:39:30 | [diff] [blame] | 16 |  |
| 18 | |
| David Tseng | dbb358b | 2021-11-24 00:01:11 | [diff] [blame] | 19 | At a glance, Chrome OS accessibility roughly decomposes into three major |
| 20 | entities, separated by large abstractions like process boundaries or api layers. |
| 21 | |
| 22 | Through a slightly different lens, the three entities can also be thought of as the three major stakeholders in any accessibility interaction: |
| 23 | - accessibility feature: an assitive technology (AT) developer can produce services targeting a specific audience. For example, screen readers such as ChromeVox. The extension system serves as a rich runtime with apis to transform the user interface into something a user can use. |
| 24 | - frameworks: an operating system developer abstracts away low-level details to provide for a clean AT developer experience. In Chrome OS, this framework mainly comes by way of extension apis, though other internal api surfaces exist in C++ for Chromium developers as well. The apis are backed by libraries and executables run in native or sandboxed contexts. |
| 25 | - renderers: app developers and web page authors have their content executed and drawn on-screen. This is where all user interactions such as key events end up. The developers and authors here also expose accessibility information for an accessibility tree. |
| 26 | |
| 27 | ### Renderers |
| 28 | A renderer as used by this document, is any component that draws to the |
| 29 | screen. Some examples include Blink (for web content), ARC++ (for Android apps). |
| 30 | |
| 31 | Each of these renderers store an in-memory representation of their user |
| 32 | interfaces. For Blink, this is the DOM or the render tree. For Android, this is |
| 33 | the application view tree. |
| 34 | |
| 35 | In almost all modern renderers, there is an equivalent parallel structure, built |
| 36 | off of the source of truth UI tree, called the accessibility tree. It is here |
| 37 | where Chrome OS accessibility extracts all it needs. |
| 38 | |
| 39 | The accessibility tree is constructed and updated as the user interface changes |
| 40 | in any way. Every single text change, scroll, load, and more gets |
| 41 | represented. Likewise, every type of data provided by either a developer or the |
| 42 | underlying framework is consumed and stored. This includes things like the |
| 43 | rendered text, styling information, bounding boxes, tree relationships, and much |
| 44 | more. |
| 45 | |
| 46 | When this construction finishes or updates with new content, the renderer will |
| 47 | serialize the tree and prepare it for export out of process if needed. That |
| 48 | moves the data to the framework entity below. |
| 49 | |
| 50 | For the web, PDF, and Android, renderers exist out of process. For the Aura |
| 51 | windowing system, along with its Views toolkit, the accessible tree lives in the |
| 52 | Ash process. |
| 53 | |
| 54 | #### Deep dives |
| 55 | - [Chrome web accessibility](../browser/how_a11y_works.md) |
| 56 | |
| 57 | ### Frameworks |
| 58 | Here are a few key api surfaces on which accessibility depends. |
| 59 | |
| 60 | #### Public apis |
| 61 | - [chrome.automation](https://developer.chrome.com/docs/extensions/reference/automation/) |
| 62 | - [chrome.tts](https://developer.chrome.com/docs/extensions/reference/tts/) |
| 63 | - [chrome.ttsEngine](https://developer.chrome.com/docs/extensions/reference/ttsEngine/) |
| 64 | |
| 65 | #### Private apis (only available in component accessibility extensions) |
| 66 | - [chrome.accessibilityPrivate](https://source.chromium.org/chromium/chromium/src/+/main:chrome/common/extensions/api/accessibility_private.json) |
| 67 | - [chrome.brailleDisplayPrivate](https://source.chromium.org/chromium/chromium/src/+/main:chrome/common/extensions/api/braille_display_private.idl) |
| 68 | - [chrome.speechRecognitionPrivate](https://source.chromium.org/chromium/chromium/src/+/main:chrome/common/extensions/api/speech_recognition_private.idl) |
| 69 | |
| 70 | #### Libraries and system executables |
| 71 | - [BRLTTY](https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/main/app-accessibility/brltty/) |
| 72 | - [Chrome OS text-to-speeech](https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/main/app-accessibility/googletts/) |
| 73 | - [Espeak text-to-speech](https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/main/app-accessibility/espeak-ng/) |
| 74 | |
| 75 | #### Deep dives |
| 76 | - [How text-to-speech works on Chrome OS (coming soon)](how_tts_works.md). |
| 77 | |
| 78 | ### Accessibility features |
| 79 | See [How accessibility features work (coming soon)](how_a11y_features_work.md) for more details. |
