Google Git
Sign in
chromium / chromium / src.git / main / . / docs / accessibility / browser / ia2_to_uia.md
blob: 2d96adcc0ced5a8d863397fcea723d54c5f9f2a8 [file] [log] [blame] [view]
Victor Feif535fe972020-10-02 06:28:25[diff] [blame]1# Runtime two way IAccessible2 to UI Automation elements look up via unique id
2Assistive technologies (ATs) who currently rely on IAccessible2 (IA2) that want
3to take advantage of the UI Automation (UIA) features at runtime can convert an
4IA2 element to an UIA element via a unique id and directly access UIA's API.
5This enables ATs who want to gradually transition from IA2 to UIA to experiment
6with individual UIA elements at runtime without switching entirely to UIA.
7
8
9To look up an UIA element through a unique id, an AT can utilize
10`IUIAutomationItemContainerPattern::FindItemByProperty()` with the custom UIA
11unique id property and the element's unique id as parameters.
12To look up an IA2 element from an UIA element, an AT can simply
13utilize IUIAutomationLegacyIAccessiblePattern::GetIAccessible() and
14then query for IAccessible2 interface. The unique id is not needed to
15look up the IA2 element from UIA element.
16
17## Convert an IA2 element to UIA element via unique id
18An IA2 element can be converted to an UIA element at runtime via a unique id
19that is shared between the two APIs.
20
21*Note: For the purpose of brevity and clarity, the code snippets below do not
22include clean-up of COM references neither does it have error handling.*
23
24~~~c++
25 #include <uiautomation.h>
26 #include <uiautomationclient.h>
27
28 // Consider the following HTML:
29 // <html>
30 // <button>button</button>
31 // </html>
32
33 // Register custom UIA property for retrieving the unique id of IA2 object.
34 // {cc7eeb32-4b62-4f4c-aff6-1c2e5752ad8e}
35 GUID UiaPropertyUniqueIdGuid = {
36 0xcc7eeb32,
37 0x4b62,
38 0x4f4c,
39 {0xaf, 0xf6, 0x1c, 0x2e, 0x57, 0x52, 0xad, 0x8e}};
40
41 // Create the registrar object and get the IUIAutomationRegistrar
42 // interface pointer.
43 IUIAutomationRegistrar* registrar;
44 CoCreateInstance(CLSID_CUIAutomationRegistrar, nullptr, CLSCTX_INPROC_SERVER,
45 IID_PPV_ARGS(&registrar));
46
47 // Custom UIA property id used to retrieve the unique id between UIA/IA2.
48 PROPERTYID uia_unique_id_property_id;
49
50 // Register the custom UIA property that represents the unique id of an UIA
51 // element which also matches its corresponding IA2 element's unique id.
52 // Custom property registration only needs to be done once per process
53 // lifetime.
54 UIAutomationPropertyInfo unique_id_property_info = {
55 UiaPropertyUniqueIdGuid, L"UniqueId", UIAutomationType_String};
56 registrar->RegisterProperty(&unique_id_property_info,
57 &uia_unique_id_property_id);
58
59 // Assume we are given the IAccessible2 element for button, and we want to
60 // retrieve its corresponding UIA element for the final result.
61 IAccessible2* button_ia2; /* Initialized */
62
63 // Retrieve button IA2 element's unique id, which will be used to look up the
64 // corresponding UIA element later.
65 LONG unique_id_long;
66 button_ia2->get_uniqueID(&unique_id_long);
67
68 // Assume we are given the Window Handle hwnd for the root.
69 UIA_HWND hwnd; /* Initialized */
70
71 // Instantiating an IUIAutomation object.
72 IUIAutomation* ui_automation;
73 CoCreateInstance(CLSID_CUIAutomation, NULL, CLSCTX_INPROC_SERVER,
74 IID_PPV_ARGS(&ui_automation));
75
76 // Retrieve the root element from the window handle.
77 IUIAutomationElement* root_element;
78 ui_automation->ElementFromHandle(hwnd, &root_element);
79
80 // Retrieve the ItemContainerPattern of the root element.
81 IUIAutomationItemContainerPattern* item_container_pattern;
82 root_element->GetCurrentPatternAs(UIA_ItemContainerPatternId,
83 IID_PPV_ARGS(&item_container_pattern));
84
85 // We also need to convert the retrieved IA2 element unique id from long to
86 // VARIANT.VT_BSTR to be consumed by UIA. For demo purpose, I utilize
87 // std::string here as an intermediary step to convert to VARIANT.VT_BSTR.
Victor Fei63c5b2f2021-07-29 21:16:41[diff] [blame]88 std::wstring unique_id_str = std::to_wstring(unique_id_long);
Victor Feif535fe972020-10-02 06:28:25[diff] [blame]89
90 VARIANT unique_id_variant;
91 unique_id_variant.vt = VT_BSTR;
92 unique_id_variant.bstrVal = SysAllocString(unique_id_str.c_str());
93
94 // Retrieving the corresponding UIAutomation element from the unique id of IA2
95 // object.
96 IUIAutomationElement* button_uia;
97 item_container_pattern->FindItemByProperty(nullptr, uia_unique_id_property_id,
98 unique_id_variant,
99 &button_uia /* final result */);
100~~~
101
102## Convert an UIA element to IA2 element.
103Converting an UIA element to an IA2 element is a lot more straightforward and
104does not require the shared unique id. Consider the same example above.
105~~~c++
106 #include <uiautomationclient.h>
107
108 // Assume we are given the UIAutomation element for button, and we want to
109 // retrieve its corresponding IAccessible2 element for the final result.
110 IUIAutomationElement* button_uia; /* Initialized */
111
112 // Retrieve the LegacyIAccessiblePattern of the button UIA element.
113 IUIAutomationLegacyIAccessiblePattern* legacy_iaccessible_pattern;
Victor Fei63c5b2f2021-07-29 21:16:41[diff] [blame]114 button_uia->GetCurrentPatternAs(
Victor Feif535fe972020-10-02 06:28:25[diff] [blame]115 UIA_LegacyIAccessiblePatternId,
116 IID_PPV_ARGS(&legacy_iaccessible_pattern));
117
118 // Retrieve the IAccessible element from button UIA element.
Victor Fei63c5b2f2021-07-29 21:16:41[diff] [blame]119 // Note: According to UIA doc, GetIAccessible returns NULL if a client
120 // attempts to retrieve the IAccessible interface for an element originally
121 // supported by a proxy object from OLEACC.dll, or by the UIA-to-MSAA Bridge.
122 // https://docs.microsoft.com/en-us/windows/win32/api/uiautomationclient/nf-uiautomationclient-iuiautomationlegacyiaccessiblepattern-getiaccessible
Victor Feif535fe972020-10-02 06:28:25[diff] [blame]123 IAccessible* button_iaccessible;
Victor Fei63c5b2f2021-07-29 21:16:41[diff] [blame]124 legacy_iaccessible_pattern->GetIAccessible(&button_iaccessible);
Victor Feif535fe972020-10-02 06:28:25[diff] [blame]125
126 // Use QueryService to retrieve button's IAccessible2 element from IAccessible
127 // element.
128 IServiceProvider* service_provider;
129 IAccessible2* button_ia2;
130 if (SUCCEEDED(button_iaccessible->QueryInterface(
131 IID_PPV_ARGS(&service_provider)))) {
132 service_provider->QueryService(
Victor Fei63c5b2f2021-07-29 21:16:41[diff] [blame]133 IID_IAccessible, IID_PPV_ARGS(&button_ia2 /* final result */));
Victor Feif535fe972020-10-02 06:28:25[diff] [blame]134 }
135~~~
136
137## Docs & References:
John Palmer046f9872021-05-24 01:24:56[diff] [blame]138[Custom UIA Property and Pattern registration in Chromium](https://chromium.googlesource.com/chromium/src/+/main/ui/accessibility/platform/uia_registrar_win.h)
Victor Feif535fe972020-10-02 06:28:25[diff] [blame]139
140[UI Automation IItemContainerPattern. It is used to look up IAccessible2 element
141via a unique id](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationclient/nn-uiautomationclient-iuiautomationitemcontainerpattern)
142
143[UI Automation Client](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationclient/)