Skip to content

Extend testdriver.js and wptrunner to test AAMs (platform accessibility APIs) #53733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 25 commits into
base: master
Choose a base branch
from
Open

Extend testdriver.js and wptrunner to test AAMs (platform accessibility APIs) #53733

wants to merge 25 commits into from

Conversation

spectranaut
Copy link
Contributor

@spectranaut spectranaut commented Jul 11, 2025 •
edited
Loading

This PR adds an extension to testdriver.js in order to test accessibility API, as defined by the ARIA, Core-AAM and HTML-AAM specifications. The RFC to extend testdriver.js in this way can be found here. The decision to open a PR with these changes was made at TPAC 2024.

The change in this CL:

  1. Tests in the core-aam directory: These tests are marked as tentative because some work may need to be done on the design of the API endpoint, and these tests are only run when supplied with --enable-accessibility-api. So on regular runs of wpt.fyi, they will fail, for now, as you can see with:
  2. Changes to wptrunner:
    a. The option --enable-accessibility-api has been added. This option will install the appropriate python requirements for accessibility API tests and turn on accessibility for Firefox and Chromium browsers. If this option is not supplied, tests may fail with Accessibility API testing not enabled. The primary reason for the flag is that on linux, there are system requirements for the python requirements, so ./wpt run will fail in environments where the system requirements are not installed. For now I think it's better to only run the tests intentionally/behind a flag.
    a. The product name ("chrome", "firefox", etc) is passed back to the executor level. This is necessary because accessibility APIs are exposed externally to the browser, and we need to find the browser in the accessibility API by name. Ideally, this will be done by PID (to allow for parallelization), but that requires some changes to chromedriver and safaridriver.
  3. executorplatformaccessibility.py and tools/wptrunner/wptrunner/executors/platformaccessibility: these executors run tests against platform specific accessibility APIs, so there is an executor for Linux, Windows, and macOS. The Windows and macOS ones are proof of concepts and will need more work. The Linux one is fully featured and can run all of the tests added in this PR.
  4. Changes to the docker system installations and updating the docker image, to test the linux runs of these tests. You can see the tests pass in the linux environment for Chrome and Firefox with the commit below: Switch CLI arg --enable-accessibility-api to true by default. Here are passing tests in the CI: https://wpt.fyi/results/core-aam?label=pr_head&max-count=1&pr=53733

To run tests on Linux:

On Debian distros:
apt install libatspi2.0-dev libcairo2-dev libgirepository1.0-dev

# Chrome
./wpt run --enable-accessibility-api  --log-mach-level debug --log-mach - chrome core-aam/role/blockquote.tentative.html

# Chromium
./wpt run --enable-accessibility-api  --binary <chromiumbinary>  --force-accessibility  --log-mach-level debug --log-mach - chromium core-aam/role/blockquote.tentative.html

# Firefox
./wpt run  --enable-accessibility-api  --no-headless firefox core-aam/role/blockquote.tentative.html

Notes about macOS and Windows

On Windows and macOS, even with the following dependencies, these tests will fail with the message "Fail: Test not implemented".

Thanks to @alice for the macOS code and thanks @jcsteh, the code to get IA2 interface was very inspired by the firefox test infrastructure: https://searchfox.org/mozilla-central/source/accessible/tests/browser/windows/a11y_setup.py

Running the tests on macOS

To run against Firefox, you need to turn on VoiceOver the screen reader.
To run against Chrome, you need to include --no-headless.

@spectranaut spectranaut requested a review from a team as a code owner July 11, 2025 19:10
@wpt-pr-bot wpt-pr-bot added core-aam infra testdriver.js wg-wai_aria wpt wptrunner The automated test runner, commonly called through ./wpt run labels Jul 11, 2025
@jcsteh
Copy link
Contributor

jcsteh commented Jul 14, 2025 

# To run against Firefox, you need accessibility turned on. You can run on the Orca screen reader or `export GNOME_ACCESSIBILITY=1` and use `--no-headless`.

I wonder if we can make the name --force-renderer-accessibility more generic (e.g. --enable-accessibility or --force-accessibility) and implement it for various browsers as required? For Firefox, we can use the pref accessibility.force_disabled=-1 (I know, it's horrible, I'm sorry). I'm not quite sure where we'd wire that up, but I see that we do set extra prefs here, so we have the capability to do that.

@spectranaut
Copy link
Contributor Author

I wonder if we can make the name --force-renderer-accessibility more generic (e.g. --enable-accessibility or --force-accessibility) and implement it for various browsers as required? For Firefox, we can use the pref accessibility.force_disabled=-1 (I know, it's horrible, I'm sorry). I'm not quite sure where we'd wire that up, but I see that we do set extra prefs here, so we have the capability to do that.

Amazing, thanks @jcsteh! I didn't know about that Firefox option, I always used the gnome accessibility trick. I changed the CLI arg to --force-accessibility and it works with both chrome and firefox now, the PR description has been updated.

@spectranaut spectranaut force-pushed the acacia-first-core-aam-tests branch from 762b8de to 356e3c4 Compare July 15, 2025 22:14
@spectranaut spectranaut requested a review from a team as a code owner July 16, 2025 23:12
@spectranaut spectranaut force-pushed the acacia-first-core-aam-tests branch from 62fc8e5 to b937113 Compare July 22, 2025 18:40
@spectranaut spectranaut force-pushed the acacia-first-core-aam-tests branch from 5b98700 to fbf3fdf Compare July 30, 2025 21:08
@spectranaut spectranaut force-pushed the acacia-first-core-aam-tests branch 2 times, most recently from 7c3363d to 0f4b380 Compare August 1, 2025 01:13
@spectranaut
Add python requirements for windows and mac
e13b205
@spectranaut
Fix requirements for macos and python 3.8
87dbd37
@spectranaut
Fix requirements for linux
947538e
@spectranaut
Update Docker dependencies
3fffddb
@spectranaut
Update tests to use new docker image
821511d
@spectranaut
Fix Flake8 errors
b0f91e5
@spectranaut
Test mistake in tag for docker image
d02bb84
@spectranaut
More flake8 fixes
4d04628
@spectranaut
Fix mypy errors
038e93f
@spectranaut
Use properly tagged docker image
507911c
@spectranaut
Change errors to warnings
4cdc951
@spectranaut
logger
13c0e71
@spectranaut
update comment
c56c351
@spectranaut
Change API name, set environmment variable for accessibility runs
093b9b2
@spectranaut
Move accessibility API requirements to their own file
8987164
@spectranaut
Only install accessibility python requirements when running accessibi…
69f778a 
…lity tests
@spectranaut
Do not fail if accessibility requirements not installed
a57acc0
@spectranaut
Switch CLI arg --enable-accessibility-api to true by default
d508e40
@spectranaut spectranaut force-pushed the acacia-first-core-aam-tests branch from 4c7afa8 to d508e40 Compare August 13, 2025 19:01
@spectranaut spectranaut mentioned this pull request Aug 13, 2025
Closed
@spectranaut
Copy link
Contributor Author

The failures are unrelated, see: #54302

@spectranaut
Copy link
Contributor Author

Ready for review :)

@gsnedders, @jgraham, @cookiecrook or @zcorpan, perhaps a couple of you could take a look?

@spectranaut spectranaut mentioned this pull request Aug 18, 2025
Open
Ms2ger
Ms2ger reviewed Sep 1, 2025
View reviewed changes
Copy link
Contributor

@Ms2ger Ms2ger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

tools/wptrunner/wptrunner/executors/platformaccessibility/executoratspi.py

:param dom_id: The dom id of the node to test.
:param test: The test statement.
:param api: The API to test.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

api is unused here; does it make sense to pass it? Or should we verify it's atspi here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So API is unused here, but it won't be unused in the "executorwindows.py" because there are two APIs on windows that need to be tested (although that is not yet implemented in windows, because the windows implementation is just stub). We need the function signatures of "test_accessibility_api" in all this executors in this folder to be the same: https://github.com/web-platform-tests/wpt/pull/53733/files#diff-4136e59af143c98357cb7a6153da10afc1fde0c16a657194efd31b1b4edd4f50

And you can see a few lines above the call to this function, we check "valid_api_for_platform"

tools/wptrunner/wptrunner/executors/platformaccessibility/executoratspi.py
Comment on lines +20 to +25
tab = find_tab(root, product, url)
while not tab:
time.sleep(0.01)
tab = find_tab(root, product, url)

return tab
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want a timeout on this, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... so far, I've been rely on the timeout within the wptrunner test infrastructure. Do you think we should have a separate (shorted I assume) timeout here?

spectranaut
spectranaut commented Sep 1, 2025
View reviewed changes
Copy link
Contributor Author

@spectranaut spectranaut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @Ms2ger ! Fixed as you suggested, but I have two questions.

tools/wptrunner/wptrunner/executors/platformaccessibility/executoratspi.py

:param dom_id: The dom id of the node to test.
:param test: The test statement.
:param api: The API to test.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So API is unused here, but it won't be unused in the "executorwindows.py" because there are two APIs on windows that need to be tested (although that is not yet implemented in windows, because the windows implementation is just stub). We need the function signatures of "test_accessibility_api" in all this executors in this folder to be the same: https://github.com/web-platform-tests/wpt/pull/53733/files#diff-4136e59af143c98357cb7a6153da10afc1fde0c16a657194efd31b1b4edd4f50

And you can see a few lines above the call to this function, we check "valid_api_for_platform"

tools/wptrunner/wptrunner/executors/platformaccessibility/executoratspi.py
Comment on lines +20 to +25
tab = find_tab(root, product, url)
while not tab:
time.sleep(0.01)
tab = find_tab(root, product, url)

return tab
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... so far, I've been rely on the timeout within the wptrunner test infrastructure. Do you think we should have a separate (shorted I assume) timeout here?

@zcorpan zcorpan mentioned this pull request Sep 2, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ms2ger Ms2ger Ms2ger left review comments

@DanielRyanSmith DanielRyanSmith Awaiting requested review from DanielRyanSmith

@foolip foolip Awaiting requested review from foolip

@jgraham jgraham Awaiting requested review from jgraham

@jnurthen jnurthen Awaiting requested review from jnurthen

@jonathan-j-lee jonathan-j-lee Awaiting requested review from jonathan-j-lee

@michael-n-cooper michael-n-cooper Awaiting requested review from michael-n-cooper

@sideshowbarker sideshowbarker Awaiting requested review from sideshowbarker

@cookiecrook cookiecrook Awaiting requested review from cookiecrook

At least 1 approving review is required to merge this pull request.

Assignees

@jonathan-j-lee jonathan-j-lee

Labels
ci core-aam docker infra mypy.ini Taskcluster testdriver.js wai-aria wg-wai_aria wg-wai_pf wpt wptrunner The automated test runner, commonly called through ./wpt run
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@spectranaut @jcsteh @Ms2ger @jonathan-j-lee @wpt-pr-bot