1.19 Accessibility Testing for AI Agents

CertiQA helps teams test selected accessibility-related behaviours in AI agents, add those checks to the agent tests they already run, and track whether the checks regress over time. The results are one input to your own accessibility and inclusion work. They are not a certification and not a statement that your product meets any law or standard.

You already have accessibility testing for web pages (see 2.10 Accessibility (WCAG) Scanning: the A11Y_SCAN and KEYBOARD_AUDIT steps, the OFF / RATCHET / STRICT gating modes and the Accessibility Health Center). This page is about AI agents, which raise three different questions. Keep them apart: each has its own checks and none of them is a score for the others.

Three questions, three sets of checks

Question	What CertiQA offers	Where it is described
Can the agent operate an application through accessible controls?	Accessibility-constrained browsing (the agent targets elements only by visible text or role and accessible name), keyboard-only browsing (no pointer, no direct targeting; key presses report where focus went), and a paired run that compares a normal run with a constrained or keyboard-only run	2.10, sections on keyboard operability and agent runs
Is the page the agent ends on accessible?	An opt-in axe-core scan of the page an agent or hybrid case ended on, reported with the case’s other accessibility findings; the advisory ACCESSIBILITY judge rubric for generated content	2.10, and 1.3 for judge rubrics
Does the agent’s conversation work for people with disabilities?	Scripted conversation checks: confirm before acting, accept a correction, ask instead of guessing, repeat on request, recover from a rejected interpretation, route to a person, consistent choices	1.18 (experimental, unreviewed)

The first two ask whether an automated agent can use a page. A pass is a resilience signal, not evidence that a disabled person can use it, and it is not a WCAG conformance result. The third asks about the agent as the thing a person talks to. It is text only: nothing here tests speech recognition, speech output or any assistive technology.

Add accessibility checks to the tests you already run

You do not need a separate accessibility suite. Accessibility checks are ordinary steps in an agent case, marked as accessibility checks, so they run in the same run as your functional steps and need no extra judge calls.

• Per step: how a failure counts. off skips the check, report records a failure without failing the test, ratchet fails the test only if that check passed in one of the project’s last 20 earlier runs (so you can adopt a check on an agent that does not meet it yet and still be stopped from getting worse), and enforce fails the test like any assertion.
• Per run: an override. The run option a11yMode (--a11y on the command line, “Accessibility checks in agent cases” in the run dialog) can skip, soften, ratchet or enforce every marked step in that run. It never re-enables a check its author switched off, and it never touches your ordinary steps.
• Where to add them. In the app, open an agent or hybrid case, choose Add accessibility check in the add-step menu, pick a check and its options, the mode and where it goes. Through MCP, call build_conversation_checks with mode: "append" and then insert_agent_case_steps. Four of the seven conversation checks can be added to an existing case; the other three need their own case, and the tools say why.
• Failures cannot hide. A failure that did not fail the test still shows in the run report (a badge on the run header) and in the conversation checks report.

Reports you can keep

Report	What it shows	How to open it
Run report	Each case’s result, accessibility checks that failed without failing the test, and any accessibility findings	The run in the app; get_run_report
Operability comparison	The same cases run normally and constrained (or keyboard-only), side by side, with the targets behind failed click, type and select calls	“Compare” in the run dialog; start_operability_comparison and get_operability_comparison
Conversation checks report	Each check group’s own result, separate from its case’s result, compared with the previous result or a run you pin; checks that were removed; no score	“Conversation checks report” on a run; GET /api/v1/projects/{projectId}/runs/{runId}/conversation-checks-report

A group is called a regression only when its checks are configured the same as in the baseline. If the checks, their modes or their options changed, the report says setup changed instead, because a check that was loosened or switched off can hide a regression.

Switches at a glance

What	Run dialog	CLI flag	API field (POST /api/v1/projects/{projectId}/trigger)	MCP trigger_run
Accessibility-constrained agent browsing	Accessibility-constrained agent browsing	--accessibility-constrained	accessibilityConstrained	accessibilityConstrained
Keyboard-only agent browsing	Keyboard-only agent browsing	--keyboard-only	keyboardOnly	keyboardOnly
Scan the page the agent ends on	Scan the page the agent ends on (axe-core)	--a11y-scan-after-agent	a11yScanAfterAgent	a11yScanAfterAgent
How accessibility checks count in this run	Accessibility checks in agent cases	--a11y off|report|ratchet|enforce	a11yMode	a11yMode

The first three apply to agent and hybrid cases only; a run with none of them is refused rather than silently ignoring the option. A paired comparison counts as two runs.

Where to start

1. Run a few of your existing agent cases with “Accessibility-constrained agent browsing” or “Keyboard-only agent browsing” and read the comparison. Treat a difference as a lead: each side is one run of an LLM-driven agent, so it can be variance.
2. Add one or two conversation checks to a case you already run, in report mode, and read the report after a few runs.
3. When a check is stable, move it to ratchet, then to enforce if it should block a build.

What this does not do

• It does not certify an agent, and it does not decide which laws or standards apply to your product. The European Accessibility Act, the ADA and others have different scopes; WCAG is about web content. These checks can contribute test evidence to your own programme, nothing more.
• The conversation checks are experimental and were not reviewed by an accessibility specialist. Validate with disabled users or your own accessibility consultants.
• Automated checks find some problems. Screen-reader behaviour, how a person experiences the agent, and most of what makes an interface usable still need people.