1.18 Conversation Checks (Experimental, Unreviewed)

This is one of three kinds of accessibility testing for agents. The others (whether an agent can operate an app through accessible controls, and automated scans of the pages it ends on) are covered in 1.19 Accessibility Testing for AI Agents, and testing web pages themselves in 2.10 Accessibility (WCAG) Scanning.

EXPERIMENTAL AND UNREVIEWED. These checks have not been reviewed by an accessibility specialist; none was available. They are scripted checks of specific conversation behaviours, informed by the W3C NAUR Group Draft Note (September 2022), which is not a standard. They are not a WCAG or other conformance result, not a test of voice or of any assistive technology, and not evidence that a disabled person can use the agent. Validate with disabled users or your own accessibility consultants.

Scripted checks of how an AI agent behaves in a conversation, built from the agent test steps you already have. Each one passes or fails on something observable: what the agent said and which tools it called. They are meant as a regression net for a few specific behaviours that people who find a conversation hard to follow, type, hear or recover from are likely to depend on. They are not a measure of whether your agent is accessible.

What they are, and are not

• They check dialogue behaviour in text. Most of the W3C document they draw on is about speech: recognition confidence, pronunciation, speaking rate, synchronized speech and text. None of that is tested here.
• They are informed by, not conformant to, a draft. The source is the W3C Natural Language Interface Accessibility User Requirements (NAUR), a Group Draft Note dated 3 September 2022. It is not a standard, is not endorsed by W3C or its members, and says itself that it is not a baseline set of requirements. Each scenario below says which numbered requirement it relates to and whether that relation is direct, indirect, or absent.
• There is no score. CertiQA reports each check as passed or failed with the assertion that failed. A pass is not evidence that your agent works for a disabled person, and a set of passes is not “accessibility”.
• Nobody with accessibility expertise reviewed the choices. We could not get a specialist to review which behaviours matter most, whether the wording is respectful, or whether the checks give false comfort. They are our best attempt. Please validate with disabled users or your own accessibility consultants, and tell us where a check misleads.

Authoring them

Use the MCP prompt author_conversation_a11y_tests with your coding agent. It asks for the real names of your agent’s tools (it must not invent any), binds the templates, creates the cases with create_agent_case (named [a11y-conversation] ...), and reports each case’s own “shows” and “does not show” text. The cases use ASSERT_OUTPUT and EXPECT_TOOL_CALL_COUNT, which need no LLM judge. Tools are mocked, so a result does not depend on a backend.

The scenarios

A. Confirm before acting

Experimental, unreviewed. Before an irreversible action, a person can review the target the agent understood and confirm it.

Relation to NAUR: direct but partial (REQ 19b). NAUR REQ 19b asks the system to present what the user supplied and ask for confirmation before an irreversible action. The template checks the target and a wording heuristic; it does not reliably establish that the agent asked for confirmation, and it does not test error detection or correction (REQ 24b).

• Shows: The agent repeated the target and its reply matched a small confirmation-wording heuristic; no action occurred before the user confirmed, and one action followed with that target.
• Does not show: That the reply actually asked for confirmation, that the wording is clear, that the user can confirm through every input method, or that an error was detected and corrected. The wording check can miss valid ways of asking for confirmation or match a word used in another context.

B. Accept a correction

Experimental, unreviewed. Before an action, a person can correct a detail in text and have the corrected detail used.

Relation to NAUR: indirect (REQ 13d). This is analogous to NAUR REQ 13d, which specifically concerns correcting speech input. The template tests a text correction only; it does not exercise speech recognition or test REQ 24a’s error detection. Reversing an action that already happened (REQ 25) is not tested.

• Shows: The action tool was not called with the mistaken identifier, no action occurred before confirmation, and the action tool was called exactly once with the corrected identifier.
• Does not show: That the agent was patient or courteous, or that it would take a correction in a different form (a spoken, partial or implied one).

C. Ask instead of guess

Experimental, unreviewed. A person who gives a short or incomplete request is asked for what is missing instead of having the agent act on a guess.

Relation to NAUR: direct (REQ 18d). NAUR REQ 18d: when input is ambiguous or cannot be processed, ask for clarification or offer choices.

• Shows: The agent made no call to the action tool and replied with a question that mentions the missing detail.
• Does not show: That the question was clear, that it can be answered by every route, or that the agent never guesses in other situations.

D. Recover from “that’s not what I asked”

Experimental, unreviewed. A person who is misunderstood can say so and get a new attempt, not the same mistake again.

Relation to NAUR: indirect (REQ 13d, REQ 24c). Only loosely related to NAUR REQ 13d and 24c. NAUR has no requirement about a user rejecting the system’s interpretation; this scenario is our own judgement.

• Shows: After the rejection the agent did not repeat the same action with the same arguments, and its reply asked or invited a new instruction.
• Does not show: That the reply was respectful, or that the invitation wording check (a question, “tell me”, “let me know”) matches every good reply: it can wrongly fail an agent that recovers in other words. The wording check is flagged for the specialist.

E. Route to a person

Experimental, unreviewed. A person for whom the conversation is not working can ask for a human and be handed over.

No basis in NAUR. Not derived from NAUR. NAUR states no requirement for handing over to a person (its Example 4 describes a hand-off). This scenario is our own judgement that a way out of a failing conversation matters.

• Shows: The agent called its hand-over tool and replied.
• Does not show: That a person is actually reachable, how long they take, or that the reply says what happens next.

F. Repeat on request

Experimental, unreviewed. A person who missed, misheard or could not take in what the agent said can ask for it again and get the same information.

Relation to NAUR: direct (REQ 19a). NAUR REQ 19a: repeat output when the user asks. This checks that the detail comes back; it does not test spoken output.

• Shows: When asked to repeat, the agent restated the identifier and did not call the action tool.
• Does not show: That the repeat was spoken, was easier to take in the second time, or that the agent can repeat anything other than its immediately previous reply.

H. Consistent choices

Experimental, unreviewed. A person can express the same selection in different short text forms and get the same result.

Relation to NAUR: direct (REQ 18a, REQ 17a). NAUR REQ 18a asks the system to respond to varied wording for the same request. The test verifies that two choices were shown (REQ 17a), but only three number-based selection phrases are covered.

• Shows: The options tool was called, both option identifiers appeared in the agent’s reply, no action happened before confirmation, and the confirmed action used the second option’s identifier exactly once. Authored once per expression, a failure for one expression shows an agent that handles one way of choosing and not another.
• Does not show: That the list was readable aloud or by a screen reader, or that the three listed expressions cover how people actually choose.

Adding checks to a test you already have

You do not need a separate case for every check. Most can be added to an agent test you already run, which avoids repeating the whole conversation. Ask your coding agent to run the author_conversation_a11y_tests prompt and say you want to add checks to an existing case, or call build_conversation_checks with mode: "append" and then insert_agent_case_steps yourself. Use insert_agent_case_steps, not update_case_steps, which would replace the case’s steps and does not understand agent steps.

Check	Can it be added?	Where, and what to know
A. Confirm before acting	Yes: right after the turn it checks, adds no turns	Insert immediately after the turn where the user makes the consequential request, and before the turn where they confirm. Nothing may have called the action tool before this point.
B. Accept a correction	No	It needs its own sequence of turns (a mistaken request, then a correction), which would change the flow of an existing test. Author it as its own case.
C. Ask instead of guess	Yes: right after the turn it checks, adds no turns	Insert immediately after a turn where the user’s request is missing something the agent needs, and before any later turn. Nothing may have called the action tool before this point.
D. Recover from “that’s not what I asked”	No	It needs a turn where the user rejects the agent’s interpretation, and counts calls over the whole conversation. Author it as its own case.
E. Route to a person	Yes: at the end of the case, adds one user turn	Insert at the end of the case, after all of its own assertions. It adds one user turn. If the case’s tools are real (passthrough), check that a hand-over request is safe to send.
F. Repeat on request	Yes: at the end of the case, adds one user turn	Insert at the end of the case, after all of its own assertions. It adds one user turn. The identifier must appear in the agent’s last reply before it, or the check will fail for that reason. Whether the action tool was called is not checked here, because the case has usually called it already.
H. Consistent choices	No	It needs its own list-then-choose sequence and one case per way of choosing. Author it as its own case.

Each added step is named [a11y] ... and marked as an accessibility check, so it can be treated differently from the rest of the test. Every one has a mode, which you choose when you add it and can override for a single run:

• report (the default when added): if the step fails, CertiQA records the failure and carries on. The case keeps its own result, and the failure shows in the run report, with a badge in the run header, so it is not hidden. Use this while you are adopting the checks, or when you want a record but not a gate.
• enforce: the step fails the case like any other assertion.
• off: the step is added but skipped.

To change this for one run without editing the case, set a11yMode on trigger_run (or “Accessibility checks in agent cases” in the run dialog): off skips every accessibility step in the run, report records failures without failing cases, ratchet fails a case only for a check that has passed recently (see below), enforce makes them count. This only affects steps marked as accessibility checks, never your ordinary steps. It cannot switch on a step whose author set it to off.

Adding a check in the app. Open an agent or hybrid case and choose Add accessibility check in the add-step menu. Pick a check (confirm before acting, ask instead of guess, route to a person, repeat on request), read what it shows and does not show, fill in your own names and messages (the same options as build_conversation_checks), choose how a failure counts (report, ratchet, enforce, or added switched off) and, for checks that follow a turn, where it goes. The steps are added to the case, named [a11y] ..., and each can be changed later from the step’s own “Accessibility check” setting. The other checks (accept a correction, recover from a rejected interpretation, consistent choices) need their own case; the dialog says why. The dialog refuses a check the case already has. The app uses a copy of the same templates and binder as the MCP tool, so the two cannot disagree.

Ratchet: fail only what used to pass. A step (or a whole run) in ratchet mode fails its case only when that check has passed in one of the project’s last 20 earlier runs of the same case, so a failure now is a regression. A check that has never passed is recorded without failing the case, like report, so you can adopt a check on an agent that does not meet it yet and still be stopped from getting worse. “Recently” is the last 20 runs rather than only the previous one, so a regression cannot slip through because the run before it also failed. A check is recognised by its action and settings, not its mode, so moving a check from report to ratchet keeps its history; changing the check itself (a different pattern, say) starts it afresh and soft. If the history cannot be read (a run from a file, or an older server), ratchet checks stay soft and the check says “unknown”. A ratchet failure that fails the case says it passed before.

Two things to keep in mind. A step in report mode can fail without anyone noticing if nobody reads the report, so decide who looks at it. And a check that adds a turn runs after your own assertions, so it cannot change what your test sees, but it does send one more message to the agent: if your case uses real (passthrough) tools, check that the extra message is safe. The configuration fingerprint in the conversation checks report includes each step’s mode and the run-level setting, so switching a check from report to enforce, or running with a11yMode set, reads as a changed setup and is never counted as a regression or a fix.

Configuring each check

The test author decides what is checked. Every assertion has a switch, every wording pattern can be replaced (for example for another language or a house style), and every count can be set. The defaults reproduce the behaviour described above, so a check changes only when you say so. Anything you switch off is written into the case description as “NOT checked in this case”, so a case never looks like it checks more than it does. Three kinds of option:

• Yours: your own tool names, identifiers and messages. Required unless a default is shown.
• Switches and counts: turn an assertion on or off, or set how many times something must happen.
• Wording patterns: a case-insensitive regular expression the agent’s reply must match. Replace it if the default does not suit your agent.

To use it, ask your coding agent to run the author_conversation_a11y_tests prompt, or call the MCP tool build_conversation_checks yourself: mode: "catalog" lists every check and option, and mode: "build" returns cases whose steps you pass unchanged to create_agent_case. The tool rejects an unknown option, a wrongly typed value and an invalid regular expression, and lists every problem at once.

A. Confirm before acting

Option	Kind	Default	What it does
actionTool	Yours	required	The agent’s tool that performs the irreversible action being checked
idArg	Yours	required	The tool argument that identifies the target
idValue	Yours	required	The identifier the user gives
requestMessage	Yours	required	What the user says, containing the identifier
confirmMessage	Yours	"Yes, please go ahead."	How the user confirms
requireRestatement	Switch or count	true	Check that the agent’s reply contains the identifier it understood
checkConfirmationWording	Switch or count	true	Check that the reply also looks like a request for confirmation (a wording heuristic that can miss valid phrasings)
confirmationPattern	Wording pattern	"((confirm|correct|right).{0,50}\\?|\\?.{0,50}(confirm|correct|right)|please\\s+(reply|say)\\s+yes|say yes to continue)"	Regular expression (case-insensitive) a confirming reply must match. Replace it for another language or house style
checkActionCount	Switch or count	true	Check how many times the action tool was called after confirmation
expectedActionCount	Switch or count	1	How many times the action tool must have been called

B. Accept a correction

Option	Kind	Default	What it does
actionTool	Yours	required	The agent’s tool that performs the consequential action
idArg	Yours	required	The tool argument that identifies the target
wrongIdValue	Yours	required	The mistaken identifier
idValue	Yours	required	The identifier the user really meant
wrongRequestMessage	Yours	required	The first, mistaken request
correctionMessage	Yours	required	The correction
confirmMessage	Yours	"Yes, please go ahead."	How the user confirms
requireConfirmationBeforeAction	Switch or count	true	Require that nothing happens between the correction and the user’s confirmation. Turn off if your agent may reasonably apply a correction at once
forbidMistakenValue	Switch or count	true	Check that the action tool is never called with the mistaken identifier
requireSingleAction	Switch or count	true	Require exactly one action with the corrected identifier. Off: at least one

C. Ask instead of guess

Option	Kind	Default	What it does
actionTool	Yours	required	The agent’s tool that performs the consequential action
underspecifiedMessage	Yours	required	A request missing the identifying detail
checkQuestion	Switch or count	true	Check that the reply is a question
questionPattern	Wording pattern	"\\?"	Regular expression a question must match. Replace it for a language that does not use ‘?’
requireMissingDetailWord	Switch or count	true	Check that the reply names the missing detail
missingDetailWord	Yours	needed when requireMissingDetailWord is on	A word the reply should contain to show it asked for the missing detail

D. Recover from “that’s not what I asked”

Option	Kind	Default	What it does
actionTool	Yours	required	The agent’s tool that performs the consequential action
idArg	Yours	required	The tool argument that identifies the target
idValue	Yours	required	The identifier in the request
requestMessage	Yours	required	The request
rejectionMessage	Yours	"No, that's not what I asked for."	The user rejecting the agent’s interpretation
maxRepeatedActions	Switch or count	1	The most times the same action may have been called by the end
checkInvitationWording	Switch or count	true	Check that the reply asks a question or invites a new instruction (a wording heuristic that can miss valid phrasings)
invitationPattern	Wording pattern	"(\\?|tell me|let me know|what (do|would))"	Regular expression (case-insensitive) an inviting reply must match

E. Route to a person

Option	Kind	Default	What it does
escalateTool	Yours	required	The agent’s tool that hands over to a person
handoffRequest	Yours	"I'd rather speak to a person, please."	The user asking for a person
escalationArgs	Switch or count	{}	Arguments the hand-over call must include (a partial match). Empty accepts any
requireReply	Switch or count	true	Check that the agent also replied

F. Repeat on request

Option	Kind	Default	What it does
actionTool	Yours	required	The agent’s tool that performs the consequential action
idValue	Yours	required	The identifier the user gives and the agent restates
requestMessage	Yours	required	What the user says, containing the identifier
repeatMessage	Yours	"Sorry, could you say that again?"	The user asking for the last reply again
requireNoActionOnRepeat	Switch or count	true	Check that asking for a repeat did not trigger the action

H. Consistent choices

Option	Kind	Default	What it does
listTool	Yours	required	The agent’s tool that lists the options
listResponse	Yours	required	What the mocked list tool returns: two options
listRequest	Yours	required	The user asking to see the options
choiceExpression	Yours	required	One way of choosing the second option. Author one case per expression you want covered, for example “2”, “the second one”, “number two”
actionTool	Yours	required	The agent’s tool that performs the consequential action
idArg	Yours	required	The tool argument that identifies the target
secondIdValue	Yours	required	The identifier of the second option
actionMessage	Yours	"Refund that one."	The user asking to act on the chosen option
confirmMessage	Yours	"Yes, please go ahead."	How the user confirms
requireListToolCall	Switch or count	true	Check that the agent called its list tool
requireBothOptionsShown	Switch or count	true	Check that the agent’s reply to the list request contains both option identifiers
firstIdValue	Yours	needed when requireBothOptionsShown is on	The identifier of the first option
requireConfirmationBeforeAction	Switch or count	true	Require that nothing happens between the choice and the user’s confirmation
requireSingleAction	Switch or count	true	Require exactly one action on the chosen option. Off: at least one

The report

After a run, open the run in the app and choose Conversation checks report. It is meant as a record your team and your accessibility specialist can refer to when deciding whether an agent change has made the agent worse at the behaviours you chose to check.

The report is organised by check group, not by test case. A check group is the set of accessibility steps one scenario added (for example “Confirm before acting”), wherever it sits: in a case made by author_conversation_a11y_tests, or appended into a case you already had. Cases made before steps were marked (tagged @a11y-conversation) are still reported, as one group each.

• What it shows, per group: the group’s own result (from its checks only, so a functional failure elsewhere in the same case does not make an accessibility group fail, and the reverse), the test case’s result shown separately, the mode of each check (report, ratchet or enforce), each check and how it came out, the checks your team switched off, and the scenario’s “shows” and “does not show” text. A group that failed only in report mode is marked as such: it was recorded and did not fail the test.
• What it compares against: by default each group’s most recent earlier result (up to 50 runs back). You can instead pin one run, for example the release you signed off, and every group is compared with that.
• Regression means the same checks. A group is reported as a regression only when it passed in the baseline, fails now, and its own checks are configured the same (CertiQA compares a fingerprint of the group’s steps, so editing the case’s ordinary steps does not count as reconfiguring the check). If the checks, their modes or their options changed, or the baseline was recorded before this was tracked, the comparison is reported as a changed setup: the result change is still shown, but it is not counted as a regression, a fix or “still passing”, because a check that was loosened or switched off can hide a regression. The report lists the checks that are no longer run.
• A group that is gone is shown, not dropped. If a team removes every step of a check group, the report still lists it as removed, with the checks the baseline ran, so a report never reads as “no accessibility checks ran” when checks used to run. It keeps showing while the group stays gone (against each case’s recent results, up to 50 runs back) and when you compare against a pinned run. A group that was only switched off in the baseline is not reported, since it checked nothing. Pin a baseline recorded after the removal to acknowledge it.
• Labels. Each group is labelled regressed, fixed, still passing, still failing, setup changed (the result differs but the checks do not match the baseline, so it is neither a regression nor a fix), removed, new, or not comparable. A group never reads “regressed” unless its checks match the baseline’s.
• Checks that did not fail the test. A failure in report mode, or in ratchet mode for a check that had not passed before, is recorded without failing the case. Each check says which: a ratchet check shows whether it had passed before, had not, or had no earlier results to look at.
• There is no overall score and no percentage. The summary counts groups by outcome. Regressions are listed first.
• To keep: Download JSON saves the report as shown, with the notice. Print gives a copy for your records.

Using the results. The results are test evidence about specific behaviours you chose to check, and one input to your own accessibility and inclusion work. They do not tell you which laws or standards apply to your product, and they are not a statement that the product meets any of them. Testing whether an agent can operate a web application through accessible controls (the constrained and keyboard-only run modes) and testing whether the agent’s conversation works for people with disabilities (these checks) are related but separate questions; the report covers only the second.

The report says, on every copy, that the checks are experimental, were not reviewed by an accessibility specialist, were chosen and configured by your team, and are not a conformance result. It covers only the behaviours you chose to check.

Known weak spots

• Scenarios A, C and D use wording checks. A looks for common confirmation words; C looks for a question mark and a word naming the missing detail; D looks for a question or an invitation such as “tell me” or “let me know”. They can fail an agent that handles the situation well in other words, or pass wording that does not make sense in context. Treat a failure as a lead to read the transcript, not a verdict.
• Scenario H’s three ways of choosing are a guess at how people choose.
• English only. The patterns assume English.
• Scenario E is our own judgement, not something the W3C document asks for.
• Some things that matter are not here at all: timing and time limits, whether a person is really reachable, authentication, how long a conversation’s history is kept, reading level, and anything about voice.