1.17 Seam Cases: Real Systems, Seeds & Database Reads

Test the seams between your agent, your UI and your own systems in one case: let some tools hit your real system, seed data before the agent acts, read the outcome back through an API or an approved database query, and have Copilot write only what can run for your account.

What a seam case is

Most bugs in an agent product live between the parts: the agent calls your booking service with the wrong field, your service writes the wrong row, the UI shows something else. A seam case puts those parts in one test. It can send a message to the agent, let one tool reach your real system while the rest stay mocked, check what your system actually did, and then look at the screen, all with the same variables, vault and trace.

Part	Step	Runs on
Public or flaky services stay mocked	MOCK_TOOL_RESPONSE	Any runner
One tool reaches your real system	PASSTHROUGH_TOOL	Any runner, if the address is reachable from it. A private address needs a self-hosted runner.
Seed data before the agent acts, read it back after	API_REQUEST	Any runner, same reachability rule
Read an end state that has no API	DB_QUERY	Self-hosted runner only
Check the screen	Browser actions and ASSERT_BROWSER_STATE	Any runner with a browser target

Letting Copilot write a seam case

Not available: authoring is done through your own AI

Copilot and AI Discovery (server-side test generation) are shelved. Instead, connect your own AI assistant to the CertiQA MCP server and use the author_certiqa_tests prompt to write seam cases. The Copilot steps below describe the shelved flow and are kept for reference.

Open the project’s Agent Under Test, then AI Discovery. Besides the usual hybrid option you can mark seams, and Copilot builds the steps around them.

1. Open “Seam cases: let some tools and data be real”. It appears when the agent has tools.
2. Tick each tool that should hit your real system. Give the path under your environment’s API base URL (for example /api/refunds), the method, and optionally the name of a vault secret to send as a Bearer token. You never type the secret value, and you never type a host: Copilot writes the address as ${BASE_API_URL} plus your path.
3. Optional: upload your system’s OpenAPI file. With it, Copilot can add API_REQUEST steps to seed the records the agent will act on and to read the outcome back, using only operations in the file. If a real tool changes data and nothing reads it back, Copilot is asked to add a read-back.
4. Optional, self-hosted runners only: upload the approved-query list from your query service (see below). Copilot can then add DB_QUERY steps for those queries.
5. Start discovery, review the drafts, and add the ones you want.

Copilot only writes steps that can run for your account

• On hosted runners it is told never to write a database read, it rejects any it still produces, the query-list upload is not offered, and the API refuses one.
• Some steps depend on where your runs execute and Copilot cannot tell: a real tool or a seed/read-back request goes to your environment’s address, which might be public or private. For a hosted account these are still written but marked with an asterisk (*) next to the action, with a note and one footnote on the draft: on a hosted runner they work only if that address is reachable from the internet. Move the project to a self-hosted runner, or point the environment at a public staging address, if you see the mark and the run cannot reach it.
• Self-hosted accounts get no mark: their runners can reach private networks.

Reading an end state from your database

When the system has no API that shows what the agent changed, use a DB_QUERY step. It never takes SQL and CertiQA never connects to your database. It names an approved query, and a small query service that you run inside your network answers it. The service owns the connection, a read-only database role and the list of approved queries, so a test can only run what you already reviewed, and parameters are bound by the service rather than pasted into a statement.

Setting	Where	Meaning
DB_QUERY_URL	Project vault, a run variable, or the runner’s environment	Address of your query service, for example http://query-service.internal:8099
DB_QUERY_TOKEN	Same	Bearer token the service requires (optional on the CertiQA side, always required by the reference service)
CERTIQA_DB_QUERY_MAX_ROWS	Runner environment	Most rows a query may return (default 1000). More fails the step; nothing is truncated.

Target:  order_status
Value:   {
  "params": { "orderId": "${ORDER_ID}" },
  "expect": { "status": "REFUNDED" },
  "sensitiveColumns": ["card"],
  "outputVariable": "OS"
}

Value field	Meaning
params	Named parameters. String values may use ${…} expressions and the vault. They are sent as data and bound by the service.
expect	Column values every returned row must have. If you give expect and no row count, at least one row must come back, so an empty result can never pass.
expectRowCount	A number or a comparison: 1, “>0”, “>=2”, “<3”.
requiredColumns, columns	Columns that must exist, and per-column rules (required, pattern, oneOf, numeric, unique, maxLength, date).
sensitiveColumns	Columns whose values are masked in messages and scrubbed from traces.
outputVariable	Stores the rows as JSON in NAME, the count in NAME_COUNT, and the first row’s columns as NAME_column for later steps or the next message to the agent.

The report shows the query name and the number of rows, never the rows. Redirects are not followed, so the token only goes to your service. A query over the row cap, a timeout or a non-JSON answer fails the step with the reason.

A tested reference service for PostgreSQL ships with the engine in examples/db-query-service. Put your approved queries in a file, run it next to the database with a read-only role, and set DB_QUERY_URL and DB_QUERY_TOKEN on the runner.

cd examples/db-query-service && npm install
cp queries.example.json queries.json   # your approved queries
DATABASE_URL=postgres://reader:[email protected]:5432/app \
QUERIES_FILE=./queries.json QUERY_TOKEN=$(openssl rand -hex 24) npm start

# the list to give Copilot (names, parameters and columns, never SQL)
curl -H "Authorization: Bearer $QUERY_TOKEN" http://127.0.0.1:8099/queries > queries-catalog.json

Protection	How
No free-form SQL	The statements live in your queries file. A request carries only a name and parameter values, and unknown or extra parameters are refused.
Bound parameters	Values are passed as $1, $2… never joined into the statement. A value like ’ OR ‘1’=’1 is just data.
Read-only	Every query runs in a READ ONLY transaction, so even a data-changing statement hidden in an approved query is refused. Use a database role that can only SELECT: that role is the real control.
Bounded	Statement timeout (default 5 s) and a row cap (default 1000). Over the cap is an error.
Authenticated	A bearer token is required; the service will not start without one and listens on 127.0.0.1 unless you change HOST.
No leakage	GET /queries never returns SQL. The database’s own error text stays in the service log; the runner sees only “query failed”.

Runs that cannot run are refused up front

A run that contains a DB_QUERY step is checked when it is dispatched, before it is created or charged. If it would go to a hosted runner it is refused with the reason, instead of starting and failing at the step. The message tells you which of two things to do:

Your account	Message says	What to do
Hosted runners only	The account runs on CertiQA-hosted runners, which cannot reach your network.	Use self-hosted runners (a bring-your-own-cloud plan), or remove the step.
Self-hosted runners, none online for this project	No self-hosted runner for this project is online.	Start your runner and run again.

The check covers suites, single cases and CertiBlocks, whether the run comes from the app, CI, a webhook or a schedule. A run that goes to your own runner is never held up by it.

Good to know

• Your database is never reachable from CertiQA, and CertiQA stores no database credential. The only secrets involved are the query-service token (vault or runner environment) and whatever your service holds.
• The older SQL value source (valueSource SQL with SQL_PROXY_URL) still exists but has no read-only guarantee, caps or masking. Prefer DB_QUERY.