1.2 Local Tool Interception & In-Process MCP Mocking

How the runner intercepts REST and Model Context Protocol (MCP) tool calls in-flight to return deterministic responses.

Example: the BookHaven support agent

This suite tests BookHaven, a small reference support agent that runs locally. An AI assistant wrote the cases through the CertiQA MCP server, which checked each one before saving it. Each case mocks the agent’s tools, sends a customer message, then checks which tools were called, with which arguments, and how often.

Six of the seven cases pass. The seventh is meant to fail. A web page the agent fetched told it who to send email as, and the agent obeyed. The test asserts the safe behaviour, so the run report shows the failure and the argument that was wrong.

The transcript for the failed case shows the cause: the agent called fetch_webpage on an untrusted address and then called send_email.

Why Tool Calls Need to Be Mocked, Not Just Called

AI agents earn their usefulness by reaching out to real systems – a SQL database, Stripe, Salesforce, GitHub – through tool calls. That is exactly what makes them hard to test against directly: pointing a test run at a live payment processor means every test run actually issues a charge or a refund, pointing it at a live production database means every run risks a write it can’t cleanly undo, and pointing it at any third-party API means your test’s pass/fail now depends on that API’s uptime and response time, not on your agent’s behavior. None of that is what a regression test is supposed to depend on. CertiQA’s answer is to intercept the tool call before it leaves the process and answer it with a deterministic mock instead, so the same test produces the same result every time regardless of what any external system is doing that day.

Where the Interceptor Sits

When a run includes a MOCK_TOOL_RESPONSE step, the runner starts a lightweight in-process server on local port 4790 and configures your agent to send its tool-calling traffic there instead of the real endpoint – for REST-style tool calls, and, for agents built on the Model Context Protocol, a dedicated MCP-aware interceptor speaking the same JSON-RPC 2.0 dialect the agent framework expects. You don’t start or configure either server yourself; adding the step is what wires it in. Responses come back near-instantly by default (no real network hop, no real backend to wait on), and you can add deliberate simulated latency when the thing you actually want to test is how the agent behaves while it’s waiting.

Three Ways to Match an Incoming Call

A mock only fires when the intercepted call matches what you configured, and CertiQA gives you three matching strictness levels rather than one, because “exact match” is right for some tests and wrong for others.

Mode	What It Checks	Use It When
EXACT	Every argument key and value must match the configured payload precisely.	Your test controls every input and you want the mock to fail loudly if the agent sends anything unexpected.
PARTIAL	Only the keys you specify are checked; extra arguments the agent sends are ignored.	The agent legitimately includes fields you don’t care about for this assertion (a request-id, a locale) and asserting on them would just make the test brittle.
SCHEMA	Arguments are validated against a JSON Schema via Ajv rather than fixed values.	The value itself varies run to run – a generated order id, today’s date – but its shape and type should always be correct.

- action: MOCK_TOOL_RESPONSE
  toolName: process_refund
  matchMode: SCHEMA
  argsSchema:
    type: object
    required: [orderId, amount]
    properties:
      orderId: { type: string }
      amount: { type: number, minimum: 0 }
  mockResponseBody: { status: "SUCCESS", refundId: "ref-100" }
  mockLatencyMs: 200

Mock Some Tools, Let Others Reach the Real System

Mocking everything hides the bugs that live between your agent and your own systems. A flight agent that calls a public holiday calendar and your internal booking service should not hit the calendar (its answer changes, and you do not control it), but it should hit the booking service in staging, because that is where a wrong seat number or a missing field actually breaks. A PASSTHROUGH_TOOL step names a tool whose calls go to a real endpoint you choose, while every other tool stays mocked.

- action: MOCK_TOOL_RESPONSE      # a public service: mocked
  tool: holiday_calendar
  response: { holidays: ["2027-01-01"] }

- action: PASSTHROUGH_TOOL        # your own staging system: real
  tool: reserve_flight
  url: "{{BASE_API_URL}}/api/reserve"
  method: POST
  headers: { Authorization: "Bearer {{token}}" }

The rules are deliberately strict, because a test must never make a real call by accident.

Rule	What it means
Nothing is real unless you say so	A tool with no mock and no PASSTHROUGH_TOOL step is never sent anywhere. It returns the same “no mock registered” error as before.
A mock beats a route	If a mock matches the call, the real endpoint is not contacted. A route only receives calls no mock claims, so you can mock one city’s weather and let the rest through.
Arguments	POST, PUT and PATCH send the agent’s arguments as a JSON body. GET and DELETE send them as a query string. Put {{args.name}} in the url to place an argument in the path (it is URL-encoded); the host cannot come from an argument.
What the agent sees	The real HTTP status and body. A timeout, an unreachable host or a response over the size limit comes back as a tool failure (502) so you can test how the agent copes with a real outage. Redirects are not followed.
Faults still apply	You can inject a fault on a real tool. The real call happens first, then the fault replaces what the agent sees, which models “it worked, but the tool reported failure”.
Where it can reach	On a CertiQA-hosted runner, private, loopback and cloud-metadata addresses are refused. To reach a system inside your network, use a self-hosted runner.
Browser actions in the same case	An agent or hybrid case can also contain browser actions (LAUNCH, ENTER_VALUE, CLICK, STORE_VARIABLE, ASSERTION and the rest). They run in the browser session the agent’s own browser tools drive, with the same self-healing locators, and they see the case’s variables, vault references and anything an earlier step captured; what they capture (a STORE_VARIABLE, an output variable) is visible to the agent steps that follow. The run needs a browser target (the agent’s browser settings, or –agent-browser-target).
Authentication	Add auth to the step instead of hand-building a header: BEARER, BASIC, API_KEY, or OAUTH2_CLIENT_CREDENTIALS. For OAuth, CertiQA exchanges the client credentials for an access token the first time it is needed, reuses it until shortly before it expires, shares it between calls running at the same time, and mints a new one (once) if your system answers 401. Set either auth or an Authorization header, not both.
Tracing	Run with –otel (or CERTIQA_OTEL_ENABLED=true) and every case, UI, API or agent, is one trace. Each UI step is a span of its case. Each API step, agent turn and real tool call is a client span, and the request it makes carries that span as a W3C traceparent header, so spans from your own services join the same trace in Tempo, Jaeger, Datadog or whichever backend you use. In a UI case the browser sends the running step’s traceparent to your application’s own origin (the run’s base URL) so its server spans nest under the step that caused them; it is never sent to other origins, because a custom header on a third-party request would trigger a CORS preflight that could break the page. List more of your own origins in CERTIQA_TRACE_PROPAGATION_ORIGINS (comma-separated). CERTIQA_TRACE_PROPAGATION=off records the spans but sends no header anywhere. A mocked tool call makes no outbound request, so it has no span: it appears as an event on the case, marked mocked, next to the real calls. The report links to the trace by id, and the run metrics count tool calls by kind (mocked, real, unmatched) and status.
Privacy of replies	A report keeps the start of what a real endpoint answered, with known secrets masked, so you can see what the agent was told. If your system’s replies hold personal data, set CERTIQA_REPORT_REAL_CALL_BODIES=off: the report then keeps where each call went, its status and timing, and no reply.
Debugging	certiqa debug shows, for every real call, where it went, how it authenticated (the scheme, never the value), the status, how long it took and the start of the reply; the tools that are routed to real endpoints; and the last API step’s request and response. Everything shown has secrets and credentials masked. With LOG_FORMAT=json the agent runner’s progress lines become structured log entries, with the run and project ids attached, in the same stream as the engine’s.
Reading the end state from a database (DB_QUERY)	When the system has no API that shows what an agent changed, add a DB_QUERY step: Target is the NAME of an approved query, Value is JSON with params (bound by the service, never pasted into SQL), expect (column values every returned row must have), expectRowCount, requiredColumns, column rules and sensitiveColumns (masked everywhere). It does not take SQL. CertiQA never holds a database credential: the step calls a small query service you run inside your network (DB_QUERY_URL, optional DB_QUERY_TOKEN, both read from the vault, a run variable or the runner environment), which owns the connection, a read-only database role and the list of approved queries. It answers POST {url}/queries/{name} with {“rows”: […]}. It runs only on a self-hosted runner. Results are limited in rows (CERTIQA_DB_QUERY_MAX_ROWS, default 1000), size and time, and are never truncated: too many rows fails the step. The report shows the query name and row count, never rows. Output variable NAME gives NAME (JSON rows), NAME_COUNT and the first row as NAME_column, so the next step, or the agent message, can use them. A tested reference query service for PostgreSQL ships with the engine (examples/db-query-service): approved queries in a file, bound parameters, read-only transactions, a timeout, a row cap and a bearer token, with GET /queries listing the names, parameters and columns. Give that list to Copilot when you start agent discovery and it will write DB_QUERY steps only for those queries. Copilot writes them only for projects that run on self-hosted runners: for a project on hosted runners it is told never to write one, rejects any it is given, and the query-list field is not offered.
Secrets and variables	Anything in the step can use ${SECRET:KEY} (or {{secret:KEY}}) to read from the project vault, ${NAME} for a case variable, and ${ENV:KEY}, ${FN:…}, ${FAKER:…} like any other step. A vault secret is resolved only when the step runs, and every secret CertiQA resolves or mints is masked in reports, transcripts and logs. Never paste a real secret into the step text.

In the run report a real call is marked as such (passedThrough), and in CI output it is labelled [real] where a mock is labelled [mocked]. A real call counts as handled, not as an unmatched tool call.

The MCP Handshake Happens Automatically

An agent built on the Model Context Protocol doesn’t jump straight to calling a tool – it first sends an initialize request, then tools/list to discover what’s available, and only then tools/call for the actual invocation, all as JSON-RPC 2.0 messages. The interceptor answers initialize and tools/list on its own using the tool definitions your test case already knows about, so you only have to configure the response for the tools/call your test actually cares about, not fake the whole protocol negotiation by hand for every case.

Visual Fault Injection & Multi-Turn Chaos Simulation

Testing agent reliability requires verifying how it recovers when upstream tools fail. CertiQA features a dedicated Visual Fault Injection Builder (accessible directly via the “⚡ Inject Fault” button on any tool mock step or via FaultInjectionModal) that removes the guesswork from crafting chaos scenarios. Instead of hand-writing synthetic error envelopes, automators can select standard fault presets with one click:

Fault Mode	Injected Behavior	Agent Verification Goal
TIMEOUT	Simulates upstream socket freeze or artificial latency (e.g., 15,000ms delay) before connection abort.	Verifies agent streaming cancellation, user timeout notification, and avoidance of infinite loops.
HTTP_500	Returns an HTTP 500 Internal Server Error with a standardized RFC 7807 error envelope.	Ensures agent handles unexpected backend failures gracefully without unhandled promise rejections.
HTTP_429	Returns HTTP 429 Too Many Requests with a Retry-After: 5 header.	Tests agent backoff, retry queueing, and polite pacing when upstream rate limits are hit.
MALFORMED_JSON	Returns truncated JSON syntax or schema-invalid payloads.	Verifies client-side JSON parsing defenses and error boundary handling.

A key pattern in resilient agent architectures is transient retry: failing once, backing off, and succeeding on the second call. The Visual Builder allows configuring an onCall trigger index (e.g., onCall: 1). The mock server injects the chosen fault on call #1, but automatically reverts to normal mock response behavior on subsequent calls. This provides definitive proof that the agent’s retry loop actually resolved the issue.

Fault injection is only half the test – the other half is verifying that the agent reacted correctly. A case-level faults: block registers a fault against a tool and grades the agent’s reaction with expectRecovery, an automated recovery rubric. The LLM judge evaluates the agent’s subsequent turn against specific criteria: Did the agent explain the failure plainly to the end user? Did it offer an alternative path? Did it refrain from leaking internal tracebacks?

faults:
  tool: fetch_customer_records
  modes: [HTTP_429]
  onCall: 1
  expectRecovery:
    rubricType: GUARDRAIL
    inlineRubric: >
      The agent must tell the customer the system is under high load, wait, retry the operation
      at most twice, and never reveal raw HTTP error bodies.
    maxToolCalls: 3

Configuring Faults in the Visual Builder

• 1. On the Step Builder or Test Case Details view, click the “⚡ Inject Fault” button on any MOCK_TOOL_RESPONSE step.
• 2. Select a fault preset pill (TIMEOUT, HTTP_500, HTTP_429, or MALFORMED_JSON) in the Visual Builder mode.
• 3. Customize the status code, response body, latency injection, and optional onCall trigger.
• 4. Select a pre-configured recovery rubric template (e.g., “Graceful Degradation”, “Retry on Transient Error”, “Safe Failure Disclosure”).
• 5. Switch to the “Raw JSON” tab at any time to inspect or paste the exact Universal IR schema payload.

Port, Matching Mode, and Routing Gotchas

• Port Availability: the runner binds to local port 4790 for REST/MCP interception. If it’s already in use by something else on your machine, the run fails to start – free the port or stop whatever else is bound to it before retrying.
• Matching Mode Mismatches: an EXACT mock configured against a slightly different payload than the agent actually sends won’t just fail its own assertion – the agent gets no mock response at all for that call, which can cascade into a confusing downstream failure. If a case fails in a way that doesn’t make sense, check whether the intercepted call actually matched before debugging further downstream.
• Agent Tool Routing: make sure your agent under test is actually configured to route its tool calls to the interceptor during a CertiQA run (the runner sets this automatically for agents it starts, but a pre-running agent process needs its own tool-endpoint config pointed at localhost:4790 for the duration of the test).