1.9 Interactive Agent Playground

How to chat with agents live, inspect reasoning tokens (), and export exploratory sessions to permanent automated tests with 1 click.

An Exploratory Sandbox Before Anything Gets Locked Into a Test

Writing an automated Case means committing to specifics up front: an exact user message, a rubric describing what a correct answer looks like, maybe a latency ceiling. Before you’re ready to commit to any of that, you usually need to just talk to the agent – try a few phrasings, see how it handles a follow-up question, notice what tools it reaches for on a case you hadn’t thought of yet. The Playground is that step: a live chat surface with no assertions and no pass/fail, so exploring doesn’t require authoring a single step first. (Each message you send costs 1 credit, and it is given back if your agent fails to answer.)

What the Playground Shows You That a Finished Run Report Doesn’t

Each reply arrives whole, with the time the agent took shown beside it, which surfaces a slow agent immediately, well before it would show up as a failed latency ceiling in an automated Case. (The Playground does not stream token by token.) Every tool call the agent makes is inspectable inline – the exact arguments it sent, not a summary of them – so you can catch an agent calling the wrong tool, or the right tool with a malformed argument, while you’re still in exploration mode rather than after wiring up an EXPECT_TOOL_CALL assertion that then mysteriously fails.

When the underlying model exposes them, reasoning tokens are shown inline too, wrapped in tags the same way the model itself emits them. This matters beyond curiosity: a wrong final answer and a wrong reasoning path are different bugs with different fixes, and if an LLM-as-a-Judge rubric later comes back with a verdict you don’t trust, reading the reasoning tokens for that exact prompt in the Playground is usually the fastest way to tell whether the agent reasoned correctly and phrased the answer badly, or reasoned incorrectly and the judge caught something real.

Turning a Conversation Into a Permanent Test

Once a chat session actually demonstrates something worth protecting against regression – a tricky edge case, a bug you just found, a conversation a customer might plausibly have – exporting it converts the full turn sequence, including every tool call that happened along the way, into a draft AI Agent test case on the Step Canvas. The user turns arrive pre-populated as SEND_USER_MESSAGE steps; you then add the assertions (an LLM judge rubric, an EXPECT_TOOL_CALL, a budget ceiling) that turn “this happened once” into “this must keep happening,” which is faster than re-typing the conversation from scratch on the canvas.

Working in the Playground

• 1. Open Project Settings -> Agent Under Test and choose the “Playground” tab for the registered agent – this points the Playground at the same invoke URL and credentials any automated Case for that agent would use.
• 2. Type a message in the chat prompt and press Enter. Continue the conversation as many turns as you need; nothing here is scored or recorded as a run.
• 3. Click “📥 Export Session as Test Case” once the conversation shows something worth locking in, then attach assertions before saving it into a suite.

It’s a Scratch Pad, Not a Run

• Nothing you do in the Playground appears in Run Reports or affects CI – it exists specifically so exploration has no consequence for your test results, which is also why it’s a reasonable place to try adversarial or edge-case prompts before deciding whether they belong in the OWASP Red-Teaming sweep or an ordinary Case.
• Reasoning tokens are only available when the underlying model actually emits them – not every provider or model exposes a block, so their absence for a given agent is a model-capability fact, not a Playground bug.

Limits and What a Failure Looks Like

• CertiQA calls your agent from its own servers, so the agent’s address must be reachable from the internet. An agent on a private network needs a self-hosted runner (the Playground cannot reach it).
• A reply larger than 2 MB is refused. A turn waits for your agent at most 60 seconds per request (your Agent Under Test turn timeout, capped at 60), and at most 90 seconds in total when mocked tools keep the conversation going.
• When you mock tools, the Playground answers the agent’s tool calls for up to 5 rounds. If the agent is still calling tools after that, the turn stops and says so, instead of showing an empty reply.
• A conversation can hold up to 200 messages and 400,000 characters; start a new one after that.
• If your agent is down, times out, answers with an error status or sends something that is not the expected JSON, the Playground shows the reason in the chat (for example that the agent answered HTTP 500, or did not answer within the timeout) and gives the credit back. Any credential your agent echoes back in an error is removed from what you see.
• The Playground speaks OpenAI-compatible, Anthropic Messages, REST webhook and A2A agents. Mocked tools are answered automatically for OpenAI-compatible agents; for the other three, the tool calls are shown and the mock you set is displayed next to them. If you set a model name on the Agent Under Test, it is sent with the request (Anthropic agents require one).