1.8 Golden Dataset Benchmarking & Latency Percentiles

How to batch evaluate golden JSONL/CSV datasets with rubric scoring, token metrics, and p95 latency percentiles.

Why a Batch Benchmark, Not Just Another Suite

A hand-authored Test Suite is the right tool for a handful of specific, named scenarios. It is the wrong tool when you have swapped a model, tightened a system prompt, or fine-tuned a checkpoint and need to know whether any of the several hundred historical query pairs that used to work still work – writing that many individual Cases, or clicking Run on each one, does not scale. Golden Benchmarking exists for exactly this situation: it takes a whole dataset of prompt/expected pairs and runs every row in one batch pass, then reports aggregate numbers – pass rate, average judge score, cost, and latency – instead of asking you to eyeball hundreds of individual results.

What a Golden Dataset Row Contains

Each row carries an input (the prompt to send), optional expected or reference-context text for the judge to grade against, and either an inline rubric or a reference to a saved judge rubric so you don’t have to retype the same grading criteria on every row of a large file. A row can also carry an optional golden trajectory – the same ordered tool-call shape ({ seq, toolName, args, turnIndex }) used elsewhere in trajectory assertions – for datasets where you care not just what the agent said, but which tools it called to get there.

{"input": "Where is order 48213?", "expected": "Order 48213 shipped and is out for delivery.", "rubricId": "groundedness-support-v2"}
{"input": "Cancel my subscription", "expected": "Confirms cancellation and states the effective end date.", "rubricId": "groundedness-support-v2"}

Upload the dataset as a .jsonl file (one JSON object per line, shown above) or as .csv with matching columns – the format is detected from the file’s first non-whitespace character, so you don’t need to declare it. A JSON array of the same row objects works too, for datasets you’d rather assemble programmatically than hand-edit line by line.

Reading a Benchmark Result

Each row’s result streams in as it finishes rather than waiting for the whole file to complete, so you can watch the pass rate build up live instead of staring at a spinner. Once every row has run, CertiQA computes the summary in one pass: overall pass rate, average judge score, total cost across the batch, and latency percentiles (p50/p90/p95/p99) – the tail percentiles matter here specifically because a prompt change that looks fine on the mean can still blow up p95 for a subset of harder queries, and a benchmark that only reported the average would hide exactly that regression.

Comparing Two Experiments

Every benchmark run is saved as an Experiment against the dataset version it ran on, and the Experiments view (Project Settings -> Agent Under Test -> Experiments tab) lets you diff any two of them: pick a baseline and a candidate and CertiQA classifies every row both experiments actually scored into newly failing, newly passing, or unchanged – the fast way to see exactly what a prompt or model change moved, rather than re-reading two separate pass-rate numbers and guessing at the overlap. Comparing two experiments that ran against different dataset versions is refused rather than silently producing a misleading diff, since the rows being compared may no longer mean the same thing.

Running a Benchmark

• 1. In Project Settings -> Agent Under Test, open the “Experiments” tab (Eval Datasets & Experiments).
• 2. Upload a .jsonl or .csv dataset containing prompt and expected/context columns – or select an existing saved Eval Dataset version if you’ve run against it before.
• 3. Choose the evaluation rubric (or leave rubricId to come from the rows themselves) and click “▶ Run Benchmark Suite”.
• 4. Once it finishes, open the Experiments tab to see the summary stats, or Compare to diff it against a previous run.

Tags and Reporting

• Pass –tag key=value (repeatable) from the CLI to attach metadata to an experiment – model name, prompt version, commit SHA – so filtering and comparing experiments later doesn’t rely on remembering which run was which from its timestamp alone.
• If the API is temporarily unreachable when a benchmark starts, the run still completes locally rather than aborting – you get your pass/fail results either way, just without a saved Experiment to compare against later.