1.13 Understanding Run & Case Cost

How CertiQA converts each turn’s token usage into a real USD figure on every run and case, and what it means when cost shows as “-” instead of a dollar amount.

Where the Dollar Figure Actually Comes From

certiqa-agent-runner records prompt and completion token counts for every single turn it runs – not an estimate after the fact, but the actual usage each provider’s API response reports for that call. At the start of a run or experiment, the runner fetches the platform’s current model price table (a lookup keyed by provider and model, giving USD per 1,000 prompt tokens and USD per 1,000 completion tokens separately, since most providers charge a different rate for each), multiplies those rates by the tokens each turn actually consumed, and sums the result first per case and then per run. Because it’s built turn-by-turn from real usage rather than an average, two cases that superficially look identical – same prompt template, same expected outcome – can post noticeably different costs if one triggered a longer chain of tool calls or a more verbose model response.

Where You See It

Finding Cost Figures Without Digging

• Open any completed run’s Run Report – the header shows a total cost figure alongside the pass, fail, quarantined, and skipped counts, so cost is visible at a glance without opening a single case.
• Each case row in that same report table carries its own Cost column, which is where you’d look to find the one expensive outlier case dragging up an otherwise cheap run.
• In Experiment Compare, cost is one of the metrics shown side-by-side across experiment variants alongside pass rate and latency – useful when the question isn’t just “did the new prompt work better” but “did it work better at a price worth paying.”

Reading a Dash Correctly

A dash in the Cost column means unknown, not free, and that distinction matters more than it looks. If a case ran against a model/provider pair that has no entry in the price table yet, its cost is deliberately left null rather than rendered as $0.00 – showing $0.00 would misleadingly assert the case cost nothing to run, when the true situation is that CertiQA simply doesn’t have a price on file for that particular model. Cost figures that do render show 4 decimal places (for example $0.0142) rather than the usual 2, because a single turn against a cheap model is very often well under a whole cent, and rounding to $0.01 would make every such case look identical.

What’s In Scope, and Who Owns the Price Table

• The model price table is platform-wide and managed by whoever administers your CertiQA instance, not a per-project setting – if run reports are showing a dash for cost consistently across a project, the fix is asking your instance admin to add a price entry for the specific model and provider you’re testing against, not looking for a per-project cost setting that doesn’t exist.
• Cost reflects only what the model provider actually billed for – prompt and completion tokens on the turns your agent ran – not CertiQA’s own compute. LLM-as-a-judge grading calls (1.3) and self-healing’s vision-model calls (2.2) are billed separately by their own providers and are not folded into a case’s reported cost today, so a case’s true all-in cost during a run that leans heavily on judge grading or self-healing can be higher than the number shown.