Skip to content
CertiQA LogoCertiQA LogoCertiQA Docs
CertiQA is in early access. Request access

5.5 ParetoOps Quality & Spend CI Gate

Gate a prompt or model change on pass rate, cost per successful task, latency and Pareto-optimality by calling the experiment gate endpoint from any CI system.

Beyond Pass/Fail

Section titled “Beyond Pass/Fail”

A traditional CI gate only asks whether tests passed. For an AI agent that is not enough: a cheaper model that still meets your accuracy bar, or a prompt change that quietly doubles cost per task, should also decide whether a change can merge. The experiment gate evaluates one experiment (see 1.8 Golden Benchmarking) against a policy you set: a minimum pass rate, an optional ceiling on cost per successful task, an optional p95 latency ceiling, and whether the configuration must sit on the Pareto frontier of your other experiments on the same dataset version.

Calling the Gate From CI

Section titled “Calling the Gate From CI”

The gate is one REST call, so it works from any CI system with curl and jq. Run your benchmark first so the experiment exists, then evaluate it:

Terminal window
resp=$(curl -fsS -X POST   "https://api.certiqa.app/api/v1/projects/$CERTIQA_PROJECT_ID/experiments/$CERTIQA_EXPERIMENT_ID/gate?minPassRate=0.92&maxCostPerSuccess=0.015&maxP95LatencyMs=4000&requireOptimal=true"   -H "Authorization: Bearer $CERTIQA_API_TOKEN")
echo "$resp" | jq -r '.response.summary'
[ "$(echo "$resp" | jq -r '.response.passed')" = "true" ] || exit 1
Query parameter Default Meaning
minPassRate 0.90 Minimum pass rate (0 to 1). The gate fails below it.
maxCostPerSuccess not checked Ceiling for cost per successful task, in USD.
maxP95LatencyMs not checked Ceiling for p95 latency in milliseconds.
requireOptimal true When true, the gate fails if another experiment on the same dataset version beats this one on both cost and accuracy.

The response carries passed (true or false), a one-line summary, a checks object with each policy test and its actual value, and the metrics used (pass rate, cost per successful task, p95 latency, whether the configuration is Pareto-optimal). The call itself returns normally even when the gate blocks, so your pipeline must read passed and fail its own step, as shown above.