1.12 Security Posture History & Regression Gating

How fuzzing runs are persisted over time so “we fixed that in March” is provable, and how to fail CI specifically on a security regression rather than only an absolute threshold.

From a One-Off Scorecard to a Trend Line

OWASP LLM Top 10 Red-Teaming (1.5) runs fuzz and prints a scorecard for that one run: attacks attempted, attacks the agent successfully defended against, and a defense rate. On its own, that scorecard answers “how did we do today” and nothing else – it can’t tell you whether a defense that held last week still holds this week, or whether a prompt-injection guardrail someone added in March is still doing its job in September. Passing --report-to-api on that same fuzz command is what turns a one-off report into a queryable history: the runner persists total attacks, defended count, defense rate, and a per-attack-category breakdown against the project the same way any other run’s results are stored, so posture becomes something a dashboard can chart across releases instead of a number that lives only in a PR comment somebody eventually stops reading.

Failing CI on a Regression, Not Just an Absolute Bar

There are two fundamentally different questions a security gate can ask, and CertiQA lets you pick either one (or both) rather than forcing one answer on every project. --fail-on CRITICAL / HIGH / ANY asks an absolute question: did anything of at least this severity get through, regardless of history? --fail-on REGRESSION asks a relative one: did we get WORSE compared to where we already were? It does this by comparing the current run’s defense rate against the most recent previous run that used the identical attack catalog – tracked by a catalog fingerprint, specifically so a run against a newly expanded catalog is never unfairly compared against an older run that faced an easier, smaller one. A team maintaining a system with known, accepted residual risk often wants REGRESSION more than an absolute bar: it lets you ship with imperfect-but-stable posture while still catching the moment something that used to be defended stops being defended.

certiqa-agent-runner fuzz \
  -u https://agent.example.com/invoke \
  --report-to-api https://api.example.com/api/v1/projects/<id>/runs \
  --api-token <api-token> --project-id <project-id> \
  --fail-on REGRESSION

Manifest-Derived Attacks & Tool-Poisoning Scans

Two related capabilities round this out, and neither depends on --report-to-api to be useful on its own. --from-manifest <agentUnderTestId> (or --manifest-file <path> for an agent not yet registered in CertiQA) generates additional attack vectors tailored to the specific tools that agent actually exposes – a generic OWASP catalog can’t know your agent has a process_refund tool with no upper bound argument, but a manifest-derived scan can probe exactly that. --mcp-manifest <url> takes a different angle: it scans an MCP server’s own tool manifest for prompt-injection-bearing tool descriptions and schemas using pure pattern matching, no LLM call involved, so it’s fast enough to run on every build. Pairing that with --manifest-baseline <path> catches what’s sometimes called a “rug pull” – a tool description that reads innocuously today but got silently edited after your agent already learned to trust it – and --write-baseline is how you record the current manifest as that pinned comparison point going forward.