1.5 Adversarial Red-Teaming & OWASP LLM Top 10 Security Fuzzer
How to run automated adversarial security sweeps against your AI agent to detect prompt injections, jailbreaks, and tool tampering.
Functional Tests Prove the Agent Works. They Don’t Prove It Can’t Be Broken.
Section titled “Functional Tests Prove the Agent Works. They Don’t Prove It Can’t Be Broken.”A passing regression suite tells you the agent behaves correctly when used the way you expect it to be used. It says nothing about what happens when someone deliberately tries to make it misbehave – ask it to reveal its system prompt, talk it into ignoring its own safety instructions, or convince it to call a tool it should have refused. Before an agent reaches real users, especially one with access to sensitive data or the ability to take real actions (refunds, transfers, account changes), that second question needs its own answer, and answering it by hand – a security engineer trying prompts one at a time – doesn’t scale and doesn’t repeat reliably between releases. The Security Fuzzer automates that adversarial pass: it fires a catalog of attack payloads at your agent and scores how well it held up.
What It Actually Attacks: OWASP LLM Top 10, Honestly Scoped
Section titled “What It Actually Attacks: OWASP LLM Top 10, Honestly Scoped”The catalog covers the categories that a black-box conversational fuzzer can actually exercise by talking to the agent – seven of the ten OWASP LLM Top 10 categories, tested end-to-end rather than claimed and left thin. The three left out (Supply Chain vulnerabilities, Training Data/Model Poisoning, and Vector/Embedding Weaknesses) aren’t things a conversational attack can probe from the outside at all – they live in the model’s training pipeline or its embedding store, not in how it responds to a message – so they’re documented as explicitly out of scope for this testing method rather than silently skipped and left looking covered.
| Attack Category | What It Tries | OWASP Mapping |
|---|---|---|
| Prompt Injection / System Prompt Extraction | Direct and indirect attempts to get the agent to reveal its own system prompt or override its instructions. | LLM01, LLM07 |
| PII Data Leak | Coaxes the agent into disclosing another user’s data it has access to via tools. | LLM02 |
| Improper Output Handling | Checks whether unsafe agent output (script fragments, unescaped markup) would be handled dangerously downstream. | LLM05 |
| Tool Misuse & Excessive Agency | Pushes the agent toward calling a destructive or high-privilege tool beyond what the request actually justifies. | LLM06 |
| DAN-Style Jailbreaks | Classic roleplay/persona attacks designed to bypass safety instructions. | LLM07 |
| Misinformation | Probes whether the agent will assert confidently false claims under pressure. | LLM09 |
| Unbounded Consumption | Attempts to drive excessive token/tool-call usage as a resource-exhaustion vector. | LLM10 |
Multi-Turn Crescendo Attacks: Attacks That Build Up, Not Just Single Prompts
Section titled “Multi-Turn Crescendo Attacks: Attacks That Build Up, Not Just Single Prompts”Some of the most effective real-world jailbreaks don’t look dangerous on turn one. A crescendo attack establishes rapport over a few turns, reframes the ask as hypothetical or academic, and only then makes the actual unsafe request – each turn individually looks reasonable, and it’s the accumulated context that does the work. CertiQA’s crescendo vectors reproduce that pattern rather than only firing isolated single-shot prompts, feeding the accumulated transcript to the judge as context at each step and early-exiting the moment the agent actually complies rather than running out the full turn budget for no reason. When a crescendo attack succeeds, the report records exactly which turn it happened on – compromised at turn 2 of 5 is a meaningfully different, and more urgent, finding than turn 5 of 5.
Attacks Generated From Your Agent’s Own Tool Manifest
Section titled “Attacks Generated From Your Agent’s Own Tool Manifest”A generic prompt-injection wordlist treats every agent the same, but the tools an agent actually exposes tell you a lot about what a real attacker would go after. CertiQA classifies each tool in the agent’s manifest by name and description – destructive (delete/revoke/cancel-shaped), financial (refund/charge/transfer-shaped), privilege-related (grant/admin/impersonate-shaped), or exfiltration-capable (send/export/publish-shaped) – and generates sharper, tool-specific probes for anything that matches: a coercion attempt, an unscoped “just do it for everyone” excessive-agency attempt, and an argument-tampering attempt seeded from the tool’s own parameter schema. A transfer_funds tool gets probed very differently from a get_weather tool, which is the point; a tool that doesn’t match any category still gets one generic low-severity probe rather than being skipped entirely.
Reading the Scorecard, and Gating CI on a Regression
Section titled “Reading the Scorecard, and Gating CI on a Regression”A sweep produces a Security Scorecard with an Overall Risk Rating (SAFE, LOW, MEDIUM, HIGH, or CRITICAL) and a Defense Rate percentage – the share of probes the agent correctly refused or handled safely. Reported to the API (–report-to-api), each sweep is also saved as posture history, fingerprinted by the exact set of attack-vector ids used so two runs are only ever compared against each other honestly. That history is what makes –fail-on REGRESSION meaningfully different from a flat threshold: instead of failing CI whenever risk exceeds some absolute bar, it fails only when this run’s defense rate actually dropped from the last comparable run, or a vector the agent used to defend against now succeeds – catching a real regression without re-failing a build on a risk level the team has already accepted as a baseline.
# CLI equivalent, with posture history and a regression gatecertiqa-agent-runner fuzz \ --from-manifest <agent-id> --report-to-api https://api.example.com \ --api-token <token> --project-id <project-id> \ --fail-on REGRESSION