1.7 Production Trace Ingestion & Automated PII Redaction

How to convert live production failure logs from Langfuse, Arize, or Helicone into automated regression tests with automated PII masking.

Turning a Production Incident Into a Regression Test

When a customer hits a real bug in a live agent, the fastest way to make sure it never ships again is to turn the exact conversation that broke into a permanent CI test case – but the raw session log that captured it usually contains the customer’s own data: order numbers, phone numbers, sometimes a pasted card number if the agent handled billing. Hand-copying that trace into a test file means either leaving that data in your repository or manually scrubbing it turn by turn, which does not scale and is easy to get wrong once. The Trace Synthesizer does both steps – reproduction and redaction – as one deterministic pass.

It accepts raw session exports from Langfuse, Arize Phoenix, and Helicone directly, detecting which of the three shapes you pasted automatically rather than asking you to pick, plus a plain flat JSON array of turns for any other observability tool or hand-assembled log. Whichever format it started from, the output is the same: a deterministic CertiQA test case (SEND_USER_MESSAGE / MOCK_TOOL_RESPONSE / EXPECT_TOOL_CALL steps reconstructed from the trace’s turns and tool calls) ready to save into a suite and run in CI going forward.

Redaction Happens Before the Case Is Written, Not After

The PII scanner runs regex-based detection for the sensitive value classes most likely to show up in a support conversation – credit card numbers, SSNs, phone numbers, and similar credential-shaped strings – and replaces each match with a labeled placeholder before the synthesized case is ever saved. That ordering is deliberate and is the same rule CertiQA applies anywhere a production trace is persisted anywhere in the product: redact first, persist second, never the other way around, so the raw sensitive value never exists in a saved test case even transiently.

Synthesizing a Case From a Raw Trace

• 1. In Project Settings -> Agent Under Test, open the “Online Eval” tab and click “📥 Import traces”.
• 2. Paste the raw JSON session log from Langfuse, Arize Phoenix, Helicone, or a generic OpenTelemetry export – the format is auto-detected, so you don’t need to tell CertiQA which tool it came from.
• 3. Leave “Enable Automatic PII Redaction” checked (it is on by default) unless you have already scrubbed the trace yourself upstream.
• 4. Click “Synthesize Test Case” (or use the CLI synthesis below), review the generated steps, and save into your target suite.

The same operation is available from the CLI for scripted or bulk synthesis, useful when you want to turn a batch of exported incidents into cases without going through the UI one at a time:

certiqa-agent-runner synthesize \
  --input ./production-trace.json \
  --source auto \
  --output ./regression-cases/checkout-refund-bug.yaml

Things Worth Checking Before You Commit the Output

• Regex-based redaction catches the shapes it knows to look for; it is not a general PII classifier. Skim the generated case for anything that looks like it should have been masked and wasn’t, especially free-text fields the customer typed themselves rather than structured data the agent returned.
• If auto-detection guesses the wrong export format for an unusual or heavily customized log, pass –source explicitly (langfuse, arize-phoenix, helicone, or generic) instead of relying on detection.
• A synthesized case is a starting point, not a finished one – tool-call arguments and expected responses come straight from what happened in production, so add or loosen assertions the same way you would for any hand-authored case before treating it as a strict regression gate.