9.2 Promoting Production Traces to Eval Datasets

How to turn one interesting production trace – a drift-flagged failure, a customer-reported oddity – into a permanent, PII-redacted row in a golden eval dataset with one click.

From “This One Trace Was Wrong” to a Regression Guard

Trace Ingestion & PII Redaction turns a batch of production traces into a whole new test suite. Promotion is the narrower, faster sibling of that: when a single trace – often one an online eval policy just flagged, or one a customer escalated – looks worth locking in as a permanent regression case, promote it directly into an existing golden eval dataset version as a new row, without re-running the batch synthesis flow.

What the Promoted Row Actually Contains

A promoted row takes the same shape as any golden dataset row – an input and an expected or reference-context field a judge or matcher can grade against – but where a hand-authored row’s expected text is someone’s idea of the ideal answer, a promoted row’s is populated from what the agent actually said in the live trace. That makes it a snapshot regression guard by default: the case alerts if the agent’s behavior on that input ever changes, rather than grading against a prescriptive ideal. If you want it to grade for correctness rather than just stability, edit the expected field afterward into what the response should say – promotion gives you a starting point pulled from reality, not a finished graded case.

{
  "input": "Why was my refund only partially processed?",
  "expected": "Refund of $42.10 was applied to the original payment method; the $8.90 shipping fee is non-refundable per the return policy.",
  "sourceTraceId": "trace_88f1c2",
  "promotedFrom": "policy_checkout_drift"
}

sourceTraceId and promotedFrom aren’t graded fields – they’re provenance, so a row in the dataset can always be traced back to the exact production trace and drift policy that surfaced it, which matters when someone later asks why a specific case exists at all.

How to Promote a Trace in App

• 1. From the Online Eval scored-trace list (or the trace’s own detail view), click “Promote”.
• 2. Choose the target eval dataset and version.
• 3. Confirm – the trace’s input and output are extracted and PII-redacted (same redaction rules as trace ingestion: emails, credit cards, SSNs, phone numbers, API keys, JWTs) before being written as the new row.

Two Things Worth Understanding Before You Rely On It

• Each trace can be promoted once – re-promoting an already-promoted trace is rejected rather than silently creating a duplicate row.
• Extraction is best-effort for multi-agent traces: it reads the root agent’s recorded prompt/completion attributes, so a swarm whose framework doesn’t populate those OTel GenAI fields well will promote a thinner row than a single-agent trace would.
• Dataset versions are immutable snapshots, so promoting into a version that’s already been benchmarked doesn’t retroactively change any Experiment that already ran against it – the new row is only picked up starting with the next benchmark run against that version.