8.1 How to Quarantine Flaky Tests in Test Health (`/health`)

How to quarantine unstable or under-investigation test cases so they execute but do not fail CI/CD builds.

Coverage You Don’t Want Blocking a Build

When a test case becomes flaky – an in-progress backend feature, a shared environment with real timing noise, a third-party sandbox that’s occasionally slow – deleting it is the wrong move, because you lose its coverage history and have to start from zero once the underlying issue is fixed. Quarantining it instead keeps the case running and collecting telemetry on every execution, while stopping its failures from blocking CI/CD builds or pull requests until someone actively decides it’s healthy again.

Putting a Flaky Case Into Quarantine

• 1. Go to “Test Health” in the left sidebar (or navigate to /health).
• 2. Locate the flaky test case in the stability scorecard list.
• 3. Click “Quarantine” on the case row in the scorecard or Run Report.
• 4. Enter the quarantine reason and a linked Jira ticket (e.g. “Investigating payment gateway timeout”) – the reason is what shows up next to the case for anyone else who finds it quarantined and wonders why.
• 5. Save or confirm the quarantine dialog.

What to Expect in CI/CD

• During CI execution, a quarantined test still runs and its result is still recorded – quarantine changes what happens to the build, not whether the case executes.
• If a quarantined test fails, the run gate still passes: certiqa-run.sh exits 0 and the failure stays visible in Run Reports and Test Health as quarantined, so the build stays green without hiding it. Only active (non-quarantined) failures exit 1.
• Once the underlying issue is fixed, return to /health or the Run Report and click “Lift” – otherwise a genuinely fixed case keeps silently passing under quarantine instead of counting as a real, gating pass.

Flake Governance for AI Agent Cases

AI agent cases feed the same stability scorecard and auto-quarantine policy as UI cases, but their flake signal is computed differently. A UI case is flagged flaky when Playwright needed a retry within a single run; an agent case has no equivalent within-run retry concept (Repeat Count/pass^k, see Statistical Repeat Evaluations, is a deliberately different, within-run statistic). Instead, an agent case’s flake score is 1 minus its pass rate over its last 10 runs – so a case that alternates pass/fail across recent runs scores high and can cross the same auto-quarantine threshold as a flaky UI case. A case that fails consistently also scores high here (it reads as “recently unreliable” rather than strictly non-deterministic) – if a case is quarantined this way, check whether it is actually broken versus genuinely flaky before re-enabling it.

Run-over-Run Significance

Every run report shows a “vs Previous” column comparing each case’s outcome to its own previous run, using a two-sided Fisher exact test on the two runs’ pass/trial counts (a repeated case’s actual repeatCount trials, or a single pass/fail observation for a non-repeated case). A small swing – say 4/5 passing then 3/5 – is marked “not significant”: the confidence bounds overlap too much to call it a real change. A large swing – 5/5 then 1/5 – is marked as a REGRESSION (or IMPROVEMENT the other way), with the p-value shown on hover. This exists specifically so a noisy 1-repeat difference doesn’t get chased as a regression, while a genuine reliability drop still gets flagged.

Quarantine Buys Time, It Doesn’t Diagnose

Quarantine stops a flaky or broken case from blocking a build – it doesn’t tell you why the case is unreliable. Once something is quarantined, use whichever of the diagnostic playbooks below actually matches the failure shape rather than assuming it’ll fix itself.

• Agent Tool Mismatch Diagnostics — Ranked causes for an agent case failing on tool-call timeouts or argument mismatches.
• LLM Judge Calibration — When an agent case’s instability is really the judge, not the agent.
• UI Selector Drift & Iframe Timing — When a UI case’s instability is a timing race or a real markup change.