Test Quarantine
Automatically isolate consistently failing tests so they stop blocking CI and the team can focus on real failures.
What is Test Quarantine?
When a test fails repeatedly across multiple builds according to a configured rule, UReport automatically marks it as quarantined. Quarantined tests are tracked in the Quarantine page and can be surfaced in dashboards and reports so teams know which tests are unreliable.
The key benefit is that quarantine is rule-driven and automatic — no one needs to manually flag flaky tests.
Automatic Detection
Rules evaluate every build and quarantine tests that cross the failure threshold — no manual action needed.
Auto-Resolve
Once a quarantined test passes consecutively for the configured number of builds, it is automatically resolved.
Exempt Support
Exclude known-flaky or irrelevant tests from being re-quarantined. Exemptions can be reversed from the Quarantine page.
| Action | Effect | Re-quarantined? | Use Case |
|---|---|---|---|
| Resolve | Clears the active quarantine flag | Yes — if the test keeps failing in future builds, the rule will re-quarantine it automatically | "I think this is fixed, let's see if it holds" |
| Exempt | Excludes the test from all quarantine rules | No — the evaluator skips exempt tests | "This test is known-flaky and we don't care about it" |
| Un-exempt | Removes the exemption, restoring eligibility for quarantine | Yes — the evaluator will re-quarantine if the test keeps failing | "We fixed it, let the rule evaluate it again" |
Quarantine Rules
Rules are configured per Product/Type in Product Lane Settings under the Quarantine Rules section. Each rule defines when to quarantine and when to auto-resolve.
Threshold
Defines how many failures trigger quarantine.
- Mode: Total — X failures out of the last N builds
- Mode: Consecutive — X failures in a row
- Multiple conditions can be combined (OR logic)
Auto-Resolve
Defines how many consecutive passes clear quarantine.
- Default: 3 consecutive passes
- Configurable per rule
- Evaluated on every new build
Scope & Filter
Limit which builds and tests the rule applies to.
- Scope: version, team, browser, platform, stage
- Name pattern: regex on test UID
- Relation filters: match by component, tag, etc.
Each Product Lane setting has a global Builds window (default 10). This is how many recent builds are evaluated when checking the threshold. Only builds with a pass rate above the global Min Pass Rate are counted — this prevents mass quarantines during environment outages.
Example: builds = 10, min_pass_rate = 70%, threshold = 3 total failures. A test is quarantined only if it failed in 3 or more of the last 10 builds that had ≥70% pass rate overall.
A rule's scope controls which builds it applies to. There are two patterns:
| Rule type | Scope defined | Applies to |
|---|---|---|
| Broad rule | None (all scope fields left empty) | All builds for the product/type. Each build is evaluated against its own scope history — a team=A build compares against team=A history, a Chrome build against Chrome history, etc. |
| Scoped rule | One or more fields set (e.g. browser = Chrome) |
Only builds where those fields match exactly |
Both types can coexist. A test can be quarantined by a broad rule and separately by a scoped rule — each creates an independent quarantine record scoped to the build's context.
Example
You have two rules:
- Rule A — no scope → fires for Chrome, Firefox, and Safari builds
- Rule B — scope:
browser = Chrome→ fires for Chrome builds only
Rule A still fires for all builds, but each browser's consecutive failures are counted independently. A Chrome streak is not broken by a Firefox build running different tests — only Chrome builds contribute to Chrome's failure count.
A test that crosses Rule B's threshold is quarantined only in the Chrome scope.
The qualifying build window is always filtered to match the current build's scope (team, version, browser, etc.). This ensures that builds from different scopes don't interfere with each other's failure counts.
Use broad rules as the default baseline. Use scoped rules to add stricter monitoring for a specific lane (e.g. a browser or environment known to be flakier than others).
The quarantine evaluator runs automatically after every build status calculation. It is fire-and-forget — it never blocks the build result. The sequence is:
Build completes
A build is submitted and its status (pass/fail counts) is calculated.
Rules evaluated
For each enabled quarantine rule, the evaluator checks all failing UIDs in the current build against the historical window.
Quarantine or resolve
Tests crossing the threshold are upserted as quarantined. Already-quarantined tests that now have enough consecutive passes are auto-resolved.
Status Reference
| Badge | Meaning |
|---|---|
| Quarantined | The test is currently quarantined and still failing. It will be skipped or flagged in reports. |
| Exempt | The test has been excluded from quarantine rules. It can be un-exempted from the Quarantine page to restore eligibility. |
| Resolved | The test was quarantined but has since passed enough consecutive builds. It is no longer quarantined but remains in the history. |
Using the Quarantine Page
Navigate to Administration → Quarantined Tests. By default the page shows only active quarantined tests. Use the status toggle to also show resolved and exempt entries.
You can search by UID, product, or type using the search bar, and sort by any column header.
Click on the rule name in the Rule column. A dialog will show the rule's threshold conditions, resolve-after passes, scope, and name pattern filter (if configured).
Active quarantined tests show two action buttons in the Actions column:
- Resolve (checkmark) — Marks the test as resolved. If the test continues failing in future builds, the rule will re-quarantine it automatically.
- Exempt (ban icon) — Excludes the test from all quarantine rules. Use this for known-flaky tests that should not block the team.
- Un-exempt (undo icon) — Shown on exempt tests. Removes the exemption so the test is eligible for quarantine again.
Resolved tests no longer show action buttons. Exempt tests show only the Un-exempt button.