AI Root Cause Analysis
Instantly understand why any test failed — root cause category, confidence score, plain-English explanation, and a suggested fix, powered by the LLM provider of your choice.
How it works
When a test fails, click Analyse with AI in the test detail panel. UReport sends the error message, stack trace, step context, and prior run history to your configured LLM. The model returns:
Root Cause Category
One of six categories — Code Defect, Test Flakiness, Environment Issue, Configuration Error, Test Data Issue, or Infrastructure Issue.
Confidence Score
0–100% confidence in the category. Low scores indicate ambiguous failures worth reviewing manually.
Explanation
Plain-English description of what likely went wrong and why, based on the error and execution context.
Suggested Fix
Actionable recommendation — a config change, code fix, or environment check to address the root cause.
Flakiness Signal
One of four values: consistent, intermittent, regression, or new. Signals the failure pattern observed across runs.
Related Patterns
A string array of patterns identified across similar failures in the same suite. Surfaces common error themes for systemic issues.
Results are cached by error fingerprint (a hash of the error message and stack trace). If ten tests share the same failure, only the first triggers an LLM call — the rest receive the cached result instantly at no extra cost.
Caching makes AI analysis cost-effective at scale. A large build with many duplicate failures costs the same as a single LLM call.
Yes — UReport automatically deduplicates sibling analyses. When a test is analysed, UReport checks all other tests in the same build that share the exact same error message and stack trace. Those siblings are immediately marked as analysed with the same result, without triggering additional LLM calls.
This makes AI analysis highly cost-efficient in builds with many duplicate failures. Clicking "Analyse with AI" on one test can resolve dozens of identical failures at once at no extra cost.
When a failing test has steps, the LLM receives a window of steps centred on the failed step: the 5 steps immediately before the failure and the 3 steps immediately after. This gives the model both the lead-up and any cleanup/teardown context without bloating the prompt with the entire test run.
Submitting steps (via your reporter's includeSteps option) significantly improves analysis quality. Without steps, the model only sees the error message and stack trace.
After viewing the AI result, you can:
- Confirm — agree with the root cause category
- Override — select the correct category if the AI was wrong
Feedback is recorded per analysis. Over time, this data lets you track how accurate the AI is for your specific test suite and failure types.
Setup
Go to Settings → AI Analysis
Open the Settings page (admin only) and navigate to the AI Analysis tab.
Choose a provider
Select the LLM provider that fits your infrastructure:
| Provider | Credential required | Notes |
|---|---|---|
| AWS Bedrock (Claude) | AWS access key + secret + region | Recommended for enterprise — stays within your AWS boundary |
| Anthropic Claude API | Anthropic API key | Direct API — good if your team already uses Claude |
| OpenAI | OpenAI API key | Defaults to gpt-4o-mini |
| Google Gemini | Gemini API key | Defaults to gemini-2.0-flash |
| Ollama | None (self-hosted URL) | For local or air-gapped deployments |
Enter credentials and save
Paste the API key (or AWS/Ollama config) and click Save. The setting takes effect immediately — no server restart required.
Only admins can configure the AI provider. The Analyse with AI button is visible to all users once a provider is configured.
Using AI Analysis
Open a failing test
Click on any failing test in the Launches view to open the test detail panel.
Click "Analyse with AI"
The button appears in the detail panel for failed tests. The analysis typically completes in 2–5 seconds depending on your provider.
Review the result
The panel shows the root cause category badge, confidence bar, plain-English explanation, and suggested fix.
Provide feedback (optional)
Confirm or override the category to help track AI accuracy over time.
The Analyse with AI button is only shown for failed tests. Passed and skipped tests do not have an Analyse option.
MCP Server
UReport ships a built-in Model Context Protocol (MCP) server at POST /mcp. Connect any MCP-compatible AI assistant and query your builds, test results, and pass rate trends in plain language — no browser required.
MCP is not Claude-only. Any compatible client works: Claude Desktop, Claude Code, Cursor, Windsurf, Cline, Continue.dev, and others.
Get your API token
Log in to UReport, click your avatar → Edit Profile → API Token → Generate Token.
Add UReport to your MCP config
In your MCP client's configuration file (e.g. .mcp.json for Claude Code, mcp.json in Cursor, etc.), add:
{
"mcpServers": {
"ureport": {
"type": "http",
"url": "http://your-ureport-host:4100/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_TOKEN"
}
}
}
}
Restart your AI client
After saving the config, restart the client. UReport tools will appear in the available tool list.
Replace http://your-ureport-host:4100 with your actual backend URL. In local development this is typically http://localhost:4100.
| Tool | Description |
|---|---|
list_presets | List all saved presets. Call this first — preset names are required by list_builds and get_statistics. |
list_builds | List recent builds matching a saved preset. |
search_builds | List recent builds using explicit filters: product, type, version, browser, platform, team, stage. |
get_statistics | Pass/fail counts and average pass rate for builds matching a saved preset. |
search_statistics | Pass/fail stats using explicit filters (same shape as search_builds). |
get_relation_fields | Discover available tags, components, teams, and custom field keys for a product/type before filtering tests. |
get_tests | Get individual test results for a build. Filters: status, name, tag, component, team, custom field key/value, file, path. |
- "Show me the latest nightly build results"
- "What's the pass rate trend for the PythonAPI preset over the last 20 builds?"
- "Find all failing tests for partner code AC in the latest nightly run"
- "Search for XRAY-768 across all tests"
- "Show me tests tagged with Payment Auth/Capture that failed on Safari"
- "List all builds for product PLAYWRIGHT on STAGE"
- "Which builds have failures today?"
- "Show me all tests that timed out in the last regression run"
The AI chains tool calls automatically — for example, it calls list_presets to get valid preset names, then list_builds to find the latest build, then get_tests to fetch failures — without you specifying IDs manually.
Root Cause Categories
| Category | Description | Common examples |
|---|---|---|
| Code Defect | A bug in the application under test | Null pointer, assertion mismatch, wrong HTTP status code |
| Test Flakiness | The test itself is unreliable | Race condition, hard-coded sleep, non-deterministic test data |
| Environment Issue | Infrastructure or environment problem at runtime | Service unavailable, DB connection failure, out-of-memory |
| Configuration Error | Wrong config value or missing setting | Wrong base URL, missing env var, misconfigured timeout |
| Test Data Issue | Bad or missing test data | Missing fixture, stale seed data, data isolation failure |
| Infrastructure Issue | CI/CD or platform-level problem | Disk full, network partition, container OOM kill |