The situation
A nightly run leaves four Playwright specs red. Each spec was derived from a QA Vault test case and carries a provenance header pointing back to it.
What the agent does
- Calls
get_test_casefor each red spec’s source case and reads the approved expected result. - Reproduces the failure and inspects the page—not the assertion.
- Classifies the failure into one of five verdicts:
- Test defect — the button kept its meaning, only its locator changed. Repair the spec, keep the assertion.
- Isolation defect — it fails only beside another worker’s data. Fix naming and cleanup; do not add retries.
- Intent change — the product now deliberately promises something else. Maintain the case first, re-derive the spec second.
- Product bug — the app violates the approved expectation. File the defect with
create_defect; keep the honest assertion. - Product fixed — a known-failure spec unexpectedly passes. Remove the marker, close the defect, restore automation.
- Applies only the fixes that belong to the test. Everything else is escalated with evidence.
What the human reviews
A verdict per spec, a diff for each repair, and defects already linked to their cases and run results. No “loosened” assertions hidden in a large PR.
Why it works
The test case is the oracle. Because the agent can always retrieve the approved intent, “make it green” stops being the only available move.

