The situation

A nightly run leaves four Playwright specs red. Each spec was derived from a QA Vault test case and carries a provenance header pointing back to it.

What the agent does

  1. Calls get_test_case for each red spec’s source case and reads the approved expected result.
  2. Reproduces the failure and inspects the page—not the assertion.
  3. Classifies the failure into one of five verdicts:
    • Test defect — the button kept its meaning, only its locator changed. Repair the spec, keep the assertion.
    • Isolation defect — it fails only beside another worker’s data. Fix naming and cleanup; do not add retries.
    • Intent change — the product now deliberately promises something else. Maintain the case first, re-derive the spec second.
    • Product bug — the app violates the approved expectation. File the defect with create_defect; keep the honest assertion.
    • Product fixed — a known-failure spec unexpectedly passes. Remove the marker, close the defect, restore automation.
  4. Applies only the fixes that belong to the test. Everything else is escalated with evidence.

What the human reviews

A verdict per spec, a diff for each repair, and defects already linked to their cases and run results. No “loosened” assertions hidden in a large PR.

Why it works

The test case is the oracle. Because the agent can always retrieve the approved intent, “make it green” stops being the only available move.