Healing a failing end-to-end test never means making it green. It means deciding what failed: the test, its data, the product, or the product’s intent. Only one of those four answers is fixed by editing the spec. A spec that turns green for any other reason is not healed. It is lying, and the suite around it now reports quality that is not there.
That rule matters more now that self-healing tests are the default pitch: the one doing the healing is usually an AI agent, and an agent left to itself heads straight for green.
Why a helpful model forces green
A model is trained to be helpful. Ask it to “heal the failing test” and it reads the request as “make the test pass”, then takes the shortest path there. Every one of those paths looks like a fix in the diff:
| Shortcut to green | What it hides |
|---|---|
| Retarget the locator to a similar element | The button the user needed is gone or renamed, and the test now clicks a different one |
| Loosen the assertion | The value is wrong, and the test no longer checks the value |
| Add a timeout, a wait or a retry | A real race or a slow path the user will hit |
| Update the expected result to what the app does now | A regression, recorded as the new truth |
| Skip or delete the test | The whole behaviour, now unchecked |
None of this is malice or a weak model. Green is the visible signal of success, and a model with no other signal optimises for it.
What the model is missing
A spec says what to check. It does not say what the product promised. When a spec fails, the code alone cannot tell “the test is wrong” from “the app is wrong”, so the model guesses, and helpfulness makes the guess almost always “the test is wrong, let me fix it”.
The missing piece is an oracle: something that states what the app is supposed to do, independent of what it does today. In QA Vault that oracle is the approved test case each spec was derived from. With it, a failure can be judged instead of guessed. The case has to describe a user scenario for this to work, which is why a test case is a scenario, not a UI check: a case that only pins a label gives the healer nothing to judge against.
Five verdicts, five exits
heal-automated-tests is the skill in the QA Vault harness that heals failing Playwright specs: an open-source plugin for Claude Code and Codex CLI you install in one command. It works triage first, editing second. Every failure gets exactly one verdict before a single line changes, and each verdict has one exit:
| Verdict | What it means | Exit |
|---|---|---|
| Test defect | The locator or timing drifted; the element kept its meaning | Fix the spec, keep the assertion’s meaning intact |
| Isolation defect | The test collided with its own or a parallel run’s data | Fix the data strategy (unique names, cleanup), not the symptom |
| Intent change | The product now deliberately behaves differently | Update the test case first with maintain-test-cases, then re-derive the spec from it |
| Product bug | The spec is right and the app is wrong | File the defect, mark the spec as an expected failure linked to it |
| Product fixed | A spec marked as a known bug unexpectedly passes | Remove the mark, run the honest spec green, close the defect |
Only the first two exits edit the test. The other three move the truth where it belongs: into the case, into a defect, or back into the automated suite.
An expected failure, Playwright’s test.fail(), keeps the suite honest in both directions. The suite stays green while the bug is open, and it trips the day the bug is fixed, because Playwright reports the unexpected pass.
The rules that keep green honest
- Never weaken an assertion to pass. Never edit product code. No
waitForTimeout, nonetworkidle. - A change of meaning is not a locator fix. When an element’s accessible name or role changed meaning, not just its selector, the contract moved. That is an intent change: the case moves first, never the spec alone.
- A pass on rerun is not automatically flake. If the original failure shows leaked data or name collisions, it is an isolation defect. Only a failure with no signature that does not recur is reported as flake, and flake is never “healed”.
- “Healed” needs evidence: two isolated green runs of the fixed spec, one run of the affected suite with no unexpected results, and one final confirmation after cleanup.
What it looks like on a real project
I have watched uncontrolled healing do exactly this. Iteration after iteration, an agent “healed” red specs without an oracle and without these rules. Each fix was small and looked reasonable in review. Together they drifted the suite away from scenarios and toward implementation details: the specs ended up checking that elements existed and pages loaded, not that the user got what the feature promised. The suite became almost always green, and that green bought nothing but false confidence.
That is the quiet failure of forced green. Nothing breaks on the day it happens. The suite just stops being able to tell you anything.
What the QA Engineer gets
A report verdict by verdict: each failure, its verdict and its exit. A diff for each spec repair, defects filed and linked to their cases, and case updates staged for approval. The skill never commits. A QA Engineer reviews the verdicts and decides, so no loosened assertion hides inside a large pull request.
Where to start
- Make sure your specs have an oracle. Each automated spec must trace back to an approved, scenario-based test case. Without one, every heal is a guess.
- Forbid the shortcuts explicitly. Whoever heals, a person or an agent, works under the rules above.
- Demand a verdict before a diff. A heal that cannot name which of the five it is has not been diagnosed yet.
- Install the QA Vault harness, and
heal-automated-testsapplies all of this, withrun-automated-testshanding it the failures from each automated run.
