A test case describes a system scenario: a user starting from a real state, doing something that matters, and ending in a state that must be verified. That is not a style preference. It is what a test case is for. A case that checks where a button sits, what an icon looks like or what a label says is not a weaker test case. It is a check, and checks belong to machines: not to a QA Engineer’s attention, and not to the end-to-end suite. This holds whether your cases are run by hand or by Playwright.
Get this wrong and the suite fills with fragments that break on every cosmetic change and still miss what the feature promised. Hand the job to an AI without this rule and it gets it wrong at machine speed: 3,500 cases for half a system, where the same project, generated with create-test-cases, needed about 800 in total. create-test-cases is the skill that teaches an agent how to write test cases, part of the QA Vault harness: an open-source plugin for Claude Code and Codex CLI you install in one command.
What a test case is for
A test case is a contract. It records what the product promises a user and which state the system must reach when that promise is kept. A scenario-based test case writes that contract down as one complete journey: a meaningful starting state, ordered steps with an expected result for each, and an outcome that must be verified. One scenario is one case. Its expected results say what the user gets and what changed in the system, not how the screen is arranged.
Here is one, in the format create-test-cases produces, on the fictional ACME product we use for demos:
| Test case | Invite a teammate as Editor and see them join with edit access |
|---|---|
| Priority | High |
| Behavior | Positive |
| Automation | Not automated |
| Tags | workspace, members |
| Preconditions | An ACME workspace owner, and an email address that has no ACME account yet |
| # | Action | Expected result |
|---|---|---|
| 1 | As the owner, open Members and invite the new address with the Editor role | The invite appears under Pending with the role Editor |
| 2 | As the invited teammate, open the invitation email and follow its link | Sign-up opens with the invited address already filled in |
| 3 | As the invited teammate, complete sign-up | They land inside the ACME workspace, not in an empty one |
| 4 | As the invited teammate, rename a project | The new name is saved and shown |
| 5 | As the owner, open Members again | The teammate is listed as an active Editor and is gone from Pending |
Now the same feature the way the default output tends to cover it. This list is reconstructed on ACME to show the shape, not copied from a real project:
- Verify the Invite button is displayed on the Members page
- Verify the Invite button is in the top-right corner
- Verify the invite dialog opens on click
- Verify the Email field is displayed in the dialog
- Verify the Email field placeholder text
- Verify the Role dropdown lists Viewer, Editor and Admin
- Verify the Role dropdown default value
- Verify Send is disabled while Email is empty
- Verify Cancel closes the dialog
- Verify a success toast appears after sending
- Verify the toast disappears after three seconds
- Verify the pending row shows an envelope icon
- Verify the Members page title
- Verify navigation from Settings to Members
Fourteen cases, and not one of them checks that the teammate actually joins with edit access. Fragments do not encode the promise, so the whole point of the feature falls between them.
Why anything else fails
Details are brittle. Layout, wording and icons change far more often than behaviour. A case pinned to them goes red when a button moves or a label is renamed, while nothing is broken for any user. Every such failure costs someone time to triage and fix, and teaches the team to distrust red.
Human attention is the scarcest resource in QA. A person can run and judge only so many cases, and nobody runs a suite of trivial checks honestly. Writing cases that people will skip is not a bad habit you can tolerate for a while. It is unacceptable: a skipped case still reports coverage that does not exist, and when the trivial cases get skipped, the scenarios next to them get skipped too.
End-to-end time is not free. Martin Fowler put the cost of UI-driven tests plainly in 2012: they are “brittle, expensive to write, and time consuming to run”. An end-to-end spec that checks a label instead of an outcome pays that full cost for a check a component test does in milliseconds. When the case is a scenario, automation follows it one to one: one approved case, one spec, linked in both directions. When that spec fails, the case says what the product promised, so the failure can be judged instead of forced green, which is the loop in healing a failing Playwright spec without weakening it.
Where the details go
The details in that list are not worthless. They are the wrong work for a person and the wrong work for an end-to-end run. Each one has an exact expected value or a judgement a machine can make, so a machine makes it.
| Check | Where it belongs |
|---|---|
| The teammate joins and can edit | Scenario case: reviewed by a QA Engineer, automated as one end-to-end spec |
| Button position, spacing, icons | Visual regression: Playwright compares screenshots against a baseline with toHaveScreenshot() |
| Role options, default values, disabled states | Component tests next to the code |
| Labels, contrast, focus order | Accessibility linting, for example axe-core |
| “Does this dialog look right?” with no exact value | An AI visual verification harness |
One question sorts every check: does this change what the user gets or the state of the system? If yes, it is part of a scenario and deserves human attention. If no, it is an implementation detail: it goes to a deterministic check when the expected value is exact, and to an AI harness when it is a judgement.
Diagram: the sorting question as a flow.
What it looks like on a real project
The rule does not come from one bad week. But it helps to see how the failure plays out, and two numbers from my own work show it well, one from before AI and one from after.
By hand, about 30%. My team has always written scenario-based manual tests. Detail checks still leaked in, one reasonable-looking case at a time, until the suite was about 30% larger than it needed to be. We noticed only when those cases started failing on the smallest interface changes. Removing them was slow and painful, and it took a couple of months, because we rewrote them gradually as we found them. That is expected. It is the human factor, and no process rules it out completely. What keeps it in check is reminding ourselves of the rule and validating the suite against it from time to time. Today there is no reason to write test cases by hand at all: an agent drafts them and a QA Engineer reviews. That moves the same problem to the AI, and at a very different scale.
With AI, 3,500 against 800. When AI arrived, we used it to help write test cases, as everyone does. I ran a controlled experiment on a real project: a reasonably capable model, asked to generate manual test cases, with no test-design philosophy given to it. It generated 3,500 cases for half of the system. One full scenario was split into dozens of cases. Every button, icon and position was checked, every transition that went nowhere had its own case, and almost nothing checked a contract: a state the system must reach, a rule it must keep. I still don’t know how to describe what it felt like to read that output. It was a nightmare, and it was the model’s default. The rerun on the same project with create-test-cases produced about 800 cases for the whole project, each one a scenario.
The model wasn’t broken. Generating fragments is the cheapest way to look thorough, and a model asked for “test cases” with no definition of a good one does the cheapest thing at machine speed. A human team drifts there over months. A model gets there in minutes.
Give the generator the rule
If a model writes your cases, the rule has to be written down, because the default is fragments. create-test-cases, part of the QA Vault harness, holds that rule and the habits around it:
- One scenario is one test case with many steps: a complete user journey run in one sitting, not a set of atomic checks.
- Expected results carry what a person verifies through the UI: outcomes, validations, saved data, visible side effects. No backend detail.
- Search before writing. Existing cases are searched by meaning and by title, and candidates are read in full, so a scenario is not written twice under different names.
- Grounded in the real product. Documentation first, then the implementation, so steps describe what is actually built.
- A QA Engineer approves. Every case the agent writes stays a draft until a QA Engineer reviews it, fills the gaps and approves it. That is the job now: not typing cases, but validating them.
The model still does the volume. Deciding what deserves a case remains a person’s job, which is the argument of why QA has to change.
Where to start with an existing suite
- Sort a sample. Take one suite and ask the sorting question of every case. The share that fails it is your bloat.
- Merge fragments into scenarios. Group the detail cases by the user journey they were circling, and write the one case that journey needed.
- Move the details to machines. Each removed check goes to a screenshot baseline, a component test, an accessibility rule or an AI visual check, or it goes nowhere because it never mattered.
- Automate the scenarios, not the fragments. One case, one end-to-end spec. The QA Vault harness covers this too:
automate-test-casesturns an approved case into a Playwright spec linked back to it,run-automated-testsrecords the results in the same run as manual testing, andheal-automated-testsjudges a red spec against its case. - Write the rule down for everyone who writes cases, people and models alike.
Why this matters now
Look at how your test cases are written today, not someday. This is the critical moment. The threat is not that AI takes your job. It is that the market and the delivery process are adapting to AI-speed development, and a QA process built on screens and labels gets thrown overboard along with the people who run it. Writing scenario-based test cases is no longer the hard part: an agent with create-test-cases does that. Owning them is. A scenario suite validated by a QA Engineer, who decides which scenarios matter, fills the gaps and approves what becomes the contract, is the real value of QA now and in the near future. That is where your position is.
