AI for QA is a harness in which agents draft, search, automate and run, and a QA Engineer judges what they produced. Nothing is approved automatically. That is not caution about a young technology. It is what keeps a test suite true to the product once most of its artefacts are no longer written by hand.
The sentence is easy to agree with and easy to get wrong. The wrong version treats the human as a safety net, someone who is there in case the agent fails. The right version treats judgement as the product of the process: the agent produces candidates, and a QA Engineer turns candidates into a contract. A test case is a statement of what the system is supposed to do. Only a person who understands the users can sign it.
How to use AI in software testing without the magic
“AI QA engineer” is a category that vendors sell and I do not believe in. An agent that bug-bashes your product and files what it finds is a useful machine, but nobody owns the result. Who decided which of its two hundred findings matter? Who said this scenario is the one that must hold on release day?
The useful shape is different. The agent and the QA Engineer work on the same artefacts with the same tools: test cases, suites and runs in one vault, reached through an MCP server that the agent calls and the person sees in the UI. The agent does everything that has a known procedure. It reads the product and drafts scenarios. It searches for cases that already cover a behaviour before writing a new one. It turns an approved case into an automated spec, runs the suite and sorts failures into their causes. The QA Engineer does the things that have no procedure: deciding what matters, recognising when a drafted case misses the point, judging whether a failure is the test’s fault or the product’s, and approving which cases become the approved suite the product is checked against.
Everything the agent creates arrives as a proposal. A person reviews it and approves it, and only then does it count. That is the whole method. The rest of this page is why it has to be that way and what it looks like on a real project.
Why an agent left alone makes the suite worse
I say “worse”, not “incomplete”. An unreviewed agent does not merely leave gaps; it degrades the suite over time. Three properties of current language models do this, and none of them is fixed by a better prompt.
It cannot feel what the users need. A model knows the product from its code and its documents. It does not know that the month-end export is the feature the finance team lives by, or that nobody has used the legacy import in a year. Sometimes it guesses priorities well, sometimes it does not, and it cannot tell you which time is which. Priority is exactly the input a QA Engineer carries and the agent lacks.
It optimises for the shortest acceptable path. Models are trained to finish the task with the least effort that still looks like success. Give one a precise skill that says “search existing test cases by meaning and by title, then open each candidate case before calling it a duplicate”, and in a long session it will, at some point, run one search and move on. The skill is good. The model still simplifies it. Somebody has to be watching the output to notice, and to correct it early, before the shortcut becomes the session’s habit.
It repeats whatever pattern already exists. This is the one that compounds. A model is excellent at continuing a pattern it can see. If the vault holds cases the agent itself wrote badly last week, those cases are now the pattern. The next batch copies their shape, their shallowness and their blind spots, with more confidence, because for the model an artefact that exists in the system is an artefact that was accepted. Without a person validating each generation, the depth of understanding in the suite does not stay flat. It regresses, one imitation at a time.
Put the three together and the conclusion is not “the agent needs supervision”. It is that human validation is the only thing that makes the suite converge on the product instead of on the agent’s own earlier output. I wrote about the wider shift behind this in why QA has to change; this page is about the mechanics.
Where the judgement goes
A QA Engineer’s time is the scarce resource, so it goes to decisions, not to typing. In the harness those decisions sit at fixed points of the cycle, and the agent brings each one to the person as a question rather than settling it alone.
| Step | The agent | The QA Engineer decides |
|---|---|---|
| Understand the product | Reads the docs and the implementation, notes behaviour that looks unintended | Whether a flagged behaviour is intended or a bug |
| Find existing test cases | Searches the vault’s test cases by meaning and by title, reads the candidate cases | New test case, update an existing one, or skip |
| Draft | Writes scenarios in the vault’s structure | Which scenarios are right, which miss the point, which are missing |
| Approve | Presents the drafted set with its open questions | Which cases enter the approved suite |
| Automate | Proposes which approved cases can be automated, writes and verifies the specs | Which cases are worth automating |
| Run and triage | Runs the suite, separates flake from failure, gives each failure a cause | Whether the cause is right and what follows from it |
| Maintain | After a product change, finds every case it touches and proposes the updates | Whether each case is really affected, whether the new behaviour is a redesign or a regression, and approves each update |
Diagram: the same cycle as the table, as two lanes. Every arrow from the agent lane ends in a decision on the QA Engineer lane; the two highlighted boxes are the approvals.
Two things about this table matter more than its rows.
The agent never approves. Not “rarely”, not “only for small changes”: the approval step belongs to a person, by design, and the harness is written so that the agent brings its work to that step instead of past it.
The agent is the first pair of eyes, not only a drafter. When it reads an implementation to write cases against it and sees a destructive action without confirmation, it says so, and the person rules. Noticing is the agent’s job. Deciding is the QA Engineer’s.
How this came about
The harness did not start as a philosophy. It started as a few prompts.
Before QA Vault existed I worked with agents in exploratory mode, giving them a prompt or a loose skill and watching what came back. A lot of it was wrong in ways that were easy to miss at a glance. Cases came back with steps and expected results tangled together, or as a paragraph where a table was needed. The search for existing cases was weak: the agent asked a keyword question, got nothing, and wrote a duplicate of a case that was already there under a different title. Each failure became a rule in a skill. The skills needed a shared vault to work against, so the vault came next. The vault needed a procedure for how its artefacts get written, and that procedure is where “the agent drafts, the human decides” stopped being a habit and became the principle.
I mention the history because it is evidence for the thesis, not its source. Every rule in the harness exists because an agent, left to itself, did the simpler thing. The principle would hold even if the history were different: a contract about product behaviour has to be signed by someone who can be asked why.
On a real project
Task Tracker is a small Kanban app I keep as the system under test for QA Vault. Its project in the vault holds eighty-odd approved cases across six suites, from account access to the board’s WIP limits. Here is what one cycle looks like with the harness on it.
As the QA Engineer, I ask for test cases for the invitations flow. The agent reads the project’s documentation and the implementation of invitations, then searches the vault for test cases already covering that behaviour and shows me two candidate cases, one of which really overlaps. I decide to update that one rather than write a second. It proposes a suite structure that follows how the product is organised and asks me to confirm it. Then it drafts: scenarios in the vault’s shape, with a short list of things it saw while reading the code that a QA Engineer would pause at, such as an invitation that can be revoked with no confirmation. I rule on each: this one is intended, that one is a bug and gets filed as a defect instead of becoming an expected result. The approved set lands in the vault. Later, the agent proposes which of those cases are worth automating; I pick four, and it writes and verifies the specs. When a release breaks one, it runs that spec, tells me the locator moved and the behaviour did not, and fixes the spec. Had the behaviour changed, it would have stopped and asked, because then the case itself is in question and that is my call.
Nothing in that cycle was typed by me, and every decision in it was mine. That ratio is the point.
The tool that holds the rule
The harness is qa-vault-skills, an open-source plugin for Claude Code and Codex CLI. Each skill covers one situation of the testing cycle (writing cases, searching, maintaining after a change, automating, running, healing a failing spec) and knows where the QA Engineer stays in control. All of it works through the QA Vault MCP server, so the agent’s search is the same semantic search a person uses in the UI, and the artefacts the agent drafts are the ones the person reviews.
Two other pieces make the judgement cheaper to exercise. Project rules put the facts every test depends on, roles, environments, domain constraints, into the agent’s context before it drafts, so fewer drafts fail for reasons a person has to explain twice. And writing cases as scenarios rather than UI checks keeps what a person reviews at the level where their judgement applies: what the system is supposed to do, not which button is labelled what.
Where to start
Do not start with the install. Start with reading.
Take the last twenty test cases written for your product, by an agent or by a colleague, and read them as a user of the product would. Which of them check something that matters? Which ones look like a copy of the one before? That reading is the judgement everything else depends on, and you already have it.
Then put that judgement to work on the agent’s output using the QA Vault harness. Install it, point it at one feature you know well, and ask it to draft test cases. Reject what deserves rejecting. The first batch will show you how the method works better than any description.
Will QA be replaced by AI?
Not by this.
Nobody is asking a QA Engineer to type faster than a model. The job is to do the one thing a model cannot do: take responsibility for the decision that this is what the product is supposed to do.
Development already runs at agent speed. Testing keeps up when the agent does the volume and a person does the judging. The teams that set this up now decide what “approved” means on their product. The teams that wait end up with a suite nobody ever reviewed: generated, run, and green, with no one who can say whether it checks the right things.
