Impact analysis in software testing is finding every test case whose promised behaviour a change touches. That is a question of meaning, not of words. A case written three years ago never uses the vocabulary of this week’s spec, so a search for the feature’s name finds the obvious cases and quietly misses the rest. A missed case does not fail loudly. It stays out of date, keeps passing, and confirms nothing.

Why keyword search misses the tests that matter

One behaviour is described in three languages. The spec speaks product (“a user can now belong to several workspaces”), the bug report speaks defect (“invite link opens the wrong workspace”), and the test case speaks user (“accept an invitation and land in the team’s projects”). None of them share the words that a title search needs.

So a keyword search returns the cases whose titles happen to match and nothing else. The ones it misses are the expensive ones: an ownership transfer, a deep link into a workspace, a role check on a shared project. They were never tagged with the feature, because when they were written the feature did not exist.

How search by meaning finds them

In QA Vault every test case can be found through two faces. The first is its own text: title, description, preconditions and steps. The second is its semantic profile, written by the agent that authored the case: what the user encounters, the other ways a person would ask for it, and the technical terms involved. A query in product, defect or engineering language scores against both, and whichever face fits better wins. A jargon query such as “least-privilege enforcement” still lands on a case written in plain product language.

Impact analysis then runs as a deliberate search, not a single query:

  1. Describe the behaviour, not the feature name. “Journeys affected when one user can join several workspaces”, not “multi-workspace”. A two-word query finds its target only about half the time.
  2. Read the relevance labels honestly. Every hit is marked strong or related, and the server returns only what it judged relevant. It never bluffs a match, but an empty answer is not proof that nothing is affected either.
  3. Widen from the strongest hits. find_related_cases pulls the neighbours of a case that is certainly affected, and suggest_cases_for_defect starts from a bug instead of a query.
  4. Check the literal side too. A title search catches what is named after the feature; meaning catches what is not. Either one alone misses cases.
  5. Read every candidate in full. A relevance label is a starting point, not a verdict. Only the steps and expected results show whether a case is affected or just shares vocabulary.

search-test-cases is the skill in the QA Vault harness that runs this search: an open-source plugin for Claude Code and Codex CLI you install in one command. The underlying tools are listed with the rest of the QA Vault MCP tools.

Search before you write

The same search guards the other direction. Before an agent creates a case, the authoring skill searches by meaning and by title and reads the matches, so the same scenario is not written twice under different names. A vault without duplicates is also one where impact analysis finds one case per behaviour instead of three contradicting ones.

What it looks like on a real project

On every project I have worked on, finding the cases a change affected by keyword search was routine, and routinely incomplete. The cases it missed surfaced later, as tests that passed against behaviour the product no longer had. That gap is one of the problems that made me start building QA Vault.

What the QA Engineer gets

A list of the cases the change touches, each one read rather than guessed, with the strong and the related ones marked. That list is where the next step starts: bringing those cases in line and selecting the regression test cases for the release.

Where to start

  1. Write the change as behaviour. One or two sentences about what a user can now do or no longer do.
  2. Search by meaning and by title, then widen from the strongest hits.
  3. Read before you decide. Affected means the steps or the expected results no longer describe the product.
  4. Keep the semantic profiles current. When a case’s behaviour changes, its profile is rewritten with it, or search starts finding the old behaviour.

And if five search steps for every change sound exhausting: relax, none of them are yours. An agent runs every query, chases every neighbour and reads every candidate in full, without once asking for a coffee break. You get the list, and the final say.

Try the instrument below: each example asks a question the way a person would, and lands on cases written in other words.