Mutation Testing

Mutation Testing

20 min3 min readCode playground

Lesson at a glance

Explain how mutation testing grades a test suite, run StrykerJS with Vitest on core logic, and turn survived mutants into the specific tests that were missing.

1How it works
2Running StrykerJS with Vitest
3Reading the survivors
4Hand-picked versus generated mutants
5Where it fits

No video here — just the real thing: a written lesson, runnable examples, a quiz and practice.

The last exercise left you with an uncomfortable fact: coverage said 100%, and the suite caught nothing. You need a measurement that asks the real question, would these tests notice if the code were wrong? Mutation testing answers it by trying. You’ve already been doing it by hand: every exercise in this course ran your tests against “mutants”. A tool does the same thing systematically, for every line.

How it works

The tool makes a mutant: a copy of your code with one small change, the kind of slip a person makes. < becomes <=, + becomes -, Math.min becomes Math.max, a condition becomes true, a block is emptied.
It runs the tests that cover that line against the mutant.
If any test fails, the mutant is killed: your tests would catch that bug. If every test passes, it survived: that bug would ship.

It repeats this for hundreds or thousands of mutants. The mutation score is the percentage killed.

Each mutant is one small deliberate bug; tests either kill it or let it survive

Coverage asks “did this line run?” Mutation testing asks “if this line were wrong, would anyone notice?” The second question is the one you care about.

Running StrykerJS with Vitest

StrykerJS is the established mutation testing tool for JavaScript and TypeScript, with a runner plugin for Vitest:

Terminal

npm install --save-dev @stryker-mutator/core @stryker-mutator/vitest-runner
npx stryker init   # interactive; or write the config yourself

stryker.config.json

{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "testRunner": "vitest",
  "plugins": ["@stryker-mutator/vitest-runner"],
  "mutate": ["src/money/**/*.js", "!src/**/*.test.js"],
  "reporters": ["html", "clear-text", "progress"],
  "thresholds": { "high": 80, "low": 60, "break": 50 },
  "incremental": true
}

npx stryker run first runs your suite once to see which tests cover which lines, then tests every mutant. The HTML report shows your source with each mutant marked killed or survived. thresholds.break fails the run below that score, and incremental reuses results for code and tests that didn’t change since the last run, which matters a lot for speed.

Reading the survivors

The statuses you’ll see most:

Killed: a test failed. Good.
Survived: every covering test passed. A missing or weak assertion.
No coverage: no test even ran that line. Coverage would have told you this too.
Timeout: the mutant caused an infinite loop and the tests timed out. Counted as detected.

Survivors are the gold. For settleUp, a report might say:

Text

[Survived] MethodExpression   src/money/settle-up.js:21
-   const amountCents = Math.min(debtors[d].cents, creditors[c].cents);
+   const amountCents = Math.max(debtors[d].cents, creditors[c].cents);

[Survived] ConditionalExpression   src/money/settle-up.js:4
-   if (sum !== 0) throw new Error('Balances must add up to zero');
+   if (false) throw new Error('Balances must add up to zero');

Each one is a precise instruction. The first says no test uses amounts where the debtor owes a different sum than the creditor is owed, so min and max give the same answer in every test. The second says no test ever passes unbalanced input. Write those two tests and both mutants die.

Hand-picked versus generated mutants

The mutants in this course’s exercises were hand-picked: each one imitates a bug a person really writes, like a swapped argument or a forgotten await. Stryker’s are generated systematically from a fixed set of mutators, typically dozens per function, so it finds gaps you would never think to look for, and also produces some noise. Both are useful. When you review a teammate’s tests, the hand-picked habit is the quick version: ask “what is the most likely bug here, and which test would fail?” When you want a measurement across a whole module, let the tool generate the list.

Where it fits

Mutation testing is slow: hundreds of mutants, each running part of your suite. A few habits keep it practical:

Scope it with mutate to the code where bugs are expensive: money, permissions, parsing. Not UI glue.
Run it incrementally locally, and on a schedule (nightly) or on changed files in CI, rather than on every push of the whole repo.
Use the report in review: a pull request that adds logic and survivors deserves a question.

In the exercise you get a weak settleUp suite and a list of survivors, and you write the tests that kill them.

Key takeaways

A mutation testing tool makes small deliberate bugs (mutants) in your code and reruns the tests; a mutant that makes a test fail is killed, one that doesn’t has survived.
The mutation score (killed mutants as a share of all mutants) measures how many plausible bugs your tests would catch, which coverage can’t.
Each survived mutant points at a missing or weak assertion; read it, then write the test that kills it.
Some mutants are equivalent (they don’t change behaviour) and can’t be killed; ignore them rather than contorting tests.
Mutation testing is slow, so run it on core logic, incrementally or on a schedule, not on every file in every commit.

Practice

Code playground

Kill the surviving mutants

This settleUp suite passes, but Stryker’s report (copied into the starter’s comment) lists three mutants that survived it: Math.min changed to Math.max, the zero-sum check disabled, and the minus sign dropped when recording what a debtor owes. Last month’s code review found a fourth bug the suite also misses: from/to swapped in each transfer.

Write the tests that kill all four, without breaking the suite on the real code. The subject is the settleUp you built in the TDD lesson.

// Returns the transfers that settle a group: [{ from, to, amountCents }].
// balances: { name: cents }, positive = is owed, negative = owes.
// Throws if the balances don't add up to zero.
function settleUp(balances) {
  const entries = Object.entries(balances);
  const sum = entries.reduce((total, [, cents]) => total + cents, 0);
  if (sum !== 0) throw new Error('Balances must add up to zero');

  const debtors = [];
  const creditors = [];
  for (const [name, cents] of entries) {
    if (cents < 0) debtors.push({ name, cents: -cents });
    else if (cents > 0) creditors.push({ name, cents });
  }
  const largestFirst = (a, b) => b.cents - a.cents || a.name.localeCompare(b.name);
  debtors.sort(largestFirst);
  creditors.sort(largestFirst);

  const transfers = [];
  let d = 0;
  let c = 0;
  while (d < debtors.length && c < creditors.length) {
    const amountCents = Math.min(debtors[d].cents, creditors[c].cents);
    transfers.push({ from: debtors[d].name, to: creditors[c].name, amountCents });
    debtors[d].cents -= amountCents;
    creditors[c].cents -= amountCents;
    if (debtors[d].cents === 0) d += 1;
    if (creditors[c].cents === 0) c += 1;
  }
  return transfers;
}
Loading the editor…

// Stryker's report on this suite:
//   [Survived] MethodExpression       Math.min(...) -> Math.max(...)
//   [Survived] ConditionalExpression  if (sum !== 0) -> if (false)
//   [Survived] UnaryOperator          cents: -cents -> cents: +cents
// and a bug from last month's code review:
//   [Survived] from/to swapped in each transfer
describe('settleUp', () => {
  test('returns no transfers when everyone is settled', () => {
    expect(settleUp({ Ana: 0, Ben: 0 })).toEqual([]);
  });

  test('returns one transfer for two people', () => {
    expect(settleUp({ Ana: 500, Ben: -500 })).toHaveLength(1);
  });
});
Loading the editor…

Ctrl/⌘ + Enter to run · Runs in a sandbox in your browser. Nothing leaves your device.

Results

(no output)

Checks

All your tests pass against the real settleUp
Kills the Math.min → Math.max mutant
Kills the mutant with the zero-sum check disabled
Kills the mutant that drops the minus sign for debtors
Kills the mutant that swaps from and to