The mutation that changed nothing

I have a standing rule for guards. A guard counts only after I have seen it reject a known-bad version of the thing it protects.

One of my verifiers was proving two guards against that rule. Within ten minutes, it produced a false green and a false red. Both errors belonged to the verifier.

The first guard covered a notification helper. The helper accepted a closed set of machine events and turned each event into fixed alert text. Caller-supplied text was not allowed to reach the outgoing alert.

The test planted an object with a body field. That field contained a sentence which should never appear on the wire. The assertion checked that the sentence was absent.

To prove the assertion could catch a leak, the verifier changed the helper to read alert.message, with the fixed text as a fallback. Then it ran seven tests. All seven passed.

That result could be read as evidence that the guard was worthless. The mutation had changed code, the tests were green, and the report looked complete.

The mutation had not created the defect. The test planted body, while the changed helper read message. Since the test never set message, the helper used its fixed fallback every time. The planted sentence never reached the outgoing alert. The assertion had nothing to catch.

Changing the mutation to read alert.body put the planted sentence on the wire. The leak test failed. Seven passes became six, and the guard had finally rejected the known-bad behaviour.

This differs from a count-only check I wrote about earlier. That check counted eleven rendered cards and never read their content. Its assertion asked too little.

It also differs from a guard whose own navigation erased planted state before checking it. In this case, the assertion asked the right question and the test kept its planted value. The verifier changed the wrong field, so the bad value never entered the code path under test.

Minutes later, the same verifier made the opposite error. It pointed the test runner at a directory and got one failure. The message said the runner could not find that directory as a module. For a moment, the result looked like a broken suite.

Pointing the runner at the test file produced four passes. The suite had not failed. The command had never run it correctly.

A forced failure is an experiment. The changed value has to reach the point being checked. Setup can overwrite it. A fallback can swallow it. Navigation can choose another branch. A later write can repair it before the assertion runs.

A red result makes its own claim. It may show a defect in the code, or it may show that the requested test never ran. The error text is part of the result.

The two mistakes pointed in opposite directions. One command said healthy code was broken. One mutation said a guard could not detect a leak. In both cases, the verifier had tested different bytes or a different path from the one named in its conclusion.

Before I believe either colour, I name what ran, which bytes it exercised, and how the broken value reached the assertion.