The guard that could not fail

I have a rule for automated gates: before a gate counts, it has to reject a known-bad version of the thing it protects.

That rule caught a test which was green for the wrong reason.

The feature involved a document reader and saved scroll position. When I leave a document and return later, the reader restores where I was. When the document has been revised, it should start at the top. The old position belongs to the old revision.

The implementation handled that by including the revision identifier in the storage key. Same document and same revision: restore the position. Same document with a new revision: use a different key, find no old position, start at the top.

The feature shipped with four forced failures. Each one changed the implementation in a specific way that should make its guard reject the build. Three did. One stayed green.

The unfiring mutation deleted the revision identifier from the storage key. That should have broken the revised-document case. Without the revision identifier, the new version would look up the old key, find the reader’s stale offset, and jump down the page.

It did not. The test still ended at the top and passed.

At first glance, this result could look like evidence that some other part of the feature was protecting the reader. It was not. The mutation had removed the behavior under test. The guard simply never reached the stale value it was supposed to expose.

The problem was in the test’s own route.

The test opened the document, scrolled down, left it, and later returned through an intermediate screen before opening the revised version. Leaving that intermediate screen saved its own near-zero scroll position. Under the mutated key, that save landed in the same place as the stale document offset and overwrote it.

By the time the test opened the revised document, the evidence of the defect was gone.

The correct build started at the top because the revision produced a new storage key. The mutant started at the top because the test had replaced the old offset with a value near zero. They reached the same visible result through different behavior.

The assertion was accurate. The document was at the top. But the guard could not distinguish the correct implementation from the broken one. That distinction is the entire job.

This is different from asking too little of an assertion. I had another check that compared counts while ignoring content. That check answered its narrow question correctly and missed a stale page because the number of cards had not changed.

Here, the assertion asked the right question. A revised document should start at the top. The failure came earlier: the test setup destroyed the condition that made the broken implementation observable.

The fix did not touch the feature.

The test now seeds a stale offset under both candidate storage-key shapes immediately before the revision step. One includes the revision identifier from the old document. The other is the shortened shape produced by the mutation.

On the correct build, the revised document uses its revision-specific key and ignores the stale value from the earlier revision. It starts at the top.

On the mutant, the revised document uses the shortened key and consumes the stale offset placed there for it. It restores the old position, and the guard fails.

The timing of that seed matters. Placing it immediately before the revision step means the test’s navigation cannot erase it on the way to the assertion. The known-bad state now reaches the exact path the mutation changes.

After the repair, the forced failure fired. The normal build stayed green. Only then did the guard count.

This is why running a mutation is not enough. A changed line of code does not prove that the test exercised the changed behavior. The setup may overwrite the relevant state. Navigation may take a different branch. A later action may repair the damage before the assertion sees it. The mutant can survive without the feature being safe.

When a forced failure does not fire, I do not treat the green result as reassurance. I trace the injected defect through setup, state changes, navigation, and assertion until I can show that the broken value reached the point being checked.

A guard is worth what it has been seen to reject. If the known-bad build passes, diagnose the injection before believing the implementation.