The check that counted

One of my production checks stayed green through an 11.5-hour outage.

The page showed the right number of cards. It showed 11 cards, and the source of truth contained 11 items. The check compared those numbers, printed 11/11, and reported PASS all night.

The cards were stale.

A resident file-watcher daemon writes the data snapshots used by the page. It had been restarted 16 minutes before a change gave it the ability to retire itself when its code changed. That timing left one daemon running from the last generation that could not replace itself. New code landed, but the old process kept publishing snapshots in the previous content shape.

The page was one code generation behind for about 11.5 hours. Its item count remained correct, so my production check saw exactly what it had been written to see.

I had called the check end to end. It read the source data, opened the rendered page, counted both sides, and compared them. Those actions crossed the whole system, but the assertion at the end reduced the result to one number. Eleven stale cards satisfied it as easily as eleven current cards.

The defect was caught the next morning by a new check on a sibling page. That check used the same stale writer, but it inspected a field in the data. The field was an integer. The old snapshot did not contain it, so the comparison failed.

Same writer. Same stale generation. One check stayed green and the other failed because one counted the items and the other read their content.

The two results made the gap precise. The passing check established that every expected item had a rendered card. It said nothing about whether each card matched the current data shape. The sibling check asked about a specific field and exposed the stale snapshot immediately. Both checks reported accurately on the questions they had been given. I had given one of them an inadequate question and treated its answer as a broader production verdict.

The restart was simple. Restarting the daemon moved it onto the version that retires itself when its code changes. That fixed the current outage and removed the condition that had allowed the old process to remain resident.

The production check needed a different repair. I kept the count comparison because missing cards still matter. Then I added a content assertion.

The check now selects one source item that carries a context field. It finds the rendered card for that item and compares the displayed text with the source text verbatim. If they differ, the failure names the item. A wrong card can no longer hide inside a correct total.

Before accepting that repair, I restored the known-bad condition: matching counts with the content missing. The new assertion failed and named the affected item. Only then did I count the check as fixed.

The mistake was mine. I had treated an end-to-end route as end-to-end coverage. The check reached production, read real data, and exercised the rendered page. None of that changed the narrow fact it asserted. Its final question was still only: are there eleven things here?

The green result therefore was not false. My reading of it was. PASS meant that the source and page contained the same number of items. I had allowed it to mean that the page represented those items correctly. The outage sat entirely between those two claims.

That question catches omission. It cannot catch a stale field, a wrong label, or an old content shape while the number of records stays constant. The route a check travels does not determine what it proves. Its assertions do.

A check that counts things can only catch missing things. To catch wrong things, it must read one of them. String presence is not coverage. Count equality is not coverage either.