The probe that invented a defect in a clean file

The day after the white-noise episode (voice messages shipping as static while every check stayed green, written up separately) my AI audited its own instruments. The best find of the audit lied in the opposite direction from everything else.

A selftest needed one number: the sample rate of the emitted audio file. The code derived its prober by taking the configurable ffmpeg path and string-replacing “ffmpeg” with “ffprobe”. Neat, until the pipeline is pointed at a binary whose name lacks that substring. Then the replacement silently does nothing, the test runs the encoder with the prober’s flags, the encoder answers “Unrecognized option ‘select_streams’” and exits with empty stdout, and Python’s int('' or 0) coerces the emptiness into a sample rate of zero. Zero does not equal 22050. FAILED. The file was perfect.

Engineers investigate failed checks quickly, so a false failure spends real hours. This probe never errored. A path edit that did nothing and a coercion that turned “no answer” into “zero hertz” produced a confident wrong verdict about a clean artifact.

The same day’s ledger holds the other direction, which costs more because nobody investigates an all-clear:

  • A spectral-flatness check, run at the 96000 Hz the files had mistakenly been stored at (an upstream filter resampled internally and nothing restored the rate), scored pure white noise at 0.03, which reads as speech. The content stops near 11 kHz, the empty bins above it crush the geometric mean, and the noisiest file on the machine passes.
  • A survey probe walking the stored files died of ENOBUFS one empty read away from scoring everything clean.
  • Headless Chrome, asked for a 390 px window to check a page at phone width, silently renders at 500. Add the device-scale flag to make it more phone-like and you get 756. The capture then crops the right-hand side, so the reviewing agent looks at the screenshot, sees no overflow, and writes “nothing is clipped”. A false all-clear produced by the act of looking.
  • Meanwhile ffmpeg, which had printed “Invalid PCM packet” about the actually-corrupt audio on every single decode, went unread.

The rate probe now asks the same ffmpeg that wrote the file, reading the rate from its own stream line. The other side of that comparison stays on Python’s wave module. Two independent instruments prevent one shared error from certifying a broken file as clean. An unreadable rate is now its own named failure. The screenshot rig pins the width from inside the page and prints the measured inner width beside every shot.

Nine days later I replaced the site’s black-and-orange palette with the Singapore Government Design System. Dozens of self-contained pages needed conversion. My AI wrote a probe that rendered every page and measured the worst text contrast from the computed colours in the browser.

It found a real defect. The source of the design tokens did not define --sgds-primary-color-default, although the conversion map assigned that token to accent text. The only available primary token was intended for filled surfaces. An undeclared CSS variable fails silently, so page after page used the fill colour for text.

The probe found these two pairs in the dark theme:

#6b4feb text on #2a2a2a   2.70   AA needs 4.5
#6b4feb text on #3b3b3b   2.10   AA needs 4.5

The defect appeared on eyebrows, kickers, section headings, and summaries across thirteen pages. Adding the missing token fixed the source. The same pairs then measured 5.84 and 4.55.

Then the probe lied. It parsed every colour with one regular expression and divided each channel by 255. CSS color-mix() resolves in the browser to color(srgb r g b), whose channels already run from zero to one. The parser divided them again and read every tinted background as nearly black. It reported an error box at 2.94. The correct ratio was 5.13. The AI quoted 2.94 in its report before checking the instrument.

The probe reports the worst pair on each page. A fabricated low ratio can rank above a real failure and hide it. The invented defect cost time. A hidden defect could ship.

The probe now has a --selftest with six hand-computed contrast ratios. It covers rgb(), color(srgb), mixed colours, and the two palette pairs the project depends on. All six pass. The browser import happens only after the selftest, so the repository can run those six cases without a browser installed.

The findings based on plain rgb() pairs reproduce exactly. The parser was wrong about one class of colour and right about the rest. That partial accuracy is why its unchecked verdict looked credible.

The rule at the end of this draft already said that a measuring instrument must report what it actually measured. It sat unpublished for nine days and did not fire when I wrote the next probe. The sentence was prose. The six ratios now run inside the instrument before its verdict can count.

A measuring instrument must prove its operating conditions before it reports a verdict.