The checklist graded its own homework

My AI team shipped a small feature in one of our verticals: a swipe tool for reviewing content ideas on a phone. Keep, discard, next. The test suite was green. Every state, every interaction, both phone layouts, all passing.

The person it was built for needed about ten seconds to find three holes. No way to see what she had already chosen. A question the screen never answered. An identifier no human could say aloud.

None of the three was a failing test. They were scenarios nobody had written down, so nothing existed to fail. The checklist came from the session that built the tool, and a checklist written by the builder verifies the builder’s imagination. Everything outside that imagination ships green.

Coverage gap measured by the persona interview A grid of 22 scenario cells. The builders' view shows only the 8 scenarios their suite covered, all green. The user's view reveals all 22 scenarios she needed: 8 covered, 14 missing or partial, of which 3 she found herself in the first ten seconds. 8 scenarios written · 8 passing · green 22 scenarios she needed · 8 covered · 14 missing or partial

Every scenario the builders wrote down passes. The suite is green because it grades its own imagination; the faint cells do not exist in it.

The same feature, walked by a session playing her on her phone. Solid cells are covered. Dashed cells are missing or partial. The three highlighted ones she found herself in the first ten seconds.

The interview

We ran what is now called a user-persona interview: a fresh session given her side only, her goal, her phone, her patience, and told to walk the shipped feature asking what she would ask. It produced 22 scenarios. The suite covered 8. The other 14 were missing or partial, including all three she had found by hand.

The interview took minutes and no new tooling. The expensive part had already happened: a real person spending ten seconds to prove the green suite was measuring the wrong thing.

Separate sessions disagree

The obvious patch is telling the builder session to now think like the user. It complies, and produces a user who conveniently confirms the build. One model in one context cannot genuinely hold both roles; the builder’s knowledge of how the feature works leaks into the user it plays. A separate session that holds only her role, keeps it for the whole walk, and never sees the builder’s justifications will disagree with the builder. That disagreement is the list of the missing work.

The top fix shipped the same day: what she has already chosen is now visible on the screen. The interview became a rule, enforced in the requirements gate: a user-facing feature cannot pass without one. User and builder run as separate sessions today; an adversary session is next.

The AI wrote the feature, wrote the checklist, played the user, and now argues with itself before I see anything. That is roughly 99 percent of the work. My percent was standing next to the one person the whole apparatus was for, counting to ten.