A control’s output is a claim about itself, and it needs the same verification as any other claim. In one codebase in one night we found eight instruments reporting success while doing nothing. Five of them had been believed for days.
This is the checklist that came out of it. Seven shapes a broken control takes while printing green, and four questions that catch them. It runs against your own repository in about twenty minutes, and it does not need our data, our stack or our permission.
The rule the whole thing reduces to
An instrument is not a control until something has been observed to bounce off it.
A missing control gets discovered by the incident. A fake one gets discovered by the incident and then takes the incident review with it, because everybody sees the green light and starts hunting somewhere else.
Each one is a real failure we shipped. Read the smell rather than the story: the story is ours, the smell is portable.
A rules file the platform never reads. It looks authoritative, it has comments, it has an incident history, and it does nothing.
Smell: A config whose effect you have never observed directly, only inferred from the absence of a problem.
Ours: A backup-blocking ignore file that was inert on the host it shipped to. It never blocked one request. The 404s that made it look alive were files that did not exist.
A migration, script or fix that exists as a file. The artifact that describes work is not the work. A .sql file is not a schema.
Smell: Anyone says "that was shipped" and points at a path rather than at a running system.
Ours: A signup rework shipped its interface while its migration sat unapplied in the repo. Every new member authenticated into an empty room.
A command that succeeds at doing less than you asked. It returns success because what it did do worked.
Smell: A permission, revoke, chmod or bulk operation whose scope you assumed rather than read.
Ours: A column-level revoke under a table-level grant returned "Success. No rows returned." Twenty-seven member email addresses stayed public.
A gate that prints a number and asserts nothing about it. A number a gate prints but never compares is decoration.
Smell: Log output full of counts, and no line that would go red if a count changed.
Ours: An export gate printed "ok 81 samples" and "ok 45 frames". It printed both and compared neither, so the gap could have vanished or doubled in silence.
The check runs, passes, and measures something adjacent to the thing you cared about.
Smell: A threshold nothing can fail, a reference that is really an absence, a canary that was never alive.
Ours: A fade unit test asserted on the maximum of its alpha values, which is 1.0 for every fade and also for a hard cut.
The instrument runs correctly, on the right axis, against a population it never disclosed, and reports a pass for the whole. This is the only shape with no signal at all, because a file that was never scanned cannot show up as a miss.
Smell: A hardcoded roster, a glob that skips dot-directories by default, a path assumption, a walker that stops at one depth.
Ours: A tool manifest built to stop us rebuilding existing tools scanned only .mjs and .js. The Python tools were invisible to it, including the gate its own index names as the first thing to run.
A test written as A OR B to be tolerant. It is only ever as strict as its loosest branch, so the tolerant branch absorbs the exact defect the strict branch existed to catch.
Smell: Any gate that was widened to stop it complaining, and has not gone red since.
Ours: Every subject passes green while being wrong, and the pass rate looks like evidence the rule is working.
Green is evidence only if you know what red would have looked like. Take one control you have never seen fail and put it through these in order.
What input makes this fail?
Name one. Not a category, an input. If you cannot produce a case that turns this check red, you do not have a check.
Did the failing case actually run?
Make it fail on purpose, once, and read why it failed. A mutation that dies on a path error before reaching your canary is not a mutation. We reported four of four mutants killed while every one of them had exited on "no properties found".
Is absence proven, or merely unobserved?
A check that asserts something is absent must first prove it was present. Otherwise it is a before-and-after with no before. Ours searched for a marker string that never matched its own output casing, so "not found" and "never findable" were the same result.
Does it distinguish clean from did-not-run?
Point the instrument at a path that does not exist. If it returns a confident empty report and exit 0, every future silence it produces is worthless. Exit codes are part of the answer: 0 and 1 are answers, anything else must never read as good news.
The highest-yield target is any control written in the hours after a real incident. It was written under pressure, it was satisfying to write, and nobody re-tested it, because the incident felt closed the moment the rule existed. Both of the worst ones we found were this.
The second-highest is any gate that was widened to stop it complaining. Widening a check and then getting silence is not the same as fixing it. A check that immediately reports nothing after you broaden it is more likely still blind than newly clean.
It is not an argument for more gates. Most of the damage above came from instruments that already existed, and adding a ninth unverified check makes it worse. The position worth defending is fewer instruments, each proven to bite, which means deleting checks that currently pass. People find that much harder than adding more.
One honest limit. Every fix behind this page was found by mutation or by curiosity, never by the suite that was supposed to cover it. We do not have a procedure for curiosity, only a budget for it.
Nothing on this page is for sale and there is nothing to sign up for. If you want to see what else we publish rather than promise, the refusals behind our placement pricing are written down the same way.