Negative controls for AI research

Here is an uncomfortable property of verification systems.

A gate that is working correctly and a gate that has silently stopped working produce the same observable output: everything passes.

If your quote verification broke last Tuesday, what you would see is a research run where no claims were rejected. Which is also what you would see on a run where every claim was genuinely well-sourced. There is no difference in the output.

The only way to tell them apart is to send something through that should fail.

The borrowed idea

This is a negative control, and laboratories have used it for a century.

If you run an assay and it comes back negative, you have learned nothing until you know the assay can come back positive. So you run a known positive alongside it. If the known positive does not light up, the run is invalid and the negative result means nothing.

Software testing has the same idea under a different name. Mutation testing deliberately breaks your code to check that your tests notice. A test suite that passes against broken code is not a test suite, it is a decoration.

Research pipelines that use language models need the same discipline and almost never have it.

How it works in practice

On every scan, for every competitor, a deliberately false claim is planted in the input.

It is constructed to be plausible: the sort of thing a model might produce if it were paraphrasing loosely or filling a gap. Not obviously absurd, because an obviously absurd control tests nothing except whether the gate can spot absurdity.

The gate should reject it, because the quote is not present byte for byte in the cached source.

If the gate ever accepts one, the run fails loudly. Not a warning. Not a log line. The run does not produce a report.

That severity is the point. A control that produces a warning gets ignored inside a month.

What the output looks like

Across a live audit of twelve companies, all twelve planted false claims were rejected, and every competitor showed exactly one rejection.

Exactly one, and that one being the control, is the signature you want. It says two things at once:

The gate is live. It rejected something, so it is capable of rejecting.

Nothing else was rejected. So the rest of the input was clean, and the rejection count is explained entirely by the control rather than by unexplained failures.

If a competitor had shown two rejections, that would be a genuine finding requiring investigation: something in the real evidence failed verification, and I would want to know what and why before the report went anywhere.

Zero rejections would be an alarm. It would mean the control did not fire, which means the gate is not doing anything, which means every other claim in the run is unverified.

Designing a control that actually tests something

Three properties matter, and getting them wrong produces a control that passes reliably while testing nothing.

It has to be plausible. A control that says the competitor’s CEO is a horse tests whether your gate can detect nonsense. Your gate does not need to detect nonsense. It needs to detect the specific failure mode you are worried about, which is a near-miss paraphrase. So the control should be a near-miss paraphrase: a real sentence from the real page, altered slightly.

It has to vary. A fixed control string can be inadvertently special-cased, cached, or learned around. Generate it per run, per competitor, from that competitor’s actual source material.

It has to be attributable. You need to be able to tell, from the rejection log, that the thing rejected was the control and not something real. Otherwise a run with one real failure and one control failure looks identical to a clean run with a control, which defeats the purpose.

Where else this belongs

Anywhere you have an automated check whose failure would be silent.

Deduplication. Inject a known duplicate pair on each run. If the deduplication does not catch it, your lead counts are inflating and nothing about the output will tell you.

Exclusion filters. Inject a known sales solicitation. If the filter does not catch it, solicitations are getting counted as leads.

Data collection. Inject a known day of expected data. If the collector reports success while returning nothing for that day, you have a silent failure that will show up as a dip in a chart three weeks later and get attributed to seasonality.

The pattern is general: any system that only tells you when something is wrong needs something planted that is wrong, on every run. Otherwise silence is ambiguous, and ambiguous silence is the most dangerous state a monitoring system can be in.

The honest limit

Negative controls test whether a gate can reject. They do not test whether the gate is checking the right thing.

Our quote verification rejects every planted false quote reliably. It also, as we found when we audited it adversarially, catches only two of seven fabrication classes, because it verifies quotations and most fabrication does not involve quotation marks.

A control that reliably fires tells you the gate works as designed. It says nothing about whether the design covers the threat.

That distinction is worth holding onto, because a green light from a well-designed control is exactly the kind of reassurance that stops people asking the harder question. We got the green light and then went looking for what the gate could not see, which is where the second, adversarial pass came from.

Controls verify the mechanism. Only adversarial review verifies the coverage. You need both, and the first one makes it easy to forget the second.