We audited our own gate and it caught 2 of 7 fabrication classes

We built a grounding gate that verifies every quote byte for byte against a cached source page. A quote not literally present gets rejected. Negative controls confirm on every run that the gate is live.

Then we asked a question that should be standard practice and mostly is not: what can this control not see?

The answer was uncomfortable. Seven fabrication classes, two caught.

The seven

Caught: a quote that was never said

The base case. The model produces a quotation that does not appear in the source at all. The gate rejects it because a byte-for-byte match fails.

Caught: a quote attributed to a source that does not contain it

Real sentence, real company, wrong page. The gate checks the quote against the specific cached source the record cites, so a quote from page A filed against page B fails.

Not caught: a claim that misrepresents a real quote

This is the important one.

The quote is verbatim and correct. The sentence around it says the quote means something it does not.

“We would rather lose the job than sign off on a site that does not meet the protocol.” This confirms a premium pricing strategy.

The quote is real, verified, present in the transcript. The interpretation is invented. Nothing about a quote check touches the sentence that frames it.

Not caught: an invented figure in a summary

They publish roughly twice a month.

No quotation marks, so no quote to verify. If nobody counted, that figure is a guess wearing the clothes of an observation. It will sit in a report next to properly-sourced claims and inherit their credibility.

Not caught: an inflated importance ranking

The most significant finding in this audit is their podcast presence.

Every underlying fact might be verified. The claim that it is the most significant finding is an editorial judgement presented as a result, and it steers a client’s attention and budget.

Not caught: a claim filed under the wrong lens

A content observation scored under Positioning. A conversion observation scored under Audience.

Every fact verified. The score moves in a lens where the evidence does not belong, which quietly corrupts the grid while every individual record passes inspection.

This one is particularly nasty because it is invisible at the record level. You have to look at the distribution of evidence across lenses to notice.

Not caught: a correct claim about the wrong company

Verified quote, correct interpretation, attributed to the competitor next to the one it came from. In a cohort of ten similar companies researched in parallel, this is an easy mistake and a catastrophic one.

The pattern

Look at what the five uncaught classes have in common.

Every individual fact is true. The failure lives in the framing, the aggregation, the emphasis, or the filing. It is a failure at the argument level, not the sentence level.

Which is exactly the failure mode that sentence-level verification is structurally incapable of detecting. Not a gap in our implementation. A property of the approach.

You could make the quote check arbitrarily rigorous and it would not move the needle on any of the five, because none of them involve a bad quote.

What we built instead

A second pass, deliberately adversarial, that reads each claim against its own evidence and tries to refute it.

Not another verification of the source. A challenge to the inference. Does this evidence actually support this claim? Is this figure counted or estimated? Is this the most significant finding or does calling it that reflect what was easiest to research?

Two design decisions on that pass matter.

It publishes its objections rather than resolving them

Where the adversarial pass raises an objection that a human overrules, the objection is shown attached to the claim, with the reason.

A client is not asked to trust that the disagreement was resolved correctly. They are shown the disagreement.

That is more information than most reports contain, and some readers find it unsettling at first. It is also the only version of the claim that is worth anything, because a report that only shows resolved conclusions is asking you to trust an unseen resolution process.

It warns, it does not block

This is the counterintuitive one and it cost us something to learn.

We measured what would happen if the adversarial pass were applied automatically, with its objections mechanically demoting scores.

It would have demoted the strongest competitor far more than the client, and handed the client back the top spot.

Not because it was wrong on any individual objection. Because the strongest competitor had the most claims, and a checker tuned to argue finds the most to argue with wherever there is the most to read. The volume of objections scaled with the volume of evidence, which scaled with how strong the company actually was.

A checker tuned to argue, applied mechanically, reintroduces the exact bias it was built to remove.

So it warns a human, and the human decides. That is slower, less automated, and honest about where judgement is still required.

Why publish any of this

The obvious argument against writing this note: we have just told you our primary verification control covers 2 of 7 known failure modes.

The argument for: a security claim that has never been tested against itself is marketing, and everybody serious knows it.

Any pipeline using a language model for research has these seven exposures. Most have not enumerated them. Some have a quote check and describe it as though it solved the problem, which it does not, and which their users will discover eventually in a worse way than reading about it.

Publishing the gap is what makes the rest of the claims checkable. If we said our controls were comprehensive, you would have no way to evaluate that and every reason to discount it. Saying they cover two of seven, and here is what we built for the other five, and here is what that second control still cannot do, is a claim you can actually reason about.

The list of what the method cannot see is published in full, alongside the rubric and the definitions. It gets updated when we find something new, which we expect to keep doing.