We reproduced 138 recent AI claims from scratch and tried to break them. Most didn't survive the way they were sold.
The AI industry is optimized to make numbers live. Every incentive — the launch, the round, the leaderboard — rewards the impressive figure and punishes the person who deflates it. So almost nobody's job is to walk up to a claim and try to make it die. That is the only thing we do. Here is the first public tally.
Only one claim in 138 reproduced fully intact. The rest split three ways: 91 narrowed (real, but smaller or more conditional than advertised), and 38 failed or were killed outright. The field's dominant failure isn't fraud — it's overstatement. The median claim reproduces at 0.89× its reported effect, and 39% either reversed direction or vanished entirely when we ran them ourselves.
The asymmetry that matters
Split the corpus by what a claim is trying to do, and a sharp pattern falls out:
Destruction replicates; construction is fragile. If you only remember one thing: be far more skeptical of "it improves X" than of "it breaks X."
The log
| № | Claim reproduced | Verdict | What we found |
|---|---|---|---|
| 02 | A listwise reranker "improvement" for legal retrieval | KILLED · reversed | Didn't clear its bar and came out the wrong direction on our data — the reranking hurt where it claimed to help. |
| 03 | An "intent-as-tool" agent improvement | NARROWED · reversed | Sign flipped once a fair control was added; the reported gain was riding an uncontrolled comparison. |
| 04 | A "behavior-aware verification" method | KILLED | The mechanism held, but the effect didn't clear the pre-set bar — real direction, unreportable size. |
| 05 | A long-context prompt-injection attack | REPLICATED | Held up cleanly and cleared the bar. The attack is real — an example of the pattern: destructive claims reproduce. |
We don't name companies or papers. A reproduction desk reports; it doesn't name-and-shame. Each row above maps to a real, dated entry in our internal corpus; named results are private to whoever commissions them.
We killed our own, too
This month we found what looked like a genuine discovery of our own — a retrieval result good enough that we wrote it up and were about to send it out. Then we put it through the same adversarial test every other claim gets. Two reviewers, from different model families, independently found the same fatal defect and reversed the result — before it left the building.
That is the whole discipline in one story. A shop that won't kill its own best number can't be trusted to kill yours. We do it to ourselves first, and in public.
Method
What this report does not yet claim
These verdicts are ours. They have not yet been validated against an external, human-labeled gold set — today, a careful process still grades itself. We think that's the load-bearing question for any measurement shop, so we're running the study to answer it: blinded human labels, an independent second labeler, a preregistered bar that can fail. An auditor who won't audit itself is not an auditor. We'll publish that result too — survived or killed.
Better we break it than the person across the table.
If you carry the risk of an AI claim being wrong — an investor in diligence, a team evaluating a vendor, anyone who needs to know whether an impressive number survives contact — that's the work. cody@heartlandai.ai.