Lab / / failure
Struck as UNMATCHED, then falsified at 1.24× slower
A "1.29× faster than Meta" claim was produced from an unmatched comparison, quoted as a prize, struck, and then inverted to 1.24× slower on a matched protocol.
Measurement methodology, Negative results, BenchmarkingProteus- Machine
- UNKNOWN — not stated
- Commit
- UNKNOWN
- Original claim
- 1.29× faster than Meta
- Matched-protocol result
- 1.24× slower
This is the most instructive benchmark failure I have, and it is instructive because the sequence is documented rather than narrated.
- A ratio was produced from a comparison whose sides were not matched.
- It was quoted as a prize.
- It was struck within the source material.
- On a matched protocol it inverted: 1.24× slower.
The doctrine document names both failure modes present here — *"arithmetic standing in for a measurement"* and *"a comparison quoted as a prize"*. This is the second one.
Direction matters more here than anywhere
The original claim was a speedup: candidate ÷ baseline, higher is better, and 1.29× meant the candidate was ahead. The matched-protocol result is 1.24× slower, which is the same ratio convention reading below 1.00. Anyone who takes the two numbers as comparable without the words attached will read the inversion backwards.
I am not publishing it as a ratio. It is published as a cautionary entry, which is what the brief asked for — record rather than reconcile.
| Step | Figure | Protocol |
|---|---|---|
| original claim | 1.29× faster | unmatched — struck |
| after matching | 1.24× slower | matched protocol |