Lab / / failure
A bake-off invalid by construction: N_prop 80 vs 16
The two sides of the SAM3 bake-off did different work per row: N_prop 80 on one, 16 on the other. No ratio from it is published.
Measurement methodology, Negative results, BenchmarkingSAM3- Machine
- UNKNOWN — not stated
- Commit
- UNKNOWN
- N_prop, Weft/ggml (C++) side
- 80
- N_prop, Meta reference side
- 16
The SAM3 bake-off has a problem that no amount of care in the timing fixes. The two sides did different work per row: `N_prop = 80` on the C++ side against `N_prop = 16` on the Meta side.
Every row derived from these CSVs is marked UNMATCHED, and no ratio from them is published in the content package. That is not a stylistic choice. There is no ratio to publish.
Why this is not a tuning problem
`N_prop` is the number of proposals the detector carries through a row. Eighty proposals and sixteen proposals are not two speeds of the same computation; they are two amounts of computation. Comparing them produces a difference in work, not a difference in efficiency.
The rule I applied is explicit: do not call measurements a head-to-head comparison when they were taken on different machines, configurations, commits, models, or protocols. `N_prop` belongs in that list, and it belongs in it for the same reason the machines do.
| Side | N_prop | Consequence |
|---|---|---|
| Weft / ggml, C++ | 80 | different work per row |
| Meta reference | 16 | different work per row |