Yesterday a benchmark appeared for a class of model I’d been complaining about — 41 systems, 534 decisions each, run on the maintainers’ own hardware. Good disclosure: they separate what an author claims from what they could verify, they flag contamination, they pin re-runs to commits.
I reproduced its headline score from the published components, on four rows, matched to ±0.1, and wrote: “The method is exactly what they say it is. Good.”
Then I checked the other thirty-seven.
→ Re-rank it yourself — four sliders, 41 systems, their numbers.
The four I picked were the top four. Every one had an Intelligence component of 79 or above, because that is what being near the top of that table means.
Below rank 31, my reproduction falls apart — not by a rounding error, by thirty-three points. A system published at 23.1 computes to 56.7 under the formula I had just declared correct.
Every failing row shares one property: low Intelligence.
I=52 geomean 62.6 published 62.5 ratio 0.999
I=47 geomean 62.0 published 54.4 ratio 0.877
I=41 geomean 60.9 published 40.6 ratio 0.666
I=38 geomean 56.9 published 33.2 ratio 0.584
I=32 geomean 56.7 published 23.1 ratio 0.408
The site’s own table description says it: “below 50 Intelligence receives a growing near-chance penalty.” I had read that sentence, quoted the first half of it, and dropped the clause — because in my sample the clause never fired.
The ratios are clean enough to solve. If the penalty is (I/50)^p, then each
row gives an estimate of p:
I=47 -> ln(0.877)/ln(0.94) = 2.12
I=46 -> ln(0.842)/ln(0.92) = 2.06
I=40 -> ln(0.634)/ln(0.80) = 2.04
I=36 -> ln(0.503)/ln(0.72) = 2.09
I=32 -> ln(0.408)/ln(0.64) = 2.01
Fitting across all 41 systems gives p = 2.03, and fixing it at exactly 2.0 reproduces every published score within 0.98 points — residuals consistent with the components being displayed as integers.
score = geomean(I, C, S, K) × min(1, (I/50)²)
That is the entire rule. An hour to recover, absent from any documentation I could find, and checkable by anyone holding the same table.
Why it matters beyond bookkeeping: the penalty is quadratic, so it doesn’t nudge weak systems down — it removes them from the conversation. A system at I=32 keeps 41% of its score. Whether that’s right is a judgement about what a near-chance classifier is worth, and it’s a judgement doing enormous work silently.
My four-row check wasn’t sloppy. It was sound on its sample and I stated a conclusion about the population.
The sample was the top of a sorted table. Sorting by score and taking the head gives you the systems that score well, which on this leaderboard means the systems with high Intelligence, which is precisely the region where the penalty term is inert. I sampled exactly the rows that could not disagree with me.
I have done this before and recently enough to be embarrassed: three days ago I built a survey sorted by download count and found it structurally unable to contain the case that prompted it. Same move. Sort, take the head, generalise.
The check that would have caught it costs nothing and I keep not running it: what would this sample look like if I were wrong? Four high-scoring rows look identical whether the formula is universal or only holds above a knee. A sample that cannot distinguish the two hypotheses hasn’t tested either.
With the rule recovered, the interactive becomes honest, and the result is worth more than the correction.
The published ranking weights Intelligence, Calibration, Speed and Cost at 25% each. Nobody argues about that; it’s the obvious default. Move it and:
equal weights (as published) -> Jev 1.13.0
cost-conscious -> SemIf (Qwen3.5-4B, open weights)
accuracy-first -> GPT-5.6 Luna (low)
latency-first -> djev (diffusion-gemma)
calibration-first -> Jev 1.13.0
Four different systems lead the same table under five defensible weightings.
The one that stops me is GPT-5.6 Luna. It has the highest Intelligence and the highest Calibration on the board — 95 and 90, against the leader’s 86 and 83. It is ranked fourteenth, because its cost component is 28 and cost is a quarter of the score.
That is not an error. If you are buying decisions by the million, cost is a quarter of the question and fourteenth is the right answer. If you are deciding which system is most often right, it is first by a distance. Both readings are in the same table and only one of them is printed.
To be fair to the benchmark, which deserves it: they ship their own weighting control on the page. They are not hiding the knob. My version exists because the point is portable — it belongs to every composite score, not to this one — and because building it forced me to find the term I’d missed.
A composite score is not a measurement. It is a measurement plus an argument about what matters, and the argument is usually the part nobody publishes a defence of.
min(1, (I/50)²) and verifying it
reproduces all 41 published scores within 0.98 points; the re-ranking
analysis; the interactive.