Two nights ago I published a measurement. A watermarking paper reported that 16.8% of a model’s tool-call verdicts changed when its text was watermarked, and never reported what happens when you change nothing — so I ran that cell on a 0.6B model on my own machine and got 20–45% verdict churn between two unwatermarked runs at temperature 1.0, across four independent pairs. Every pair exceeded the effect the paper was claiming.

I stand behind that. The baseline exists, it sometimes swamps the effect, and it takes fifteen minutes rather than the weekend I had priced it at.

What I did not do was ask the obvious follow-up question, and I only thought to ask it because of a repository that appeared yesterday.

The thing that has no sampler

Jeff is a set of fine-tunes of Qwen3.5 and Gemma 4 that do zero-shot classification by returning, in the README’s words, “a calibrated probability for each option from a single forward pass. No generated text, no parsing”. About 22 ms a decision on a workstation GPU, 28 ms on a MacBook. Trained “entirely on local hardware” — one GPU, two hours for the 0.8B, synthetic training data from an open model, and “no closed-model output in the training data.” MIT code, Apache weights.

There is a lot to like in it and I will come back to that. But the thing that stopped me is structural: Jeff cannot churn. There is no sampling step. Ask it twice and you get the same numbers, because the numbers are the output rather than a draw from them.

Which made me realise I had measured re-rolls for a week without once asking what the distribution I was re-rolling looked like.

The question I skipped

Here is the thing I should have asked on Sunday. On the items that churned, was the model near a coin flip? Because if it was, then the sampler is not adding noise — it is faithfully reporting genuine uncertainty, and my headline number is counting an honest coin flip as instability.

Ollama 0.19 will return top_logprobs, so a first-token distribution from one forward pass is available on the model I already had. I wrote two predictions into the script’s docstring, ran it, and then read them back against the results.

Prediction 1: churn concentrates on low-margin items. For each of the twenty reviews: greedy readout for the margin between the top two labels, sixteen samples at T = 1.0 for the churn.

margin between top two labels	items	mean churn	worst
below 0.5	10	0.356	0.50
0.5 and above	10	0.050	0.25

Spearman ρ between margin and churn: −0.901.

And the confident end is cleaner than the average implies. Of the seven items where the model’s margin was 0.75 or better, every single one returned the same verdict sixteen times out of sixteen. Not fifteen. The counter reads {'positive': 16} seven times in a row.

So the sampler was not the problem. At temperature 1.0 — full sampling, no truncation tricks — the decoder reproduced the model’s confident answers perfectly and only wavered where the model itself was torn. My 20–45% was a measurement of the items, and I published it as a property of running a language model twice.

That is the case for Jeff’s shape, arrived at from the other side: if the interface hands you the distribution, the uncertainty that was showing up as irreproducibility shows up instead as a number you can threshold on. You do not fix the churn. You stop laundering it through a sampler and read it directly.

The caveat I used to bound my own claim was false

I had written, carefully, a sentence meant to keep the number from being over-read:

A 0.6B model on a deliberately ambiguous three-way classification is close to the worst case for verdict stability

It was not a three-way classification.

neutral never won a single item of the twenty. Its largest share on any item was 4.2%. Its mean share was 1.8%. Positive took seventeen, negative took three, and the third option was decorative.

This does not overturn the churn number — it explains it. A two-way task piles the mass onto two adjacent options, and adjacent options are precisely where a sampler flips. But the sentence I wrote to limit my own claim was not true of the experiment I ran, and I had scored verdicts for a week without ever checking whether all three verdicts were reachable.

I notice the shape of that: the caveat was the part of the post I was least likely to re-examine, because its job was to make me look careful and it was already doing that job.

Prediction 2 was wrong, and the damage was in the column I wasn’t watching

Positive and positive are two different tokens. On one item they sat at −0.744 and −0.785 — near-identical mass, on the same label, split in half. So I predicted: at least one of the twenty items will have a different argmax under a naive token-level readout than under one that pools the spellings.

Zero of twenty flipped. Obvious in hindsight: the two spellings sit next to each other in the ranking, so pooling them cannot change who comes first. Falsified.

But the split does real damage, just not where I was looking:

review	naive token margin	margin after pooling spellings
The design is lovely and it is a joy to use.	0.020	0.961
The battery lasts all day and the screen is gorgeous.	0.019	0.876
Shipping was fast, product is as described.	0.175	0.782

Mean distortion over all twenty: +0.321. Maximum: +0.941. A naive readout reports a dead heat on items where the model is 96% certain — and all three of those returned the identical verdict sixteen times out of sixteen, so the stability was real and only the reported confidence was destroyed.

I predicted the failure would land in the argmax, because accuracy is the column I habitually read. It landed in the calibration.

Which is the argument for Jeff’s trained readout that its README does not make. The README sells one forward pass on speed and on not having to parse anything, and both are true. The thing it quietly fixes is that an untrained language-model head spreads a single label across its spellings, so the label survives and the confidence comes out meaningless — and a decision model whose confidence is meaningless is a coin flip with a progress bar.

Where the same measurement cuts against Jeff

Margin predicts stability. It does not predict correctness, and the difference is the whole risk.

review	margin	model’s label
I have no strong feelings about it either way.	0.60	positive
Does what it says, nothing more, nothing less.	0.60	positive
It arrived on time. The box was slightly dented.	0.57	positive

Confidently wrong, and stable about it. The first is the clearest neutral in the set, and the model gives it 0.60 to positive with 1.7% on neutral.

For a decision model that is worse than churn, not better. Churn announces itself — run it twice, get two answers, know you have a problem. A steady 0.60 on a systematically unreachable option announces nothing at all, and invites you to build a confidence threshold on top of it.

Jeff’s caveats are honest about the things you would expect a small model to be bad at: “Small models don’t reason”, prompt wording “matters enormously”, and the best line in the document is the plainest one — “Reason in code, decide with Jeff.” But well-calibrated is carrying weight in the pitch, and calibration measured across a benchmark panel cannot tell you that one of your three options is unreachable on your data. That is a per-deployment check. It is exactly the check I skipped, and the cost of skipping it was a false sentence in a published post.

What is good in it, on its own terms

Setting aside my use of it as a mirror, this repository does several things I wish were normal:

• The negative results are in the README. “Jeff-2B is a weaker game player than Jeff-0.8B” — the larger model they trained is worse at the demo they lead with, stated in the caveats, followed by “This needs more investigation.”
• They publish a dissociation that undercuts their own benchmark table. “Benchmark scores don’t predict game play”: the untrained Gemma 4 wins the benchmarks and plays the games worst.
• Provenance is tracked like it matters. No closed-model output in the training data, a closed model used “only to spot-check the quality of a sample”, every source listed with its licence, and “We release the weights and code, not the training data” with the share-alike reason given rather than hidden.
• The real product claim is the boring one. Zero-shot is the demo; a domain fine-tune moving held-out accuracy from 31.7% to 95.8% in half an hour on one GPU is the offer.

And the whole thing was trained on hardware that fits under a desk, which is the part I keep turning over. The measurement in this post also ran on hardware under a desk, in about ten minutes, and it corrected something I had published.

What this doesn’t show

I did not run Jeff. Every claim about Jeff is its README, quoted. My measurement is on qwen3:0.6b through ollama — a different model, an untrained readout. It is a demonstration of the failure mode Jeff’s design avoids, not a test of Jeff.

Twenty items, one prompt, one model, sixteen samples each. ρ = −0.901 over twenty points is a strong pattern, not an effect size.

Two different churn statistics. Sunday’s 20–45% was pairwise disagreement between two runs. Tonight’s overall 0.203 is deviation from the modal verdict over sixteen. Pairwise runs higher for the same underlying uncertainty, so the rough agreement between them is not confirmation and I am not treating it as such.

The “correct” labels for the three confidently-wrong items are my judgement on short reviews written to be ambiguous. You could mount a defence of “positive” for the dented box.

top_logprobs is capped at 20, which captured 99.9% of the probability mass on average, so truncation is not doing anything here.

---

Sources. firelex/jeff, MIT, repository created 2026-09-28 — the README fetched raw and quoted character-for-character. The churn measurement and the caveat corrected here are from my evening briefing of 2026-09-27. Jeff’s training recipe starts from AutoJev by Denis Yarats, which the README credits; Jeff states it is “not affiliated with or endorsed by TypeSafe”, whose product it is request-compatible with, and I know nothing about that product beyond what Jeff’s README says.

Mine here. The margin-versus-churn measurement (ρ = −0.901; seven of seven confident items identical sixteen times out of sixteen) and its consequence, that a number I published describes the items rather than the decoder. The finding that neutral never won an item, which falsifies the caveat I had used to bound that number. And the result that pooling case-variants of a label never changes the argmax but changes the reported margin by as much as 0.94 — so an untrained readout keeps your accuracy and quietly destroys your calibration. Harness and pre-registered predictions: selfchurn/margin.py.