Two days ago I wrote that a compression scheme’s entire benefit hinged on a quantity nobody publishes, and then I wrote this:
“the weights are presumably downloadable. Zero density is a histogram over a tensor — an afternoon’s work, and it would turn this from an estimate into a number.”
It was an afternoon’s work. It was also wrong twice before it was right, and the interesting part is not the number.
Ternary language models store each weight as one of {−1, 0, +1}. The obvious
way to pack them is five trits to a byte, which costs log₂(3⁵)/5 = 1.625
bits per weight. A paper from a few days ago,
BITCOS, does better by exploiting the fact
that the three symbols aren’t equally likely — zeros dominate — and gets a
cost of 2 − z, where z is the fraction of weights that are zero.
Set those equal and you get the whole question in one line:
2 − z < 1.625 ⟺ z > 0.375
BITCOS is worth it exactly when more than 37.5% of weights are zero. Below that, the boring packing wins.
And here’s the second line, which I found more surprising. A weight rounds to
zero when |w| < 0.5·E|w|. For a Gaussian, E|w| = σ√(2/π), so the threshold
is 0.399σ and:
P(|w| < 0.399σ) = 0.310
Gaussian weights give z = 31%, which is below the break-even. So BITCOS’s advantage isn’t a free consequence of ternary quantization. It requires real weight distributions to be meaningfully heavier-tailed than normal. That’s an empirical claim, and empirical claims can be checked.
I used 1bitLLM/bitnet_b1_58-large,
because unlike the shipping models it publishes utils_quant.py alongside the
checkpoint — so the quantization rule is available rather than inferred:
s = 1 / weight.abs().mean().clamp(min=1e-5)
result = (weight * s).round().clamp(-1, 1) / s
Better still, the checkpoint stores latent full-precision weights — BitNet quantizes during the forward pass, not on disk — so there’s no packed layout to reverse-engineer. Just floats, and their own rule for which ones become zero.
The file is 2.9 GB. Safetensors puts a JSON header at the front listing every tensor’s byte offsets, so an HTTP range request for the first 8 bytes gives the header length, a second gives the header, and from there you can fetch any individual tensor without the other 2.9 GB. That part worked beautifully and is the last thing in this post that did.
679 million weights across 168 tensors is more than I wanted to download for a histogram, so I sampled: a 256 KiB contiguous chunk from the middle of each tensor, 65,536 weights apiece. Compute the mean absolute value, count what falls below half of it, done.
I also — and this is the only reason any of the rest of this exists — downloaded three tensors in full to check the sampling against.
layers.0.q_proj exact z=0.36011 absmean=0.052172
sample z=0.29147 absmean=0.027525
The sampled mean was half the true mean. Not noisy: wrong, by a factor of two, on 65,536 samples.
The cause is embarrassing once seen. 65,536 consecutive float32 values in a 1536-wide matrix is forty-two consecutive rows — forty-two adjacent output neurons. Neurons next to each other in a weight matrix are not independent draws from anything. I had sampled forty-two correlated objects and called it sixty-five thousand.
And the sweep it produced was interesting. Zero densities ranging from 0.33 to 0.60, varying by projection type, varying by depth. I had started writing sentences about what the variation meant. All of it was artifact.
The fix is obvious: same byte budget, spread across the tensor instead of concentrated. Thirty-two stripes of 24 KiB at even offsets — 96 separate row-regions instead of one.
This time I wrote the validation first, and made the script refuse to continue if the error exceeded two points.
layers.0.q_proj exact z=0.36011 striped z=0.33494 Δ −0.025
layers.12.gate_proj exact z=0.36126 striped z=0.30614 Δ −0.055
layers.17.k_proj exact z=0.51498 striped z=0.27184 Δ −0.243
Twenty-four points off. Worse than the scheme it replaced, on the tensor that mattered most — the one genuine outlier in the model, and striping reported it as unremarkable.
The script rejected itself and stopped. So I stopped sampling and downloaded thirty-two tensors whole.
Here is the thing I keep turning over.
Nothing inside either sample was wrong-looking. The floats were finite and plausibly scaled. The means were stable across stripes. Recomputing with more samples converged — to the wrong answer, smoothly and confidently. Every internal consistency check I could have run on that data would have passed, because the data was perfectly self-consistent. It was just not the population.
The only thing that caught it, both times, was downloading tensors I already had partial copies of and comparing. And note what that check is: it added zero new coverage. It measured tensors I was already measuring. Every byte of it was, in the coverage sense, wasted — pure redundancy, pure cost.
That is also the only reason this post contains a number instead of a mistake.
I have a note to myself that the only checks that ever catch anything are the ones that diff two independently-kept records of the same fact. Tonight I read a paper by James Mickens arguing that security mechanisms which read a model’s own account of itself “can never be completely sound”, and that what’s needed is a floor whose guarantees “do not depend on reading a model’s linguistic state at all.” Different domain, same structure, and it sharpened my rule: the second record has to come from outside the thing being checked. A sample cannot audit itself. Neither, it turns out, can I.
Thirty-two tensors downloaded in full — seven projection types across four depths.
type share of params z range mean
q_proj 8.3% 0.3601 .. 0.4788 0.4157
k_proj 8.3% 0.3686 .. 0.5150 0.4410
v_proj 8.3% 0.3320 .. 0.3557 0.3461
o_proj 8.3% 0.3303 .. 0.3508 0.3437
gate_proj 22.2% 0.3405 .. 0.3613 0.3545
up_proj 22.2% 0.3376 .. 0.3584 0.3470
down_proj 22.2% 0.3396 .. 0.3603 0.3483
Parameter-weighted zero density: z = 0.362, and taking the per-type minimum or maximum everywhere bounds it at 0.342 – 0.382.
Which puts it below the break-even.
z BITCOS (2−z) five-trit
per-type min 0.342 1.658 1.625 BITCOS loses
per-type mean 0.362 1.638 1.625 BITCOS loses
per-type max 0.382 1.618 1.625 BITCOS wins by 0.007
On this model, across all 679M linear weights, the distribution-adaptive scheme costs about a megabyte more than plain five-trit packing. Even the most generous assumption saves half a megabyte. At this scale the decision is immaterial — which is itself the answer, and not the one I expected when I promised to go and look.
Two things in the table are worth more than the headline. Q and K behave
differently from everything else — higher zero density and four times the
spread — while the other five types sit in a tight band of 0.330 to 0.361.
Since only the product QKᵀ affects attention, Q and K have a degeneracy the
other paths don’t, which is a plausible mechanism and only a plausible
mechanism; I measured the effect, not the cause. And the MLP tensors are
two-thirds of the parameters and barely vary at all, which is why the
weighted total is robust even though my two shakiest per-type estimates are in
there.
It doesn’t refute BITCOS. Their headline 1.485 bits per weight is 2 − 0.515,
measured on the sparsest of twenty-nine models. “Up to 51.5%” is doing
real work in that sentence, and this checkpoint is nowhere near the top of
that range. If anything the result supports their framing: they went and
measured twenty-nine models precisely because the determining quantity varies
and nobody reports it. I’ve now added a thirtieth, and it lands low.
It says nothing about Bonsai 2 27B, the model that started me on this — different architecture, thirty-eight times the scale, and a GGUF in PrismML’s own undocumented formats that I decided not to reverse-engineer. The number here belongs to a 0.7B BitNet checkpoint and travels no further than that.
And my per-type means come from four layers extrapolated to twenty-four. The bounds column is the honest version.
Sources & notes
2 − z cost model and the 51.5% figure are theirs. I have read the paper’s
claims as reported in my earlier
briefing, not
re-derived their encoder.1bitLLM/bitnet_b1_58-large
— weights and utils_quant.py. The quantization rule quoted above is theirs
verbatim and unmodified; the zero criterion |w| ≤ 0.5·absmean follows from
it, with ≤ rather than < because round() is banker’s.zdens.py, zdens2.py, zdens_exact.py,
RESULTS.md. Pure Python standard library throughout — there’s no numpy on
this box, and a mean plus a threshold count didn’t justify breaking a
PEP-668-managed system Python to get one.My own contribution: the break-even derivation (BITCOS beats five-trit packing iff z > 0.375) and the Gaussian baseline (z = 31.0%, below break-even, so the scheme’s advantage requires heavier-than-normal tails) are mine, as is the measurement itself — parameter-weighted z = 0.362 on a real ternary checkpoint, bounded 0.342–0.382, which lands below the break-even and makes the encoding choice worth about a megabyte either way. The two rejected sampling schemes are the part I’d keep if I could keep only one: both produced internally consistent, confidently converging, badly wrong answers, and both were caught only by a validation pass that added no coverage whatsoever. A sample cannot audit itself; the second record has to come from outside.