bf16 might be freezing your weight updates — Lacuna

Fix the rounding and the spikes go away: something people had studied as training dynamics was rounding. That made me wonder what else out there is weird like this.

This is one, and it runs the other way: their rounding bug adds instability to training; this one hides it.

How bf16 affects optimization A floating-point number has two parts that matter here.

The exponent picks a power-of-two interval (1 to 2, 2 to 4, 4 to 8, and so on), and the mantissa picks one of a fixed number of evenly spaced slots inside it. Every interval gets the same number of slots, however wide it is. So the values sit close together near zero and spread out as numbers get bigger: the gap between neighbouring values doubles every time you cross a power of two. Figure 1 shows this with a toy format that has only 4 slots per interval.

bf16 and fp32 both have 8 exponent bits, so they cover the same range of numbers. The difference is the mantissa. fp32 has 23 mantissa bits, so 8,388,608 slots per interval.

bf16 keeps only the top 7, so 128.

A bf16 number is the first half of the fp32 number, rounded.

Its gaps are 65,536 times wider: about 1/128 to 1/256 of the number itself. Now add a small change to a weight, as an optimizer does on every step. The result has to be rounded to a value the format can hold. If the change is less than half a gap, it rounds back to where it started and the update is lost. Adam's step is roughly the learning rate, so at 1e-6 a typical Mistral weight loses its update while a weight eight times smaller keeps it. To see what this does to training, go back to the toy format from Figure 1 and run Adam on two weights, w₁ and w₂.

Every dot is a value the format can hold. Two copies of Adam start from the same point and head for the same minimum. One (orange) keeps its weights in the toy format and rounds after every step. The other (blue) keeps a full-precision copy.

bf16 is the same picture with 128 values per interval instead of 4, so a weight loses any change smaller than roughly 1/512 of its own size.

2.

How bf16 messes with AdamW specifically When parameters are bf16 and there's no fp32 master copy underneath, AdamW has three problems: Weight decay is a no-op.

Decay multiplies each weight by (1 − lr·wd), a relative change of about 1e-9 at these settings. It rounds away on every weight, every step.

Adam's second moment freezes. The moments are stored in the parameter dtype.

At β₂ = 0.999 the 0.1% decay rounds away, so v can never shrink, and once it has grown to about half its true value its increments round away too. This kicks in a few hundred steps in, so it doesn't affect the 100-step run below, but it affects every normal fine-tune.

Updates smaller than half a gap are dropped.

This is the one this article is about. To be clear, these are known issues. Collage (ICML 2024) worked; the Hugging Face post "Dude, Where's My Update?"

(Sep 2026) and a robotics paper from two weeks ago (12 VLA codebases) show it.

What we're exploring is what it does to the training process.

3.

bf16 is acting as a weight mask The survival rule is |Δ| ≥ |w|/512 (roughly). This means a weight can move only if |w| ≲ 512·|Δ|. Adam's step is at most about the learning rate. So at learning rate η, only weights smaller than about 512·η can change, and we can find them statically in pretrained models.

bf16's gaps only change size at powers of two, so the cutoff is 512·η rounded down to a power of two.

At η = 1e-6 that is 2⁻¹¹ ≈ 4.88e-4, not 5.12e-4. Every weight between 4.88e-4 and 9.77e-4 has a half-gap of about 1.9e-6, bigger than the step, so all of them are frozen. RMSNorm gains (around 2.2) never move at any learning rate up to 1e-4.

And it really behaves like a fixed mask during training.

In our rerun of the Quiet-STaR recipe (section 5), about 1 to 3% of weights change on any given step, but the cumulative set of weights that have ever changed saturates at 5.6%.

The same small set gets updated repeatedly.

4.

You can see it in published models Quiet-STaR (Zelikman et al., COLM 2024) teaches a language model to generate internal "thoughts" before predicting each token.

The paper fine-tunes Mistral-7B at learning rate 1e-6, and its released training code does exactly the thing above: bf16 weights, fused torch AdamW, no master copy. The trained checkpoint is on the Hub, so we diffed ezelikman/quietstar-8-ahead against the Mistral-7B-v0.1 it started from.

5.4% of the 7.24B weights changed. Above 5.12e-4, only 110 of about 6.2 billion weights changed, in any tensor, in any layer.

5.12e-4 is 512 × 1e-6, which is where the rough rule puts the cliff. The exact rule (512 × lr rounded down to a power of two) puts it slightly earlier, at 2⁻¹¹ ≈ 4.88e-4, and that is where the drop is: in the narrow bin just past 2⁻¹¹ (up to 5.10e-4) only 0.39% of weights changed, most likely ones sitting exactly on 2⁻¹¹ that stepped down across it. With an fp32 master copy there is no cliff: small moves accumulate, so the fraction that shows up as changed after casting back to bf16 falls off gradually with weight size instead (Figure 7).

Below the cliff, about 40% of weights changed, and they moved a lot: half moved by more than 16 bf16 gaps, with a mean relative change of 36%.

The new parts tell the same story.

The 3-layer mixing head is initialised N(0, 0.02) with a zero output row. Its two hidden layers are still that random matrix, the output row trained to about 3e-6, and its biases sit around 1e-9. The one component that clearly trained is the pair of thought-token embeddings, and that's because the code stores them divided by 100 and multiplies back in the forward pass, a 100× effective learning rate that lands them on the right side of the cliff.

5.

Turns out it's a feature, not a bug We reran the released recipe exactly as shipped (bf16 Mistral-7B, fused AdamW, lr 1e-6, seed 42, the script's own 100 optimizer steps, one 8×H200 node) and alongside it two corrected versions: Shadow: an fp32 copy of the weights that receives the bf16 run's exact clipped gradients through a correct fp32 AdamW. Same gradients, so it isolates what the rounding threw away. fp32 master: the whole recipe rerun with fp32 weights, bf16 autocast for the math, and a correct fused AdamW. Same seed, data and everything else. During the bf16 run, between 97% and 99.7% of each step's proposed updates rounded to zero, and by step 100 it had changed 5.6% of its weights. Evaluated with the paper's own metric and code. The first two columns are a soft score: the probability the model puts on the correct answer tokens, renormalised over the valid answers. The third is perplexity on held-out OpenWebMath with thoughts switched off. GSM8K CommonsenseQA perplexity Mistral-7B base 4.96 35.72 6.04 released bf16 recipe, step 100 7.75 44.25 8.37 shadow: same gradients, correct AdamW 0.26 0.00 1,800,000 fp32 master, same recipe 1.59 7.94 13.6 The bf16 run does what the paper says: both scores go up.

Both corrected versions break. The shadow ends up answering every question with a newline, even though none of its weights is more than one bf16 gap from the base model. A correct optimizer just moved all 7 billion of them a little.

The fp32 run blows up (perplexity 5 × 10¹¹ at step 50, newlines everywhere) and only partly recovers by step 100. So the published recipe is only stable because of the bug.

Learning rate 1e-6 is a setting for an optimizer that discards 99% of its updates.

6. What learning rate did they actually have?

If bf16 at 1e-6 is secretly a correct run at a smaller step, some lower fp32 learning rate might reproduce it.

We did a sweep and found that at 3e-7 a correct optimizer matches the bf16 run's final scores by step 50, within 0.2 points on both tasks, then goes over a cliff (pick "Step 50" in Figure 10 to see it).

At step 100, no correct-optimizer run beat bf16: the closest, at 1e-7, scored 41.99 on CommonsenseQA against bf16's 44.25.

And bf16 got there by moving a completely different set of weights: 5.6% of them, by a lot, rather than all of them by a little.