Arithmetic Without Numbers

Rune article companion What happens inside an LLM when it tries to calculate with nothing but matrices.

The spiral is a simplified picture of a Fourier-style number code: one part of the vector tracks phase around a circle, while another tracks coarse position.

integer 137 phase49.3° cos0.65 sin0.76 coarse13

integer value 0 999

The question A model has no fingers If you learned arithmetic the ordinary human way, you probably learned it with a body. You counted on fingers. You grouped things into piles. You lined digits into columns. You carried a one. Later, perhaps, you used an abacus, graph paper, or a calculator. A language model has none of that. It has matrices. Tokens enter, activations flow, logits come out. And yet, if you ask a modern language model for a greatest common divisor, a multiplication, or a division with remainder, something inside that matrix-only body responds. Working vocabulary Token: one unit the model reads or prints. A token might be a word, part of a word, punctuation, or a chunk of digits. Vector: a list of numbers. A model stores each token's current state as a vector with many dimensions. Activation: the model's temporary internal state while it is processing a token. Readout: a small external model trained to recover a fact from an activation, such as the operation or an operand. Logit: a raw score for a possible next token. Higher logit means the model is more likely to print that token. Layer: one repeated processing step in the transformer. A modern model has many layers, each updating the running state. Residual stream: the main running vector passed from layer to layer, like a shared scratchpad without named variables. Attention: the part of a layer that lets one token position look at information from other positions.

MLP / feed-forward block: the part of each transformer layer that transforms one token position's vector by itself. Attention lets positions exchange information; the MLP then reshapes the local vector, often strengthening, suppressing, or recombining features already present there. Next-token prediction: the training and generation rule for ordinary language models. Given the text so far, the model scores possible next tokens, prints one, then repeats the process. Phase: position around a repeating cycle, like the angle of a hand on a clock. Helix-style number codes use phase-like geometry. GCD / LCM: greatest common divisor and least common multiple. For example, gcd(84, 36) = 12. Rune began with the debugging question behind the jargon: when a language model gives an arithmetic answer, is it recalling a pattern, running something like an algorithm, or merely producing a plausible next token?

The human contrast We learned arithmetic with bodies George Lakoff and Rafael E. Núñez argued in Where Mathematics Comes From that human mathematical ideas are grounded in embodied experience: grouping, moving, measuring, balancing, collecting, and mapping one domain onto another. Whatever one thinks of the full philosophical claim, it is a useful starting point for this story. A transformer has no fingers, no beads, no written columns, and no scratch paper. It has token embeddings, attention, feed-forward networks, residual streams, and matrices. If it learns arithmetic at all, it has to invent a machine-native version of number. Humans also do arithmetic in more than one way. We answer 7 x 8 from a memorized multiplication table. We may divide 963 / 17 by running a written algorithm. We estimate tips with shortcuts. So the first scientific problem was not just "can the model answer?" It was "what kind of answering is this?" A memorized table, a learned shortcut, and a real multi-step calculation can all print the same number.

Residual stream A vector changes as the model reads Before we can ask whether a number is memorized, computed, or merely rendered, we need one more piece of machinery: the model's running state.

Imagine reading What is the gcd of 84 and 36? one token at a time. The model does not create a neat little variable named operand_a. Instead, each token position carries a long vector of numbers. As the prompt passes through the transformer layers, those vectors are updated again and again. Some updates move information across positions: the token for 36 can affect the state near the answer position. Other updates reshape the local state: a direction in the vector may become more gcd-like, more operand-like, or more answer-like. The residual stream is the running scratchpad where those changes accumulate. This is why readouts and patches are possible at all. If the operation and operands leave traces in the residual stream, a small readout may recover them. If a state really matters, a patch may change behavior. If a state is writable, an intervention may guide the model. But those are increasingly strong claims, and the vector itself does not label which claim is true. tiny residual vector layer update: scratch vector + attention + feed-forward + = Each token position carries a vector, shown here as the little bars. A layer first uses attention to gather information from other token positions, then uses a feed-forward network, often called an MLP, to transform that information. The result is added back into the running vector. The important part is not the formula; it is that the scratch vector can contain facts such as "this looks like operand A" or "this is where the answer begins."

Next-token constraint The model emits the answer left to right Humans often compute arithmetic from the rightmost digit inward because carries start at the ones place. A language model has the opposite interface: it must print the first visible answer token before it has printed the later ones. 327 x 48 = 15696 human scratch direction69651 carry, then move left model emission direction A next-token model must commit to the first visible chunk before it emits the later chunks.

That is why answer rendering and exact computation are not the same problem. For 327 x 48, a person can multiply from the ones place, carry intermediate values, and only later write the leftmost digit. The model does not get that luxury when it is generating text. To answer 15696, it first has to choose something like 15, then 696, then stop. That matters for the helix story. As answers get longer, more digit chunks have to be represented, tracked, and emitted in order. The experiments found that these chunk readouts can remain partly readable while becoming less separated from one another. When the geometry gets crowded, the model may still have number-like structure inside, but the visible next-token decision loses resolution. A separate counting experiment made the same pressure visible in a simpler setting. The model saw four consecutive large numbers and had to print the next one. An easy case looks like 314582706123450, 314582706123451, 314582706123452, 314582706123453, ...; the next number is just 314582706123454. Llama's tokenizer split that answer into 3-digit chunks: 314 | 582 | 706 | 123 | 454. The failures appeared at carry boundaries. A deep-carry case has the shape 314582706999996, 314582706999997, 314582706999998, 314582706999999, .... The correct next number is 314582707000000: the tokenizer chunks move from 314 | 582 | 706 | 999 | 999 to 314 | 582 | 707 | 000 | 000.

In the experiment, cases like this collapsed; the best deep-carry cell reached only 18.75% accuracy, and the dominant error was simply repeating 314582706999999. In the long-continuation version, the failure became almost theatrical. Once the model missed a deep carry, 96.88% of deep cases collapsed to a fixed point: it kept re-emitting the last correct number instead of recovering. That does not prove the subtraction helix failed for the same reason. It shows the broader engineering lesson: long numeric continuations can look stable until a carry or token boundary asks the model to coordinate more state than its learned shortcut can handle.

The original dream A just-in-time compiler for arithmetic The tempting product answer is simple: if the model is bad at arithmetic, call a calculator. A parser can read the prompt What is 84 times 37?, translate it into 84 * 37, send that expression to Python, and return the result. Rune was chasing a stricter question. Could we look inside the model and find the calculation it was trying to perform? Could the model's own activations tell us the operation and operands? And if we computed the exact answer, could we put that answer back inside the model so it continued naturally? Use one concrete prompt: What is the gcd of 84 and 36? A normal tool route reads the text, extracts gcd, 84, and 36, and calls a calculator. Rune disallowed that at runtime. The route could see token IDs and internal activation vectors, but not the prompt string, regex matches, hidden operands, operation labels, or the gold answer. Only if the model's internal state supplied gcd, 84, 36 could Python compute 12. That is different from standard tool use. PAL, Program-of-Thoughts, ReAct, Toolformer-style systems, and ordinary function calling already make external computation available. Rune was not trying to beat that path on product simplicity. It was asking whether the arguments to the tool could come from the model's own hidden state rather than from the prompt text. That distinction is easy to miss, so here is the plain version.