Pangram verdict · v3.3
We believe that this document is fully human-written
AI likelihood · overall
HumanArticle text · 1,815 words · 6 segments analyzed
TurboQuant: A First-Principles Walkthrough Compressing AI vectors to 2–4 bits per numberwithout losing accuracy. Modern language models store large tables of high-dimensional vectors: KV caches, embeddings, attention keys. TurboQuant compresses each coordinate of these vectors to 2–4 bits with provably near-optimal distortion, no memory overhead for scale factors, and no training or calibration. This page explains how it works. The single load-bearing idea: in high dimensions, a random rotation turns every input vector into one whose coordinates follow a known fixed distribution. A codebook designed once for that distribution can then be reused for every input. Everything else on this page is the construction that puts this observation to work. §0 · Primer: jargon decoder Eight ideas the rest of the page is built on. Each mini-demo below covers one concept used later. Skip the ones you already know. §0.1 · VectorA list of numbers. An arrow in space. A vector is an ordered list: [0.3, −1.2]. Geometrically it is an arrow from the origin. A d-dimensional vector is an arrow in $d$-space, hard to picture past 3-D, but the rules are the same. ↕ drag tipcoords[0.70, 0.50] length0.86§0.2 · Length ‖x‖ & Inner Product ⟨x,y⟩How much one vector points along another.Length = $\sqrt{x_1^2+x_2^2+\dots}$. Inner product $\langle x,y\rangle = x_1 y_1 + x_2 y_2 + \dots = \|x\|\|y\|\cos\theta$. The inner product reaches its largest positive value when the two arrows point in the same direction. It drops to zero when the two arrows are perpendicular. It becomes negative when the arrows point in opposite directions, with its most negative value when they point exactly opposite. ↕ drag either tip‖x‖1.00 ‖y‖1.00 ⟨x,y⟩0.00 angle90°§0.3 · Mean Squared ErrorWhy we square the mistake. Error is the distance between a guess and the truth. Scoring a guess by the signed error lets positive and negative errors cancel, which means the score does not penalise being off.
Squaring forces every error to count as a positive number and gives big errors a larger penalty than small ones. The guess that minimises the mean of squared errors is the data’s average: it is the unique number that minimises the sum of squared distances to the points. The average has a property we will use in §7. It lies between the data’s most extreme points, so its magnitude is smaller than at least one of them. When a quantizer compresses a whole bin of values down to the bin’s average, the stored value is smaller in magnitude than the bin’s largest values. The reconstruction is a shrunken version of the input. An inner product against a shrunken reconstruction comes out smaller than the same inner product against the input. Guess0.00mean of data0.00 MSE at guess1.00 MSE at mean1.00§0.4 · Unbiased vs Biased EstimatorNoisy is fine. Systematically off is not. An estimator is a procedure that takes data and returns a guess $\hat\theta$ for an unknown truth $\theta$. Repeat it on fresh data and the guesses form a cloud. The cloud can fail in two independent ways. Variance is one: individual guesses are noisy. Bias is the other: the procedure is wrong even after averaging many guesses. An estimator with $\mathbb{E}[\hat\theta]=\theta$ is unbiased; the cloud’s centre sits at $\theta$ regardless of the cloud’s width. The bullseye below shows both failure modes. Bias is the distance from the cloud’s centre to the crosshair. Variance is the width of the cloud. The two quantities are independent of each other. §7 runs the same bullseye against the MSE quantizer of §6, and the cloud’s centre lands away from the crosshair. §8 runs it against a different estimator whose cloud centres on the crosshair. Mode shots0 mean of shots– bias–§0.5 · RotationA rigid spin. Preserves lengths and angles. A rotation matrix $R$ spins space. The key property: $\|Rx\|=\|x\|$ and $\langle Rx,Ry\rangle=\langle x,y\rangle$. Rotation only changes the basis the coordinates are written in, not the geometry.
↕ drag tipAngle θ0°‖x‖ before → after1.41 → 1.41 preserved?yes§0.6 · Where bell-curves come from (CLT)Add up many small randoms → Gaussian. The Central Limit Theorem says that summing enough independent random numbers produces a distribution close to a bell curve. The shape of each individual term in the sum does not affect the limit. A sum of coin flips converges to the same Gaussian shape as a sum of uniform draws or a sum of skewed draws. A rotated coordinate is one of these sums: it is a weighted combination of every coordinate of the original vector, with random weights. After a random rotation, each new coordinate is therefore approximately Gaussian, which is the property TurboQuant relies on for every input. Terms in sum n1Source source shape±1 coin converged?no, n too small§0.7 · Life in many dimensionsCoordinates of a random unit vector are all small. Pick a random point on a unit sphere in $d$ dimensions. In 2-D any coordinate is possible. In 100-D, almost every coordinate is close to $\pm 1/\sqrt{d}$. This is measure concentration, and it is the core fact TurboQuant exploits. Dim d2std of x₁≈ 0.71 1/√d0.71§0.8 · Quantization, in one dimensionSnap every number to the nearest of $2^b$ levels. This is what $b$ bits per number means. With $b=2$ you get 4 levels, $b=3$ gives 8. The gap between levels is your worst-case error. Adding one bit halves the gap, so the squared error drops by 4× per bit, the $4^{-b}$ factor that shows up later. Bits b2levels4 gap Δ0.667 max error0.333 ■ CHEAT SHEETEight ideas, one sentence eachVector: ordered list of numbers / arrow from the origin. Length & inner product: the norm $\sqrt{\sum x_i^2}$ and how much two vectors point the same way. MSE: average squared error. Unbiased: the average of many estimates equals the truth. Rotation: change of basis that preserves lengths and angles.
CLT: sum of many independent randoms converges to a Gaussian. High-D concentration: coordinates of a random unit vector in $d$-space cluster near $\pm 1/\sqrt d$. Quantization: snap each number to one of $2^b$ levels; one extra bit quarters the squared error. §1 · Vector quantization What is vector quantization, really? You have a vector $\mathbf{x}\in\mathbb{R}^d$, say $d{=}1536$ floats from an OpenAI embedding. You want to store it using $b$ bits per coordinate (total $b\cdot d$ bits), then later recover an approximation $\tilde{\mathbf{x}}$ close to $\mathbf{x}$. Closeness is measured by MSE distortion $D_{\text{mse}} = \mathbb{E}\big[\,\|\mathbf{x} - \tilde{\mathbf{x}}\|_2^2\,\big]$ or inner-product error $D_{\text{prod}} = \mathbb{E}\big[\,|\langle\mathbf{y},\mathbf{x}\rangle - \langle\mathbf{y},\tilde{\mathbf{x}}\rangle|^2\,\big]$ The second one matters because attention scores and nearest-neighbor queries are all inner products. We would like the estimator to be unbiased: $\mathbb{E}[\langle\mathbf{y},\tilde{\mathbf{x}}\rangle] = \langle\mathbf{y},\mathbf{x}\rangle$. ■ KEY WORDSMSE distortion: average squared error between the true vector and its reconstruction, primer §0.3.Inner product $\langle y, x\rangle$: how much two vectors point the same way, primer §0.2. This is what attention computes.Estimator: a rule (here: quantize, then decode) that returns an approximation $\hat s$ of a true number $s$.Unbiased estimator: across many queries, the average of $\hat s$ equals $s$. Individual estimates can be noisy; the mean is on target. Primer §0.4. The obvious quantizer For each coordinate, pick the closest of $2^b$ evenly-spaced levels in $[-1, 1]$. That is $b$ bits per number. The same rule runs in 2D and 3D first, where the geometry is visible, before the high-dimensional version below. First, in 2D Drag the tip of the vector.
The vector snaps to the nearest point of a $2^b \times 2^b$ grid. The green arrow shows the original input. The blue arrow shows where the input is quantized to. The red segment between them is the reconstruction error $\mathbf{x} - \tilde{\mathbf{x}}$. Bits b2Preset: ↕ drag tip‖error‖ / ‖x‖–levels per axis4grid points16 Same trick in 3D A $2^b$-level grid on three axes gives $2^{3b}$ snap points. Drag the canvas to orbit the view. The spike preset shows where the construction breaks: the input lies near one axis and falls between two grid levels, which is where the reconstruction error is largest. Bits b2Preset: ↕ drag tip · ↻ orbit‖error‖ / ‖x‖–levels per axis4grid points64 Now at scale (d up to 128) The same rule applied to every coordinate of a high-dimensional vector. You cannot see the grid anymore, but the per-coordinate errors are still there. Bits b3Dimension d64Input: original $x_i$ quantized $\tilde{x}_i$ ‖x − x̃‖² / ‖x‖²–levels per coord–bits used– Select the spike input. The naive quantizer's grid is spaced evenly over $[-1, 1]$. The input has almost all of its magnitude in a single coordinate, whose value falls between the two grid levels nearest to it and so reconstructs poorly. The remaining coordinates are near zero and consume most of the levels despite carrying little of the input's information. ■ TAKEAWAY · NEXT §2where the gap shows up A fixed grid produces small reconstruction errors on inputs whose coordinates are roughly uniform in magnitude, and large reconstruction errors on inputs whose magnitude is concentrated in one or a few coordinates. Next: §2 shows how production systems handle the second case and what they pay for the fix. §2 · Why naive fails The adversarial coordinate, and why production systems pay a tax Real embeddings are rarely flat. Trained models produce outlier channels, a few coordinates much larger than the rest. A fixed $[-L, L]$ grid either clips the outliers or wastes resolution on the bulk.
Production quantizers (GPTQ, AWQ, KIVI, KVQuant) work around this by computing $(\min, \max)$ (or zero-point and scale) for every small block and storing those in full precision as side information. The catch. To decode any block you also need its scale and zero-point, two float16 numbers (32 extra bits) stored next to every 16–64 quantized values. Walk through one case: a block of 32 numbers at 3 bits each is 96 payload bits, plus 32 metadata bits, which works out to 4 bits per number, not 3. Smaller blocks of 16 numbers push it to 5 bits per number. The advertised 3-bit scheme is really a 4–5-bit scheme once you count everything. TurboQuant matches this worst-case quality while storing zero per-block metadata. ■ DEMO · feel the catch same b bits/value, three strategies A 64-dimensional vector whose coordinates are mostly small, with one large outlier shown in red. Three quantizers reconstruct the same vector at the same b-bit budget. Strategy A uses a single fixed grid for the whole vector. Strategy B adapts the grid per block, at the cost of a float16 header per block. Strategy C rotates the vector first and then applies a single fixed grid. The metrics report the RMSE of each reconstruction and the effective bits-per-value once the metadata cost is included. Outlier magnitude4.0Bit budget b3Block size s16Source vector (outlier in red, dashed lines = fixed grid range)A. Fixed grid [−L, L]one global range, b bits/value, no header. Outlier clips.RMSE–bits/value–overhead0%B. Per-block scale + zerofloat16 scale+zero per block (dashed dividers). Outlier fits, header taxes you.RMSE–bits/value–overhead–C. Rotate → fixed gridrotation smears the spike across all 64 coords. One global grid works, no header.RMSE–bits/value–overhead0% Read the storage line.