The 4-bitter Lesson: Balancing Stability and Performance in NVFP4 RL

Introduction

RL Training Simulator. The simulator models an asynchronous RL system where samplers continuously generate rollouts while the trainer updates the policy. Policy mismatch arises from both off-policyness (stale rollouts) and quantization error, accumulating into policy drift that eventually degrades reward if it exceeds the optimizer's correction capacity. RL & Efficiency Knobs. Off-policy, weight sync, batch size, and horizon control the degree of asynchrony and policy staleness. MXFP8 and NVFP4 improve training and rollout efficiency, while dequantized backward, BF16 last 15%, and shared experts improve numerical stability. How to use it. Explore the throughput–stability tradeoff by varying algorithmic and systems knobs. Increase asynchrony or lower precision to improve utilization, then observe how stabilization techniques recover the stability margin. The goal is to identify configurations that maximize throughput while keeping policy mismatch below the critical threshold where drift accelerates and reward collapses.

The essence of RL is to teach a model through action and consequence; at humans&, we use RL to train models that understand the long-term impacts of their interactions with people. In an RL loop1Specifically a policy gradient algorithm., a model acts, gets a reward, then updates the actions' likelihoods. However, in real-world RL training for LMs, we want to update the policy from observations as soon as they're available, even as other rollouts are still being sampled. For the long-horizon multiplayer rollouts central to our mission, we may even take dozens of training steps as a model completes one rollout. This introduces the tug-of-war between throughput and stability in RL. On one hand, we want to train on as many examples as quickly as possible2Make each training step faster, each rollout faster, or overlap trainers and samplers more.. On the other hand, most techniques to increase throughput cause the sampled policy and the trained policy to diverge, potentially slowing learning and destabilizing training. Quantization exemplifies this tradeoff: while low precision formats enable faster communication and computation on hardware, quantization hurts stability. Fast and accurate low precision formats like NVFP4 paired with hardware support3Allowing up to 9x more operations per second than 16-bit training on NVIDIA Rubin GPUs.

Dense Tensor Core PFLOPS per GPU from NVIDIA's HGX platform specs (B200/B300 from 8-GPU HGX systems, Rubin from the HGX Rubin NVL8 table):GPUBF16FP8FP4B2002.254.59B3002.254.513.5Rubin (NVL8)417.535 have driven large throughput increases in both model training and inference separately. However, there is no stable, hardware-native 4-bit4While INT4 QAT recipes exist to simulate 4-bit quantization, these have used 16-bit activations (Kimi K2 Thinking), and MXFP4 recipes like in DSv4 have used MXFP8 activations (DeepSeek-V4 on Day 0). Our approach uses NVFP4 quantization for both weights and activations. RL recipe in the open-source community, largely because the sampling and training instabilities compound in RL. In a long-running collaboration with the open source community, we have developed and shared a low-precision RL recipe preserving higher-precision training dynamics. In this recipe, we needed to address instability from the forward pass due to policy quantization errors, from the backward pass due to gradient mismatches, and at the intersection of both due to a small set of particularly sensitive weights. Below, we explain how we addressed each and validated the final recipe. Note this effort could not have happened without our amazing collaborators at RadixArk and NVIDIA, and their work across the training, inference, and RL stacks.

Baseline: A starting recipe with stable training dynamics For consistent comparison, unless otherwise noted, all experiments in this report use the Qwen3-30B-A3B model trained with 8k sequence length on DAPO-math-17k datasets. We start with a baseline recipe that uses the NVFP4 format. NVFP4 is a 4-bit floating-point format that improves throughput and memory efficiency on newer NVIDIA GPU architectures. It uses hierarchical block scaling: each block of 16 values has an FP8 E4M3 scale while the full tensor has a global FP32 scale. Together, these scales can recover a higher-precision value.

Why is the NVFP4 pretraining recipe not sufficient for RL?

NVIDIA's NVFP4 pretraining recipe is the natural starting point for this work. It uses a mixed-precision strategy, in which most operations are performed in FP4 precision, while numerically sensitive components remain in higher precision. However, pretraining and RL have different failure modes. In pretraining, the gradient signal is dense and repeatedly averaged over a large number of tokens. A main goal is to avoid quantization bias that would perturb the direction of optimization. Stochastic rounding helps by making the gradient quantization approximately unbiased. The RL setting has a different bias-variance tradeoff. The policy-gradient is already a noisy estimator because it depends on the sampled rollouts, advantage estimation, reward estimates, KL regularization, and policy staleness. Therefore, it is not sufficient for the quantization method to be just unbiased. Quantization noise must be small enough that it does not degrade the per-update true policy gradient signal. Therefore, we prioritize interventions that decrease quantization error in the policy and improve the accuracy of the gradient computation. Baseline recipe For the baseline recipe, we quantize only the MoE layers and keep all other layers in higher-precision BF16. In DeepSeek-V3-style architectures, MoE experts account for 97%5Each expert contains 3 projection matrices of size 7168 x 2048, giving about 44M parameters per expert. With 256 experts and 1 additional shared expert per MOE layer, that's about 11.32B parameters per layer. Across 58 layers, that's 656B parameters, which is approximately 97.8% of total parameters. of the total parameters so aggressively quantizing MoE layers results in most of the memory benefits. We run the forward pass in NVFP4, while keeping the backward pass in BF16 precision. This is a conservative starting point to give us rollout and memory benefits of FP4, while avoiding FP4 in the backward pass. For weights, we use the standard NVFP4 format with FP8 per-block scales and a single FP32 global scale. For activations, we do not use a global FP32 scale computed over the entire tensor.

As the Cursor Composer 2 technical report points out, the global scale can create two issues:

The same token can be quantized differently depending on which other tokens are present in the batch. If the tensor contains multiple positions from the same sequence, a later token can influence the shared scale used by an earlier token, creating a leakage path from future tokens to past tokens.

To solve this problem, similar to Cursor Composer 2, we use per-token activation scaling where each token computes its own FP32 activation scale across the hidden dimension. This keeps quantization local to each token, and avoids a separate calibration step, since scales can be computed directly from the activations observed during the forward pass. Also, this fine-grained FP32 scale helps with the lower quantization error compared to the per-tensor FP32 scale. During rollout, this per-token scale computation is fused into the activation quantization kernel. For each token row, the kernel computes the FP32 scale, the FP8 E4M3 scales for 16-value blocks, and packs these quantized values into FP4. This fused implementation reduces extra memory movement and kernel launch overhead.

Implementation

The per-token NVFP4 recipe required changes across the stack and we have open-sourced our implementation:

TransformerEngine row scaled recipe cuDNN kernel for row-scaled grouped GEMM quantization FlashInfer inference kernel for MoE per-token quantization SGLang support for per-token NVFP4 MoE

Improving Gradient Stability We observed gradient norm spikes during training with our open-source per-token NVFP4 recipe. This recipe is intentionally conservative with precision in backward pass: although the forward pass uses NVFP4 operands, the backward pass keeps weights and activations in BF16 precision. This is much more stable than running the full backward path through coarse-grained NVFP4, but it introduces a mismatch between forward and backward pass which causes occasional gradient spikes. The issue is that the backward pass is no longer differentiating the same function that was used in the forward pass. For example, in a simple linear layer, the forward pass evaluates y = x · Q(w_bf16), where Q(.) denotes quantization6Quantization function behaves like a staircase function that's not differentiable at step change values.

Quantization usually involves clipping and rounding i.e clip(round(w/s), qmin, qmax).. The backward pass behaves as if it is differentiating y = x · w_bf16. This decision makes backward pass more stable, but also means that the backward pass is unaware of the clipping and rounding decisions made in the quantization function. To mitigate this chain-rule inconsistency while preserving the stability of the BF16 backward pass, we use dequantized backward. Instead of the BF16 precision in the backward pass, we use the BF16-dequantized value of the exact quantized tensor used in the forward pass. In case of the linear layer example, that means y = x · DQ(Q(w_bf16)), where DQ denotes dequantization. Here, the backward pass still uses BF16 operands, but the operands reflect the same NVFP4 quantization decisions that were used in the forward pass. Note that our recipe applies dequantized backward operation on both weights and activations.

Results Validation with MXFP8: To validate our changes, we first tested this fix with the MXFP8 recipe. The figure below shows how the gradient norm behaves under 3 different recipes:

a) quantized forward and backward, shown as mxfp8-mxfp8 b) quantized forward and BF16 backward, shown as mxfp8-high-precision-bf16 c) quantized forward with the chain-rule violation corrected using dequantized weights in the backward pass, shown as mxfp8-dequantized-bf16

Our method leads to cleaner and less noisy gradients compared with MXFP8 backward throughout all stages of the training. Compared with high-precision BF16 backward, our method helps prevent the gradient norm from increasing and is consistently stable. Validation with NVFP4: In comparison to MXFP8, NVFP4 complicates training dynamics. On one hand, our method helps stability by providing less biased gradients because the backward pass matches the quantized forward pass. On the other hand, NVFP4 quantization can increase gradient variance due to its coarser representation grid. To understand this empirically, our first setup uses SGD optimizer instead of Adam because SGD makes it easier to isolate our method's effect on the true gradients.