Pushing the Limits of KV Cache Compression · zartbot

TL;DR When DeepSeek-V4.1 Flash was released, I thought it might just be a post-training iteration version... but after using it for a while, I found it reached nearly 420 Tokens/s in speed, and then Cui said all DeepSeek-V4 Pro models would be taken offline... suddenly I felt this was no small matter... until the Technical Report 《DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression》[1] was fully released, only then did I realize it should be called DeepSeek-V5 Flash... As the paper title states, the purpose of DeepSeek-V4.1 Flash is to push KVCache compression to the extreme. The main reason is that Long-horizon Agent Workflows cause the Context to grow longer and longer, while various tool calls bring heavy prefill computation pressure. The storage pressure of KVCache in HBM and external SSD is very high, all of which are reasons that make Scaling impossible. Therefore, a series of optimizations were made on the model architecture, especially in the compression of KVCache and the computation optimization of Prefill. Prefill computation optimization: Drawing on YOCO, the entire model has 40 layers, and only 20 layers are needed during Prefill. Therefore, the Prefill activated parameters are only 8B, and the Decode activated parameters are 16BKVCache compression: Engineering-wise, KVCache compression is divided into several dimensions: head count compression similar to GQA, then block-based compression like CSA, and the cross-layer compression of CSA2 in this paper. At the same time, the indexer computation of Sparse Attention is also optimized. Finally, there are some numerical precision optimizations, for example DS41F adopts FP4 KVCache. Finally, under the premise of maintaining high-quality task completion by the model, KVCache is further compressed by 4x: In addition, the original writing of the paper is somewhat complex, especially the description of CED. In fact, if we redraw a diagram centered on KVCache and combined with the perspective of computer architecture, it seems to become clear all at once. It can be seen as a kind of Recursive Transformer architecture, a way of modifying Q and reusing KV during the recursive process. Regarding the Recursive Transformer architecture, you can refer to 《On the Future Transformer: Loops Are Not What You Need》. Next, we will conduct a detailed interpretation and analysis according to the chapter structure of the technical report. This article is the first in this series, analyzing the model architecture in detail, and the more critical content is in Chapter 3. 1. Overview 1.1 Why KVCache compression is needed First, the report states that in recent years Long-horizon Agents have made ultra-long-context processing an increasingly important model workload. Supporting this type of workload not only requires efficient processing of long sequences, but also requires persistent storage, reuse, and transfer of large KVCache. Therefore, KVCache management has become a fundamental capability of model deployment, while also bringing significant challenges in computation, storage, and communication. Then it goes on to introduce the DeepSeek-V4 architecture, which processes by combining a Sparse Attention that fully covers the context with a Sliding Window Attention (SWA) that covers the local window. Although advances related to Sparse Attention have significantly reduced the computational cost of long sequence processing, persistent storage and data movement have gradually become more prominent bottlenecks. In long contexts, the usage of the Global KV Cache will dominate, and being persisted for prefix reuse, it will heavily occupy Host memory capacity and SSD capacity, and will also place high demands on the interconnect bandwidth for KVCache movement. These limit service throughput, increase deployment cost, and ultimately hinder the deployment and promotion of agents toward longer task spans and broader application scenarios. Therefore, further reducing the key-value cache footprint is crucial for alleviating storage and communication bottlenecks and reducing long-context serving costs. DeepSeek-V4.1-Flash is a multimodal mixture-of-experts model designed for more aggressive KVCache compression. DeepSeek-V4.1-Flash has a parameter scale of 552B, natively supports multimodal input, and supports contexts of up to 1 million tokens. It adopts a Causal Encoder-Decoder (CED) architecture, in which the Decoder's Global KVCache is obtained by projecting the Encoder's final hidden states. This design makes the model activate 8B parameters per token during the Prefill stage and 16B parameters during the Decoding stage, which is especially cost-effective for input-dominated Agent scenarios. Although DeepSeek-V4.1-Flash is significantly larger than DeepSeek-V4-Flash, at the same sequence length, its required runtime KVCache storage is only about 1/4 of the latter, and its persistent KVCache storage is only about 1/8 of the latter. In addition, the overall performance of DeepSeek-V4.1-Flash is superior to DeepSeek-V4-Flash. These compressions for KVCache mainly come from the joint optimization of model architecture, cache precision, and deployment strategy. For DSv4, it is a model with SWA as the backbone and enhanced by global compressed attention (CSA/HCA). Based on this perspective, the DeepSeek team carried out a series of optimizations. First, it is worth noting that they abandoned the block-based high-compression-ratio structure like HCA, and instead carried out more optimizations on CSA, forming CSA2. The main optimizations compress the KV Cache from three dimensions: In the channel dimension, a 512-dimensional latent vector is used to share the representation of the keys and values required by each attention head;In the sequence dimension, the Encoder merges 2 adjacent positions into 1 cache entry through channel-wise learned weights, while the Decoder retains per-position entries;In the layer dimension, multiple layers share the same global KV, and the whole network retains only 3 copies of Encoder cache and 1 copy of Decoder cache. Combined with FP4 quantization, the storage growth of the global main KV and the Indexer is about 890 bytes per token. 1.2 Overview of model architecture The overall model architecture is as follows: The paper reports that the backbone parameters are about , the Engram parameters are about , and the activated parameters per token for prefill and decode are about and respectively. The core is to use CED to reduce long-context Prefill computation, use CSA2 to reduce attention and KV cache overhead, and then combine Engram conditional memory with DSpark speculative decoding. The model has layers in total, hidden dimension , and vocabulary size . Each layer contains attention and MoE, organized through mHC residual connections. The entire attention mechanism is divided into two modules, Encoder and Decoder, forming a Causal Encoder-Decoder (CED) architecture. The key of CED is that the Decoder's global KV comes from the Encoder's end representation, and subsequent Decoder layers share these KVs. Therefore, most positions of a long prompt only need to pass through the first 20 layers, The key CSA2 among them adopts a mechanism of local sliding window + global sparse retrieval + cross-layer KVCache reuse. The sliding window size is 128, attention uses Q heads, sharing a -dimensional KV latent, of which RoPE is -dimensional and NoPE is -dimensional. Q uses a low-rank projection of rank , and the output projection is divided into groups, each of rank .