Qwen3.8 27B at 256K: 50 TPS on a 24 GB GPU

I gave Qwen3.8's MTP drafter another 69.2 MiB of precision. Throughput fell from 50.44 to 37.02 tokens per second. That result sums up the whole experiment: the best local inference setup is rarely made from the individually "best" parts. I wanted a dense 27B model, its full 262,144-token context, multimodal input, maximum useful quality, and speculative decoding on an NVIDIA RTX PRO 4000 Blackwell SFF with 24 GB of VRAM. The server also had to survive real agent work after printing model loaded. The experiment followed a hunch I had written about earlier: careful operation may matter as much as moving to a larger model. The finished system averages 50.44 tok/s in the current ten-run production series. On a strict runtime A/B, the custom llama.cpp build reaches 55.40 tok/s versus 45.42 for clean master, a 21.97% gain. Against target-only greedy decoding, embedded MTP moves 21.19 to 59.46 tok/s, or 2.81 times the throughput. At the far end of a genuinely occupied 256K cache, it still produces 12.61 tok/s without an out-of-memory failure. Those numbers came from different gates and should stay separate. Combining them into one heroic speedup would make a better headline and a worse benchmark. The winning setup came from the fit between the quant, drafter, CUDA kernels, memory layout, and workload. No component won on its own. The target was deliberately unreasonable Qwen3.8 27B is a 64-layer dense model. Its repeating pattern contains three Gated DeltaNet layers followed by one full-attention layer, giving 48 recurrent layers and 16 conventional attention layers. It has a native 262,144-token context, a one-layer MTP head, and a separate 27-layer vision encoder. The hardware is lopsided in a useful way: GPU0: RTX PRO 4000 Blackwell SFF, 24 GB GDDR7 with ECC, a 192-bit memory interface, 432 GB/s peak memory bandwidth, 24,467 MiB reported capacity, and sm120a. It holds the target, embedded MTP, recurrent state, graphs, and the 256K KV cache. GPU1: RTX 2000 Ada, 15,996 MiB, sm89. It holds the F16 multimodal projector and other auxiliary services. Runtime: Debian 13, CUDA 12.9.86, GCC 14.2, dual-architecture CUDA build. Only the 16 full-attention layers grow a conventional KV cache with sequence length, which makes 256K less absurd than it first appears. With Q4 K and V, that cache costs roughly 4.25 GiB before allocator overhead. DeltaNet adds recurrent state and checkpoints instead. Four checkpoints were the useful minimum; the default 32 spent memory I needed elsewhere. NVIDIA quotes 432 GB/s of peak bandwidth. That is a hardware ceiling rather than an application metric from llama.cpp, but it matters here. Autoregressive decode repeatedly streams quantized weights, and the 16 attention layers add increasingly expensive KV reads as context fills. This is why the same profile averages about 50 tok/s on the production task and 12.61 tok/s at the far end of a 261.5K-token cache. The original plan was simple: estimate the capacity, select a quant, then benchmark it. The machine immediately taught me that capacity estimates are just admission tickets. The real test begins after loading. The first winner was Q4_0, and it was the wrong winner I began with public GGUFs at 40K context. Q4_0 was surprisingly strong. Target-only decoding reached 22.40 tok/s, and MTP with n_max=3 reached 44.95. It beat smaller Q3_K_M and nominally smarter Q4_K_M variants because file size and quant label do not describe the CUDA kernel that actually runs. QuantTarget onlyMTP n=3Acceptance Q3_K_M17.00 tok/s31.34 tok/s83.98% IQ4_XS, iMatrix20.63 tok/s34.40 tok/s64.87% Q4_022.40 tok/s44.95 tok/s80.40% Q4_K_M17.57 tok/s26.15 tok/s66.86% Then quality testing spoiled the easy answer. On a short, identical WikiText-2 control, IQ4_XS scored 6.1175 perplexity while Q4_0 scored 6.3798. Q4_0 led the speed table. Hermes needed a main model, though, and that quality trade felt too expensive for a few hundred milliseconds. I would have been using a 27B model as oversized autocomplete. The opposite extreme failed too. Q4_1 reached 6.1127 PPL, marginally ahead of IQ4_XS, but its memory footprint made 256K plus F16 vision uncomfortable. The useful point was somewhere between a fast blunt quant and a precise file that left no room for the rest of the system. Loading 256K proves almost nothing Early capacity tests looked excellent. Q4_0, MTP, Q4 KV, four recurrent checkpoints, and the F16 projector all allocated at 262,144 context. That still did not answer the question I cared about. I filled the slot with 261,500 input tokens, generated another 256, and then reused the hot cache. No truncation. No OOM. The first Q4_0 profile decoded at 12.06 tok/s near the end of the cache, compared with 44.95 around 40K. GPU usage sat at 99 to 100%, while the server used roughly one CPU core. The bottleneck was the 16 full-attention layers reading a huge occupied KV cache, not a secret CPU fallback. This changed the benchmark method for every run that followed. "262K loaded" was banned from the results table. A long-context claim had to include actual token fill, post-fill VRAM, hot decode, truncation state, and an output hash. The ready-made NVFP4 quant failed the quality gate Blackwell has native FP4 hardware, so a ready-made NVFP4-MEDIUM GGUF looked like the obvious route. Its bulk target matrices used NVFP4, with a Q8 output head, Q6 embeddings, and an IQ4_XS MTP layer. It reached 40.46 tok/s and fitted the complete 256K plus vision profile with about 1,055 MiB free. Its PPL was 6.4949. Worse than Q4_0. The conversion recipe was the problem. Attention and DeltaNet weights from the source FP8 checkpoint had been expanded and requantized into NVFP4 along with the large, tolerant matrices. Native arithmetic made the file quick, while indiscriminate low precision damaged sensitive parts of the model. Hardware format support does not tell you where to spend the bits. That failure gave us the design for a custom quant: use NVFP4 for the bulk, then protect only the tensors that our own workload says matter. I calibrated the model on how I actually use it The calibration corpus started with 5,472 messages from 296 Hermes sessions. I placed that material before the generic corpus so the 153,600 processed tokens represented coding, Polish and English conversation, infrastructure work, tool calls, and the awkward mixtures my agents really see. A secret scan ran before calibration. No PEM keys, provider tokens, GitHub tokens, Slack tokens, or email addresses were present. llama-imatrix collected importance data for 497 target weights. NVFP4 does not consume an iMatrix directly during block quantization, so I used the matrix as a map: large tolerant tensors stayed native NVFP4; selected attention, DeltaNet, and FFN tensors moved to Q5_K or Q6_K; embeddings became Q6_K; the output head stayed Q8_0. The first 5.14 BPW hybrid was the quality winner at 6.0967 PPL. It was also slow at 34.19 tok/s and too large to keep the desired projector on GPU alongside 256K. Good experiment. Bad production model. The second build was tighter: Size: 16,321.38 MiB, 5.01 BPW. Bulk matrices: native NVFP4. Sensitive target tensors: selected Q5_K and Q6_K using the Hermes iMatrix ranking. Embedding and output: Q6_K and Q8_0. Embedded MTP: native NVFP4. PPL: 6.1197, versus 6.1127 for Q4_1. The 0.11% gap is far below the error of this short control. This became Qwen3.8-27B-Hermes-iMatrix-NVFP4-Balanced.gguf. It preserved the measured quality of the Q4_1 reference, ran faster, and left enough room for the actual serving stack. MTP had a trapdoor at n=8 The early sweep suggested n_max=3. Values 4 through 7 got slower as rejected draft work accumulated. Then n=8 jumped to 49.31 tok/s. MTP n_maxTPSAcceptanceCombined process VRAM 343.1178.73%18,352 MiB 437.7962.22%18,502 MiB 729.1042.95%18,950 MiB 849.3148.33%19,100 MiB 949.0243.95%19,250 MiB 1243.3134.34%19,700 MiB 2030.6019.97%20,900 MiB The curve is jagged. Eight candidates hit a favorable batch and kernel shape. Nine was no faster, and each extra candidate cost about 150 MiB. With the full 256K allocation, n=9 at ubatch=256 failed on one more 162 MiB CUDA graph buffer. Reducing ubatch to 128 made it load, but throughput fell to 52.70 tok/s and 32K prefill suffered. N=10 failed on another 81 MiB. Even an experimental scheduler pool lost to a 31 MiB allocation. I kept n=8 because it was the last fast point before the allocator started biting. More accurate MTP made the system worse I wanted a controlled rival for the NVFP4 MTP choice. A patched MTP-aware iMatrix run processed 300 Hermes-history chunks and added all eight MTP matrices. For the comparison, all 851 non-MTP tensors were verified byte-for-byte identical to production. Only the eight MTP weight tensors changed. MTP weightsExtra sizeMean TPSAcceptanceResult Production NVFP4baseline50.44148.329%keep iMatrix Q5_K50.625 MiB48.73346.751%-3.39% Q5_K with critical Q6_K69.219 MiB37.02433.065%-26.60% The higher-bit drafter may be closer to the BF16 source model. In production it had one job: predict this quantized target. The NVFP4 errors in the MTP head happened to align better with the NVFP4-heavy target, so its exact proposals survived more often. Standalone precision lost to quant-drafter alignment. This changed how I treat drafter quality. The drafter and target form one quantized system, and their interaction decides acceptance and throughput. A standalone quality score for either half misses it. DSpark had a whole second GPU and still lost I also tested Qwen3.8-27B-DSpark, a 1.36B diffusion drafter with a Markov head and confidence head. A llama.cpp patch allowed the sidecar to run on GPU1 while the target stayed on GPU0.