A Tale of Two HTTP/2 Codecs

Same config, same traffic, and the CPU graphs stepped up by roughly a fifth. This is the kind of chart that earns you a calendar invite with no agenda, so we decided to get ahead of it and started digging in. Bisecting versions pointed at Envoy v1.34, which is where Envoy switched its default HTTP/2 codec from nghttp2 to Google's oghttp2. We weren't the first to notice: users had been reporting 15–25% latency regressions since that release, and v1.37.0 eventually flipped the default back, with a comment in the source promising to try again "once performance aligns with nghttp2." That fixed our customer. It did not fix our curiosity. Why doesn't it align? What does an HTTP/2 codec even spend its time on? This post is the investigation. There's a sequel coming where we make the fast codec faster still, but first things first. The best of codecs, the worst of codecs Envoy has two implementations of HTTP/2 behind a runtime flag, though nobody could say which was which without a flamegraph: nghttp2 - Tatsuhiro Tsujikawa's C library, the HTTP/2 workhorse since 2013 and Envoy's original codec. Lean C structs, caller-provided buffers. oghttp2 - Google's C++ codec from the QUICHE family, which shares code with the HTTP/3 stack. It became Envoy's default in v1.34, and that's when the regression reports started. We benchmarked both on four microarchitectures (Intel Sapphire Rapids, AMD Zen 4, AWS Graviton4, Google Axion). Setup, briefly: same-host loopback (h2load → Envoy → Go h2c backend, or Envoy serving direct_response), every process pinned to disjoint physical cores, Envoy at --concurrency 1, so RPS/core is a pure CPU-efficiency number. The user reports reproduce almost exactly: nghttp2 beats oghttp2 by 15–25% RPS/core on header-heavy proxied traffic, on every host, and by 7–19% under heavy connection churn. So the regression is real, portable, and lives somewhere in the codec. Time to open the profiles. PSAIf you build Envoy yourself, check your optimization flags. Envoy's .bazelrc does not default to -c opt - a plain bazel build //source/exe:envoy-static produces a fastbuild (debug) binary that looks completely functional and benchmarks 15–32× slower per core than the same source at -c opt. Every number in this post is from -c opt builds. We found this out the embarrassing way. Where an HTTP/2 codec spends its day The flamegraphs for both codecs are dominated by the same job: header decompression. Every HTTP/2 request that crosses a proxy arrives with its headers compressed, and the proxy has to undo that - twice per hop, decode and re-encode - before it can route anything. It's a few hundred bytes of work per request, which sounds like nothing until you multiply it out: tens of thousands of requests per second per core, a dozen-plus header fields each, so the per-field decode path runs a few hundred thousand times a second. The compression scheme is HPACK (RFC 7541): two mechanisms working together. The first is indexing: a fixed 61-entry static table covers the universal headers (:method: GET is a single byte), and both sides maintain a synchronized dynamic table of recently seen header fields - so the second time you send cookie: session=abc... it costs one or two bytes instead of the kilobyte it costs literally. The second mechanism handles everything that can't be an index hit: literal strings are Huffman-coded. First-time headers, unique values (request IDs, tokens, the cookie your ad vendor rotates on every request), and most of what crosses a fresh connection goes through the Huffman path. A proxy at the edge sees a lot of fresh connections and a lot of unique values, so this path is hot.

A graduate student named David Huffman picked the paper, struggled with it for months, and by his own account had just thrown his notes in the trash to start studying for the exam when the solution surfaced. Everyone before him had built codes from the top down, splitting the symbol set into halves. Huffman inverted it: start from the bottom - take the two rarest symbols, merge them into one, repeat until a single tree remains. Rare symbols end up deep in the tree with long codes; frequent symbols end up shallow with short ones.

And because letters live only at the leaves, no code can be the prefix of another - hover a letter to trace its path from the root. The leaves, top to bottom, spell what a decoder does when it reaches one. HPACK ships one fixed Huffman code for header text, its frequencies baked into the RFC from real header corpora captured a decade ago, so both ends of a connection always agree on it. Lowercase letters and digits sit near the top of the tree at 5–7 bits; rare bytes hang thirty levels deep. Typical header text comes out around 6 bits per byte. Now notice the property the tree structure gives you for free: no codeword can be the beginning of another, because symbols only live at the leaves. So the bit stream needs no delimiters at all. Read bits until they trace a path to a leaf - that's your symbol. Start again from the root. The stream carries its own boundaries, provided you start at the right place. It also tells you where the cost lives. Encoding is table lookups and bit-appending. Decoding is that walk: left on 0, right on 1, emit at a leaf, jump back to the root - for HPACK's code, an average of ~6 dependent branch-and-load steps per output byte, which is a lot of machinery for one character of a cookie. Here's the encoding half on a real header value - step through it character by character and watch the codes pack into bytes with no regard for byte boundaries: Huffman, unpacked - “application/json”FIG.

Which is exactly what makes decoding the interesting half of the problem. nghttp2 does something much better, and has since 2014: it decodes with a precomputed finite-state machine. A decoder mid-stream can only be in finitely many states - one per internal node of the Huffman tree, and HPACK's canonical tree has exactly 256 of them, so the state fits in one byte. You can precompute, for every (state, next-n-bits) pair, where you end up and which symbols (if any) you emit along the way. Decoding becomes: take the next n bits, index a table, maybe write a byte, repeat. No bit-buffer, no branching on code lengths. The machine below decodes a real header value under the real RFC 7541 code, reading two bits per lookup so the whole state table fits on screen. Step through it - the table and the tree are the same thing wearing different clothes: Fast prefix decoding - “application/json” at n = 2FIG. 03The same 88 bits, read 2 at a time - 44 lookups instead of 88 branch decisions. Click a chunk or step with ←/→ (keyboard works too). Chunks ignore character boundaries; the state carries the overlap.