We Replaced mmap with io_uring in Our Rust Query Engine. It Got Slower.

We copy them to local NVMe and read large (~3–5 GB) Arrow IPC files. We chose Arrow IPC for simplicity and speed — its memory and disk layouts are identical, so decode cost is minimal, and mmap gives us zero-copy reads natively supported by arrow-rust.

Two disk configs during the investigation: 2× NVMe LVM-striped (~5.5 GB/s fio ceiling) and 32× NVMe RAID-0 (~21 GB/s fio ceiling). Kernel 5.15 during investigation, 6.x in production. The Production Symptom At lighter loads, mmap worked well — fast, serving queries from raw events in seconds. The trouble started under heavier concurrency. Some latency increase under load is expected — more queries competing for the same CPU. But we saw p95s and p99s spike well beyond what linear scaling would predict, with rows scanned per core dropping sharply even after accounting for concurrency: OS page cache shrank — each pod consumed more memory as private allocations, less as shared cache A huge number of page faults p95 spiked from ~30s to 150s+ under real concurrent load Adding pods made it worse, not better That pointed at mmap page-cache thrashing under memory pressure. Controlled Benchmark: 1 Pod vs. 4 Pods To isolate the effect, we ran a controlled test: 1 pod vs. 4 pods on the same host, same concurrent query load. We expected 4 pods to win — more parallelism, better isolation. We were wrong.

For 14-day queries — long enough to fill the page cache — 1 pod beat 4 pods by a real margin: 41% faster at max, >20% at p95. The mmap page cache lives on the host and is shared across pods, so the 4 pods weren’t fighting each other for CPU — they were fighting for page cache.

perf record on the same run showed 100% lock contention at the kernel level. The core issue: mmap’s page cache is implicit shared state. Every process on the host shares one cache, one lock hierarchy, one eviction policy. No single pod controls the resource that matters most for read latency, and as concurrency rises, everyone’s slice of it shrinks. A Storm of Page Faults We didn’t want to just guess, so we dug into the mmap mechanics and the page fault stats.

When you mmap a file, the application gets a region of memory backed by the file on disk. Here’s what happens on access: The CPU touches a Virtual Memory Address (VMA) with no physical page attached, and throws a page fault. The kernel handles the exception: looks up the VMA, checks ownership, and acquires a lock, since another thread might be modifying or unmapping that virtual space concurrently. Linux 6.4+ has a fast per-VMA lock; earlier kernels fall back to the slower mmap_lock. Once the kernel confirms the VMA is file-backed, it triggers a file fault — a minor fault if the bytes are already in warm page cache from read-ahead, or a major fault if it must trigger physical I/O. Recommended reading: mmap_lock scalability (LWN) and per-VMA locks (LWN, Suren Baghdasaryan’s design). Under heavy page cache contention, read-ahead runs out of room and major faults spike.

Here’s what that looked like — pidstat on one process during a stressful run: 23:26:46  RSS = 652 GB (87.88%) 23:27:47  RSS = 734 GB (98.91%)   ← peak, nearly all RAM 23:27:48  RSS starts dropping     ← kernel begins evicting 23:28:05  major faults appear: 571/s, 1352/s, 975/s RSS grows to 98.91% of RAM → the kernel has no choice but to evict pages still needed → evicted pages get touched again → a major fault storm as they’re read back from disk. Early on, read-ahead keeps faults mostly minor and fast; as concurrent queries pile up, read-ahead stops keeping up and major faults spike. Minor faults, meanwhile, ran sustained in the millions per second: 23:27:09   1,255,709 minor faults/sec 23:27:41   2,124,327 minor faults/sec 23:27:46   2,354,383 minor faults/sec Each minor fault touches a cache line via atomics — at 2 million faults/sec, that’s enough to thrash L1/L2 entirely, which is deadly for an application that leans on large cache-resident lookup tables. Faults can also trigger TLB shootdowns, and CPUs only hold a few thousand TLB entries.

Virtual address space, meanwhile, had grown to ~3 TB from mmap’ing so many Arrow files: Start:  3,125,750,740 KB (~2.9 TB virtual) Peak:   3,209,184,828 KB (~2.98 TB virtual) Modern kernels handle large VMA trees, but not for free — every fault does a VMA lookup, and every lookup takes the mmap lock (fast path notwithstanding). Context switches told the same story: cs = 2,106,576/sec cs = 2,025,726/sec cs = 1,944,397/sec cs = 1,524,279/sec Over 2 million context switches/sec, versus 14K/sec on a warm-cache run — 150x more. Every thread was constantly blocking on page faults, getting descheduled, and rescheduled once pages arrived. Perf Top and Off-CPU Analysis To confirm the link between page faults and lock contention, we compared perf top on cold vs. warm runs of the same query: Function Cold run Warm run __filemap_add_folio (kernel) 78.0% not in top kernel spinlocks 0.96% 0.76% CPU/data processing 4.96% 45.08% __filemap_add_folio adds a page to the page cache.

Off-CPU time via bpftrace (actionable time only, excluding idle Rayon threads): Futex: 30.9% (1,172s) — threads blocked on synchronization, queued behind another thread’s page-fault handler Preempted: 29.3% (1,109s) — surprisingly high for 12 threads on 192 cores; the kernel’s page-fault work (readahead kthreads) was preempting our worker threads Disk I/O: 6.9% (262s) — actual NVMe latency was small next to the machinery above it mmap_sem: 0.9% (33.5s) — the explicit VMA lock; small only because it captures the wait, not the cascading futex wakes from threads queued behind it The picture: under load, page-cache thrashing and kernel-level lock contention — not disk I/O — were the bottleneck. This isn’t unique to mmap; any buffered I/O path can hit similar page-cache and lock contention. The Fio Ceiling fio with the io_uring engine — 4 processes, iodepth 32, 4 MiB blocks, O_DIRECT: READ: bw=20.2 GiB/s (21.7 GB/s) All 32 NVMe drives at ~99.75% utilization md0 util = 99.95% What mmap actually delivered, peak, from vmstat during the stressful runs: 3.44 GB/s — about 16% of what the hardware could do. That gap was the size of the prize. Enter io_uring io_uring has earned its hype.

Beyond async kernel I/O, part of its promise is direct user I/O that bypasses the page cache entirely — the thing causing most of our problems above. Worth reading: “io_uring for high performance DBMS” — a good overview of optimizations, though focused on traditional DBMSes with 4KB page buffers, so many don’t translate.

We used compio, a Rust-native io_uring wrapper (executor + futures + reactor built around io_uring). The first cut leaned on compio’s async futures — one future per Arrow column read, all 40 columns (8 batches × 5 columns) submitted concurrently, awaiting completions to yield decoded Arrow buffers.