Task Failed Successfully: Saturating NIC and Disk Bandwidth

0. “Task Failed Successfully” #The AI era has arrived faster than most of us expected. Agentic coding has completely changed the way I work day to day. To be honest, I haven’t written a single line of code at work in quite a while. Yes, it is true. NOT A SINGLE LINE!! And yet, that hasn’t stopped the code from running across clusters with hundreds of HPC servers at peak performance.Of course, not writing code (or even not fully reviewing it) does not mean we are just randomly poking around, like monkey typing. We still need to analyze requirements, refine the design with the agent, build demos, run mock experiments, study the results from small-scale tests, iterate on the problems we find, and maintain a complete, solid testing process, blah blah blah.Monkey TypingHowever, with AI and agentic coding, everything has become faster. Sometimes, code is churned out faster than we can fully understand it. And sometimes, it is even faster than AI can understand it. Yes, you read that right. And this post comes from one such example.After I gave my agent the prompt to optimize the performance of my system, the AI quickly took it from roughly half throughput to full saturation. But its explanation of why it worked was completely wrong. It was a classic case of task failed successfully.Task Failed SuccessfullyThis post doesn’t talk about why the AI “failed successfully”. It is a walkthrough of the analysis and debugging process behind this system performance optimization.1. Optimize a Demo with 1 NIC and 8 disks #Let’s turn the system into a simple abstraction to focus on the performance optimization rather than the complex business:A single thread issues 1 MiB random direct I/O reads across 8 NVMe drives, then sends the data to a remote host via RDMA WRITE. Now, saturate the NIC bandwidth.More specifically, each drive can deliver up to 7 GiB/s of read throughput, and the NIC provides 400 Gb/s of network bandwidth. All devices are attached to the same NUMA node. The worker thread is pinned to a non-CPU0 core.

The host runs with the IOMMU in passthrough mode, and none of the I/O devices involved are translated through the IOMMU.For the implementation, I (actually it was my AI agent) built a very simple event loop: the client sends read requests to the server; the server polls the RDMA CQ for incoming requests, submits the reads through io_uring, polls for the resulting CQEs, and then sends the data back via RDMA WRITE.Simple Demo TopologyThis demo setup is very straightforward and rules out almost all sources of interference. Other than the NIC we are trying to saturate, every component has plenty of headroom: the NIC’s theoretical maximum throughput is 46.6 GiB/s, each drive averages less than 6 GiB/s of read throughput, total IOPS stay below 50,000, and the CPU has more than enough capacity as well.Now that everything is in place, let’s look at the results.inflightGiB/savg µsp50 µsp90 µsp99 µsp99.9 µs48.87440430519632759815.565014866167849551622.69688670850111814363222.53138613841696193422726422.1528212819313333663691Surprisingly, the system already hits a bottleneck at an I/O depth of just 16, with aggregate throughput reaching only about half of the NIC’s bandwidth. And the CPU utilization reached 100%.It was clear that there must have been something wrong, so I profiled the system with perf at an I/O depth of 16. Here is the flamegraph.Simple Demo Flamegraph (iodepth=16)As the flamegraph shows, most of the CPU time is spent in io_submit_sqes, which accounts for 81.62% of the total CPU cost.

Because the demo uses Direct I/O, every I/O submission requires the kernel to construct DMA metadata from the user-space buffer for the block device to consume. The most costly parts of this path are:__bio_iov_iter_get_pages: Turn iov into bio pages.pin_user_pages_fast: Translates a user-space virtual address range into an array of struct page pointers, and pins those pages so they cannot be reclaimed, migrated, or swapped out while the device is performing DMA.bio_set_pages_dirty: Mark the buffer pages dirty. With Direct I/O, the NVMe device DMA-writes data directly into the pages backing the user-space buffer. Those pages must then be marked dirty so that the VM does not treat them as clean pages.folio_*: It updates VM state associated with the folio, including its reference count, dirty state, mapping, locking, and reclaim-related state. In the Linux VM, a folio is a unified abstraction for a physically contiguous set of pages.In a word, the wide frame of io_submit_sqes represents the cumulative cost of preparing user memory for Direct I/O DMA. Each SQE contains only a user-space pointer and length. The kernel must walk the page tables, find and pin the backing struct pages, build bio_vec entries, update folio state, and submit the resulting bio.Most of that work is paid per page. A 1 MiB read backed by 4 KiB pages touches roughly 256 pages, turning one logical read into hundreds of page-table lookups, page pins, folio updates, and bio-vector operations. At 20,000 to 50,000 reads per second, the system processes roughly 5 to 13 million pages through GUP (Get User Pages) per second. If the virtual address range is backed by heavily fragmented physical memory, there may be a comparable number of folio metadata updates and atomic refcount/pincount updates, along with potential cross-core cache-line ownership transfers.Therefore, if we can avoid paying the cost of processing the user-space buffer on every I/O, we should be able to improve performance. Fortunately, liburing provides a way to do exactly that. io_uring_register_buffers(3) lets us register I/O buffers ahead of time, moving this metadata preparation work out of the per-I/O path.

More specifically, io_uring_register_buffers(3) performs the following work up front:Validates the iovecs up front, checking address ranges, lengths, alignment, and count limits.Performs GUP on the buffers, translates the user-space virtual addresses into the corresponding struct pages / folios, and pins those pages for the lifetime of the registration.

Constructs and retains kernel-side buffer metadata, building io_mapped_ubuf for each registered buffer.These are exactly the major costs we just observed in the flamegraph! Let’s try it. In the demo, we introduce a 64 MiB read arena and divide it into 1 MiB slots, matching the I/O size. At startup, we register the 64-slot read arena as 64 io_uring fixed buffers through io_uring_register_buffers(3), with one iovec per slot. For each read, we switch the opcode from opcode::Read to opcode::ReadFixed and set buf_index to the corresponding slot. This allows the I/O path to use registered buffers. Here are the results:inflightGiB/savg µsp50 µsp90 µsp99 µsp99.9 µs49.09429421519641767816.784654475907759661628.08556514774111914253239.877837051229175421846446.0013581248208931954214As the I/O depth increases, throughput continues to rise. At an I/O depth of 64, it nearly saturates the NIC bandwidth.inflightBaseline GiB/sREAD_FIXED GiB/sΔBaseline p99 µsREAD_FIXED p99 µs48.879.09+2%632641815.5616.78+8%7847751622.6928.08+24%111811193222.5339.87+77%193417546422.1546.00+108%33663195Compared with the baseline, throughput is similar at low I/O depths, where the CPU has not yet become the bottleneck.

By an I/O depth of 16, the baseline is already showing CPU pressure; beyond that point, per-I/O buffer handling becomes fully CPU-bound. READ_FIXED removes this bottleneck, allowing throughput to continue scaling until it saturates the NIC.The flame graph also confirms this.Simple Demo Flamegraph (iodepth=16, with READFIXED)2. Scale to a Larger Deployment #With the simple demo resolved, we can move on to a larger-scale demo that more closely reflects a real-world deployment.In the larger-scale demo, the client consists of a single node equipped with 8 * 400 Gb/s NICs. The server side consists of 4 nodes. Each server has 2 * NUMA nodes, and each NUMA node has 1 * 400 Gb/s NIC and 8 * the same NVMe drives used in the previous demo.The I/O size increases from 1 MiB to 1,028 KiB because an additional 4 KiB is needed to store metadata.The event loop is similar to the earlier version, but to better approximate a real workload, the server verifies the CRC of each read before sending it back via RDMA WRITE. For CRC computation, it uses crate crc-fast, which includes an implementation optimized with AVX-512 VPCLMULQDQ instructions. It can deliver roughly 50 GiB/s of checksum throughput on a single core.While the CRC computation throughput is just enough for this workload in isolation, a CPU core must also run the event loop and handle other application logic. One worker thread per NUMA node is therefore no longer sufficient. Instead, each NUMA node is served by multiple worker threads that share the same index and the eight local NVMe drives. The index is sharded as well, preventing a global index lock from becoming a performance bottleneck.Each worker thread connects to every NIC on the client node. Each connection has its own QP, while all QPs owned by the same worker share a single CQ for completion processing.Large Demo TopologyTo make the discussion below clearer, I will use the following terms for the different levels of sharding on the server side:TermExplanationClusterThe four-node deployment.

NodeA physical server. Each node contains two NUMA nodes, with a total of two 400 Gb/s NICs and sixteen NVMe drives.Shard1 * NUMA node within a physical server. Each shard owns 1 * 400 Gb/s NIC and 8 * NVMe drives, and is served by multiple worker threads.Worker threadA CPU-pinned worker thread. Worker threads within the same shard share the NIC, index, and local NVMe drives. Each worker thread connects to every client NIC through a dedicated QP per connection, while sharing a single CQ across all of its QPs.Now that we have a clear picture of the larger-scale demo (hopefully), let’s put it under load! Here are the results; T represents the worker thread count per shard, and all throughput numbers are in GiB/s.Configiodepth=16iodepth=32iodepth=64T=4181.9204.9209.2T=8183.2207.1209.8T=16181.4206.0211.2In theory, the system should be able to reach an aggregate throughput of 372.5 GiB/s across all 8 * NICs. In practice, however, we achieved only about half of that throughput.Here comes a new bottleneck!Next, let’s work through the flame graph step by step to identify the bottleneck.2.1 Ruling Out the Impact of iou-wrk #To locate the bottleneck, let’s look at the new flamegraph, captured from a T=8 run with an I/O depth of 64.Tips: The full flamegraph is too large to embed here. You can download the SVG and open it directly in a browser to explore the details.Large Demo Flamegraph, T=8, iodepth=64Although the multi-shard flame graph is fairly complex, it splits quite clearly into two clusters: 16 iou-wrk threads on the left and 16 worker threads on the right.When we captured the flame graph for the simple demo, there was only a single worker thread, so we profiled only that thread and did not include the iou-wrk threads.