Pangram verdict · v3.3
We believe that this document is a mix of AI-generated, AI-assisted, and human-written content
AI likelihood · overall
MixedArticle text · 1,950 words · 6 segments analyzed
Two virtual machines on the same host, configured identically, running the same workload. One of them is 20% slower than the other, consistently. Nothing is wrong with the workload, nothing is wrong with the host, no contention from other tenants. The slow one's memory just happens to be on the wrong side of an interconnect from the CPUs running it, and there is no in-guest knob the operator can turn to fix it.That is the story this series is about. Edera shipped a stack of changes that make Xen-based virtualisation NUMA-aware end to end - inside the guest, through the paravirtual I/O drivers, into dom0, and back to the hypervisor's view of host hardware. Some of those pieces are, as far as we can tell, the first implementations anywhere. To explain why any of it matters, we have to start with what NUMA actually is.Where NUMA came from: From UMA to Multi-Socket ServersNUMA stands for Non-Uniform Memory Access. The defining property is in the name: memory access cost on a NUMA machine is not uniform. Which CPU is doing the access matters, and which physical memory bank the data lives in matters, and the cost depends on the relationship between those two things.The historical opposite of NUMA is UMA, Uniform Memory Access, where every CPU reaches every byte of memory through the same memory controller at the same cost. UMA was the world of single-socket commodity servers and many embedded systems. It is conceptually simple and it scales up to a limit - that limit being roughly "as many cores as you can fit on one chunk of silicon with one set of memory channels".What happens when you need to push past that limit? UMA scales until it stops, and "stops" has a few intertwined reasons. Electrical trace lengths get long. Signals take longer to propagate end to end, and at the data rates a modern memory bus runs at, longer traces also force you to slow the bus down to keep signal integrity - which piles latency on top of latency. (This is the same physics behind the L1 / L2 / L3 cache hierarchy: L1 is small partly because keeping a cache small and physically close to the core is what keeps it fast.)
A single memory controller cannot feed an arbitrary number of cores at full bandwidth; pin counts on the package put a hard ceiling on how many memory channels you can wire to one socket. Bus contention between many CPUs trying to read memory simultaneously stops being negligible. The industry's answer was to give each socket (or die) its own memory controller and stop pretending memory is one symmetric pool. The machine becomes NUMA. Each node has its own CPUs and its own slice of memory; CPUs can still read from any node, but reads to a remote node have to cross an interconnect, and the interconnect is fast in absolute terms and slow relative to a local DRAM access.UMACPU 0CPU 1CPU 2CPU 3memory controllermemoryone pool, shared by all CPUsevery access has the same costNUMAnode 0CPU 0CPU 1memory controllermemorynode 1CPU 2CPU 3memory controllermemoryinterconnectlocal access is cheap; remote access crosses the interconnectIt helps to think of the interconnect as a bridge between two adjacent towns.
On an empty road, the crossing is only marginally slower than local traffic; at rush hour, when everyone is trying to use the same bridge at once, the cost is whatever the queue happens to be. We will come back to that distinction when we talk about how NUMA looks on a busy production workload versus a quiet microbenchmark.A short chronology to anchor where this came from. Commercial NUMA showed up in the 1990s in big-iron systems aimed at a small set of buyers who needed more cores than one bus could feed: SGI's Origin line, Sequent NUMA-Q, Compaq's AlphaServer GS series, all with bespoke node-to-node fabrics. NUMA arrived in commodity x86 in 2003 with AMD's Opteron, which gave each socket its own memory controller and connected sockets via HyperTransport. Intel caught up in 2008 with Nehalem and the QuickPath Interconnect (QPI), after a long run of front-side-bus chipsets that were genuinely UMA (all CPUs sharing one bus to a northbridge, no per-socket memory at all). Today's Intel parts use the Ultra Path Interconnect (UPI), the descendant of QPI; today's AMD parts use Infinity Fabric, the descendant of HyperTransport. Each generation lifted the absolute bandwidth substantially, but the ratio of remote-to-local DRAM cost has stayed stubbornly in roughly the same range.You might be tempted to think, looking at all of this so far, that each socket is one NUMA node and be done with it. For most of the decade after Opteron, that was right. Operators learned NUMA as "the topology mirrors the sockets" because at the per-socket core counts of the era a single memory controller could feed the whole socket without trouble. The mental model worked.That stopped being true around the late 2010s. AMD's Zen-based EPYC pushed per-socket core counts high enough - and made the manufacturing tradeoffs of one giant monolithic die unattractive enough - that the package moved to a small constellation of chiplets: several Core Complex Dies (CCDs) inside one socket, each with its own slice of L3 cache and its own slice of the memory controller, glued together by Infinity Fabric.
The wrinkle is that Infinity Fabric is now used inside a socket between CCDs, not just between sockets: it is the same fabric, doing the same job, at a smaller scale. A single EPYC socket can therefore present several NUMA nodes to the OS, depending on the part and the BIOS configuration - the "Nodes Per Socket" (NPS) knob, NPS1 through NPS4 on current generations.Intel's version of the same idea is Sub-NUMA Clustering (SNC). Off by default but available in BIOS on recent Xeon parts, SNC partitions the on-die mesh of memory controllers and presents the result as multiple NUMA nodes per socket."One socket, one node" was a useful first approximation, and it stopped being correct on commodity hardware a few years ago. Greater-than-two-socket x86 servers have become rare outside specialized niches; the interesting NUMA complexity on modern systems comes from inside a single dual-socket box, where four to eight nodes is now routine.How big is "slow"?A precise number is platform-specific and people argue about the precise methodology, but a useful rule of thumb is that a remote DRAM access on a modern server costs roughly 1.5x to 3x what a local one costs. The bandwidth is also lower, sometimes by a similar ratio. Those numbers are what you measure with a microbenchmark on an idle machine.node 0CPU 0CPU 1memory controllernode 0 memorynode 1CPU 2CPU 3memory controllernode 1 memoryinterconnectlocalremoteOn a real workload, the gap is usually worse, because the interconnect is shared. If many cores are doing cross-node accesses at once, they contend for the same interconnect bandwidth, so per-request latency goes up. This is the mechanism behind a lot of "why is this server slow at high load" mysteries. The microbenchmark says cross-node is 2x worse. The production workload sees 4x or 5x worse because everyone is fighting over the same pipe.The corollary is worth mentioning: NUMA effects do not just slow the affected workloads down, they make their performance less predictable. The same request can take 80 nanoseconds or 250 nanoseconds depending on what else is on the interconnect at the moment.
Average latency is a useful number but tail latency is where NUMA pain usually first shows up.Interleaving: sacrificing peaks for floorsThere is a well-known way to flatten the cost curve without doing any of the work to understand where memory should live: memory interleaving. Allocate pages round-robin across all nodes, regardless of which node the requesting task is on. On Linux this is numactl --interleave=all; some databases and JVMs do something equivalent in their own allocators; a few hypervisors do it implicitly for guest memory when they have nothing smarter to fall back on.A BIOS-level version of the same idea exists too. Most dual-socket server firmwares have a "Node Interleaving" or "Memory Interleaving Across Nodes" toggle that spreads physical memory across the sockets at cache-line or page granularity. With it on, the OS sees one big UMA pool instead of a NUMA topology, and whatever NUMA-aware code it has becomes inert: there is no topology left to be aware of.That toggle is most common on dual-socket boxes, but the deeper problem is the same everywhere: interleaving scales badly. The more nodes you have, the longer the worst-case path between any core and any memory bank gets, and the more pressure the interconnect comes under to carry all that round-robin traffic. The trade interleaving makes - predictable mediocrity over unpredictable mix - only gets steeper as the topology widens.the same task on node 0; its pages, placed three waysNUMA-awarefast and consistentevery access is localNUMA-obliviousunpredictableno pattern, varies by runinterleavedconsistent, but mediocre"makes everything equally bad"page on node 0 (local)page on node 1 (remote)Interleaving makes the expected cost of any single allocation roughly equal to the average of all the node-to-node costs on the machine. On a two-node host, that means about half your accesses are local and about half are remote, by construction. Latency variance collapses, because the workload no longer wins or loses based on which page it happens to touch.A former colleague had the cleanest one-line summary of this trade: interleaving makes everything equally bad. It is consistency at the cost of peak.
A workload that could have fit entirely on a single node, hitting only local memory, is now paying remote-access cost on half its allocations every time.This is a real tool and there are real reasons to reach for it. NUMA-oblivious software running on a NUMA host gets predictable performance from interleaving, and "predictably bad" is often easier to operate than "unpredictably mixed". Memory-bandwidth-bound workloads that span the whole machine anyway can even come out ahead, because they get to aggregate the bandwidth of every memory controller instead of saturating one.But it leaves a lot on the table. A workload that does know its own access pattern, running on a stack that gives it correct topology and honours its placement requests, can do substantially better than interleaving on both latency and throughput. The whole point of the work this series is about is that the Xen guest no longer has to pick between "blind and hope" and "blind and interleave". The guest can know where it is, and then act on it.Two affinities, not oneA lot of developers go their whole careers without thinking about NUMA. If your deploy target has been laptops, single-socket workstations, or the kind of cloud VM that has been carved out of a single host node, you have effectively been on a UMA machine the whole time, and NUMA-aware tooling has lived at the periphery of your view at best. It shows up the day the workload moves onto a bigger box, usually with no warning louder than "this is weirdly slow", and the standard developer reaction is to discover an entire second axis of placement that nobody mentioned.The first axis, the familiar one: if you have ever pinned a process to specific CPUs with taskset or sched_setaffinity, you already know one half of the picture. Tasks have an affinity for CPUs. The kernel scheduler tries to keep them where you put them. CPU affinity is the half that tends to show up in tutorials and blog posts.The other half is memory affinity, and it comes up far less often. On Linux it is controlled through mbind(), set_mempolicy(), and the libnuma helpers like numa_alloc_onnode(), all of which decide which NUMA node an allocation comes from; the numactl(8) man page is the practical entry point.