NUMA Part 4: Closing the Xen dom0 I/O Gap

In the Part 3 of this series, we walked through the Xen paravirtual I/O architecture and what makes it expensive on a Non-Uniform Memory Access (NUMA) host: foreign-mapped pages with no NUMA metadata, kthreads landing wherever the scheduler happened to put them, an information gap between the dom0 kernel and the host's actual topology. We closed that part on a promise that the fix has two pieces - a structural one and a per-component one - and that part 4 would deliver both.This part makes good on that - and along the way it covers two things we did not plan to fix. One we hit before the real work could start: dom0's own memory placement was so skewed that synthesising a topology for dom0 without fixing it first would have been an exercise in lying to a kernel that had no memory to back the lie with. That one is a prerequisite, not a detour - everything else in this part is built on top of it. The other we hit after the work was nominally done: a memory-placement bug in our own toolstack that had been silently broken for as long as multi-vnode vNUMA had existed in our tree, and only ever surfaced under a memory-bandwidth benchmark. That one is the genuine detour, and it waits until the end.Let's start with the prerequisite.The First Discovery: dom0's Memory Was in the Wrong PlaceStock Xen, when booting dom0, has to trim the host's BIOS memory map (the E820) down to the subset of memory dom0 is supposed to own. The trim walks the host E820 in physical address order, marking regions as dom0's RAM until it has set aside enough to satisfy dom0_mem. On a single-node host, this is harmless; dom0's memory comes from the only node available. On a multi-socket host, the same code is a real problem. The host's RAM regions are typically grouped by NUMA node in physical address order, so walking the E820 in order means dom0 gets all its memory from the lowest-address nodes first.The worst case is severe. On a 128 GiB, 8-node host with dom0_mem=35% - the configuration our lab box runs - dom0 takes 100% of the lowest-address nodes' RAM and 0% of the others.

Nodes 0 through 2 are almost entirely consumed by dom0; nodes 3 through 7 have no dom0-owned pages at all. Without dom0 being able to even attempt an allocation on the remote nodes, making dom0 NUMA-aware would have been synthesising a topology the kernel could never act on - the SRAT we generated would have described one node where 60% of dom0's memory lived alongside six nodes where it had nothing at all.Our trim replaces this in-order grab with a proportional one, and it is one of the places our tree diverges from upstream Xen. It runs in two passes: pass one sums total host RAM; pass two gives each region a proportional share of the dom0_mem budget. The arithmetic is Bresenham-style, with rounding remainders accumulating so the total ends up exactly equal to dom0_mem. Dom0's memory ends up spread across every host node in proportion to that node's share of physical RAM.With dom0 actually owning memory on every host node, the rest of the work could begin.Making dom0 NUMA-awareThree Pieces of NUMA Topology, for Any Xen DomainRecall from part 2 that a NUMA-aware Xen domain sees three pieces of topology:The ACPI System Resource Affinity Table (SRAT), which maps CPUs and memory ranges to NUMA nodes.The ACPI System Locality Information Table (SLIT), which gives the inter-node distance matrix.The x2APIC IDs in CPUID, which encode the per-CPU package/core/thread identifiers.Any Xen domain that wants accurate NUMA awareness needs all three to be present and consistent. Upstream Xen does most of this work for PVH and HVM domUs when the toolstack asks for a vNUMA layout via XEN_DOMCTL_setvnumainfo: SRAT and SLIT are synthesised to match the requested topology. Upstream Xen does not synthesise the x2APIC IDs in CPUID from the vNUMA layout, however. That third piece has been an Edera extension since before the work this post describes began, and we extended it further along the way to handle non-power-of-two vCPU counts in multi-node domUs.Dom0 is a different story. Until the work described in this post, upstream Xen synthesised none of the three for dom0.

Dom0 saw a single flat NUMA node regardless of how many physical nodes the host had. The work this section covers extends the same three-piece synthesis to dom0, under one specific set of conditions.The conditions: dom0 must be PVH; it must boot with dom0_vcpus_pin=1 so each dom0 vCPU is hard-pinned 1:1 to a specific host pCPU; the dom0 vCPU count must equal the host pCPU count so that every pCPU has a corresponding dom0 vCPU; and the host must have more than one NUMA node. Those conditions also have a useful side effect for the implementation. Because every dom0 vCPU N is pinned to pCPU N, the vNUMA node assignment for vCPU N is simply the host's NUMA node for pCPU N. There is no layout algorithm to design, no placement decisions to make at boot, and no edge cases for partially-covered host topologies, because dom0 covers the whole host by definition. The implementation works out to a direct walk over the host's cpu_to_node table.When the conditions are met, dom0 gets:A synthesised SRAT describing one proximity domain per host NUMA node that dom0 spans, with the appropriate CPU set and the appropriate memory regions in each.A synthesised SLIT sliced directly from the host's own SLIT - the distances dom0 sees are the real inter-node distances, not the default "10 local, 20 remote" stub that Linux substitutes when SLIT is absent.x2APIC IDs in CPUID encoded to match the synthesised topology, using the same machinery that handles the domU case.After this, numactl -H inside dom0 finally shows the real topology - eight nodes where it used to report one, each with the CPUs and memory that actually belong to it. Storage daemons inside dom0 see the real topology. Container runtimes inside dom0 see the real topology.And the kernel's own automatic NUMA balancing finally has something to act on: with an accurate map it samples where each task runs and, over time, migrates that task's pages onto the same node, healing remote accesses on its own. That self-healing does not exist against a flat one-node view - there is nothing to balance - so it is one more thing the topology work unlocks.

It is a fallback rather than a plan, though: a sampling heuristic with its own cost, and the kernel's guidance is to switch it off for workloads that are already pinned - which is exactly what the later sections do to the I/O path.The dom0 shell of every Edera deployment now operates on accurate information for the first time.It's also worth noting that dom0 NUMA-awareness really matters on mixed nodes. Plenty of Edera deployments are not all-Edera: the same machine runs some pods as ordinary containers under the default runtime - directly in dom0 - and others under the Edera runtime, each sealed into its own zone. Those ordinary containers are dom0 workloads like any other, and before this work they ran NUMA-blind. A non-Edera pod could end up slower on an Edera host than it had been on the same box as plain Linux, purely because turning the machine into a Xen dom0 hid the topology its workloads used to see. Making dom0 NUMA-aware closes that gap: the containers that stay in dom0 get the topology they would have seen on bare metal, so adopting Edera does not quietly tax the workloads that were never sandboxed in the first place.Why SRAT, SLIT, and CPUID All MatterIt is tempting to assume the ACPI tables alone are enough. The kernel reads them at boot, the NUMA infrastructure picks up the topology, you are done. For some consumers that is enough. The hwloc library, which most NUMA-aware userspace tooling is built on, derives its view of the topology from the kernel's sysfs view (which the kernel in turn populates from the SRAT and SLIT). Get those two right and the sysfs surface is populated correctly, and hwloc is happy.Other consumers go further, and skipping CPUID quietly breaks them. The LLVM OpenMP runtime, used by parallel-loop-heavy workloads in compiled languages, derives its socket and core topology from the x2APIC CPUID leaf rather than from sysfs. On a Xen guest where CPUID has been left at the default single-socket layout but SRAT and SLIT are correct, OpenMP concludes the machine is single-socket and assigns its worker threads accordingly - which on a multi-socket host means stuffing every worker into one node's worth of cores.

The kernel scheduler will rebalance over time, but OpenMP's initial assignment is set in stone, and the resulting placement is bad enough that benchmark numbers visibly suffer.(LLVM OpenMP does have a compile-time option to use hwloc for topology discovery instead of reading CPUID directly, but it is not enabled by default and we have not found a binary distribution that turns it on. The CPUID path is what production builds run against.)Conveniently, the runtime will show you its reasoning. Run an OpenMP workload with KMP_AFFINITY=verbose using the LLVM OpenMP runtime, and the runtime prints both the topology it discovered and the thread placement it chose; on an Edera guest, where the x2APIC IDs are synthesised to match the real layout, both come out correct - the same whether the workload runs in dom0 or inside a domU.The lesson generalises beyond OpenMP. Making the topology visible to all of its consumers is the only way to make it actually useful. SRAT alone, SLIT alone, CPUID alone - none of those are enough on their own, because different consumers in different ecosystems read different sources. Doing two out of three would have meant choosing which class of workload to quietly disappoint.Bugs Found While Building dom0 vNUMAGetting the synthesis right turned up a handful of bugs we had not expected.The first was a node-numbering bug. Xen renumbers proximity domain identifiers during boot into a dense internal node ID space, in SRAT scan order. The first version of the dom0 vNUMA build code used those internal Xen-side node IDs as the vNUMA node index, which then leaked into the SRAT we generated for dom0 and into the encoded APIC IDs. The result: a dom0 vCPU pinned to a pCPU on host proximity domain 1 might show up inside dom0 as belonging to "socket 3" if Xen's renumbering had landed proximity domain 1 into Xen's internal slot 3. Tools running inside dom0 would target the wrong physical socket if they were keying off proximity domain numbers. The fix was to index by host proximity domain directly, not by the Xen-internal node ID, so the SRAT, APIC IDs, and SLIT all use the same host-firmware numbering consistently.