Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,445 words · 2 segments analyzed
Hello you fine Internet folks,NVIDIA has published a 45-page whitepaper explaining Vera, its first server CPU built around the company’s own Olympus core. On paper, Vera is a fascinating chip with an 88-core monolithic compute die, Olympus being a 10-wide Arm v9.2 core that has value prediction, a graph prefetcher, 2 MB of private L2 per core, 164 MB of shared last-level cache, and eight LPDDR5X memory interfaces promising 1.2 TB/s.Unfortunately, NVIDIA also spends a good part of the paper trying to turn those interesting design choices into a morality play about x86. Traditional simultaneous multithreading is drawn as time-slicing, a configurable NUMA topology is presented as an unavoidable 32-node maze, four SPEC components become “agentic benchmarks,” undefined performance-counter ratios are promoted as causal proof, and an unlabeled pictogram becomes a 1.8x reinforcement-learning result.The frustrating part is that Vera does not need this help, with early independent testing suggesting Olympus is genuinely formidable. The whitepaper’s strongest case is the hardware; its weakest case is the story wrapped around it, so let’s pull that story apart.Before getting out the cheese grater, let’s talk about the good stuff. Olympus is a very wide out-of-order Arm core.Its front end can decode ten instructions per cycle and handle up to two taken branches per cycle. NVIDIA describes a neural branch predictor, value prediction, memory renaming, a large instruction window, six 128-bit SVE pipes, four load pipes, two store pipes, a 96 KB L1 data cache, and roughly 10-cycle access to a 2 MB private L2. Eighty-eight of those cores sit behind a 3.4 TB/s coherency fabric and a distributed 164 MB system-level cache.Looking closer at the core, the value prediction is one of the more unique additions that Olympus has. This has been a research area for a long time and what value prediction allows Olympus to do is if the core correctly predicts a result, dependent instructions can keep moving instead of piling up behind a long-latency operation.
Researchers have discovered that Apple uses value prediction in their cores and AMD talked about how in Family 17h (Zen 1 and 2) they could predict the value of some floating point instructions. However, AMD’s Family 17h implementation was quite limited, while Olympus appears to have a broader value-prediction implementation closer to Apple’s.However, the graph prefetcher is not unique to NVIDIA. Intel has a similar mechanism called Data-Dependent Prefetcher that has been in shipping silicon since at least 2022. Intel’s newest datacenter CPU, Granite Rapids, also has an Array of Pointers prefetcher which “treats the data prefetched for a constant stride load as a pointer and may issue prefetch requests to the memory addresses corresponding to the pointer’s value.” This is fundamentally the same producer-consumer idea that NVIDIA describes for its graph prefetcher. Intel’s implementation is fairly constrained, so NVIDIA’s implementation may be able to deal with more complex chains than Intel’s implementation. So while Vera’s Graph Prefetcher may be an implementation that can deal with more workloads, producer-consumer prefetching is not a new idea.Nor is a “neural branch predictor” a new idea. Back in 2012, AMD implemented a perceptron branch predictor in the Piledriver microarchitecture and continued to use a perceptron-based branch predictor in Zen 1. However, starting with Zen 2, AMD used a perceptron BPU only for its initial direction prediction, with a TAGE predictor overriding it because it delivered a 30% reduction in mispredictions. With Zen 5, AMD has likely fully committed to TAGE predictors, if it had not already done so with Zen 3 or Zen 4.Moving to the SoC side, with how beefy the Olympus core is, NVIDIA has given Vera an equally beefy memory subsystem. Vera pairs eight SOCAMM2 LPDDR5X modules with up to 1.5 TB of capacity and 1.2 TB/s of bandwidth. NVIDIA claims the populated memory subsystem only consumes approximately 50 watts. A conventional EPYC or Xeon platform can offer higher-capacity DIMMs which are easier to replace, but it pays for that flexibility in board area and power.Most importantly, we have more than NVIDIA’s results to look at. In May, Michael Larabel at Phoronix ran an early Vera system against current Arm and x86 servers. Across the NVIDIA-permitted test set, Vera’s geomean was 10% above a 5 GHz EPYC 9575F, 1.55x a Xeon 6980P, and 1.63x Grace which makes Vera the most performant Arm server CPU we have seen in public testing. There are major caveats with the testing, such as NVIDIA choosing the permitted workload scope and not allowing frequency or power monitoring. The system that Phoronix tested was pre-production and the test window was one day which puts a fairly hard limit on what they could test irrespective of the limits NVIDIA placed. This means that broader coverage will have to wait until Vera can be found in the wild rather than just in NVIDIA’s labs.Still, the result is strong enough that we can reject the explanation that the charts in NVIDIA’s whitepaper are all fantasy. Olympus appears to be a fast CPU core, so now we can ask whether the whitepaper proves what NVIDIA says it proves.Here is the first major technical error in the document.Figure 5 contrasts “Traditional SMT (x86)” with NVIDIA’s Spatial Multithreading. The x86 side depicts the branch predictor, decode, execution, load/store, and memory stages alternating between two threads. The caption says Vera avoids “opportunistic time-sharing” by partitioning resources across its two hardware threads.NVIDIA’s diagram gives a misleading impression of how SMT is usually implemented, both on x86-64 and other ISAs. SMT implementations share various stages in the execution pipeline by either selecting a thread to service every cycle, or by behaving in a thread-agnostic manner. Fetch, decode, and allocate typically service threads on a per-cycle basis, while the execute and memory access stages are thread agnostic and can service micro-ops from both threads in the same cycle. Stages that threads arbitrate for do not leave resources unused when both threads can be fed, as NVIDIA’s diagram suggests. Static partitioning and per-cycle selection would provide the same average throughput to both threads in the absence of per-thread stalls. If there are stalls, per-cycle selection can give otherwise unused throughput to the un-stalled thread.Hypothetical example of decode stage activity for a processor that statically partitions decode for SMT, and an 8-wide one where decode selects a thread to service every cycle. Per-cycle thread selection can efficiently hide stalls in one thread, while static partitioning leaves throughput on the tableThe same idea applies to thread agnostic stages like execute and cache access. Each thread is permitted to utilize as many execution units or cache ports as it can feed. In contrast, statically partitioning resources as NVIDIA suggests could lead to one thread being compute bound and unable to use half of the core’s execution resources because they’re reserved for the other thread.Text in NVIDIA’s paper emphasizes “determinism, isolation, and quality of service” as advantages for NVIDIA’s Spatial Multithreading approach. Performance is conspicuously not called out. QoS may be a more important consideration than throughput for NVIDIA’s target market, and Spatial Multithreading may not be a bad design point. But NVIDIA’s figure makes it look like vertical space represents time, and gives a misleading impression that Spatial Multithreading is meant to give larger performance gains than traditional SMT.By reducing resource interference between threads, Spatial Multithreading improves determinism, isolation, and quality of service compared to traditional SMT approaches. The result is a CPU architecture that can run large numbers of concurrent agent tasks while maintaining more consistent latency and throughput. - NVIDIA’s Vera whitepaperVera’s actual SMT performance is unknown of course, and a lot of variables go into SMT gains besides partitioning strategies at fetch, decode, execute, and memory access. Out-of-order resources like the reorder buffer, register files, and memory ordering queues can be duplicated, statically partitioned, watermarked, or competitively shared. Partitioned structures were split between the two logical processors in multi-threaded mode and recombined for one thread in single-thread mode, which was documented in 2002. Various SMT implementations use different strategies for each structure, and those choices can have significant implications for SMT gains.Also something to note is that it apparently takes 10,000 cycles for an Olympus core to transition back to the single-thread mode once the sibling thread on that core is done. This means that software will have to be very aware about launching a second thread on an Olympus core due to the penalties incurred not only from the partitioning scheme but also from the delay of swapping back to a single thread. It’ll be interesting to see what strategy Vera uses to partition its out-of-order resources, and how its SMT performance compares to that of other modern cores. NVIDIA’s whitepaper gives no information on that.