Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,306 words · 9 segments analyzed
For the past 3 years, graphics processing units (GPUs) have dominated the large language model (LLM) conversation. In traditional chatbot applications, central processing units (CPUs) provide a fraction of the total compute per request, while GPUs do the heavy lifting. However, inference isn't a single model answering a single question. A growing reliance on tool calls, multistep reasoning, and orchestration across small, specialized models changes the math on where compute should live.
Intel has called out this shift noting that the CPU-to-GPU ratio is moving from 1:8 in training workloads to 1:1, and in some cases 4:1 in agentic deployments.Here we'll examine why the assumptions making GPUs the obvious choice for LLM inference are being renegotiated, what's driving renewed demand for CPU-based serving, and what the data says about where the industry is heading.What CPUs are good atAt their core (no pun intended), CPUs and GPUs solve fundamentally different problems.A modern GPU contains tens of thousands of cores designed to execute the same operation on thousands of data elements simultaneously.
This makes them extraordinarily fast at the dense matrix multiplications that dominate the forward pass of a transformer for inference. During training, and during high-concurrency batched inference, that parallelism translates directly into throughput: greater tokens per second (TPS), and many more requests served per dollar of compute.Modern CPUs, by contrast, have anywhere from one to hundreds of cores optimized for sequential, conditional, and branching logic. They're particularly fast at a single operation moving through a complex decision tree. They have direct access to main system memory, and they're the natural execution environment for the orchestration layer wrapping any model. Tool dispatch, code execution, Python runtimes, sandboxes, input/output (I/O), and the agent loop control flow all sit in the CPU.These architectures, while different in nature, aren't competitors; they work best in tandem. The real question isn't which architecture is better, but which workload belongs where.This division of labor comes down to how we measure their work: FLOPS versus instruction latency.GPUs live and die by floating-point operations per second (FLOPS). Because AI models are massive webs of decimal numbers being multiplied and added, a GPU's job is to brute-force trillions of these matrix calculations simultaneously. It's built entirely for raw mathematical throughput.CPUs, by contrast, specialize in instruction latency. Instruction latency measures how fast a single core can execute an unpredictable chain of diverse commands. A CPU core excels at the rapid logic shuffling needed to parse JavaScript Object Notation (JSON), handle network I/O, or check security permissions.If you force a GPU to run a chaotic Python runtime, its massive FLOP capacity sits idle, choked by constant task-switching. If you force a CPU to crunch an LLM's math, it works perfectly, but takes ages because it lacks the massive parallel pipelines. The GPU is the mathematical muscle; the CPU is the logic engine directing the show.The traditional inference stack: Where CPUs were passengersWhen we think of the traditional inference stack, we think about the chatbot serving application. Here, the CPU often plays a supporting role.A request arrives at the API server, where the CPU tokenizes and schedules it. The CPU handles all these initial tasks. It then hands the request to the GPU for the forward pass, which dominates the compute budget. The GPU runs attention, feeds forward layers, and samples the next token with a brief CPU sync to retrieve it and update the scheduler. It then repeats until an end-of-sequence token is emitted.
Finally, the CPU collects the output and returns it to end the session.In this model, the CPU acts as a receptionist, doing the coordination and organizational work, while the GPU does the computational heavy lifting. The ratio of CPUs to GPUs in AI data centers during the training era reflected this: roughly 1 to 2 CPUs for every 8 GPUs, a ratio dictated more by GPUs throughput than by CPU demand. Teams provisioned CPUs to keep GPUs fed. This setup is slowly shifting away from the de facto standard, as advancements in CPU inference increase global efficiency.The first driver: How agents are changing the workloadWith the rise of agentic AI comes a new inference profile magnifying the need for CPU-specific compute."Agentic AI" conjures up images of integrated agent harnesses, like Hermes and Openclaw, and developers with hosts of agents tied into their coding assistants. While these use cases are becoming more popular, agentic AI isn't as niche as it may seem at first glance. In fact, many people aren't using the "traditional" process of querying a chatbot. Popular AI assistants like Claude and ChatGPT are backed by enormous models that often incorporate "reasoning" stages into their more complex responses. During reasoning, the "chatbot" becomes a full agentic system.
A single user-visible task may decompose into dozens of individual model calls, each of which is short, context-dependent, and conditionally branching.In an agentic system, the model doesn't issue a single prompt and wait for a response.
It generates a plan of action, executes tool calls, compiles all the returned data, and culminates in completing an action or returning an answer to the user. The CPU is in charge of parsing that output, figuring out which tool to invoke, making the API calls or running the code, collecting the result, and feeding it back. Then the loop repeats—sometimes dozens of times per user request, with parallel sub-agents adding more CPU coordination work on top. The model quickly becomes one component of a larger reasoning loop. The ratio of organizational work to pure computation shifts, making the CPU a much bigger player, and potential bottleneck, in the full process.In a collaboration between researchers at Georgia Tech and Intel, researchers found that in agentic workloads, CPU-side tool processing accounts for 50–90% of total end-to-end latency.
Intel's CEO Lip-Bu Tan, at Computex 2026, framed the shift precisely: "for reinforcement learning, orchestration, and agents, the CPU is a much better fit."The numbers from Intel's Q1 2026 earnings call quantify how far this has already moved:Training workloads: ~1 CPU per 8 GPUsInference workloads: Already shifted to ~1 CPU per 4 GPUsAgentic workloads: Converging toward 1:1, with some customers reporting 4 CPUs deployed per GPUIntel's Q1 2026 Data Center and AI segment revenue came in at $5.1B, up 22% year-over-year, with demand running ahead of supply. Intel has deprioritized consumer chip production to redirect fab capacity to server Xeon parts.Arm's own analysis forecasts the shift more starkly.
Traditional AI data centers require approximately 30 million CPU cores per gigawatt (GW) of capacity. Arm CEO Rene Haas estimates in the AI agent era, that figure rises to 120 million CPU cores per GW. The orchestration demands of agentic workloads drive this proposed 4× increase.The second driver: Smaller, localized modelsSeparate from the shift agentic AI is cultivating, a second structural change is pushing inference toward CPUs.
This change is the move toward smaller, domain-specific models deployed close to where data is generated.We can attribute this change in tides to 4 main pressures on providers:Latency: A round-trip to a GPU cluster in a cloud data center takes time. For real-time applications like voice interfaces, autonomous systems, and industrial monitoring, that latency budget is measured in milliseconds. Local computing removes the network roundtrip.Privacy: Sending data off-device or off-premises to a cloud provider means data leaves the organization's control. Healthcare, finance, legal, and many other enterprise workflows increasingly cannot tolerate this. While this doesn't guarantee your data is being picked apart and auctioned off to the highest bidder or used for training, a local or CPU-resident model running on existing on-premises server infrastructure processes data without it ever leaving the facility, completely mitigating that risk.Cost: A single GPU can cost upwards of $45,000. For workloads that don't require the throughput of a GPU, that cost is unjustifiable. CPU inference on already-provisioned server infrastructure has a near-zero marginal hardware cost.Offline availability: Edge deployments frequently operate without reliable internet connectivity.