Why harnesses matter for energy efficiency

By Chad Gibson, Co-founder & CEO Key findings Harnesses cache differently. We saw 1.6% to 11.1% uncached input across clients. "Cached" is the share of input tokens served from the prompt cache instead of computed again. At 92% cached, 8 tokens per 100 are uncached, versus 3 at 97%: about 2.7 times the prefill work. A cache miss roughly doubles a request's energy. Here, a miss means a request with little or none of its prompt cached (under 10%). On GLM-5.3, it's about 2.5 times as much. Deep sessions make misses expensive. A re-read means recomputing that part of the prompt from scratch. Past 128k tokens, one turn that re-reads more than half its prompt costs as much prefill as about 600 normal turns at that depth. Most of this is fixable. Every cache break in our lab tests came from a harness editing what it had already sent, forcing a re-read from the first difference onward. At Neuralwatt, we can see how the agent harness calling a model affects the energy it uses. I've been looking through a month of traffic on our cloud, and the harness seems to decide a surprising share of how much work our GPUs do. We usually talk about inference efficiency in terms of the model (how many active parameters, its attention mechanism) or the hardware (H200 vs B200 vs B300). The client deserves a closer look. Almost all of our traffic is agentic, with far more input than output. A typical coding-agent turn sends 100k tokens or more of conversation and gets back a few hundred tokens. Most of that input is the previous turn's prompt with a little added at the end. If the earlier text is unchanged, the server already has its key-value (KV) cache and only has to process the new part. This is prompt caching, our biggest source of efficiency. Serving a cached token is far cheaper than computing it again. Caching needs an exact prefix. A cache break happens when the harness changes something it already sent. We then recompute everything from the first difference onward. A change near the beginning, even a timestamp, means recomputing nearly the whole prompt. A change near the end costs little. The harness controls how the prompt is built. Same model, same price, different cache rates Every request tells us which client sent it. In the 30 days from September 8 to October 7, 2026, we saw hundreds of different clients, including open-source coding agents such as OpenCode, Oh My Pi (omp) and Pi. The mix also included editor integrations, the OpenAI SDKs and custom apps. OpenCode alone is used by about one in four of our customers. For the most widely used clients, here is the share of input tokens we could not serve from cache: Figure 1. Uncached input by client. Every client here caches the large majority of its input. Those small differences still matter: a client at 97% cached sends us three uncached tokens per hundred; a client at 92% sends eight. That is 2.7 times as much prefill work for the same conversation. Hit rate also depends on the workload, not only the harness. Longer sessions score higher, because each turn's new text is a smaller share of the prompt, and about half of deepseek-harness's lead comes from users who run very long sessions. A cleaner measure of the harness is how often a turn breaks the cache: about 1 turn in 100 for deepseek-harness, against about 3 in 100 for OpenCode and Pi. We see both OpenCode 1.x and 2.x in our traffic, so the bar combines them. 2.x looks better alone (2.2% uncached against 3.0%), mostly because of the model mix. On the same model, they cache about equally well. 2.x fixes most of the 1.x cache breaks we found in the lab. That difference doesn't show up in production, and we don't yet know why. What a miss costs in energy We bill on measured energy, so we can see the cost directly. Over the week of October 1 to 7, 2026, we compared requests on the same model with the same prompt size (about 50k tokens) and similar output length. Here is the extra energy each request used compared with a fully cached one: Figure 2. Extra energy from a cache miss. A mostly uncached request takes roughly twice the energy of a cached one of the same size. On GLM-5.3 it is closer to two and a half times. Prefill also competes with generation for the same GPUs, so wasted prefill slows down everyone else's responses on that server. Cache misses get expensive deep in a session We followed individual sessions turn by turn on our three busiest models for the week of October 6 to 13, 2026. We looked at every turn that landed on the same server within five minutes of the previous one, where the cache should have been warm. Most cache breaks were small: a few thousand tokens near the end of the prompt got recomputed. Past 128k tokens of context, about 3 turns in every 1,000 re-read more than half of the prompt from scratch. Those few turns account for 75% of all the re-read tokens at that depth. Each one re-reads about 230k tokens on average. A normal turn at that depth adds a median of about 370 new tokens, so a single one of these costs as much prefill as roughly 600 ordinary turns. The delay is noticeable too. Here is the median time to first token for a normal turn compared with a turn that had to re-read most of its prompt: Figure 3. Time to first token deep in a session. I pay more attention to how deep into a session the cache breaks than how often it breaks. Changing the beginning of the prompt once, early on, costs little. Do it at turn 200 of a long session and you get a large, visible pause and a large energy cost. Long agent sessions account for most of our tokens, so this adds up. We can't always tell from the server side whether a harness edited the prompt or the cache was evicted under memory pressure. These numbers include both. Our lab tests below show that harnesses cause the early-prompt edits we could reproduce; eviction likely accounts for some of the rest. What breaks the cache I looked for the causes in two places: the lab, where we could see every token, and production, where we only see request metadata.