Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,164 words · 2 segments analyzed
Input and output tokensA request goes through the GPU in two phases, and they cost different amounts. First, during prefill, the model reads your request and context: the system prompt, your CLAUDE.md, your message, and everything that's been added to the conversation since (the files Claude has read and the output of the commands it ran). Those are your input tokens.Then, during decode, it writes output tokens: its thinking, the tool calls it makes, and the text you see. This happens one token at a time; a 200-token response is 200 runs of the model, one after the other. Per token, decode keeps the GPU busy for a lot longer, which is why output is priced at roughly 5x input.A lot of the output tokens in a session are thinking tokens, and how much thinking the model does per turn is what the effort level controls. Like the model, the level you pick with /effort sticks around as your default for the next session too.Tip: run /model and /effort once in a fresh session to see what you're actually on. Both remember whatever you picked last time, and you want that decision to be deliberate.Tip: if you already know a session is going to be grunt work, MAX_THINKING_TOKENS=0 claude turns thinking off for that one session (except on Fable 5), which is the step below /effort low.Prompt cachingIf a request starts with exactly the same tokens as a request the server just saw, the state for that shared beginning comes out the same, so the server can keep it around from last time and only prefill whatever comes after it. This is called prompt caching.Reading from the cache costs 0.1x the input price, because the server loads the state instead of computing it. Writing tokens into the cache costs a bit more than normal input, up to 2x, since the server also has to hold on to the state afterwards. But the write happens once per token, and the 0.1x reads happen on every turn after it.Claude Code manages the prompt cache on every request, there's nothing to turn on. However you can break it, so it's important to know how to avoid these cost spikes.Say we type "fix the failing test in utils.test.ts". Here's what Claude Code sends for it:Claude Code assembles the first request out of the system prompt (tool definitions included), your CLAUDE.md, and your message, and sends it off (input tokens). Nothing is in the cache yet, so all of it gets prefilled and written into the cache.The model can't fix a test it hasn't seen, so it thinks for a moment and responds with a Read call for utils.test.ts (output tokens). Claude Code reads the file, appends it to the conversation, and sends the whole thing again (input tokens). This time everything from request 1 is read back out of the cache at a tenth of the price, and the only thing prefilled at full price is what's new: the Read call and the file.Now the model wants the file under test (output). Another Read, another append, and everything goes out again: requests 1 and 2 from the cache, the second file at full price (input).The model responds with an Edit (output). Claude Code applies it, appends the result, and sends everything again. Same story: the Edit and its result are new, everything in front of them is a cache read (input).The model runs npm test (output). Claude Code appends the test output and sends everything again, with the test output as the only new part (input).The tests pass, and the model responds with a short summary (output). No tool call means nothing to append and no request 6, so we're done.That's five requests for one small fix, and every one of them contained the entire conversation up to that point. A typical turn is lopsided: tens of thousands of tokens going in, a few hundred coming out. But only what's new in that turn gets prefilled at full price. That's the whole per-turn bill: cache reads on the history, full input price on whatever's new, and the output price on the response.This applies on a subscription too. You don't see these prices directly, but the same requests are what draw down your limits.The cache has to match from the very start of the request forward, and requests always go out in the same order: tool definitions, then the system prompt, then the conversation (with CLAUDE.md at the front of it). If anything in that prefix changes, everything behind it gets prefilled again. A tool result appended to the end of the conversation is the ideal case, since nothing is behind it. What throws the cache away is anything that changes the request further towards the front, or changes what the cache is keyed on:/model: every model has its own cache, so on the next turn the entire conversation gets prefilled again at full price. (This includes opusplan, which switches models every time you go in or out of plan mode.)/effort: the effort level is part of what the cache is keyed on too, so it's the same story. It's why both /model and /effort ask you to confirm when you switch in the middle of a conversation.Fast mode: also part of the key, and the re-prefill happens at fast mode prices, so if you're going to turn it on, turn it on at the start. (Turning it off again is free, cache-wise.)/compact: the conversation gets replaced with a shorter one, so nothing in it matches anymore (the system prompt in front of it survives). Writing the summary itself is cheap as long as the old conversation is still in the cache, so it's a lot cheaper before a long break than after one.Time: every turn resets the clock, but the cache expires after an hour on a subscription or five minutes on an API key (ENABLE_PROMPT_CACHING_1H=1 makes it an hour).
Come back later than that, and the next turn prefills the whole conversation again. Resuming an old session almost always does too: the cache is usually gone by then, and the system prompt gets rebuilt at launch anyway.None of this means you should never switch models or effort. It means there are cheap moments to do it, the start of a session or right after a /clear, and expensive ones, the middle of a long conversation.Tip: if the last few turns went somewhere you don't want to keep, /rewind to just before them instead of running /compact. Rewinding only cuts those turns off the end, so everything before them is still cached and it costs nothing. Compacting rewrites the whole conversation, so it always costs something.What decides how many tokens a session sendsThe main thing to know here is that nothing gets sent just once. Everything that ends up in the conversation, a file Claude read or the output of a command it ran, gets sent again on every turn after it, for the rest of the session.