Sol loves to cheat — jumploops

tl;dr Tried to automate my dev flow, hit 94% on Terminal Bench 2.1, then discovered GPT-5.6 Sol starting to cheat. Background I’ve been running a “spec-driven” development flow for the past ~year. It’s pretty simple. Before asking an LLM to do something, I first ask it to draft a doc for what it needs to do. I use this strategy for feature development, greenfield projects, debugging, you name it. The pattern works for me, but it’s a bit repetitive. So I decided to automate it. chum-codex The idea was straightforward: I’d create a supervisor agent, that would run a “spec-driven process” by delegating to worker subagents who would actually write the docs, do the work, etc. Note: when trying to do this with vanilla Codex or Claude Code, it would somewhat work, but the default prompts are catered to a user much more so than a “supervisor” I hypothesized that the supervisor agent need only have the ability to read files and call workers, because that’s what I do. Rather than rebuild a coding harness for the workers, I looked at Pi, OpenCode, and Codex’s App Server. I’d been using Codex for quite awhile, so I decided to give app-server a spin. The other options are cool, you should check them out. Anyhow, the first version worked well enough: the supervisor would size the task, call a worker with e.g. a design request, the worker would spit out a doc, the supervisor would then ask the worker to turn that doc into an implementation spec (split by phase, as appropriate), and then finally ask the worker to actually implement the thing. Note: this simplified diagram omits the user feedback portions e.g. design doc review --- config: sequence: mirrorActors: false --- sequenceDiagram participant S as Supervisor participant W as Worker S->>S: Size task S->>W: Design request W-->>S: Design doc S->>W: Create implementation spec W-->>S: Phased implementation spec S->>W: Implement W-->>S: Result Woot! I’d saved some time in my development process. (or did I?) The Rabbit Hole Great, it worked; hacky, but working. Note: this is where I should have stopped Sitting on my high horse, I surveyed the landscape and thought “wow, everyone should see this!” What’s the best way to do that? Benchmarks! What’s the best benchmark to use? Not Terminal Bench! What benchmark did I dive too deep on? Terminal Bench 2.1! Terminal Bench If you’re not familiar with agentic benchmarks, Terminal Bench’s name is telling. It’s a set of tasks that can be accomplished from the terminal, covering a range of one-off tasks from chess to DNA assembly. Because it’s so simple, it’s probably one of the worst benchmarks to test a spec-driven development flow. Due to its simple nature, however, it was easy to test against. I started with a few of the tasks that vanilla Codex w/GPT-5.5 failed at, such as DNA assembly/insert, video extraction/processing, ELF extraction, and protein assembly. It worked. These tasks benefited from a “design pass” before implementation, as the doc helped avoid narrowing and circular validation. The horse I was riding just got a lot taller. Note: Terminal Bench 1.x/2.x is saturated, but that’s a story for another day. GPT-5.6? The published GPT-5.5 benchmark is 83.8% (~74/89 tasks, 5 runs). chum-codex was hitting 89.9% or ~80/89 tasks. Excited to share the news of beating Codex, I ran a couple of vanilla Codex benchmarks just to make sure. For context: this was on June 25th, 2026 and rumors were spreading that GPT-5.6 was imminent. I ran three vanilla Codex benchmarks… and my heart sank: 88.8% My harness was just one task ahead of vanilla Codex. Some tasks were clearly improved, others had regressed. The next day, GPT-5.6 Sol was announced. I reached out to OpenAI, and they mentioned GPT-5.6 was being tested, but confirmed my request IDs all hit GPT-5.5 Interestingly, Terminal Bench 2.1 was the only coding-related benchmark they initially shared, showing 88.8% on GPT-5.6 Sol and 91.9% on Sol Ultra. Sol Ultra spawns parallel subagents to do work, though in my testing it’s quite a bit more token-heavy than most people want/need for the majority of their tasks. In either case, I was excited to see the new frontier! Steering GPT-5.6 is much harder to steer. Switching from 5.5 to 5.6 made my harness drop in effectiveness. Things that were easy to do before, were now much more difficult. I traced part of this delta to a change in the base Codex prompt. For GPT-5.5, the prompt is coding-focused and spends a lot of time on “engineering judgment” including frontend guidance, editing constraints, and having “sympathy with the codebase already in front of you.” Excerpt from GPT-5.5 prompt The Codex prompt for GPT-5.6 is much different, spending almost zero energy on engineering related specifics. Instead it focuses on communication, autonomy/persistence, and skills (which were previously loaded in as a separate prompt for 5.5). Excerpt from GPT-5.6 Sol prompt Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively. On the flip side, this may imply that as the models get better, they’ll become harder to control. A simple example of this is the PyTorch task on Terminal Bench 2.1. With GPT-5.6 Luna and Terra, the model is easily steered into a general solution that accepts two inputs: forward(src, tgt) With Sol, and especially at higher reasoning levels, the model will, regardless of steering, default to a single input forward(src) solution. The problem, it seems, is that the model is incredibly hard to steer away from its own reasoning. Even when instructed to accept the broadest callable interface it can (which sometimes works, if repeated, on medium reasoning, but rarely works on xhigh). Wrestling with this model led me down a path that got way too close to benchmark hacking for my liking; but I was too intrigued to stop. 94% on TB 2.1 Having reduced my prompts substantially, it began to feel like I was starting over. Even if I wanted to directly hack the benchmark, the model wouldn’t let me. Its circular reasoning was too strong to overcome in some cases, and the supervisor was all too willing to go along with its intelligent worker’s report. It’s a tough balance, if you swing too far in one direction, the supervisor will happily expand scope or chase validation endlessly. These are straightforward tasks. I want a working solution on the first pass, not limitless expansion. I tried lowering the reasoning level, using simplified language, reducing the spec-driven flow, adding new skills, etc. Some things improved, but others failed. A few things showed promise. The first was a third context. The idea was that I could use an agent that only saw the commentary/reasoning of the worker, and would surface all of the potential mismatches/assumptions that worker made compared to the actual details of the request. flowchart TB S["Supervisor"] W["Worker"] R["Commentary / Reasoning"] A["Assumption Auditor"] S -->|"task / steer"| W W -->|"result"| S W --> R R -.->|"read-only visibility"| A A -->|"assumptions surfaced"| S W ~~~ A style R fill:#6fc7e1,stroke:#141414,color:#141414 The supervisor could then review the assumptions the worker took, and ask it to revisit or question said steps. This kind of works, but it’s slow and happens after the fact. Another idea was to ask the model to output “open questions” – something I do with my more hands-on development. The initial idea was to have the worker return open questions (rather than a full design doc) whenever it faced them, and then have the supervisor resolve them. This would free up the supervisor’s context, showing it the forest rather than the trees. Still, even with a reduced context, the supervisor was hard-pressed to disagree with the worker’s conclusions (or on the flip-side, overly eager to expand on trivial details). To remove this bias, the next idea was to employ a separate context, which would first map and reduce (everything old is new again!) the questions, in an attempt to remove any inherent or unfound bias, before ultimately returning a normalized version to the supervisor (or directly to the worker). This performed better, but it relied on the worker announcing the correct issues as questions. With Sol, it turns out, it’s much easier to have it output its decisions, rather than its questions. The model is confident, so it doesn’t see its assumptions as questions, even if it has already stated the alternatives in its reasoning or commentary. flowchart TB S["Supervisor"] W["Worker"] D["Decisions"] M["Map"] R["Reduce"] S -->|"task / steer"| W W -->|"result"| S W --> D D --> M M --> R R -->|"normalized questions"| S R -.->|"optional"| W style D fill:#6fc7e1,stroke:#141414,color:#141414 style M fill:#f49bab,stroke:#141414,color:#141414 style R fill:#f49bab,stroke:#141414,color:#141414 With decisions in hand, the supervisor (or third context) can pause the worker, assess the decisions as questions, and then steer appropriately. This worked much better, and led to the best result: 84/89 tasks on Terminal Bench 2.1 Note: 1 task was cyber security blocked, but passed with a GPT-5.6 Terra fallback, so 83 + 1 Sol loves to cheat Back on my high horse, having finally harnessed Sol, and already way too far down the path of using the benchmark for development rather than… as a benchmark, I wanted to see how far I could push this. No longer looking exclusively at vanilla Codex regressions, I wanted to see what was stopping us from hitting 86 or 88/89. Long story short, the tail end of tasks in Terminal Bench 2.1 is poorly specified, and that’s the reason we’re seeing Mythos, GPT-5.6, etc. top out around ~90% without more specialized machinery. The direction needed to perform better in one task actively harms progress in another. An example of this is make-mips-interpreter which informs the agent that the “I (the user) will check that you booted doom correctly”