Resisting Cognitive Surrender

Working Notes: a commonplace notebook for recording & exploring ideas. Home. Site Map. Subscribe. More at expLog. — Kunal This particular note is around exploring cognitive surrender: uncritically accepting whatever agents output without actually thinking, exploring or considering alternatives. Working this way is extremely fast, not particularly taxing, and unsurprisingly seductive. And yet I find myself repeatedly questioning if this is actually a good way to apply agents: often enough I find myself making suboptimal decisions, accruing technical debt and reducing my familiarity with the system I’m working on instead of improving it. As we collectively rethink how we work I’ve been finding it useful to read and share other people’s workflows, and consciously write about and examine my own. This particular post focuses on finding the right balance of trust while working with agents. Cognitive surrender is seductive One of the virtues of being a great programmer is to be lazy. After more than a decade of training myself to look for the shortest and simplest path it’s rather attractive to be able to simply hand over the problem. If I can spin up a software factory, why wouldn’t I? There’s also a lot of peer pressure (or game theory) at work: if any of my colleagues have seemingly surrendered, then I’m only going to look worse if I spend too much time trying to do silly things like understanding what I’m working on. And, of course, while everyone else is a slop-cannon generating code faster than they can read it I can build a meticulously designed software factory manufacturing perfect designs. The sheer volume of work also appears remarkably productive – we’ve all talked about just how many more PRs we ship every day. This tends to get compounded by the agents’ expressed confidence & conviction. Perhaps an inadvertent consequence of RLHF, but models tend to be astonishingly confident in what they recommend (and perhaps even more blasé while admitting they made a mistake). This makes it too easy to take their answers at face value, with whatever social programming I seem to have evolved with. OSS projects – particularly those facing time pressure – seem to accrue tremendous complexity quickly, to the point that it feels like the only way to interact with a project in any sane time frame is by heavily relying on agents to navigate and modify and test the code. This is also a self-reinforcing feedback loop: broken windows multiply rapidly with agents on the loose. [1] Technical debt and incidental complexity have a fairly attenuated feedback loop: complexity introduced by slop doesn’t bite immediately; it’s far more subtle and compounds over time when I’m not paying attention. Beyond all of that, I have to mention the hype cycle. We all should be skipping Netflix to watch talks by Anthropic engineers on making loops (or the agentic workflow du jour). [2] Why not surrender? It’s not entirely obvious that it’s a bad idea to surrender: if it’s easier and better, then why not take that path? So far, my personal answer is that I need to take the easier path consciously and intentionally, with enough effort to mitigate downsides. Part of what makes me deeply uncomfortable is suffering from a form of Gell-Mann amnesia while working with agents. When I have expertise and the overall design direction I can generally simplify the output significantly, controlling for complexity. When I don’t they seem to be the smartest engineers in existence – far beyond the more squishy, organic variety – and all I can do is follow in awe. And then the agent gets stuck creating a directory and I have to reset all my assumptions. For the current state of the art (October 2026) I generally find maximal use of agents leads to a significant explosion in complexity: something that mostly works, with extremely surprising edge cases and bugs. These are generally the projects with hundreds of thousands of lines of code shipped rapidly, breaking the scale supported by most software development tools, so you also need to revisit those – and this compounds rapidly. I have a strong aversion to incidental complexity, and generally refuse to build, maintain or extend it: it’s fine for a proof-of-concept, but it must end there. For systems that need to keep working, steering the design aggressively towards simplicity and modularity is what lets me build something I can actually operate (with and without agents). As models continue to improve rapidly, I expect the previous argument to go away as the models learn and can express a broader sense of engineering taste, good design and quality. The other reason – which I unfortunately expect to persist – is that my personal understanding and mechanical sympathy for systems I maintain erode extremely rapidly, or never get a chance to develop. In the early days when I would build things by hand I’d be able to quickly diagnose any bug or misbehavior based on my familiarity with building and maintaining the code: the more I rely on agents, the more my only option is to spin up one more session and hope; and the automated fix then further reduces my understanding in a self-reinforcing loop. Without context, I no longer have much meaningful judgement and can’t engage – or handle it in context. The Ironies of Automation (1983) is referred to often in articles similar to this one. I’d recommend directly reading the paper if you haven’t: it’s fully relevant to the decisions we have to make today. The paper warns that it becomes even harder to deal with tasks an agent fails at because we never manage to get any practice with the easier tasks; nor do we get a chance to internalize what a system feels like under standard operating circumstances. At the end of the day, I’m responsible and accountable for what I build, not the agents. Surrendering agency and not thinking basically makes me accountable without any meaningful ownership. There’s also the small matter of pride in the quality of my work.[3] Solutions The sombre opening shouldn’t lead you to believe that I’m not excited about the sheer potential unlocked by AI, and the opportunities to build things that I would never have been able to without it. It’s more about resetting expectations, explicitly working through the tradeoffs involved and choosing which ones are acceptable. And finding ways to maintain the quality and craftsmanship I care about. There are still tradeoffs The first step for me has been to internalize that I can’t truly have my cake and eat it: I can go terribly, inhumanly fast at building things but I need to accept that there’s no way to retain the mechanical sympathy, depth and confidence I used to have without putting in the effort. Time that used to be only counted as building a system also involved comprehension; agents can drive the time to build to zero but the next bottleneck – comprehension – hasn’t moved much so far. The hype cycle occasionally can make it hard to communicate this explicitly: a lot of the communication around agentic loops tends to assume that there are no downsides whatsoever. Choose your vibe limit & triage Several months ago I defined my vibe limit as the complexity of the change I can make with sufficient confidence using agents. I like the risk/complexity curve enough to post it again: The next step is figuring out where I land: there are several projects where good enough is fine, or the problem is sufficiently sandboxed that I’m happy to live with whatever the agents generated. On the other hand, if the system is important, fragile, can’t be sanity checked easily or isn’t particularly well isolated I need to directly spend my time. Aggressive prioritization, triage, and constantly recalibrating myself become really important in this ecosystem: I should explicitly choose where to surrender agency and where to spend my time (not unlike leading a team, except the pace is much faster). Constantly reviewing my predictions and the outcomes helps a lot in recalibrating, particularly with new model drops. Watch out for automation bias I also find myself needing to be much more mindful while working. A warning sign I watch out for these days is if I find myself repeatedly tapping “yes, continue” in a session. At that point either I should figure out what it would take to completely delegate the problem to Claude instead of trying to micro-manage the agent – or realize that I stopped thinking and I need to check the decisions being made. Completely delegating the problem generally means building a better sandbox or feedback loop, or figuring out ways to be able to sanity check much more easily before shipping to production. Alternatively, if I need to engage much more deeply I’ll use some of the other tactics described below. Force yourself into the loop The premise of augmenting myself by working in the loop with powerful thinking machines was a large part of my decision to join my current company. This generally comes down to directions you can give the model to reduce the size of the steps it takes, or explicitly choosing to drive instead of passively observing. For projects I deeply care about, choosing to drive instead of letting the model drive is one way to intervene in every single decision. It’s much slower to type things out, but it does force me to engage with every decision and gives me an opportunity to override it. In fact, for this post I’m using a Codex session to edit: help me tighten the post, point out the small typos, punctuation errors, or half-finished sentences that tend to embarrass me after I explicitly share a post publicly – but I make all changes by hand. In the AGENTS.md for the notebook: Never edit files directly, always wait for the author to request or confirm changes, or let him make the edits directly.