advanced-context-engineering-for-coding-agents/wsff.md at main · humanlayer/advanced-context-engineering-for-coding-agents

Why Software Factories Fail or: harness engineering is not enough NoteI run a company (HumanLayer) building tools in the human/agent collaboration space, so what I'm gonna say below may be a tad biased. Perhaps in spite of that, I hope you find the subject helpful or at the very least that you find it as interesting as I do. -Dex i guess we doin loops now We're all racing to put AI coding into production. A lot has been said about loop engineering, and the prevailing wisdom is that we should probably write more loops.1 StrongDM wrote about their lights-off software factory where no human reads code and no human writes code. The narrative goes something like this: You are the bottleneck. The models are good enough. Code is free. Just ship more stuff. Ryan Lopopolo of OpenAI wrote about this in February and gave a talk in April about OpenAI's software factory, Symphony. These people are all really dang smart and I have a ton of respect for them. But the most cynical take here would be to call this yet another excuse to pump more VC money into the slop cannon. it's uh...it's going Our friend Mario got up at AI Engineer Europe and begged us to slow down -- because companies that have no business having outages due to coding-agent mishaps, are, well... having outages due to coding-agent mishaps. As Matt Pocock put it, codebases are falling apart faster than they ever have before. I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. edit - there is some conversation with the team on hacker news on July 23 - sounds like we might get a more formal update soon! The folks at Faros AI put out a report: since we2 all picked up these AI coding tools back in January and February, pull-request review quality is way down. More comments, longer comments, and tons of PRs getting merged with no review at all. Incidents are way up. Bugs per developer are way up. This report is more of a correlation signal than a verifiable smoking gun3, and the whole point of this post is to be wary of slop data, but it feels directionally valid based on what I've seen. "

You're holding it wrong" (you're not) A lot of people will tell you that this is a skill issue -- that if you're not getting good results, that's your fault. But however you're choosing to...erhm...hold it, I guarantee you're being told that if token-maxxing isn't working for you, it's a skill issue. You just need to spend more tokens. Let go of reading the code. And if you're just getting there, I promise it's part of the progression. I thought this way last summer too. Unfortunately for my ego, some dumb stuff I decided to say about "how to hold it better" got recorded and now has about a million cumulative views on YouTube. I am not trying to brag here, I share this only to establish that I've been going deep on the best ways to use coding agents for a long time now, and have discovered some things that many others have found genuinely useful. Advanced Context Engineering for Coding Agents No Vibes Allowed -- Solving Hard Problems in Complex Codebases Everything We Got Wrong About RPI Anyhow, The promise of all this online "just token harder" yapping we've been forced to endure is, succinctly: with enough harness engineering, we can get the best of both worlds: 10 to 100x faster, high quality, and nobody ever has to do that thing we all hate called code review All we have to do is configure more linters and sprinkle some magic words like "adversarial review" onto enough PR review bots, and our software will happily build itself without incident. This is not a skill issue What I'm gonna try to convince you is that no amount of harness engineering or loopsmaxxing can solve what is fundamentally a model-training issue. To grapple with this, I had to dig into how coding models are actually trained and evaluated - with respect to both the RLVR and the benchmark side of things.

In this post I'm gonna run through: Software factories date back to 1968, how have they evolved, and how has AI changed them Why models can generate mountains of slop despite ace-ing benchmarks (even the brand new "frontier" benchmarks) In spite of this, you can move pretty fast without setting your codebase on fire I'm gonna try to cut through the hype of every daily-emerging skills plugin and the ai-psychosis-tokenmaxxing advice pandemic, and talk in general terms about the types of things that work without referencing any particular skill or framework. Video Version: this post is based on (and expands upon) my keynote at AI Engineer World's Fair 2026. Thanks to @addyosmani, @CyrusNewDay, @HamelHusain, @zeeg, @dillon_mulroy, @nayshins, and @jeffreyhuber for feedback on this post. An aside: this has nothing to do with vibe coding Addy Osmani detangled this thing that is worth highlighting: A developer vibe-coding a side project a dozen people will ever run, and a team keeping a ten-year-old enterprise system alive for another quarter, share almost no constraints worth naming, and most of the advice in circulation is really one of those two people telling the other how to live. If you love vibe coding, please, go on vibing. I still vibe code lots of things, I just also maintain lots of production software (and through HumanLayer, help 1000s of other engineers do the same), so the rest of this is aimed at folks solving hard problems in complex codebases. I hear the word brownfield a lot to talk about this split. Historically that meant some ten-year-old Java thing, but at the pace we can ship now, it feels like an agent-built codebase starts to struggle after maybe three to six months -- you start to slow down, and the way you approach adding new things has to change. A brief history of the software factory I've been building and studying software factories my whole career, but I only learned this recently: the term traces all the way back to a NATO conference in 1968 -- the same one that gave us "software engineering."

The only other bit I find super interesting since then is that the US Department of Defense wrote a 31-page pdf about how the DoD needs to start using jenkins better or something. The 2022 software factory Let's ground our "software factory" definition around 2022, right before AI. In a typical software factory: People decide what to build -- engineers, PMs, leadership driving the vision It goes in a tracker -- Linear, Jira, whatever: a state machine of what needs to happen Someone grabs a ticket and builds it -- probably does some manual/automated testing while they're at it Pull request -- automated checks, a human reviews the code, maybe someone pulls it down to test Anything wrong? Loop back to "someone builds the thing" Ship to prod -- and it makes contact with users Add monitoring -- there's an entire industry built around paging an engineer at 3am when something breaks Users complain -- ask for things, find bugs, file feature requests → back to the team to add to the tracker wsff-boxes-2x.mp4 And on and on. We haven't even hit AI yet, and there are already several loops in this picture. front-loading alignment The thing teams figured out decades ago: building takes hours or days, and so does review. So we front-load the work -- planning, architecture proposals, sprint planning -- together, as a team. That means: less rework, because we aligned before anyone wrote code less time reviewing every line, if you've ever read a long-but-well-done PR, you know how fast the review goes when it's close-to-perfect We'll come back to this later - let's look at what happens when you bring agentic coding into the picture. The agentic software factory Now every company and their mother -- Ramp Stripe WorkOS Brex has spent the better part of this year explaining how they built an agent factory that ships on the order of 75% of their code. The agentic factory looks mostly like swapping "someone builds the thing" → "an agent builds the thing" -- there's some stuff here like orchestration, a harness, a sandbox, a model, computer use, etc. I won't go in depth on those details because quite frankly I'm sick of reading about it and I'm sure you are too.

When the agent builds the thing: Building drops from hours or days to minutes or hours. Review still takes hours or days. A human still has to read the code and test the change. So review is now the bottleneck. So you speed review up too: Agentic code review, to catch style, bugs, security. Agentic regression testing, to poke it from the outside with browsers and computer use and maybe send you a cute little video when it's done Review is faster now, but it's also probably still the bottleneck. But we can do more loops. Next you might route incidents into the factory. Instead of paging someone at 3am, they wake up to a PR that maybe already fixes it. We can also route user feedback into the factory. People ask for stuff, it gets built. At which point the job is two questions: how much can you stuff into the queue, and how fast can you review and test what comes out? Which brings us to the lights-off software factory. The lights-off software factory Dan Shapiro coined this term and Simon Willison wrote about StrongDM's implementation of it -- where we no longer read the code. You look at your beautiful software factory. It's ruined by that annoying little code review step and you say: you know what, that thing where a human reads every change? No thanks. So you drop it, and you put the effort somewhere else: Invest in testing and letting the agent test its own work Invest in sandboxes and orchestration Invest in automated review Invest in monitoring Invest in rollout Invest in collecting feedback signals from users And now the job really is just one question: how much stuff can we ask the agent to build? How much of the ocean do we want to boil? This is going to go great (its not) I'm going to posit something potentially controversial: the lights off factory does not work. Let's get into why software factories fail. We tried this In July 2025 we went full lights-off. Just read the specs and the tickets, background agents for all the small/medium stuff, the whole thing. If you've tried this seriously for a few months, you already know how it ends. You find at least one issue gnarly enough that the agent can't solve it -- even with your most advanced prompting and workflows.