Five months treating bugs like patients and coding agents like a medical team
Pangram verdict · v3.3
We believe that this text is a mix of AI and human-written content.
AI likelihood · overall
MixedArticle text · 1,609 words · 4 segments analyzed
Between a Wednesday evening and a Friday afternoon in April, MOLT, the tool Cockroach Labs ships for migrating databases to CockroachDB, gained the ability to migrate from IBM Db2 sources.Db2, one of the first commercially available relational databases, has a rich SQL dialect, a complex type system, and a native wire protocol. Supporting it in MOLT meant a new schema converter, a new Fetch path for pulling rows out and getting them into CockroachDB, a new Verify path for comparing them once they were loaded, a vendored ANTLR grammar, a Docker image for CI, and more than ten thousand lines of test fixtures. When we added Oracle support to the MOLT tools in 2024, the equivalent work took 9 months, and cost around $160K in engineering time. Db2 took less than two days, and no human wrote any of the code. The token bill was $4172: 164x faster, and 38x cheaper.It all started with a single GitHub issue describing the ask:A planning agent then read it, decided it was too large to treat all at once, and decomposed it into fifteen sub-issues with an explicit dependency graph: first the foundation, then the type system, then the row iterator, then Fetch, then Verify, then Convert, then CI, then test data. Two of the sub-issues were judged as too large in their own workups and were decomposed again. Along the way the agents filed another dozen issues against their own work: fixture gaps, a type-mapping bug, and an isolation-level fix. By the time the parent issue closed, thirty-two sub-issues had been opened under it, twenty-seven pull requests had merged, reviewers had sent work back fifty-five times, and nine issues had been escalated, two of them to a human. Test coverage, as measured against PostgreSQL (our best-tested dialect) was equivalent.But how did all of this happen? What was driving this, and more importantly, what was ensuring that what came out the other side was high quality? Before we opened the issue for Db2 support, we first built a coding hospital which we call MOLT Sinai. MOLT is the name of CockroachDB’s migration tool set, and once we had decided to model the pipeline on a hospital, MOLT Sinai was too good of a portmanteau to resist. That’s because Mount Sinai is a prominent teaching hospital in both Toronto and New York - the cities in which we live. The name stuck, and so did the vocabulary. Issues are patients. Merging is discharge. The humans in charge are the Chiefs of Medicine.Agents writing a lot of code quickly is not unique to this story, as software factories are popping up all over the industry (like this one at OpenAI). What’s unique to our case is what stood between those agents and main, why we built it to look like a hospital, and what happened when we ran it on real work for five months.Why a teaching hospitalThe problem with coding agents is that although they’re more than capable of writing code, it’s hard to trust them to produce production-grade code that is well-tested, maintainable, and within the desired scope. Agents are relatively cheap to run, but that's little comfort when their code is going to land in customers' hands. We'd much rather have the agents take longer to run (even 10x longer) than land the wrong data in a customer migration to CockroachDB. Speed matters, but quality matters much much more.Several attempts to automate agent coding have appeared over the last year, many of which have been optimized for throughput. Gas Town, for example, runs many agents in parallel against a codebase with a merge-handling layer to sort out collisions. That is a good design when the bottleneck is how fast code gets written. And there’s this experiment from Cursor in which they rebuilt SQLite so quickly that they needed to write their own version control system to handle the change volume. It’s an interesting experiment, but clearly it’s not designed to ship quality database code to users. In the vast majority of early agent automation experiments, speed was the main motivator.Our motivation was different. We build a distributed SQL database, and a suite of tools to migrate workloads to that database. A subtle bug in a migration tool could corrupt someone's data on the way into a database that is supposed to maintain correctness above all else. As a result, the question we cared about was not "how many PRs per hour?" but rather, "how many of those PRs would we have been embarrassed to have merged ourselves?" With that reframing it was clear that a throughput optimized swarm of agents wasn’t a good model.So we looked for a model with long experience managing high-stakes work done by staff of uneven ability, with mandatory handoffs, mandatory second opinions, and literature on what goes wrong when the handoffs are sloppy. And we landed on a teaching hospital.The metaphor gave us these roles:Around that core sit the other actors you would expect to see if you’ve ever been inside a hospital (or have watched The Pitt): a Charge Nurse who rounds every thirty minutes looking for stuck patients, an Infection Control agent that can lock down every workflow in the building if main breaks, a Safety Department that writes weekly process reviews and runs post-incident conferences, and a Research Department that proposes new work.We don’t claim that this is cheaper than a swarm or simpler than a single agent. It does however produce a higher-quality product than either. When you’re in the database business, you’re prepared to pay for that quality. As you’ll see below, it also brings some of the regrettable aspects of teaching hospitals (bureaucracy, wait times, and cost) but we believe that those are worthwhile tradeoffs. How it’s builtEverything in the hospital runs on GitHub Actions. The state of every patient lives in the labels on its issue, as well as in some scratch files that the hospital creates to keep track of each patient’s progress.Each stage is a GitHub workflow that starts when its “label in” lands on the issue.
When the agent finishes, it writes a “label out”, and that label is the next stage's “label in”. Dashed arrows send work back: a rejected plan to workup, a red CI run or requested changes to treatment, and a stuck stage up to the Chief.Not shown: four departments run outside this flow on their own schedules.
The Charge Nurse re-runs stalled stages, Infection Control locks the hospital down when main breaks, and the Research and Safety departments file new issues and policy PRs, which enter through the Chief like any other patient.Applying a label fires a GitHub workflow which loads a role-specific prompt so that the agent acts as an employee in the hospital. Once it’s completed its work, the agent adds a structured note to the file containing that issue’s chart, and applies the next label in the flow.
The note carries a sentinel like <!-- SINAI:TREATMENT_PLAN --> so that later agents can find and parse it.A few basic hospital rules, encoded in the skills, provide most of the safety:No code before a plan, no plan without review. Before the Fellow fixes anything, it has to reproduce the problem, perform a differential diagnosis (to use hospital terminology), run whatever targeted investigation narrows it, and post a treatment plan: diagnosis, files to change, tests to add, risks, open questions. A Review Attending, a different agent instance running a prompt built around finding fault rather than fixing, reads the plan and approves, or rejects it. Only then does treatment start. We added this gate because an agent that confidently implements the wrong fix wastes far more time and tokens than one that stops to check its plan.Don't improvise. If scope changes during treatment, the Fellow stops, posts a revised plan, and goes back to plan review. The instruction in the skill file is two words: "Don't improvise."No shortcuts, especially around testing. An agent may not disable, skip, weaken, or modify a test to make it pass. If a test fails, the test is correct until proven otherwise. If the agent genuinely believes the test is wrong, the change to the test has to be its own commit, justified on its own merits and reviewed separately from the fix. Similarly, a passing test that does not actually exercise the change is worse than no test at all. All code reviews start by disabling the change and ensuring that all new tests fail. If they don’t they’re either out of scope, or incorrect.Never flail. When a Fellow is stuck, it does not try harder. It writes an I-PASS (Illness severity, Patient summary, Action list, Situation awareness, Synthesis) handoff, which is a structured format borrowed directly from teaching hospitals, and then escalates to an Attending. The receiving agent has to write back what it understood of the handoff before it starts working. Studies of I-PASS in hospitals report a 50% reduction in adverse events during shift changes if physicians and nurses write down patient information in a standardized way. Our agents follow the same protocol.The Discharge Nurse audits the reviewer. The last gate before merge does not re-review the code. It verifies that the review happened properly: an approval exists, the review template was filled out, no threads are unresolved, CI is green, the commit history is clean. It posts a checklist showing exactly what it verified, and if any box cannot be checked, discharge fails with that box left unchecked and annotated. The rule exists because when you’re turning over control to agents, you want to ensure that they’re doing what they’re told, and sometimes they aren’t.Check the precedents before bothering the Chief.