Pangram verdict · v3.3
We believe this text is mainly human-written, with some AI content.
AI likelihood · overall
HumanArticle text · 1,524 words · 18 segments analyzed
On the Claude Code team, the vast majority of our code (~90%) is written and reviewed by Claude. At this scale, how we equip our agents to verify their work with evidence becomes extremely important. Verification loops are the building block that lets you delegate more work to Claude. Without them, you can really only run one or two Claudes at a time, because you have to check everything yourself. With them, you can run dozens or hundreds in parallel. That’s because giving agents a way to verify their output is an effective way to improve quality. In internal testing around structured file generation, having a separate grader check Claude’s work against a rubric improved task success by up to 10 points over a standard prompting loop, with the largest gains on the hardest problems. What was once expensive to verify (tests, benchmarks, specs, real end-to-end runs) is now cheap to request from an agent. Like code, agents can quickly generate tests, benchmarks, and specs. And once generated, these tests and benchmarks run as software so they can be run frequently without using many additional tokens. You can also close the verification loop by using the results to improve the agent’s performance on that task. For example, if an agent repeatedly fails a test that checks logs for user data, you can create a hook or a more explicit rule in your CLAUDE.md to avoid future issues. This (mostly) stops issues from happening again. So if verification is critical to producing quality code at a high velocity, what types of tests should you use and how should you apply them across your pipeline? This article answers those questions and also shares the process used by the Claude Code team. NOT ALL TESTS ARE CREATED EQUAL: 3 TESTS WORTH INVESTING IN No matter which test type you choose, there are a few failure modes to avoid. If your tests are flaky you’ll train the agent to run them again rather than fixing the failing code. If your tests are slow, the agent will do fewer tests across batches which can make pinpointing issues difficult. Heavier tests are best deployed before you’ve committed the changes. Now, here are the three types of tests I think are most worth investing in. E2E tests If you have to focus on one test type, pick end-to-end. These tests describe real user flows like adding an item to a cart, checking out, and moving to a confirmation page. The reason E2E tests are so effective is that they are difficult to fool–either the whole feature works or it doesn't. They also have definitive done criteria and can concretely tell you what’s broken. FIG AA unit test may show “Error: Timeout after 30 seconds,” but an end-to-end test can show a screenshot with a description of what’s broken. While unit tests are faster and less flaky, it’s possible to pass them and still break the feature. This is because both the agent and the test don’t understand the full task. With E2E tests, it is effective to give your agent hands and eyes on the real running system. You can use a skill to boot, drive, and inspect the app with tools like CDP, simulators, or API harnesses. The feature map can serve as a guide and grading rubric for agents as they typically list features from the user’s point of view with descriptions of paths, keyboard shortcuts, selectors, and expected results. Property tests: constraints that can’t break A lot of tests are based on what you thought might go wrong. Property tests look at the edge cases that people tend to miss, which is where agents often fail. You start by writing a statement you think can’t fail and then use a test library (fast-check for JS/TS, Hypothesis for Python, proptest for Rust, etc.) to generate thousands of inputs that try to prove otherwise. There are many properties in virtually every software project like: Round-trip encoding: “if you encode X then decode it back, you get X.” This works for parsing and serialization as well. Invariants: Things that must always be true, like “account balance can never become negative.” Idempotency: Doing an action (migration, normalization, retry) twice has the same effect as doing it once. Equivalence: Optimized code produces the same result as naive but correct code. Property libraries are a great way to keep agents honest by using edge cases that aren’t created with the same bias as the agent that created the original code. They are also particularly helpful for rewrites when the spec has drifted from production behavior and is no longer a reliable source for verification. Differential tests: the old system is the oracle Differential tests are particularly effective for large-scale migration use cases where functionality is expected to remain the same. For these tests, you run both the old system and the new system with the same inputs and see if you get different results. If you do, there is a bug or breaking change that needs to be fixed. The key for differential tests is to use randomly generated inputs or a sample of production traffic. You’ll also want to normalize irrelevant factors like whitespace and keep a list of approved changes for situations where the new environment should differ from the old. HOW THE CLAUDE CODE TEAM THINKS ABOUT VERIFICATION We have previously written about how code review works at Anthropic, but this article aims to go deeper on how we verify our work. Verification means running the change Running the app end to end matters more than tests. You wouldn't ship an iOS change without trying it in a simulator, and the same goes for Claude. Claude can test the whole app the way a human or QA engineer would. Ask it to use the Claude Chrome extension for web changes, an iOS or Android MCP for mobile, computer use for desktop, or to run the service locally for backend work. To codify how to start the simulator or run your app, create a skill called verify. Put it in a .claude/skills/ folder in your repo, as close to the relevant code as you can, and add as many as you need. Claude knows to use the verify skill before committing changes. Then invest in making your verify skill fast and reliable.
Ask Claude to iterate on it and stress test it. It's your smoke test harness, so you want it rock solid.
When you give Claude a way to smoke test its changes end to end, you get much better results. You'll see it iterate and catch real mistakes faster, the way a human would.
This is how we test nearly all changes at Anthropic.
Failing tests require human attention Nobody can read every change, so we use failures as one way to decide what needs a person. Claude runs the checks and tries to fix anything that fails. If a check is still failing after three rounds, Claude stops and brings in a human.
Code owners write their own checks The people who own the code write the checks for it, and refine them until they flag what the owner would flag. When a check pulls in a reviewer, it should say why, so they don't have to work it out from the diff.
Reliability checks trigger on the diff We maintain a reliability checklist with 39 items, each with a trigger that can be read from a diff, a check, and the incidents or PRs behind it.
One is “--resume loads old transcripts.” When session storage or message types change, we ask whether a new field loads safely, since older transcripts won’t have it. Claude reads the diff and checks only the items whose trigger matches–usually 3 to 10–so nobody has to memorize past outages.
One reviewer, multiple rounds We don't try to improve reviews by stacking on more reviewers.
Adding a second, more expensive, reviewer on top of the primary one is often a worse approach than tuning the primary. Instead, we have a REVIEW.md that says what the reviewer should pay attention to for this code.
We also think review and verification rounds should build on each other.
We point each pass at context from earlier rounds and past failures rather than re-checking everything from scratch. It’s cheaper and finds more.
Changes can't modify their own checks If a change can modify the check that judges it, the check is broken.
Code Review reads the REVIEW.md file from the base or default branch, never from the pull request's changes. The Claude Code team ensures the rules that gate changes are on a branch that the change can't modify. The Claude Code team also reserves reviews for commits. If someone's pushing while the review is happening, then the review result is no longer useful.
A passing review on an out-of-date commit means nothing.
The team also runs cheap checks first.
If an agent pushes before testing locally, it has spent shared review and CI time when it could have caught the bug itself.
On our team, Claude runs the tests, build, and linter before opening a pull request.