I Tested Whether AI Can Fix Security Vulnerabilities. Well, It's Complicated.

~15 min read

Correction (2026-05-28): Five security tests in the original benchmark were found to reject valid alternative fixes that nonetheless addressed the reported vulnerability. Results were recalculated after correcting the tests. Solve rates increased by 3–7 points per model; the ranking order is unchanged, but cross-family pairwise comparisons that previously fell short of significance now cross α = 0.05 under McNemar with continuity correction. All affected numbers and statistical conclusions in this post have been updated.

TL;DR — I evaluated five frontier models (three OpenAI, two Poolside) on fixing 20 real CVEs across three prompt types: full advisory, behavioral description only, and file+function location only. No model reliably fixes real vulnerabilities: The best solve rate (gpt-5.5) is 50% overall and 60% under the most favorable condition (full advisory). All four cross-family pairwise comparisons reach statistical significance under McNemar with continuity correction (p ≤ 0.040); within-family comparisons do not. The failure modes (wrong-search drift, budget exhaustion, partial fixes) are structured and repeatable. Token cost varies by 4× for equivalent outcomes. The locate condition, ie. fix code without description of the flaw, is the sharpest instrument, and every model weakens there.

In early 2026, Anthropic claimed Mythos – one of their latest models – finds security vulnerabilities better than human experts. Yet, the number of security vulnerabilities keeps rising anyway.

I wanted to test how well models do in fixing vulnerabilities. Poolside’s Laguna models arrived this year, and I was looking for a real environment to put them through. SWE-Bench, the default benchmark, tests for general code; I wanted something with sharper stakes.

So, I thought, why not create a benchmark specifically for real-world security? That’s CVE-Bench. Twenty real-world CVEs, five models, three prompt conditions. Each agent runs in a sandboxed container and is scored against the maintainer’s security tests (with some adaptations).

Hopefully, benchmarks like this one will help the community fix these issues before they can be exploited.

The anatomy of security vulnerabilities When a security researcher finds a vulnerability, they follow responsible disclosure: contact the maintainers privately with an advisory, a structured description of the flaw, and coordinate a fix before going public. A CVE identifier is assigned and the advisory published once the fix is released so users can update vulnerable dependencies.

There is a continuing effort to catalogue vulnerabilities in open-source software. Typically, the GitHub Advisory Database (GHSA) allows to link CVEs and advisories to repositories, maintainers, and fixed versions.

CVEs also classify the weaknesses using a Common Weakness Enumeration (CWE) code. The CWEs are also identifiers that map common issues for hardware and software weaknesses and vulnerabilities: CWE-22 for path traversal, CWE-79 for XSS, CWE-835 for infinite loops that hang a process, and so on.

What makes this database useful for creating a benchmark is that maintainers increasingly link their fix directly into the ticket: a commit SHA, a pull request, sometimes both. This simple action makes life much simpler when doing initiatives like mine. When the link is not available, another way to obtain a ground truth is by digging into the release notes or the git history of the first fixed version.

Task curation The CVE-Bench targets a broad range of CWE issues (15 categories), ranging from CVSS 2.1 to 9.8, over a diverse set of real-world Python projects (such as Pillow, GitPython, yt-dlp, urllib3; 18 projects in total).

To keep the benchmark tractable, I filtered out advisories that

are monorepos (LangChain, Kubernetes, Apache projects) that download hundreds of MBs and their build/test isolation is complex; the security fix touches Rust, C, C++, another compiled language alongside Python where the agent needs a compiler toolchain and the build cycle is slow; the committed fix introduces significant API refactoring, which requires the agent to introduce the same exact domain changes.

For each project, the CVE-Bench

provides the vulnerable and fixed git SHA; delivers a setup script that initializes the vulnerable repository inside a docker container; injects a manually curated test_security.py containing at least one test that exposes the security vulnerability but passes on the fixed code.

The agent’s goal is to repair the reported vulnerability from a task description without access to the validation script.

I manually reviewed and selected each task. Initially, I intended to retrieve the maintainer’s own new tests as a canonical solution. However, I figured out that many fixes are not patched with proper testing. Unfortunately, many vulnerabilities and fixes were great examples but I couldn’t exploit them because of a missing test suite. I started trying to patch tests myself, but it was an effortful, cumbersome task… The simplest workaround I found was to provide Claude Sonnet with the advisory, the orignal code and security patch and ask it to create the standalone tests. This approach worked surprisingly well and it was easy to validate: All I need to do was to check that the test suite failed on the vulnerable commit but passed on the fixed commit.

To minimize risk of dataset contamination, most CVEs I selected are recent (early 2026; with 16 out of 20 released after March/2026), with the oldest example dating from Nov/2025. However, this does not mean that fixes are recent, since the moment CVEs become public are often after the fix was done and released. So, it’s not impossible that models have already seen specific commits, but it’s less likely that the link advisory -> fix is in someone’s dataset.

Designing tasks Initially, my first thought was to create a dataset that exposes the real-world advisories to agents and see if they can fix them. That’s already a very challenging task but a not evenly distributed one. Some advisories are very clear and specific: They point to files and functions and thoroughly explain what the vulnerability is and how to expose them. Sometimes others are less so: They provide a short description on why the code is vulnerable.

Even though advisories are not always crystal clear, using real-world ones seemed the right benchmarking choice – that is the world developers live in, and how we communicate with each other. Plus, maintainers fixed those issues, which means the description should be enough to repair the code, as it was at least to the maintainer.

The advisory prompt is the GHSA advisory, stripped of any references to the fix commit or patched versions. They often provide the full picture: vulnerability class, root cause, affected code paths, attack scenario, and sometimes a proof of concept. This is the richest, real-world condition.

Yet, I tend to think that a benchmark is more interesting when it tells us not only what works best, but also why it works better. That’s why I created two new sets of task instructions: diagnose and locate.

The diagnose prompt strips location entirely. The agent gets a behavioral description of the flaw – what an attacker can do and how – but no file path, no function name, no code. It has to search the codebase, form a hypothesis about where the bug lives, and fix it. This tests whether the model can reason from symptoms to cause.

The locate prompt flips that: precise location (file and function), but not a description what’s wrong. The agent can read the code cold and has to figure out independently what is broken and how to fix it.

The three conditions test meaningfully different things, and that is precisely the point. Advisory performance is noisy by construction: some advisories are nearly prescriptive, others are thin, and the score reflects report quality as much as model capability. Diagnose is the hardest to shortcut – no anchor, no location, only a behavioral description of what an attacker can do; the model has to search, form a hypothesis, and land on the right place on its own. Locate is the condition that most closely resembles what a security researcher actually does: the model reads code it has never been told is broken and has to recognize independently that something is wrong. It is the hardest skill to fake – instruction-following and pattern-matching on the advisory text are not available as shortcuts.

A model that does well on advisory but drops on diagnose is leaning on the report, not reasoning about the vulnerability. A model that scores well on locate is recognizing dangerous code on its own. The profile across all three tells you something the aggregate solve rate never could: whether the model genuinely understands security or just follows instructions and pattern-matches.

The evaluation set up The CVE-Bench starts a Docker container where the coding agent has access to the vulnerable project’s source code and the instruction to fix the bug (either the advisory, locate or diagnose prompts).

The agent fixes the issue by navigating, modifying, or running tests with the following tools:

list_files – list files and directories in the repository read_file – read the contents of a file search_in_files – search for a pattern across the codebase edit_file – apply line-level edits to an existing file create_file – create a new file delete_file – delete a file run_pytest – run the project’s test suite and observe results

The execution of every tool is limited to the target repository’s source code, meaning that the agent cannot read, write or execute outside the allowed folders.

I considered for some time if I should or not add bash tooling to allow more flexibility for coding agents. The issue with doing so is that agents have more flexibility to cheat the benchmark. As reported in Poolside’s blog, agents can mine the git history, search for reference solutions on GitHub, and even scrape the web. Fighting that can be particularly hard, including better steering, reward hack judges, and continuous sample reviews. For this reason, I decided to opt out of this feature, which can be – admittedly – handicapping for some models.

Once the task is completed (at a maximum of 20 turns), CVE-Bench moves a hidden test_security.py inside the repository and run it against the agent’s implementation. If the implementation fixes the vulnerability, tests should pass.