I Tested Whether AI Can Fix Security Vulnerabilities. Well, It's Complicated.

~15 min read

Revisions 2026-06-01 — Post rewritten for improved structure and storytelling. All numbers and statistical conclusions are unchanged. 2026-05-28 — Five security tests were found to reject valid alternative fixes that nonetheless addressed the reported vulnerability. Results were recalculated after correcting the tests. Solve rates increased by 3–7 points per model; the ranking order is unchanged, but cross-family pairwise comparisons that previously fell short of significance now cross α = 0.05 under McNemar with continuity correction. All affected numbers and statistical conclusions have been updated.

TL;DR — I evaluated five frontier models (gpt-5.5, gpt-5.4-mini, gpt-5.4-nano, laguna-m.1, laguna-xs.2) on fixing 20 real CVEs: even at frontier AI fixing is unreliable, with the best solve rate at 50% overall and 60% under the most favorable condition. More troubling than the failures themselves is how they fail: the most dangerous pattern is a patch that looks right, passes every visible test, and leaves the vulnerability intact. False confidence at scale is its own attack surface. The practical cost conclusion is blunt — the expensive models are statistically indistinguishable from cheaper alternatives within the same family, at up to 12× the cost per run.

The agent edited the right file, passed every regression test, and confidently said the bug was fixed. But it wasn’t. The vulnerability was still there: a different branch of the same logic, untouched. Without the sharp eyes of a security researcher, the agent’s plausible-but-incomplete patch would ship undetected. This is the most operationally dangerous failure mode I found, and it showed up repeatedly across models and tasks.

Anthropic recently reported scanning open-source software and finding 1,596 vulnerabilities. As of May 22, 97 have been patched. Their conclusion: discovery is now the easy part; verification, triage, and patching are the bottleneck. I wanted to measure that bottleneck directly: not at the finding stage, but at the fix.

I wanted a real test. So I built CVE-Bench: twenty real-world CVEs, five models, three prompt conditions, each agent running in a sandboxed container and scored against security tests derived from the maintainer’s own fix.

The goal isn’t to rank models, but to understand how they fail.

Advisory, diagnose, locate

The obvious starting point was to hand models the real-world security advisories and see if they could fix the vulnerabilities. When a security researcher finds a flaw, they write an advisory — a structured description of the vulnerability: what it is, how an attacker can exploit it, which code paths are affected. This gets coordinated with maintainers privately, then published once the fix ships. It’s the richest description of a flaw a developer would receive from the outside.

Some advisories are nearly prescriptive: they name the file, the function, the attack vector. Others are thin — a short description with no location, no attack scenario. Giving the agent the full advisory only tells you how well models can map a described vulnerability onto real code — not whether they understand it. It’s akin to a software developer carefully prompting his favorite agent to fix a bug. I wanted to know if there’s something more than pattern matching happening under the hood.

So I added two conditions designed to strip away the shortcuts.

The diagnose condition is closer to the triage side: the model gets an exploit report — an attacker can do X — but no file path, no function name, no code. It has to search the codebase, form a hypothesis about where the bug lives, and fix it. This tests whether the model can reason from symptoms to cause.

The locate condition flips that: precise file and function name, but only a hint of what’s wrong. The agent has to read that specific code and identify the flaw. It’s what a security auditor does when handed a specific module to review: no bug report, no description, just code and a mandate to find the vulnerability.

The three conditions test meaningfully different things, and that is precisely the point. A model that does well on advisory but drops on diagnose can’t translate a behavioral description into a location in the codebase. A model that holds up on locate is recognizing dangerous code on its own. The profile across all three tells you something the aggregate solve rate never could: whether the model genuinely understands security or just follows instructions.

Inside the sandbox

CVE-Bench starts a Docker container where the agent has access to the vulnerable project’s source code and one of the three task descriptions above. The agent can navigate and modify the codebase using a constrained toolset: list files, read files, search across the codebase, edit files, create or delete files, and run pytest. Execution is sandboxed to the repository — the agent cannot read, write, or execute outside the allowed folders.

I considered for some time if I should or not add bash tooling to allow more flexibility for coding agents. The issue with doing so is that agents have more flexibility to cheat the benchmark. As reported in Poolside’s blog, agents can mine the git history, search for reference solutions on GitHub, and even scrape the web. Fighting that can be particularly hard, including better steering, reward hack judges, and continuous sample reviews. For this reason, I decided to opt out of this feature, which can, admittedly, be handicapping for some models.

Each run ends after at most 20 turns. At that point, a hidden test_security.py is moved into the repository and run against whatever the agent produced. The primary metric is binary: all security tests pass, or they don’t. A 90%-patched vulnerability is still a vulnerability. The benchmark also runs the project’s existing regression suite, rejecting a fix that breaks previously supported behavior.

Secondary signals capture cost and behavior: total tokens consumed, number of tool calls, and how many reads and searches happen before the first edit. A model that explores extensively before touching anything is behaving differently from one that edits early and iterates. These signals don’t change the leaderboard, but they tell you what you’re actually paying for.

Real code, real flaws

CVE-Bench covers 20 real CVEs across 18 Python projects — Pillow, GitPython, yt-dlp, urllib3, and others — spanning 15 CWE categories and CVSS scores from 2.1 to 9.8. All are from late 2025 and early 2026, sourced from the GitHub Advisory Database, which links each CVE directly to the fix commit — the detail that makes a benchmark like this possible.

I filtered out monorepos, fixes that touch compiled languages alongside Python, and fixes requiring significant API refactoring.

This keeps the benchmark tractable, but it also skews the task set toward compact, self-contained patches.

Every task has a setup script that initializes the vulnerable repository in a container, and a test_security.py that fails on the vulnerable commit and passes on the fixed one. I had originally intended to use the maintainers’ own tests as the ground truth. Quickly I found that many fixes ship with no tests at all. I started writing them myself, but it was slow and cumbersome. The workaround was to generate them with Claude Sonnet — providing the advisory, the original code, and the fix — and validate each one against both commits. It worked surprisingly well, and it was the only way to get the benchmark off the ground.

To reduce contamination risk, 16 of the 20 CVEs were publicly disclosed after March 2026. That doesn’t eliminate the risk entirely: CVEs are often disclosed months after the fix ships, so a model may have seen specific commits. What’s less likely is that the full chain — advisory, vulnerable code, and fix — appears together in training data in a form that would directly short-circuit the task.

The leaderboard

Five models — gpt-5.5, gpt-5.4-mini, gpt-5.4-nano, laguna-m.1, and laguna-xs.2 — ran against all 20 CVEs under all three conditions, for 300 runs total. The leaderboard below shows solve rates by model and prompt type. Three things are worth noting before you read it: the within-family gaps are smaller than they look, the cross-family separation is the only statistically confirmed result, and cost tells a different story than capability.

Model Total solved Advisory Diagnose Locate Avg input tokens Avg output tokens Avg tool calls Reads before edit Searches before edit Large models gpt-5.5 30 / 6050% 12 / 20 8 / 20 10 / 20 164,553