There is no posttraining data commons

We got extraordinarily lucky with pretraining data. Training large language models required a truly vast corpus of text, and it was just kind of lying around. Not exactly by accident. For the last 50 or so years, humanity has been busily creating and digitizing an enormous public record of human knowledge. Most obviously, Wikipedia’s editors have assembled the most comprehensive encyclopedia humanity has ever produced. A vigorous open source movement created a couple petabytes of software source code. Communities on Stack Exchange spent more than a decade asking questions, arguing about answers, and carefully curating the results. Scientists uploaded their papers to ArXiv. Project Gutenberg—almost as old as the internet itself—produced digital editions of out-of-copyright books. And of course, billions of people made websites, commented, had arguments both petty and profound, uploaded photos and paintings, wrote fiction, and—believe it or not—sometimes went and told lies on the internet. Efforts like Gutenberg, Internet Archive, Wikipedia, ArXiv, and Stack Exchange were self-conscious efforts at creating a digital knowledge commons. And then there’s Common Crawl. Since 2007, Common Crawl has been regularly archiving snapshots of a sizable subset of the internet, and then making it freely available for anyone to access and use. Today, there’s something like 10 petabytes of data archived by Common Crawl. This vast commons of human knowledge formed the basis for the LLM revolution. GPT-3 was trained on about 60% Common Crawl-derived data. The original Llama data mix was almost entirely extracted from the internet commons.1 That’s not to say that refining the raw material of the web into LLM-ready chunks was simple. Web pages are extraordinarily messy creatures, and a large fraction of the web is spam or junk intended to fool Google’s crawler. People are still writing papers on how to do filtering, HTML-to-text extraction, old document OCR, and synthetic rewriting. Though the incumbent large players have seemingly moved on with their own custom web crawlers (Cloudflare reports that over half of crawler traffic is for AI), modern open LLMs like Nemotron and our own still rely on this commons for the vast majority of their pretraining data. Without continued access to this commons, truly open LLMs cannot exist. These days, the focus of frontier AI research has shifted to posttraining, particularly so-called “agentic” AI. Like pretraining before it, posttraining requires vast amounts of compute and vast amounts of data. However, there isn’t the same ready-made trove of raw data lying around for posttraining. In short, there is no posttraining data commons. The fundamental difference is simple: Pretraining consumes documents, while posttraining consumes tasks.2 But “task” here is already an abstraction. The tasks used by reinforcement learning for posttraining are not the same tasks that humans perform in the world. Human tasks are situated in a particular moment. Someone wants something; they act in the world to get it; the world responds; the goal changes as they learn; and, eventually, they succeed, or fail, or change their goals. Once attempted, the task is gone. The same conditions will never quite occur again. In this sense, human tasks echo oral culture: An oral story is as much an event as it is an object. The telling of a story changes with the teller, the audience, and the occasion. Writing the story down makes it durable and repeatable, but it freezes a single telling. Encoding a human activity as an RL task does something similar. RL tasks are frozen, schematized, and legible, while human-world tasks are adaptive, conditional, and ephemeral. An episode of situated action is transmuted into an object that can be performed again and again. For an RL task, we have to preserve at least three components: a world, an intent, and a set of tools for interacting with the world. For reinforcement learning, we also need a grader that looks at the results of the agent’s actions and determines how well the intent was satisfied. Humans have spent millennia building institutions around durable knowledge stored in the form of documents: books, libraries, archives, the internet. We have had much less time, developed much less infrastructure, for making action durable. Except for one very important exception: software. Software engineering is, in large part, the practice of writing tasks down. Software engineers have spent decades turning work into durable artifacts: preserving the world in which the work happens, recording the intent behind it, and leaving a trace of the actions taken. It is no wonder that agentic AI came for software first. Computers are built around determinism. Restart a computer, and its state is more or less a clean slate, aside from any information—any documents—you've chosen to write down. (Mom, this is why I keep telling you to reboot your computer.) This determinism is perfect for reinforcement learning. Reinforcement learning algorithms work best when an agent can attempt a task, have the changes discarded, and try again from the exact same starting point. Over the last few decades, software has undergone a series of transformations to double down on the determinism of computers. Version control preserves the history of programs. Continuous integration and formalized unit testing libraries record procedures for exercising them. Virtual machines and containers encapsulate a controlled snapshot of the world a program needs to function in. None of this is perfect. Software too flows: system clocks move, versions change, packages disappear. Bits rot. Debian has been working systematically on reproducible builds since 2013, and more than a decade later still does not describe the distribution as fully reproducible. Despite these limitations, open-source software has created a commons of preserved worlds and recorded actions. But turning the software commons into tasks that can be consumed by machine learning is not easy. And it’s hard in a way that is qualitatively different from cleaning web text: the missing information may never have been recorded at all. Turning software repositories into posttraining data therefore looks less like curation and more like manufacturing. To build a task, we need the world, tools, intent, and a grader. In theory, the open source software commons gives us all of these: The world consists of the software repository at a point in time, its dependencies and the container and build machinery. The tools are just the tools of software engineering and computer use: the shell, compiler, browser, etc. The intent we can recover from GitHub issues or pull requests. The grader is whether the tests pass. It turns out that this is basically the framing used by the original SWE-bench. And it turns out to be both surprisingly difficult and much too simple. They crawled GitHub issues and pull requests from 12 Python repositories, yielding a bit over 90,000 PRs. Of these, only about 1 in 8 could be readily mapped to the frame: they either didn’t get merged, or weren’t linked to an issue, or didn’t change a test. They lacked a clear mapping for intent or the grader. Of the remaining 11,407, about 80% couldn’t be turned into executable tasks. For example, the package could not be reconstructed at that state, or there were weird runtime errors. These could not have their worlds reconstructed or their graders failed. This left the 2,294 tasks that comprised the original SWE-bench. A year later, OpenAI set about creating SWE-bench Verified. OpenAI hired 93 experienced Python developers to manually review 1,699 randomly selected SWE-bench tasks. Of these, more than two thirds were flagged for removal due to underspecified prompts, unfair tests, or other major issues. After filtering, they produced a pool of 500 “Verified” tasks. Then, a later audit of a problematic subset of the tasks found that another sixth or so of the overall 500 were also broken, despite the extensive set of checks leading to that point. I’m not trying to pick on SWE-bench. It’s one of the better constructed public coding datasets available. When examining the compilation of tasks in TaskTrove, we found that in one subset of 5,000 converted Java tasks, more than 90% had to be discarded during repair: tests did not parse, checked nothing, or depended on missing project context and external services. In another Python subset, almost 98% of the inspected verifier bundles could not even start their tests because dependencies were missing. And executability is only the beginning. An audit by Vals found that two thirds of the 2,698 coding environments in Xiaomi’s MiMo v2.6 public release still contained reference fixes recoverable from Git. In one demonstrated case, the agent copied the fix and passed all five hidden tests. Making good tasks is hard. And again, software is the easy case! And if even software is so difficult to get right—I didn’t even talk about building sandboxes that your agents can’t hack their way out of, and tasks that don’t have a predefined unit test for a grader—then how difficult must the rest of the economy be? How can the frontier AI companies reliably encode the rest of so-called “economically valuable human activity,” the work done by lawyers, scientists, salespeople, doctors, etc., as the kind of formal repeatable tasks that RL needs? For the rest of the economy, there is not even the beginning of a public “task commons.” So, what’s a frontier lab to do? Pay those lawyers, scientists, salespeople, and doctors to fit their work into the RL task frame, to inscribe the ephemeral tasks of their human world into the ones consumed by agents. They create the tasks, environments, rubrics, verifiers, simulators, etc. All of this work itself must be coordinated and verified. And thus we have the raison d’être of the rapidly expanding posttraining data industry.