Verification Is Not Knowledge

This is a guest essay by Jessica Hullman, Ginni Rometty Professor of Computer Science and Policy Fellow at Northwestern University. Diego Velázquez, The Fable of Arachne (c.1657)In his essay on the role that rigorous proofs play in mathematical advances, Bill Thurston writes that the measure of success in math research is “whether what we do enables people to understand and think more clearly and effectively about mathematics.” Many recent discussions of AI for science hinge on a tension between understanding and answers. Is science about humans achieving insight, or the archiving of verifiable solutions?One side prioritizes outputs. Progress is implied to be evident in verifiable proofs of longstanding conjectures and cures for diseases, whether humans were involved in their making or not. The other side argues that human understanding is paramount, because it drives integration of new findings into the canon, because it shapes and inspires the next generation, and so on. But the first side, those who emphasize correct outputs, dismisses this as gatekeeping or nostalgia.When Terry Tao posed a thought experiment in a recent talk, asking how comfortable people would be taking a cancer treatment produced by AI, complete with a formal certificate of correctness and a passed phase 3 trial, but without any human understanding how it was found, critics suggested that Tao’s reasoning missed the point. Of course a patient with stage 4 cancer would welcome the treatment.Other examples concern ambiguity in authorship. Tom Dietterich wrote about arXiv’s growing problem with papers submitted by human authors who seem unlikely to have the expertise to understand the claims. Many questioned in response why accountability – by which I mean being answerable for a claim, able to describe what it rests on and respond when it’s challenged – should matter if verification has been solved. I suggested peer reviewers reserve their expert judgment for texts that don’t get flagged as heavily AI-written, so as to prioritize review as human exchange, but some see this as an obstruction to truth-seeking rather than an attempt to preserve it.The problem is that if we all agree that outputs matter–that we do care about the solutions to longstanding conjectures, that we do ultimately want to know how to cure diseases–then understanding appears as something separate. It’s cast as a mental state that looks optional next to the cure or the proof itself, and becomes easy to dismiss as a second-order constraint that we tack on because historically we’ve required it to arrive at the solutions in the first place.When defenders of understanding don’t dispute this separation, they immediately lose the argument. It becomes easy for the side that prioritizes output to peg them as simply not trying hard enough to envision a new future. But the superficial framing of understanding that we keep falling back on obscures deeper concerns that these examples share. They are less about anyone’s mental state than about what happens when something goes wrong. Tao’s scenario turns on the possibility that the AI passed the trial by exploiting a weakness in the trial itself. A paper whose authors can’t explain its claims leaves no one to consult if those claims fail.Where along the way did we come to believe that scientific value lies in correctness alone?This intuition is ultimately less about AI than it appears. The inclination to let AI drive science is less a sign of how forward-thinking one is about our future with AI, and more a symptom of a “value-free ideal” that has shaped our thinking since the Cold War era: a view that only epistemic criteria (not ethical and social values) deserve a place in judging what the evidence shows. That it pervades our debates about AI for science is not so surprising; it’s only in the last few decades or so that rejecting the value-free ideal became the philosophical mainstream.This ideal feeds unrealistic expectations and keeps us from seeing that pushback against the rapid infiltration of AI output into the scientific record is neither anti-AI nor anti-science. The stakes are bigger than any one debate or field. The value-free ideal distorts how we go about seeking truth, causing us to mistake verified results for knowledge. It threatens the human capacities that sustain science, causing us to see criticism and accountability as inconvenient overhead rather than the mechanics of knowledge production. It leaves us confused about what understanding is, and consequently why it matters.The social contract for science represents an idealized relationship between science and society that arose after World War II in the US: scientists receive public funding and autonomy, and in return society receives the benefits of their inquiry. Built on a distinction between pure and applied science and a “linear model” in which basic research feeds applied work, the social contract replaces abstract arguments about the value of understanding with practical societal benefits, reshaping how we see science. Drawing on arguments made by concerned scientists in the late 1800s, who worried public celebration of new inventions like electric lights and the steam engine would crowd out respect for what they viewed as their more essential contributions, it also helped to forge a value-free ideal that says that at its core, science must be removed from “contextual values” like ethical or social concerns. For the social contract to gain broad societal buy-in as a model of scientific progress (and consequently shore up government investment in scientific research), we needed to imagine truth-seeking as the impersonal application of methods, immune to political or social pressures.Philosophers of science aspired to inquiry based only on “epistemic” (or “cognitive”) values. Any influence from the scientist’s intentions or values came to be seen as a risk that the right methods should minimize, laying the groundwork for today’s questions about why it should matter who asserted the claim so long as it’s correct.Many debates about AI’s role in science come down to disagreements about whether the scientific truth can be realized purely through use of the right procedures. Over the course of the late nineteenth and early twentieth centuries, there was almost a moralizing drive to eliminate as much of the scientist’s self as possible from the scientific process, so that trust in individuals could be replaced with trust in the system.AI agents set loose on the scientific literature, able to verify and intervene experimentally themselves, look like the long-awaited fulfillment of nineteenth century scientific ethics. Much of the “AI for science” rhetoric inherits what I call the “science is naturally aligned” premise: by enacting the scientific method at scale, with its various justification procedures, we can reach a place where we no longer need human judgment to produce truth.Is what is true really so separable from understanding or accountability? Math has become a centerpiece in recent debates because it gives us a case where correctness can be separated cleanly; a proof can be verified by AI without any human understanding how it works. What mathematicians still want after verification is a clue to what mechanical objectivity can’t provide.Philosophers of science now predominantly reject the value-free ideal as being appropriate or helpful. Values enter because whether we reject or accept hypotheses is contingent on how we weigh consequences. Even in abstract realms like math, there are often simply too many possible truths for values not to be brought to bear in sifting through them. Mathematicians accepted long ago that in principle a machine can enumerate the set of theorems that can be proved from a given set of axioms. Yet the scale of this space is immense, motivating people to seek clever shortcuts for exploring it.But even with automated theorem-proving at scale, not all theorems are equally interesting to people. Values are inherent in the choices that lead to certain open problems gaining widespread appeal, and in the processes by which the community comes to accept and internalize the results. Fermat’s Last Theorem initially got attention not because it was an inherently important conjecture, but because Fermat claimed to have a proof but never provided it, enticing others to try their hand. It became central when it was shown that it would follow from the modularity conjecture, which proposed a previously unanticipated link between disparate areas of math. It was considered solved after peer review exposed an error that led to a corrected version of the original proof, even though formal verification didn’t come until recently. Epistemic taste develops because without it, we face an undifferentiated flood of facts. Taste is a capacity we develop over time as we choose problems, pursue some of them, and see which yield new understanding or ideas.If AI takes over choosing, we risk losing the personal fulfillment and community cooperation that motivates many to want to develop a capacity for scientific judgment in the first place. Tao makes a similar point by framing open problems as lighthouses rather than destinations. Reaching them prematurely with AI may sterilize the field around them. If, as fields are transformed by AI, we stop producing people who can judge what’s interesting and aligned with human aspirations, we may end up with a science that fails to serve basic human interests. Its results will consume attention from those to whom it is targeted, while providing neither fodder for subsequent discoveries nor contributing to the processes that help shape the next generation.This starts to sound like motivation for a new kind of linear model for science, which ensures that human reflection and inquiry are sustained despite AI becoming better than the average mathematician. The risk of proceeding without human intervention is drift, as