Intelligence is not the main bottleneck

Edit: To be clear, I'm not saying everyone or even a majority of people in the SF tech scene think this way. But some certainly do (e.g. the opening conversation is verbatim) and this post is about a particular core scene whose views carry disproportionate weight in the discourse and in policymaking.At a recent dinner, someone from an AI lab asked me, quite bluntly , why I do what I do — why I write about the regulatory bottlenecks to medical progress, why I spend my time on policies related to clinical trials. He looked at me with something best summarized as pity.I told him what I believe: that in the age of AI, medicine will be bottlenecked more than ever by regulation and the grind of clinical trials, and that billions poured into faster pre-clinical research won’t touch that problem, unsexy as it is. I brought up housing: we’ve had the technology to build better housing for decades, yet it’s more expensive than ever, because housing is a question of political will, not pure capability.He looked at me incredulously. Surely, a smart person like me should know that AI, or better said, AGI will be hyperpersuasive soon – already on a bunch of benchmarks it exceeds professional debaters at persuasion. I said, “Hmm.” He said, “Yes, yes”, with the undertone of a man who knows things deeply, things that mere mortals like me, not being AI lab employees, are simply not well placed to grasp. And he looked at me again with that sense of pity, the way one looks at a slightly mentally impaired but cute animal awaiting its imminent slaughter.That night I lay awake, twisting and turning, wondering whether my life had any point at all. Whether every decision I’d ever made had been, in some way, a mistake. What could I have done better?But as the sun came up, I found my way back to the same thought: no matter how “intelligent” AI becomes, intelligence is often not the main bottleneck to things changing in the real world. It is hard to hold on to that conviction when people smarter than you, with access to privileged information, insist otherwise. After all, this could all just be “cope” from the naive hamster awaiting its slaughter.A scared hamster watched over by the AGIBut, I shall nonetheless stick to my beliefs.Because the people at these labs made fortunes betting on ideas that once looked insane, the world now takes nearly everything they say on faith. That, I think, is a mistake. It is not a given that AI will solve the problems most people actually care about, including medicine, unless we think about those problems clearly. And at the moment, I don’t believe we are or at least, not to the extent that we could. My case is simple, and it comes from two directions. One is what I observe in the social dynamics of San Francisco, and in how people there talk about all this. The other is what I notice in a field I happen to know something about: medicine.One of the promises most often invoked to justify AI’s risks is that it will “cure disease.” Every major AI lab CEO says it, and investors seems to agree: any biotech startup with an AI story attached commands an impressive valuation, even as more conventional biotechs struggle for funding and die. But this whole enterprise, as I have long argued, is bottlenecked by many things that have little to do with “intelligence” as such, and the degree to which that often goes unacknowledged is strange to watch.The evidence is everywhere, if you treat scientific advancement as a rough proxy for intelligence and ask whether it alone unblocks progress. Take Eroom’s Law: the number of new drugs approved per dollar of R&D has fallen for decades, even as our scientific tools have grown vastly more powerful, the exact opposite of what the existence of more raw capability would predict. Or take a company like Adaptimmune, which has brought two transformative therapies to market in rare cancers and is nonetheless fighting to stay alive, due to the cost of developing them. Or one can listen to the scientists behind “baby KJ,” the infant saved by a bespoke gene-editing therapy: they have everything they need scientifically and still cannot easily repeat it for the next child, because manufacturing costs, driven in part by regulatory requirements, stand in the way.Figure 2. Pharmaceutical R&D productivity has steadily decreased since 1960, despite advances in basic science. In the last decade, the deceleration in pharmaceutical productivity seems to have ameliorated, due to a combination of factors including an increase in predictive validity (through e.g., genetics), but also due a larger share of efforts being directed at oncology and rare diseases, where the burden for approval is often lower due to the high unmet need.In biopharma specifically, one great bottleneck is clinical trials (I know I keep banging on about this), which today consume something like seven years and cost more than a billion dollars per drug. And trials are not a mere formality. They generate precisely the kind of data that would train better models in the first place: human data, which is ultimately irreplaceable. We have extensive evidence faster clinical trials are important for biomedical innovation, both directly and indirectly, through second-order effects (like increasing the risk appetite of those working in the industry and helping better align incentive through fast feedback loops). This ranges from robust economic papers showing shorter trials massively boost investment in a disease area, all else equal, to the natural experiment of China. China is threatening to race ahead of US in biotech, with Chinese biotechs encompassing more than a half of big Western pharma licensing deals. A mere decade ago, this percentage was zero. This transformation happened while China remains worse in terms of basic science, mostly due to regulatory reforms that allow faster iterative learning using in-human data1.Trials themselves can be made shorter, more informative and better, including with AI. One area I am particularly bullish on is surrogate endpoints and biomarkers, where AI could turn discrete, coarse readouts of whether a drug is working into fast, continuous ones. This would in turn optimize trials immensely. For some indications, the gain could be as high as 10x faster and cheaper trials if the right surrogates are found. And this is not even considering the benefits of simply optimizing the trials without necessarily shortening them: imagine having the right biomarker that tells you whether a therapy is working early enough. Figure 3. One way in which surrogate endpoints help is by increasing incentives to invest in an area. I can confirm that the findings from the economics paper that shows this are true, as I could notice myself how after the FDA allowed Bone Mineral Density (BMD) as a surrogate endpoint in osteoporosis, there has been a stark increase in the appetite from big pharma in pursuing this indication.But even here the binding constraint is not entirely intelligence. Quite often, it is governance. I keep talking to companies trying to build exactly these biomarkers, and what they run into, again and again, is how hard it is to access the underlying data. Some have been waiting for a year for the NIH to release imaging datasets they can use to produce better biomarkers. If one needs to interact with the FDA to get their endpoint validated, it is even worse: I have previously written about how the validation of Bone Mineral Density (BMD) for use as a surrogate endpoint in osteoporosis trials took 12 years (!), despite the fact that the data to support it already existed in full and the analyses done were basically regressions. When people in the AI sphere do engage with regulation and governance in medicine, it often seems to happen in a rather superficial way. I have seen many times that argument that because most drugs fail for lack of efficacy, regulation can’t explain much of the slow-down in medical progress compared to basic science that we have witnessed in the last decades: after all, approving more inefficacious drugs won’t help. This analysis sounds superficially true, but it’s not. As I keep saying, when people talk about the importance of regulation in slowing down biomedical progress, they rarely mean the approval decision at the end. It is about the entire process upstream: how easily human data can be collected, and then how easily, once collected, it can actually be used.I know most about clinical trials, but this is not the only area where governance and policy slows down progress. Another example is related to how the structure of our patent system shapes how innovative our drugs are.In biomedicine we have a problem called target herding: most companies chase the same biological targets because they are de-risked, which means we explore far less of the biological space than we could. The cleanest way to see why is through risk. Two kinds matter in drug discovery. Target risk captures the biological uncertainty: is this protein causally involved in the disease, and will hitting it help a patient without unacceptable toxicity? Molecule design and optimization risk is the downstream problem, conditional on the target: designing a novel molecule against a target2 and improving potency, selectivity, safety and a myriad of other features of a chemical (e.g. small molecule) or biological (e.g. antibody). That is mostly in the realm of chemistry and is intrinsically far more tractable and predictable; given a validated target, good teams usually reach a decent molecule.Figure 4. The logic behind target herding The patent system then further disincentivizes companies from taking target risk, because it rewards novel chemical matter, not novel biology. Composition of matter is what most patents protect. That refers to a specific molecule, which means that they don’t protect the insight that a target is worth drugging at all. Target herding then follows almost mechanically from the above