The Flat Curve Society

18 min read2 days ago--Press enter or click to view image in full sizeThe Flat Curve SocietyHi-ho, we’ve reached the moment, in this movie we’re all watching together on X, where model intelligence has become dangerous. Dario predicted years ago that it would happen this year. With Fable being (briefly) shut off by the USG, it’s the first highly visible sign that we’ve crossed into treacherous waters.Which is too bad, really. I was hoping that we’d get a couple more generations of model upgrades, powerful enough to convince all remaining skeptics, before we got to one that was a security problem. But the Mythos class (Fable being the sloppily-guardrailed version they released last week) has everyone spooked.Now that we know models are getting dangerous, we can do some extrapolating.The AI race isn’t going to slow down, and AI will continue to grow exponentially in capability. Unfortunately, most of you aren’t going to see it progress anymore.I am now in the camp who believe that we are only at most two or three model generations away from AI finally being controlled like nuclear weapons. Only a few will have access to superintelligence above the classes of models we’re seeing this year. As far as I can tell, most Fortune 500 companies will either not have access at all, or it will be tightly controlled for only a small subset of the company. And it will be supervised.I think those with access to powerful frontier models will sell intelligence like a vending machine: You send them a software spec or a problem to solve, and their models implement it for you, on their servers, with your dollars. And since most companies aren’t going to want to send their code and problems to the model vendors, I think the world will learn to live with the models we do have access to.Press enter or click to view image in full sizeSuperintelligence Under Lock and KeyEvery government will restrict access, acting on its own. Nuclear weapons are scarce because it’s hard to get enriched uranium. AI is going the same way, with the chokepoint being the supply chain — something governments can actually clamp down on. China will lock superintelligence inside its own borders as hard as the USG will.

And if China ends up taking the frontier lead, it just changes where the power is concentrated, but not the overall shape of the world we’re going to be operating in.A World of Mediocre ModelsMany of us hoped OSS models would keep us on the exponential curve. They trail the frontier by roughly seven months. But they stay on that curve by training on compute which increasingly takes international-relations-level dealmaking to secure. Maybe distillation or some clever peer-to-peer training scheme keeps them in the race. But to push past Fable class they’d have to do it while the whole hardware-and-software supply chain gets locked down the way the nuclear chain was. And the frontier labs themselves are going to decline to help train the next dangerous open model.If OSS hits Fable class next year anyway, that’s great for the world. But open models are not going to blow past Fable class, not with a huge compute wall and government lockdowns looming.So again, today’s models are roughly as good as we’re going to get.As disappointing as I find that in some ways, I find it still has a lot of upside to be happy about. Because today’s models, particularly Fable-class, are plenty good enough. They will still utterly transform coding and knowledge work. It’s just not going to be a walk in the park. It will take a big, multi-year effort to pivot.I’m going to assume for the rest of this post that we will all get Fable back, and that we may even get one higher class of model before further advancements become inaccessible to all but a very few.Many of you have been expecting the hockey-stick AI advancement curve to level out soon, refusing to believe that it’s truly on an exponential curve that could lead to it being so much smarter than humans. You predicted AI would not be able to replace human engineers.In a way, you turned out to be right. A very practical way.In reality, behind the scenes, the curve is NOT flattening at all; the exponential growth will continue, and you will be able to see outwardly observable signs of it, e.g. in data center growth.But the curve will appear to flatten out for you, through two separate phenomena.The first reason is the one we already mentioned: they’re going to keep the smartest (and thus dangerous) models out of our hands. So most of us never get a chance to try them out.

And those models certainly won’t be replacing engineers, if we can’t use them.The other reason’s kind of interesting, and it took me a while to see that it’s really the same reason wearing a different hat.A World of Mediocre UsersSome people are already reporting they can’t tell the difference between Opus 4.8 and Fable 5. I’ve been calling this the “discernment horizon”: every human has a ceiling on model intelligence past which all the models start to feel about the same.But there are actually two ceilings, both instructive lenses on what’s happening.The first I’ll call the demand horizon. It’s set by the hardest problem you bring. If all you have handy are easy problems, they don’t give a smarter model any room to pull ahead — the outputs look the same because the problem never stretched either one. The demand horizon is where you can’t tell two models apart because you don’t have a hard enough problem.I call my hard problems “back-pocket evals,” and I collect them. Whenever I give a project to a model, and it can’t do the project, I add it to my pocket-eval list. Then every time a new model drops, it’s like Christmas. I try it out on all my pocket evals and see which ones it can now solve.Press enter or click to view image in full sizePocket EvalsConcrete example: No Opus-class model has been able to write the React client for my game; it’s just way too complicated and fiddly. Fable was absolutely smashing it. Easy way for me to see the difference vs. Opus. But I also have other problems that will prove too hard for Fable. I will collect them eagerly as it chomps through my work. All you need is ambition, and you can create your own pocket eval collection.So my demand horizon is super high, and will last at least three or four more model generations, if I can manage to get access to that level of intelligence, which seems unlikely. I don’t have my hopes up. But at least I will be able to tell if it’s actually that smart, using my evals.The demand horizon is benign enough, even kind of flattering: it just means your work isn’t hard enough right now.

But bring an unusually hard problem one day, and your horizon widens on the spot, as you watch the cheaper model fumble some task the expensive one nails. Like my React client.There is a darker horizon, which I think of as the discernment horizon proper. This one is set not by the hardest problem you can pose, but by the hardest answer you can judge. Past this scary line, you can’t tell whether the model is right, because checking the work is itself beyond you.I’ve been chewing on this problem since my Drunken Rants days, when I’d write about how hard it is to interview someone smarter than you. How do you know they’re not a charlatan, if they’re professing expertise in an area you know nothing about? You can’t, really.Everyone has a discernment horizon, even Dario. Past some level of capability there is no human alive who can verify the model output.Press enter or click to view image in full sizeThe Discernment HorizonThis takes us full circle to why they are starting to lock down the models. You can’t hand out an intelligence engine that nobody can supervise. It’s pointless to own because you won’t know if it’s helping you or walking you off a cliff. Superhuman means unverifiable.So the safety people see a potential weapon, and the rest of us see a tool that we can’t effectively supervise. In both cases, you don’t need or want the more powerful model. You want the safer one, even if it’s less capable.Companies also have both of these horizons. For plenty of companies, Fable is already past the demand horizon — every problem they’ve got, it handles, and a smarter model would change nothing they could measure. For the harder shops the binding limit is discernment: the AI produces work that nobody can grade. A terrible outcome, assuming you don’t want to surrender your business to AI entirely.As a result of all this, the curve is flattening for most of us. I think commodity intelligence will soon stop growing exponentially, or at least, it will appear that way, and we’ll all operate as if it’s true.I had never spent much time considering the possibility that the intelligence curve would flatten out. But now that it seems to be happening, let’s look at some of the clear and obvious implications for the industry.

SaaS is Back, BabyIt’s clearly going to be too expensive to rebuild all the SaaS at the top of the pyramid. Yes, there will be models that can do it, but access and cost will both be prohibitive.SaaS actually came rocketing back over the past month all on its own, after spending much of the past year on the ropes, pummeled from all directions by threats of in-house rewrites and fears of Claude taking it all.Then companies learned about token efficiency the hard way, with huge firms blowing their yearly budgets in months. A few months ago, everyone was planning to tell their CFO they could cancel a bunch of SaaS subscriptions and bring their dependencies in-house. No longer. Now the buy-vs-build decision is tilted heavily towards buy. If you despise your current SaaS enough, then sure, you may be motivated to rewrite it with AI. But buying SaaS has predictable costs that are usually already in the budget, whereas vibe-coding replacements could be an expensive gamble.If we see a plateau in accessible model capabilities, then the other dreams we had about AI in SaaS fade too: not just replacing it, but transforming it with agentic behaviors and monitoring. Today’s models aren’t good enough to replace a person yet (jailbreakable, confusable, etc.), so you can’t just swap an agent in for an SRE or a trained customer service rep. And the models that could reliably replace humans may be too dangerous to give to most people.So SaaS looks like it might be fine, even without agentic behaviors. It just needs to save you the money of building and maintaining it in-house.SaaS still has its problems: users subsidizing the 80% of the features they don’t use, dollars extracted from local economies to enrich Silicon Valley, enshittification creeping up the pyramid. But it remains fundamentally about crystallization of knowledge. Groups of people build stuff that’s tricky, stuff you wouldn’t want to do yourself, and rent it to you. The AI models powerful enough to replace most of that “easily” will either be unavailable or prohibitively expensive.It feels to me like the SaaS model is here to stay.AI Literacy 101Today’s models, while quite capable, are still very difficult to work with.