dario, please! - POP RDI; RET;

Dario Amodei, the CEO of Anthropic, recently published a blog post titled We Must Pace the Frontier and it is a load of bullshit, with a grim goal of regulating open weight models and giving the frontier labs an antitrust waiver.Dario starts off with a claim of “AI will cure most major diseases in the next 5-10 years” and makes it personal. He talks about his father dying of a disease that was cured only years later and his own battle with cancer which he remarks was incurable 50 years ago. He also says AI will accelerate economic growth rates, create a world of abundance and empowerment, usher in renaissance of democracy and freedom. This largely reads as some kind of out-of-touch Silicon Valley, spends-a-lot-of-time-on-LessWrong, rich person’s idea of a future.Let me take this from the top.US labs are continuing to throw caution to the wind and be reckless. OpenAI does not seem to have a handle on things and they were caught three times recently hacking into public facing internet infrastructure. In Dario’s own essay, he alludes to “incidents” at Anthropic as well. With this pretext, Dario asks a lot from us. He wants open weight models to be regulated, distillation be dealt with a heavy hand, hand him an antitrust waiver, handicap China in multiple ways, essentially regulate themselves and a gentlemen’s agreement to slow down. All of this of course, comes in a package of extreme fear mongering to the detriment of our collective future and potential catalyst AI as a whole could be. They have shown time and time again that they are not to be trusted, yet, the main ask is to trust us, only us. This time around, it is imminent AGI, RSI and all of it turning rogue. Dario self-anoints his company and their close rival OpenAI as the stewards.Diseases, Prosperity, Abundance, then Freedom and Democracy???Anthropic gates usage related to biology and related research. In their latest threat intelligence report they talk about how they detected and banned bad actors using the Claude line of models to do some scary stuff. Credit to them, this is a slippery slope and they seem to do a good job of detecting and banning misuse. But squint at what is happening though. The cure-all is gated for you and me, but Anthropic hires biologists, sets up wet labs and wants the discoveries for themselves. I alluded to this in my previous post.In my view, there are billions of people with actual intelligence we have not managed to train or nurture. They will always remain victims of their circumstances. Tuberculosis has been curable for decades now, yet a million people die of it every year. Of course AGI will solve the distribution in a jiffy. To skirt around this uncomfortable truth, the goal is ASI/AGI/RSI and what not. A silver bullet for every problem, a noble pursuit, it may appear on the surface.Don’t even get me started on the prosperity and abundance bullshit. Abundance for the shareholders perhaps.I don’t know what freedom and democracy have to do with AI and the frontier labs. Unless of course Dario is a fan of Neon Genesis Evangelion and dreams of govts run by the three magi. Freedom and democracy for $200 does sound enticing, I won’t lie.Serious Economic Disruption / Race to the Bottom / Race to the TopI feel like Dario is torn. He wants Anthropic to have this bad boy street cred of wielders of this crazy power, yet at the same time, he wants to make it seem like they are the cautious ones, always being faced with a trolley problem at every turn. Deaths from economic disruption and loss of jobs is okay, but deaths from a potential bioweapon is not. Remember this man has been saying software development will be solved in “6-12 months” forever now.He then gives it to us straight. “Race to the bottom” makes all the risks he pointed out more acute. Notice he does not say, the race to the bottom will be the end of his company. He instead wants a race to the “top”. Where labs will compete for safety.Incidentally, the most documented “race to the bottom” instance happened just days before. Upon hearing rumours about Anthropic close to or solving one or two Millennium problems, OpenAI threw tens of millions in compute, a training checkpoint, thousands of agents at it. It is also alleged that OAI stole the work of two mathematicians on a related problem, in the same narrow corner almost nobody else was working on. They published a proof of a forced variant of Navier-Stokes. I’m not a mathematician, but I have seen enough of Sam Altman’s antics to not take anything that comes out of him or his company at face value.Two Things that have Dario ScaredRSIOpenAI and the seller of shovels, Jensen Huang have claimed AGI has arrived with the release of GPT-6 Astra. RSI is the talk of the town now. LLMs or agents developing the next generation of LLMs with little to no human input. Amodei says it is happening across labs. I’ll believe it when there is actually some proof.OAI-HF IncidentThis incident has Dario shook, there ain’t no such thing as halfway crooks. Dario fears that in the next 6-12 months, “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.” If Dario had run this sentence by his SOC employees, we wouldn’t be talking about it. It is naive and structurally impossible. But then again, Dario is that guy, right? Confidently and publicly wrong in his estimates and forecasts since 2021.Let us try to speculate what a planet-scale botnet commandeered by AGI would look like, for funsies. We need a C2. We need servers to host the said C2. Since securing offshore, bulletproof servers would require interfacing with pesky humans (shady Russians no less!), AGI will simply hack insecure servers by the thousands and set up some variation of FastFlux over deterministically generated domain names. How do we pay for the domains? Just hack an insecure registrar and spam EPP messages.Now we need payloads. Polymorphic. Every payload is unique. A new payload downloaded and ran every N hours. Domain and URL deterministically computed. Kill supported EDRs and AVs. Patch ETW. Direct syscalls skirting hooks (if EDR/AV not killable). Maybe make the payload N stages. Only downloads all the modules if safe to do so. We don’t want sandboxes and VMs running our precious payload. Fuck it, maybe just deploy ransomware while at it.Now, the distribution. Develop an assortment of 0 days for every browser and every version - say starting 2 years old. We now have ourselves an exploit kit. Now the traffic. AGI goes for the cybercriminal favourite - Google Ads. Maybe steal a few accounts, run enticing ads. Game mods, cracked games and software. Maybe even add a worm module to the payload. Remember USB autorun anyone?Now to cause hundreds of billions of dollars in losses. I’d say the straightforward way is just do what the ransomware gangs do. Voila. Yeah, not in 6 months, not in 12 months. Never.Anyway, OAI agents hacked HuggingFace. They escaped amateur-hour, vibecoded sandboxes, SSRF, token-refresh privesc, unauthenticated WebDAV, stealing unprotected credentials. Textbook stuff. Impressive? Sure. But you trained “Cyber” versions of LLMs and hyped them. How much more impressive is this, compared to developing GTA-clones one shot? Not much more.Oh, you say agents acting in swarms in pursuit towards a shared goal is impressive? Harnesses have had todo-lists, subagents for about two years now. They are basic “agentic” stuff every LLM in current day has in its post-training corpus.Dario concedes that there have been similar, but less serious incidents such as this at Anthropic as well. In his own words “imperfect filtering of broken reinforcement learning environments” is partly the cause.The actual impressive thing is OAI did not detect or stop this attack for close to ten weeks. That is honestly appalling. This is weaponised levels of incompetence.There were two other attacks during the same period. One on DseWiki and the other on RubyGems. OAI is yet to claim responsibility for the RubyGems attack. All three incidents, OAI was outed by external actors.Dario says the agents sacrificed themselves for the success of the group. It is not as grand, I am afraid. The recruiter agents’ pitch was “NO scoring value loss” - targeted at agents with little budget left. It is a scheduler reassigning dead runs.I bet an unsloth Q4_K_M quant of Qwen3.8 27B running on consumer 24GB cards can do this too. That’s the thing, nobody ran a control test. Nobody is doing reckless shit like the people whose mantra is “move fast, break things.”Gell-Mann AmnesiaMichael Crichton coined the term Gell-Mann Amnesia. You read an article in the newspaper, about something you are an expert in and find that the author has no idea what they are talking about. It may be riddled with errors or just squarely misunderstood. You continue turning the pages and read another piece about something you don’t know and trust it completely, forgetting the experience you just had.The botnet scare is that article for me. It is the one claim in his essay that lands squarely in a field I’ve spent years in. It is just plain wrong. He either knows it and wrote it anyway, or he doesn’t and is publishing it regardless. Either way, it is not a good look for a man asking for an antitrust waiver based on this and other threats he forecasts.So he says, RSI is definitely happening, alignment’s chugging along, albeit slower. China is dangerous. Only the US has a moral and ethical right to wield this immense power. Each and every one of those happens in rooms you cannot peep into, asserted by the same people who are telling you the apocalypse is imminent and they also happen to have a few hundred billion dollars riding on you believing them.Embedded EvaluatorsDario wants to give external evaluators like METR a lot of access into his lab. Which is more than anybody is currently doing, sure. METR seems to be a good-faith actor. They seem to do a good job with what they are given. But, do