AI Safety Is Mostly A Sex Cult In Berkeley, California

These people shouldn’t make policy.There has been a great effort over many years to distance AI Safety discussion from its origin and intellectual center, because its center is a sex cult started by a fan fiction author. Since those involved want to be taken seriously, they have to hide that. Everyone else working in AI Safety is, in effect, a smokescreen for this core group and their ideas.We might some day have “AI Safety”, as a science or policy area, that is not primarily a sex cult in Berkeley, California, but it does not currently exist.This is Eliezer Yudkowsky. He is seen here in a meeting with Bernie Sanders, who he is advising on AI policy, and in The AI Doc, now on Netflix.This is Yudkowsky in Time Magazine in 2023, advocating in favor of risking nuclear war to prevent AI research from happening anywhere on Earth.This is very senior OpenAI employee and unofficial OpenAI spokesman Roon. Roon’s publicly stated opinion about Yudkowsky is:Yudkowsky is in and/or running what Roon describes as a sex cult.Yudkowsky’s opinions about AI are important, and Roon says he takes the things he says quite seriously.This seems to be an extremely common pair of opinions concerning Yudkowsky.Yudkowsky is sufficiently controversial that people motivated by his ideas often downplay or deny his influence. Sometimes this spans multiple layers: People who received money from Sam Bankman-Fried denying they’re Effective Altruists, Sam Bankman-Fried denying he’s an Effective Altruist when caught, Effective Altruists denying their version of AI Safety is heavily influenced by Yudkowsky’s, people who are extremely core members of the movement often call themselves “rationalist adjacent”, and so on. But Yudkowsky is provably, publicly influential on essentially all major public figures in AI. It is easy to deny not because it is hidden but because it seems so insane.Yudkowsky is the person who connected Shane Legg and Demis Hassabis to Peter Thiel so that they could secure funding to start DeepMind1, which is now Google's AI research division. Here's Sam Altman's opinion of Yudkowsky.Left, we have a schedule for the Future of Life Institute’s January 2015 AI conference in Puerto Rico, which both Elon Musk and Yudkowsky attended. Right, we have a separate, explicitly Effective Altruist event: EA Global, held in Mountain View that August, where Musk and Yudkowsky both also appeared.2 Musk would cofound OpenAI that December. Given these events and Sam Altman’s direct credit, Yudkowsky and Effective Altruism were certainly adjacent to and probably did play a critical role in the decision to start OpenAI.On the right, in order, Daniel Dewey (MIRI), Nick Bostrom (Oxford Future of Humanity Institute), Elon Musk (self-explanatory), Nate Soares (MIRI), Stuart Russell (Quite Good At AI Research, otherwise uninvolved)Dario Amodei, current CEO of Anthropic, is here in 2013 identified as a "member of the effective altruism community" on the blog of Yudkowsky's non-profit. He appears along with Holden Karnofsky, also identified as an Effective Altruist.And here Dario is arguing with Yudkowsky about the paperclip maximizer, a now-classic thought experiment, originally Yudkowsky’s, about an AI that might kill everyone.“AI Alignment” and “AI Safety” were first popularized by Yudkowsky’s non-profit, MIRI, the Machine Intelligence Research Institute. OpenAI and Anthropic have cited Yudkowsky or his non-profit at least a dozen times3, and the entire way we talk about AI today uses terminology his non-profit popularized. It is effectively impossible to even discuss these areas without, implicitly or explicitly, performing commentary on Yudkowsky’s writing.This is Jacob Coxon, the recent “whistleblower” who quit Anthropic, being asked his opinions on rationalists and effective altruists.This is Nate Silver citing LessWrong as a source of information about AI risk. LessWrong is a website started by Yudkowsky in 2009, and was originally filled only with Yudkowsky’s writing.This is Oliver Habryka, who is the executive director of the nonprofit controlling LessWrong and the rationalists’ compound, known as Lighthaven. He is here explaining how large he thinks the influence of rationalism is on AI Safety and AI existential risk discourse. He appears to be engaged in some kind of internal territorial dispute. It is possibly the only time I have ever agreed with him.This is a podcast interview with Daniel Alejandro Moreno-Gama, who attempted to firebomb Sam Altman’s house, in which he names Yudkowsky as his primary influence.Yudkowsky’s more dedicated followers seem to prefer to call themselves “the rationality community” these days, everyone else calls them “rationalists”, the entire following “rationalism”. “Rats” is also common.4This is a long denial that they are a religion. I alternate between finding these denials funny and finding them insulting.The fan fiction that launched a thousand cults is Harry Potter and the Methods of Rationality, or “HPMOR”. It is possibly the highest-impact single modern religious text, rivalled only by the Book of Mormon. Its influence may not yet have peaked. It is the holy book of the first great Internet cult.Allegedly, Effective Altruism and its AI safety efforts are a separate thing and not directly beholden to Yudkowsky. This is very difficult to believe when familiar with the facts. Effective Altruism in the Bay Area has almost perfect overlap with rationalism, and the Effective Altruist AI Safety people in the Bay set the agenda for the rest of Effective Altruism’s safety efforts.In the Bay Area, Rationalism and Effective Altruism appear to be the same people, at the same events, in the same places, who believe the things said by the same authors and leaders, with only very slight differences between spheres of influence. Yudkowsky called himself both an effective altruist (as late as 2022)5 and a rationalist until a peculiar internal schism. Scott Alexander, the second-most influential writer in the space, has usually called himself both. Often, members of these communities seem to show up in groups working for the same companies and non-profits.Frequently, Effective Altruists bring up all of Effective Altruism’s other charitable activities and its charming, benevolent members who don’t live in the Bay Area to deflect true criticisms of the Bay Area Effective Altruist community. This effectively makes their other charitable operations and workers shields and fronts for the priorities and behavior of the Effective Altruists in the Bay Area, who are interested almost exclusively in AI.Where there is any separation between the two things in the Bay, “rationalism” is people and things more closely tied to Yudkowsky, and “effective altruism” has had a few institutional centers but is currently focused around Anthropic and its CEO, Dario Amodei. They compete for attention and funding in that space. Yudkowsky’s position is that AI will definitely kill us, and Dario’s position is that AI is ten to twenty-five percent likely to kill us.Both are saying what justifies giving them and no one else money, labor and attention. Since Yudkowsky is a writer and not a scientist of any kind, he can only claim that he and his organization are important and deserve money and prestige if AI Safety is an unsolvable problem and only advocating against AI matters. Since Dario leads a company that actually makes AI, Dario must claim that “AI doom” is a very large problem that only his company is able to manage.Yudkowsky, similarly, claimed that his (non-profit) company was the best chance at “saving the human species”, back when AI companies seldom made anything and he could claim to be doing research by simply writing.6 Yudkowsky’s belief system is a successful pitch for investors, donors, volunteers and employees. It is also, politically and doctrinally, a pitch for people to give you total control over them, because if they do not, the world might end. After Yudkowsky refined this pitch modifications of it became the pitch for organizations like OpenAI and Anthropic7. This millenarian ideology appears to have become integral to employee morale and recruiting because so many research staff are believers to one degree or another.Yudkowsky is not going to stop preaching the gospel of doom because, among other reasons, doing so is the only job he has ever had. He would stand to lose his social position and the roughly $600,000 in income he is paid out of his non-profit if he changed his tune. In general, if doom prophecies are worth money, people will produce more. This system’s purpose is to attract funding and young idealists and to find ways to spend the funding and make use of the young idealists. Historically, rationalists have seemed to encourage many of their young idealists to take up sex work.8People tend to follow their incentives.People sometimes deny that Anthropic is an Effective Altruist organization. This may be technically true. Anthropic, the company, can say it is or isn’t whatever it wants to. Anthropic’s founders, senior employees, and investors, however, have numerous ties with Effective Altruism.9Dario in particular is almost perfectly on-type for a Bay Area Effective Altruist in what he says and how he says it. Dario’s often bizarre, vague, and meandering writing style alone has no real precedent besides LessWrong and its AI-focused spinoff, Alignment Forum. He has been directly involved in Yudkowsky’s community meetups since at least 2008.10 Anthropic’s strategy is very similar to the playbook Yudkowsky laid out, back before the success of deep learning and companies that specialize in deep learning made Yudkowsky’s organization no longer credible as a serious research project.This same denial of affiliation happens when it comes to the incestuous relationships between all of the other non-profits in the Bay that are meant to be focused on AI risk. They virtually all draw support from a handful of sources of funding