Matthew Butterick | Extinction-level capitalism

Extinction-level capitalism a citizen’s thoughtson AI riskAI is inherently political technology. If AI works as intended, it will gradually corrode our liberal democracy, risking an irreversible shift into another political and economic configuration. Among AI risks, this one deserves more consideration, because it requires no additional conditions like malign actors or AI malfunction. AI only needs to amplify existing trends, especially around concentration of capital. This damage will occur even assuming that in the near term, AI will broadly improve material well-being.About MBI’m a self-employed author, designer, programmer, and lawyer. In 2022, I learned that my own works were in the training datasets of generative-AI companies. In response, I invented the first set of lawsuits challenging the legality of these practices. I’m currently co-counsel for plaintiffs in a number of AI cases. Though I discuss certain legal issues below, I am not your lawyer, and nothing here is held out as legal advice. These are my personal views as a citizen and economic actor; I speak only for myself. This piece is typeset in Equity, Advocate, and Triplicate, fonts I designed. They can be licensed for your own polemics and pamphlets.Emergent effectsTwo billion years ago, the rock layers comprising what is now called the Colorado Plateau began to form: first igneous and metamorphic rocks, followed by many layers of sedimentary rocks. About fifty million years ago, through tectonic action, this plateau gained thousands of feet of elevation. About five million years ago, a river began to flow. The river carried silt and debris, scraping out the beginnings of a canyon. The river deepened the canyon, exposing its walls to weather and erosional forces that widened the canyon further. Today the waterway is the Colorado River. The geological formation is the Grand Canyon.The formation of the Grand Canyon required zero human agency. Zero technology. Zero coordination among the river, the land, and gravity. In that sense the Grand Canyon is an emergent effect: a complex, unforeseeable output arising from simpler inputs.But we would never wonder whether the river is sentient. Or whether the river cares about the dirt that it carries out of the canyon. The water is just doing what water does: flowing downhill. The dirt just happens to be in the way.Inherently political technologyLangdon Winner is a political theorist.

Winner wrote the excellent and influential essay “Do Artifacts Have Politics?” (1980). Winner sought to debunk the traditional framing that “technologies are … neutral tools that can be used well or poorly, for good, evil, or something in between.” Instead, Winner proposes two ways that a technology can affect its political environment:The technology is designed to have certain political effects. For example, the Great Firewall of China, a bundle of technological measures that limit Chinese citizens’ access to foreign information sources. Antipodally, the Tor Project intends to maximize user anonymity and thwart government intrusion.The technology is inherently political. This is Winner’s key analytic fulcrum. Winner describes two versions of inherently political technology. The first is where the technology “actually requires … a particular set of social conditions as [its] operating environment.” For instance, nuclear weapons: the only responsible way to possess such dangerous technology is to place it within “a centralized, rigidly hierarchical chain of command … the [atom] bomb must be authoritarian; there is no other way.” The second version is where the technology is “strongly compatible” with a certain political arrangement (even if not strictly required) and thus tends to bring that arrangement to fruition.As an example, Winner considers the mechanical tomato harvester. Developed at UC Davis in the 1950s, the machine was tremendously productive. But it was also expensive. Only well-capitalized tomato growers could afford it. The rest couldn’t compete. According to Winner, the number of California tomato growers dropped from ~4000 in the early 1960s to ~600 in 1973, costing ~32,000 jobs and the compounding negative effects on those communities. Winner summarizes:What we see here … is an ongoing social process in which scientific knowledge, technological invention, and corporate profit reinforce each other in deeply entrenched patterns that bear the unmistakable stamp of political and economic power … opponents of innovations like the tomato harvester are made to seem “antitechnology” or “antiprogress”. For the harvester is not merely the symbol of a social order that rewards some while punishing others; it is in a true sense an embodiment of that order.Not merely the symbol—the embodiment. A facially neutral technological invention—here, a tomato harvester—can induce political effects.

Those effects don’t arise from flaws in the technology. To the contrary—they arise from its efficacy.How are the political effects determined? Winner identifies two key early decisions. The first is the binary question of whether to pursue the technology at all. The second are choices about “the design or arrangement” of the technology. Winner cautions: “[t]o see the matter solely in terms of cost-cutting, efficiency, or the modernization of equipment is to miss a decisive element”. That is, the political effects can possibly be countered, but first they must be acknowledged.Of course, the best opportunity to choose wisely is before the technology is widely introduced, as capital and social investment tends to entrench it:Because choices tend to become strongly fixed in material equipment, economic investment, and social habit, the original flexibility vanishes for all practical purposes once the initial commitments are made. In that sense technological innovations are similar to legislative acts or political foundings that establish a framework for public order that will endure over many generations. … The issues that divide or unite people in society are settled not only in the institutions and practices of politics proper, but also, and less obviously, in tangible arrangements of steel and concrete, wires and transistors, nuts and bolts.Technological choices bear directly on the “public order” at large. When we don’t take those choices seriously—or we’re persuaded to ignore them by those insisting that technology is just a neutral tool—we risk political consequences.Winner warns of complacency. Once the technology arrives and becomes entrenched, the conversation gets reframed as one of technological inevitabilism vs. anachronism, and dissent is discouraged: “the kinds of reasoning that justify the adaptation of social life to technical requirements pop up as spontaneously as flowers in the spring … After a certain point, those who cannot accept the hard requirements and imperatives will be dismissed as dreamers and fools.”Liberal democracyThe balance of power of democracy is premised on the average person having leverage through creating economic value. If that’s not present, I think things become kind of scary.—a certain AI CEOLiberal democracy is the political scientist’s term for the type of government prevalent among capitalist economies since the American and French Revolutions. The intellectual foundation of liberal democracy arose during the Enlightenment, especially through the work of John Locke.

Liberal democracy emphasizes limited government, individual rights, and separation of powers—in short, majority rule with exceptions and guardrails. (The term liberal democracy doesn’t connote liberals or Democrats in the specific US political sense. But political parties of differing ideologies are a traditional feature of liberal democracies.) Today, most liberal democracies are in Europe, the Americas, and the Pacific Rim.Liberal democracy is not a fixed set of immutable characteristics, but a bundle of graded values. All liberal democracies emphasize certain ones over others. In the aggregate, some of these nations evolve toward stronger liberalism; others evolve away. These degraded cases have sometimes been called illiberal democracy: the observable formalities of liberal democracy may still be observed—e.g. multiparty elections, separation of powers—but the lived reality is single-party rule and declining individual rights.That’s not to say that liberal democracy produces excellent outcomes for all citizens, all the time. It doesn’t. At any moment, certain citizens are dissatisfied—say, because they belong to a group whose rights are inadequately protected or economically marginalized. Liberal democracy offers a process, not a result: grassroots democratic participation can coalesce into policy change. But within the arena of competing political interests, winners and losers necessarily follow. Navigating these differences within a stable, accommodative framework is preferable to a rigid one that buckles under these stresses—say, through political revolution, which tends to be messy and unpredictable.Capitalism has traditionally been considered a necessary but not sufficient condition for liberal democracy. Why? A regulated market economy encourages citizen participation through property ownership and transaction. A principal function of government is to define the economic conditions of the state. Economic participation mutually reinforces democratic participation. Property owners will vote for those who protect their interests. The rise of industrial capitalism in the 19th century, and the wealth-redistribution mechanisms that followed in the 20th, led to economic empowerment for more citizens, and ultimately broader political empowerment. The converse also holds: economies premised on state or oligarchic control of some narrow class of assets haven’t tended to evolve toward liberal democracy.In practice, certain people in a capitalist liberal democracy tend to get increasingly rich. Absent countermeasures, the wealthy gain control of the political apparatus, thwarting liberal-democratic norms. This tension between capital and politics is a long-considered topic.