Simulating everything, sort of: The promise and limits of world models

The Bet Experts explain how they work, what they can do, and what’s still unsettled. Credit: Aurich Lawson | Getty Images Credit: Aurich Lawson | Getty Images Over the past few years, many of us have gotten a crash course in what we now call artificial intelligence—but really, it has mostly been a crash course in large language models. Increasingly, however, LLMs are no longer the only category of AI drawing high expectations, massive funding rounds, and significant research and product development. Over the past year, we’ve seen a plethora of new announcements in a category labeled “world models,” and you’ll likely see more movement there in the coming months and years. Instead of or in addition to working with language, world models aim to lay the groundwork for AI systems that are capable of simulating the physical world, or at least a useful approximation of it. To examine what’s different and important about this idea, Ars spoke with three expert practitioners working on world models and related technologies: Vincent Sitzmann from MIT, Anastasis Germanidis from Runway, and Ben Mildenhall from World Labs. From these conversations, we learned that while LLMs-as-a-product started with an interface (chat) and then sought a use case, the big players in world models right now are arguably working in the other direction: They’re starting with specific use cases and applications in robotics, research, and asset generation, but it’s unclear exactly how the interfaces, systems, and tools will ultimately look. Table of Contents The off-ramp from LLM disillusionment As you’ll soon see, there are many parallels between LLMs and world models in terms of architecture and how people expect them to improve over time. For some, though, they’re seen as a potential answer to the limitations of LLMs, even though work on them predates that contemporary narrative. “The idea that you’re going to extend the capabilities of LLMs to the point that they’re going to have human-level intelligence is complete nonsense,” former Meta chief AI scientist Yann LeCun told Wired earlier this year. LeCun has made waves with an opinion that some working in AI and LLMs see as contrarian, but he’s actually speaking for a sizable segment of the field. See also Fei-Fei Li, the computer vision pioneer who co-founded World Labs, one of the new companies working on world models.

In a Substack post late last year, she wrote: Today, leading AI technology such as large language models (LLMs) have begun to transform how we access and work with abstract knowledge. Yet they remain wordsmiths in the dark; eloquent but inexperienced, knowledgeable but ungrounded. Spatial intelligence will transform how we create and interact with real and virtual worlds—revolutionizing storytelling, creativity, robotics, scientific discovery, and beyond. This is AI’s next frontier. LeCun and Li’s ventures are built on these ideas, so it’s not surprising they’d say these things. But you’ll also see similar sentiments from some prominent figures still working primarily with LLMs. “I think we’re in an LLM bubble, and I think the LLM bubble might be bursting next year,” said Clem Delangue, the CEO of Hugging Face—a platform that hosts repositories of LLMs of all stripes. “But ‘LLM’ is just a subset of AI when it comes to applying AI to biology, chemistry, image, audio, [and] video,” Delangue added, speaking at a conference. “I think we’re at the beginning of it, and we’ll see much more in the next few years.” A financial flurry Over just the past few months, world models have advanced from a research topic (which they still are, of course) to the basis for new commercial projects and huge funding rounds. A few key examples: In August, Google DeepMind unveiled Genie 3, a model that builds real-time interactivity on top of the foundation of a video generation model. In November, World Labs introduced Marble, a model and toolset that allows users to generate immersive environments that can be exported as 3D assets, based on input in the form of text, images, video, or other assets. Just a month later, video generation and filmmaking AI company Runway entered the fray with its announcement of GWM-1, a trio of specialized world models built on Runway’s past work with video models. Even more recently, Yann LeCun started Advanced Machine Intelligence (AMI), a company betting the farm on the notion that the real future of AI systems is in models that interact with the physical world (or at least simulate it), not just language. These and similar efforts have received substantial funding.

World Labs and AMI reportedly raised around $1 billion each in February and March, respectively, and Runway also raised $315 million in February. Some of the activity around world models is at least in part aimed at ostensibly building the foundations of AGI or superintelligence, but most people working on them are talking about practical applications: training, testing, and driving robots; generating 3D assets for game development and film production; scientific simulation and modeling; and so on. It’s important to note that “world models” is an umbrella term that is often thrown around without a clear definition, though. Defining “world model” “It’s definitely an overloaded term,” Vincent Sitzmann told me in a lengthy conversation about the research and concepts underlying world models. Sitzmann is an assistant professor at MIT who has published research on neural rendering, visual computing, and robotics. He leads the Scene Representation Group within MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). Vincent Sitzmann leads MIT CSAIL’s Scene Representation Group. Vincent Sitzmann leads MIT CSAIL’s Scene Representation Group. Credit: MIT When asked to give a definition, he simply described a world model as any model that takes in an interaction, “and given that interaction, it enables you to simulate what would happen next in some environment.” When announcing its GWM-1 family of models in December, Runway defined a world model as “an AI system that builds an internal representation of an environment and uses it to simulate future events within that environment.” Further, “the aim of general world models is to represent and simulate a wide range of situations and interactions—like those encountered in the real world,” Runway added. I also spoke with Ben Mildenhall, co-founder of World Labs, a former Google computer vision and physics researcher and co-creator of neural radiance fields (NeRF)—a method for constructing navigable 3D scenes from 2D images in a format that is differentiable and therefore useful in machine learning contexts. A group shot of the World Labs team. Left and center in the black hoodie is Ben Mildenhall; to his left, also in a black hoodie, is co-founder Fei-Fei Li. A group shot of the World Labs team. Left and center in the black hoodie is Ben Mildenhall; to his left, also in a black hoodie, is co-founder Fei-Fei Li.

Credit: World Labs “The key things that would distinguish it from an LLM are demonstrating degrees of spatial and—maybe for lack of a better word—continuous understanding,” he said. “A very distinguishing aspect of interacting with an LLM is they are turn-based.” Meaning, users type some text, there’s a pause, and then they get a block of text back. By contrast, he sees a world model as a synchronous, real-time system. “Something that would define a world model is the degree of freedom that you have in interacting with the spatial world, where you do not have this mediated linear journey of A then B then A then B then A then B,” he said. When a user or agent is utilizing a world model, they are “actually able to interact with it like it is some sort of world and you are taking continuous actions” where “there are parallel things happening at the same time.” Mildenhall’s co-founder Fei-Fei Li has written that she believes there are three criteria that define a world model: World models “can generate worlds with perceptual, geometrical, and physical consistency,” “are multimodal by design,” and “can output the next states based on input actions.” The phrase itself is not new; it has long appeared in reinforcement learning and robotics to describe models that predict environment dynamics. What is new is the attempt to scale that idea into general-purpose, generative systems trained on massive visual and multimodal data. The huge funding rounds are relatively new, too. The truth is that there are numerous approaches and definitions. “Ask Runway, ask us, ask whoever—we’re all gonna give you a little bit of a different term,” Mildenhall said. Further, it’s important to be aware that “world model” is becoming a branding term as much as a technical one. It’s used as a marketing label that can cover anything from action-conditioned video generation to 3D asset creation to robot policy evaluation. While there is conviction that one foundation model ought to be able to solve for a lot of these things, companies are using the term while solving different technical problems with varying approaches. The video foundation “Today, what most people mean when they say ‘world model’ is generating pixels, so, like, generating a realistic video conditional of the actions,” Sitzmann said. We’ve seen video generation models gain traction over the past couple of years.

Runway, for example, built its reputation as a company that makes video models and related tools that are used by filmmakers, advertisers, and others in creative fields. Now that focus has shifted, as the company has made clear its intention to expand beyond those areas to focus on world models like GWM-1 as well, with an eye toward applications in robotics and beyond in the coming years. That might seem like a jarring lateral pivot, but if you consider how these world models are being developed, it’s a natural next step. In many cases, they’re a direct extension of the work previously done on video models. “One of the moments we started realizing there was something beyond this short-form video generation task was at the point we released camera controls, so you could direct how a camera moves in the scene,” said Anastasis Germanidis, Runway’s CTO, when I spoke with him about GWM-1 back in December. “It started to feel more almost like a video game and you’re exploring this world and you move around. It feels less that I’m just creating a video and more that I’m exploring an environment that’s continuously being rendered by this model.” Runway CTO Anastasis Germanidis speaking at the Cerebral Valley Voice Summit. Runway CTO Anastasis Germanidis speaking at the Cerebral Valley Voice Summit. Credit: Newcomer AI Summits Most video diffusion models generate many frames simultaneously, denoising the sequence in one go. (Denoising is the core mechanism behind many AI image and video generators. The models begin with an image or frame of static-like random noise, and the denoising process iteratively predicts cleaner versions, guided by patterns learned during training, until the image resolves into a finished output.) This approach has advantages for coherence and quality because every frame is generated within the context not just of what past frames were but what future ones should be. That’s great for generating a fixed-length video, but it’s not a fit for the open-ended simulation and interactivity that a world model requires. There’s no way for users to meaningfully intervene with input and see an immediate result. A widely subscribed-to answer for video-based world models is autoregressive diffusion, which is still a denoising process, but the frames are denoised sequentially.