Simulating everything, sort of: The promise and limits of world models

See also Fei-Fei Li, the computer vision pioneer who co-founded World Labs, one of the new companies working on world models. In a Substack post late last year, she wrote: Today, leading AI technology such as large language models (LLMs) have begun to transform how we access and work with abstract knowledge. Yet they remain wordsmiths in the dark; eloquent but inexperienced, knowledgeable but ungrounded. Spatial intelligence will transform how we create and interact with real and virtual worlds—revolutionizing storytelling, creativity, robotics, scientific discovery, and beyond. This is AI’s next frontier. LeCun and Li’s ventures are built on these ideas, so it’s not surprising they’d say these things. But you’ll also see similar sentiments from some prominent figures still working primarily with LLMs. “I think we’re in an LLM bubble, and I think the LLM bubble might be bursting next year,” said Clem Delangue, the CEO of Hugging Face—a platform that hosts repositories of LLMs of all stripes. “But ‘LLM’ is just a subset of AI when it comes to applying AI to biology, chemistry, image, audio, [and] video,” Delangue added, speaking at a conference. “I think we’re at the beginning of it, and we’ll see much more in the next few years.” Over just the past few months, world models have advanced from a research topic (which they still are, of course) to the basis for new commercial projects and huge funding rounds. A few key examples: These and similar efforts have received substantial funding. World Labs and AMI reportedly raised around $1 billion each in February and March, respectively, and Runway also raised $315 million in February. Some of the activity around world models is at least in part aimed at ostensibly building the foundations of AGI or superintelligence, but most people working on them are talking about practical applications: training, testing, and driving robots; generating 3D assets for game development and film production; scientific simulation and modeling; and so on. It’s important to note that “world models” is an umbrella term that is often thrown around without a clear definition, though. “It’s definitely an overloaded term,” Vincent Sitzmann told me in a lengthy conversation about the research and concepts underlying world models. Sitzmann is an assistant professor at MIT who has published research on neural rendering, visual computing, and robotics. He leads the Scene Representation Group within MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). This approach has advantages for coherence and quality because every frame is generated within the context not just of what past frames were but what future ones should be. That’s great for generating a fixed-length video, but it’s not a fit for the open-ended simulation and interactivity that a world model requires. There’s no way for users to meaningfully intervene with input and see an immediate result. A widely subscribed-to answer for video-based world models is autoregressive diffusion, which is still a denoising process, but the frames are denoised sequentially. Germanidis explained it this way when describing how GWM-1 works: By generating all the frames at once… the future can influence the past. You don’t have this causal nature to what you’re creating. So the difference with the autoregressive diffusion, which is the approach that we’re taking with GWM, is you’re essentially denoising one or a few frames at a time. So you’re denoising those frames, you present it to the user, the user can provide an action that influences the next frames. This approach also has some disadvantages. It’s extremely compute-expensive, as each batch of frames requires a full denoising process, and each needs to resolve quickly enough that the user experiences a timely response to their inputs. Further, the simulation will drift and forget information from early frames over time. “If you’re in a room, you exit the room, you go back to the room, you need to make sure that all the details of the room are preserved inside the context and the memory of the model,” Germanidis said. These are wrinkles that are still being ironed out, even in the largest, most cutting-edge frontier models. “Our overall approach is that 3D consistency and that statefulness should emerge from scale and from just predicting 2D pixels,” Germanidis said. “So we should try to, as much as possible, not introduce any specific techniques that tell the model this is a 3D scene and you need to have this 3D representation that you’re using to guide the model.” This reflects a widely accepted idea in the field called “the bitter lesson,” which originated in a short but influential essay by computer scientist and reinforcement learning pioneer Richard Sutton. Sitzmann told me he believes this is also a cornerstone of the argument against making explicit 3D data or articulations of the laws of physics the basis for training world models. He argued: Through many, many—in fact, in this case, literally thousands of years of research—we have been very proud that we figured out that the world is three-dimensional and how light bounces around and makes little images on our camera, and that is the reason why we would like these algorithms to have these kind of 3D representations, to make use of all of our know-how. But then in the end, the bitter lesson would suggest that this is maybe a fallacy because at the end of the day, the thing you care about is the input-output behavior. And for a particular input-output behavior, it turns out that the thing that we conjured up in our minds is not actually the right way to do it. For this reason and others, many research groups and companies working on world models today are using a process very similar to what Germanidis described. However, not all world models are about outputting videos to a user. Some video-based models are built to output actual 3D assets or data in addition to 2D video in the form of meshes, Gaussian splats, or something else; that’s the main product focus for World Labs at the moment, though the company’s ambitions for the future are broader. Co-founder Fei-Fei Li wrote the following on this topic: World model research will inevitably drive advances in model architecture and learning algorithms, particularly beyond the current MLLM and video diffusion paradigms. Both of these typically tokenize data into 1D or 2D sequences, which makes simple spatial tasks unnecessarily difficult—like counting unique chairs in a short video, or remembering what a room looked like an hour ago. Alternative architectures may help, such as 3D or 4D-aware methods for tokenization, context, and memory. For example, at World Labs, our recent work on a real-time generative frame-based model called RTFM has demonstrated this shift, which uses spatially-grounded frames as a form of spatial memory to achieve efficient real-time generation while maintaining persistence in the generated world. Speaking to Ars, Li’s co-founder, Mildenhall, described the explicit 3D aspect of the company’s current tools and products as simply a matter of giving users an off-ramp to an export format for now. “We choose to bake out to 3D so that we can go talk to a bunch of people who have workloads today that they can make use of this persistent thing they can share, edit, integrate, render against, and so on,” he explained. World Labs’ publicly usable systems use video and image data to infer 3D structure and output explicit scene representations using techniques such as neural radiance fields (NeRFs) and Gaussian splats so that users get assets that work with traditional 3D workflows. Marble, World Labs’ product you can jump into right now, takes in an image or a video, along with an optional text prompt, and creates a space roughly the size of a typical American suburban backyard. You can then fly around in it with a keyboard and mouse or export it as a 3D asset for use elsewhere. As for why World Labs is using representations like NeRFs and Gaussian splats rather than conventional meshes, Mildenhall said: The whole point of NeRF and splatting was primitive meshes are a huge pain to integrate into ML workflows because they have an explicit discrete topology—they’re not a uniform representation… The thing with NeRF and splats is you have a notion that’s more continuous of, well, it’s sort of partially opaque over here, mostly transparent over here, and when you render with that type of volumetric-based system, it’s much easier—or it’s possible—to actually pass gradients through and backprop and train a model. Producing an asset that can be brought into current workflows gives World Labs’ product the potential for immediate application by game developers, VFX artists, and professionals in other creative and technical disciplines. Runway’s approach currently harbors an advantage for dynamic simulation, though. “A lot of the world models of today have focused on kind of navigating this static, lifeless world. And we think that actually simulating the world well and building good world models means being able to generate dynamic worlds where things happen, where you can take actions and see the consequences,” said Germanidis. “That’s the biggest differentiation around our announcement compared to some other world model announcements.” At that point, it can render locally on consumer 3D hardware. This is a big plus at a time when cloud compute is at an absolute premium amid exploding demand, and it’s a practical advantage in terms of getting the technology into users’ hands in meaningful ways right now. On the other hand, if you’re building a model for simulations in robotics, many argue that these explicit 3D representations matter a lot less than they do for content creation or asset generation. “In order to simulate performing different manipulation tasks, you need to have the robot perform actions and have the interaction with different objects be simulated well. And you need to be able to simulate failure very well as well,” Germanidis said. But let’s be clear: Whether explicit 3D structure is included in the training process or as an output artifact (or not at all), video is still the foundation for what many of these commercially oriented teams are currently doing, including both Runway and World Labs. “The underlying technical backbone has a lot in common across most of these approaches… it’s also just diffusion-based stuff,” Mildenhall told me. “It’s not like a fork at the root. It’s really like a branch in the application side.” Now take a humanoid robot operating in a home or a workplace as a contrasting example; it is a much more complex machine, with a more sophisticated range of motions and actions in the face of potentially far more varied situations and environments. There is comparatively little useful training data tailored to this purpose. You might have read news stories about AI and robotics companies offering incentives or paying people to wear cameras while doing chores at home; those companies are trying to get training data. However, that approach is unlikely to produce enough data on its own. Physics are key for robotics and other areas of physical AI. Any autonomous physical unit must demonstrate something at least analogous to an understanding of the laws of the physical world. Intuitively, it might seem odd to say that something built primarily from video data could develop anything resembling physical understanding, but many researchers and the companies building products on top of their research believe that video models appear to demonstrate a genuinely useful ability to reflect or predict real-world physics, at least when looking at their inputs and outputs. “In order to solve this problem of predicting the next frame of a video very well, you need to basically predict so many aspects of the physical world, so many aspects of how objects move, how people move in an environment, physics,” Germanidis told me. These models are typically not trained on or constrained by explicit physics rules, data, or simulations—remember, the bitter lesson. Rather, proponents of this approach say that there is a useful-enough approximation of understanding in latent space. When asked to define this term for those outside of the AI world, Sitzmann said: If you looked into your own brain, if you could—which we can’t, but if you could—you would probably not find in your own brain a mesh of your environment. You would find something that is kind of a weird thing that still somehow contains certain information about the 3D scene, but it would not be in a human-readable, human-accessible way. It would not be stored in some format like an explicit 3D scene. And so when people say latent space, that is kind of what they mean. You don’t tell the model how that 3D representation should look. Instead, kind of like your own brain, you force the model to have a certain input/output behavior—you force it to always predict the next frame of a video, and then if you make the model big enough and you let it figure it out itself, then the model will figure out itself how it wants to represent 3D scenes. “Latent” simply means not directly visible in the input or output. In this context, it usually refers to hidden internal representations—activations, embeddings, or compressed states—that the model develops as it learns to predict future frames or actions rather than to an explicit human-or-robot-readable output. Strangely, it’s not explicitly clear to human observers how the model can accurately simulate physical behaviors, but proponents of this approach argue that as long as you can prove the outputs will be useful, it’s not a disqualifying problem that we can’t fully grok the internal workings of the models. Germanidis described methodologies for testing models like GWM-1 to demonstrate their usefulness for simulation and for training the policies that control robots. First, he said you can use a model to perform a specific task in the real world and try the same inside a model like GWM-1. If there’s a correlation between the success and failure of a model in a real-world scenario compared to the same model in a simulation, you gain some confidence that you can use the model to test policies. “You can do the same with the synthetic data case,” he said. There is another approach, which is you want to test the physics of those models directly. So you construct some physical scenarios, whether it’s in solid mechanics or thermodynamics. You have a first frame that’s maybe a ball hanging from the ceiling, and you’re giving that as a first frame to a model and you’re asking basically ‘generate what happens next.’ And then you have shot that same video in actual, real-world footage. You have this trajectory of the ball, and you can have this synthetic trajectory of the ball. And you can correlate the two. Sitzmann described similar scenarios. “The way this will be proven useful is that people will increasingly implement systems that use these models for doing robot tasks in a variety of different ways. Then we have a benchmark, we run the benchmark, there you go,” he said. “This is, by the way, also how it happened with self-driving cars, right? With self-driving cars, what in the end happens is you just show empirically that it works.” While we’re still in the early days of this field, a growing body of research suggests that video and world models can already be useful for synthetic data generation, policy evaluation, and some robot-planning tasks. There’s enough to justify cautious optimism about these models, though it’s worth noting that their reliability as general-purpose physical simulators remains unproven. Synthetic data could still teach the wrong lessons if it misrepresents contact forces, surface friction, or any number of other things. Small mismatches can break policies, and closing that gap in wide-ranging scenarios is still not completely solved. Further, real-world situations will inevitably present unexpected variations. This is one of many reasons why impressive marketing videos from robotics companies showing robots—humanoid or otherwise—performing tasks previously thought impossible can be misleading. Everything from the environments to the tasks to the specific objects being manipulated can be carefully selected to show just what’s possible if all this could be sufficiently generalized—but that generalization is not yet proven. A useful world model for robotics should improve policy learning or policy evaluation on held-out tasks, in environments it was not tuned for, with failure cases included rather than edited away. It should also be compared against simpler baselines like conventional simulators. Without that, a demo can show that a model has learned enough structure to be compelling, but not that it is the most reliable or scalable way to build physical intelligence. Sitzmann made the same point in our conversation. “No one needs to be explained how to interact with an LLM,” he said. “You ask them a question, they tell you something. You ask them to code for you, they do something. For world models, maybe not as much.” Mildenhall called this representation question the real “kicker.” There is likely no one-size-fits-all interface for world models’ various future applications. In terms of human interfaces, some people may interact with what they’re generating via video game controllers and a virtual avatar or the same control scheme they use to play 3D games on a PC. Others will want a Blender– or Unreal-like scene view. And then there are the applications where the interface is for robots, software, or other AI models, not humans. Very little of this is settled. But practicality and speed-to-market won out for Runway, with a focus on meeting immediate customer needs. Germanidis said the company plans to unify the three models into one generalized model later. Mildenhall has the same attitude about the value of pursuing a unified model. “Very reasonably, there is one model where you would be running every stream of pixels through it and be getting some kind of foundational knowledge or infrastructure out of that that you can layer things on top of,” he said. “To me, it would feel like we’re doing something too bespoke if we’re seeing a lot of fragmentation and more than a little bit of fine-tuning to have to adapt to those different use cases.” In fact, Mildenhall went so far as to predict that AI tools like these will help reduce fragmentation within the existing industries doing work with 3D assets and simulation. Here’s an example: A company like Disney makes Star Wars movies and TV shows, but it is also involved to some degree in working with game studios and publishers to make Star Wars video games. In theory, you might imagine that some 3D assets used in the films could be adapted for use in the games. But that’s not how it works, for several reasons. Historically, the two use different formats and software and have different levels of detail depending on what type of machine is rendering them and whether the rendering is pre-baked or being done in real time. These assets are enormously expensive to produce, so it would be significant if the differences could be bridged. “A lot of the things that are issues I think will be solved by more automated tooling that comes with AI,” Mildenhall argued, while also noting that Epic—the company that makes the Unreal Engine and related tooling—has already been working on trying to consolidate much of this. All of this is to say that much is still up in the air. The investment is pouring in because that’s the only way to see if there’s “a there there,” as the saying goes. And there are promising signs giving researchers and commercial entities reasons to be optimistic. When I spoke with Sitzmann about competing approaches within the industry and the field—like, for example, between using explicit 3D or physics data in training versus relying on the latents in the video models—he said the latter is what he’s currently most optimistic about as a researcher but that it’s still early days. “I’m saying it’s a bet. This is not a decided thing,” he clarified. The notion that companies and researchers are placing bets—some more proverbial than others—doesn’t just apply to the question of explicit 3D or physics in training data. It also applies to figuring out how human users will see and interact with what the models generate. It applies to product decisions about how these tools will fit in current 3D workflows in fields like game development. It even applies to whether the desired output for a specific application is actually video or some other kind of data, or even whether autoregressive diffusion and video models are a key part of building working products in the world of physical AI. Sitzmann further addressed this topic in a blog post he published after our conversation, writing: I do not believe that video generative models will solve embodied intelligence. They may not even be a necessary component of the final solution. Rather, they should be seen as one of several early attempts at identifying a scalable pre-training objective for perception–action learning… However, nevertheless, things have changed: I believe that we are now at a time where tackling these questions head-on is viable. People are making these bets because the potential upside of getting it right is huge, both in terms of commercial success and solving problems. Setting aside more speculative applications of world models, such as scientific modeling or healthcare, there’s reason to be hopeful that they will have applications in robotics, manufacturing, and other areas under the physical AI umbrella, even if they are neither the entire nor the final solution. AI researchers often talk about “AI Springs” and “AI Winters”—periods when progress in the field is either booming or stagnant. The past few years have been an AI Spring, and most people see that in large language models, but the bets being placed right now make it clear that many people in the field believe this AI Spring might not end with LLMs.