Spatiotemporal World Models and Real-Time Simulation

The core shift across 2 talks

Scaling laws extend beyond turn-based text to continuous video and action streams, turning pixels-to-actions policies and next-frame predictors into real-time world simulators that internalize physics, control robots, and replace static 3D engines with interactive visual interfaces.

General Intuition and Runway pushed AI engineering past autoregressive text into continuous space and time. General Intuition argued that real-time agents must observe while acting, leveraging Medal's gameplay pipeline—scaling toward three billion videos a year and 1.5 trillion monthly controller actions—because high-APM game controllers share the exact action space of low-DOF robot teleoperation and plug neatly as reactive control layers beneath frontier LLM planners. Runway bet on compute scaling laws for next-frame video prediction, demonstrating that models trained at scale internalize geometry, gravity, occlusion, and object permanence directly from pixels—unlocking action-conditioned robotics simulators, engine-free World Interface Models with no Unity or Blender mesh underneath, interactive Continuum video portals, and visual scratchpads where coding agents simulate spatial designs alongside code. Their sharpest technical disagreement sits on supervision: General Intuition contended that explicit controller action labels transfer far better to real-time control than passive video-only latent inference because subtle inputs like aircraft rudder adjustments leave almost no trace in raw pixel frames.

“So what this means is that the action space for video games, often because of game controllers, and the action space for operations or the real world many cases are the same.”Pim de Witte · Frontier Models for Acting in Space and Time · 00:02:33

Observing while acting: gameplay controllers as the universal robotics interface

In Frontier Models for Acting in Space and Time (00:06:01), Pim de Witte argued that medal's gameplay recorder is scaling from one billion to three billion uploaded videos annually, generating over 1.5 trillion action-labeled inputs per month and matching frontier robotics training volume in half a day. In Frontier Models for Acting in Space and Time (00:07:53), Pim de Witte argued that high-APM gameplay trajectories provide far denser simultaneous multi-key supervision than sparse human teleoperation recordings, bringing high-frequency control tasks like drone micro-adjustment within reach.

In Frontier Models for Acting in Space and Time (00:10:31), Pim de Witte argued that in production architectures, real-time pixel-to-action models serve as a fast reactive control layer plugged beneath a frontier LLM orchestrator such as Gemini that handles long-horizon planning. In Frontier Models for Acting in Space and Time (00:00:09), Pim de Witte argued that autoregressive, turn-based LLMs struggle with spatial-temporal control because real-time physical and virtual agents must continuously observe incoming sensory frames while simultaneously executing actions.

“Medal at the moment sees about a billion videos uploaded per year. This year, we're on track for three billion, which lead to over a trillion and a half actions taken per month.”Pim de Witte · Frontier Models for Acting in Space and Time · 00:06:01

“You don't want to start with tell operations because the human moves take a lot less actions. It has a lot less information, so you want a model that can predict really high APM.”Pim de Witte · Frontier Models for Acting in Space and Time · 00:07:53

“we still expect you to use an orchestrator model, so you may use something like Gemini or Astro in order to do long-term planning and everything, and you'll be able to plug these into your workloads.”Pim de Witte · Frontier Models for Acting in Space and Time · 00:10:31

• Pim de Witte: Medal's gameplay recorder is scaling from one billion to three billion uploaded videos annually, generating over 1.5 trillion action-labeled inputs per month and matching frontier robotics training volume in half a day.
• Pim de Witte: High-APM gameplay trajectories provide far denser simultaneous multi-key supervision than sparse human teleoperation recordings, bringing high-frequency control tasks like drone micro-adjustment within reach.
• Pim de Witte: In production architectures, real-time pixel-to-action models serve as a fast reactive control layer plugged beneath a frontier LLM orchestrator such as Gemini that handles long-horizon planning.
• Pim de Witte: Autoregressive, turn-based LLMs struggle with spatial-temporal control because real-time physical and virtual agents must continuously observe incoming sensory frames while simultaneously executing actions.

Scaling next-frame video prediction into interactive World Interface Models

In Everything will be Simulated (00:02:30), Cristóbal Valenzuela argued that predicting sequential frames at scale forces video models to internalize real-world geometry, gravity, occlusion, and object permanence directly from pixel data without hand-coded physics laws. In Everything will be Simulated (00:02:30), Cristóbal Valenzuela argued that runway bets that compute scaling laws hold for next-frame video prediction just as they do for language models, yielding emergent visual realism that humans can no longer distinguish from real footage.

In Everything will be Simulated (00:07:00), Cristóbal Valenzuela argued that conditioning real-time frame prediction on interventions at arbitrary steps turns offline video models into branching counterfactual world simulators capable of serving as reasoning brains for embodied robots. In Everything will be Simulated (00:08:29), Cristóbal Valenzuela argued that world interface models trained on UI clicks and user actions can manipulate objects and viewpoints inside a single 2D image in real time without any underlying 3D mesh or game engine.

In Everything will be Simulated (00:10:12), Cristóbal Valenzuela argued that runway's Continuum research prototype treats a real-time world model as an operating system layer that turns structured web text into persistent, interactive video portals whose behavior adapts to clicks, queries, and window resizing. In Everything will be Simulated (00:13:49), Cristóbal Valenzuela argued that connecting a coding agent to Continuum gives it an interactive visual scratchpad to simulate spatial designs alongside code, letting humans inspect and steer the agent's imagination directly rather than through text alone.

“You're learning about physics. You're learning about object permanence. You're learning about gravity, you're learning about occlusion. You're learning about movements. You're learning about how the objects work in the real world.”Cristóbal Valenzuela · Everything will be Simulated · 00:02:30

“I think we believe in scaling loss, not only applied for language models, but also for video, and this has been true.”Cristóbal Valenzuela · Everything will be Simulated · 00:02:30

“You take a frame, you predict next frame, then X, then X, then X, but then it can act at any point in time, in that prediction, which means I can simulate any possible divergence or action in that world.”Cristóbal Valenzuela · Everything will be Simulated · 00:07:00

• Cristóbal Valenzuela: Predicting sequential frames at scale forces video models to internalize real-world geometry, gravity, occlusion, and object permanence directly from pixel data without hand-coded physics laws.
• Cristóbal Valenzuela: Runway bets that compute scaling laws hold for next-frame video prediction just as they do for language models, yielding emergent visual realism that humans can no longer distinguish from real footage.
• Cristóbal Valenzuela: Conditioning real-time frame prediction on interventions at arbitrary steps turns offline video models into branching counterfactual world simulators capable of serving as reasoning brains for embodied robots.
• Cristóbal Valenzuela: World interface models trained on UI clicks and user actions can manipulate objects and viewpoints inside a single 2D image in real time without any underlying 3D mesh or game engine.

Where text planners meet continuous visual scratchpads and real-time control

In Everything will be Simulated (00:02:30), Cristóbal Valenzuela argued that predicting sequential frames at scale forces video models to internalize real-world geometry, gravity, occlusion, and object permanence directly from pixel data without hand-coded physics laws. In Everything will be Simulated (00:02:30), Cristóbal Valenzuela argued that runway bets that compute scaling laws hold for next-frame video prediction just as they do for language models, yielding emergent visual realism that humans can no longer distinguish from real footage.

“You're learning about physics. You're learning about object permanence. You're learning about gravity, you're learning about occlusion. You're learning about movements. You're learning about how the objects work in the real world.”Cristóbal Valenzuela · Everything will be Simulated · 00:02:30

“I think we believe in scaling loss, not only applied for language models, but also for video, and this has been true.”Cristóbal Valenzuela · Everything will be Simulated · 00:02:30

• Cristóbal Valenzuela: Predicting sequential frames at scale forces video models to internalize real-world geometry, gravity, occlusion, and object permanence directly from pixel data without hand-coded physics laws.
• Cristóbal Valenzuela: Runway bets that compute scaling laws hold for next-frame video prediction just as they do for language models, yielding emergent visual realism that humans can no longer distinguish from real footage.

Where speakers pushed back and drew the boundary

No operational pattern at AI Engineer NYC went unqualified. Across the corpus, 1 counter-claim pushed back on naive versions of this shift—highlighting where judges introduce variance, where caches break, or where human judgment stays irreducible.

“There's no way of being able to tell that from the video frames. It's very difficult, so our mode of having the action data really transfers better to action sequences than it does to world models and to generate the pixels.”Pim de Witte · Frontier Models for Acting in Space and Time · 00:17:42