World Foundation Models

World foundation models (WFMs) are neural networks that simulate real-world environments as videos and predict accurate outcomes based on text, image, or video input. Physical AI developers use WFMs to generate custom synthetic data or downstream AI models for training robots and autonomous vehicles.

World models are generative AI models that understand the dynamics of the real world, including physics and spatial properties. They use input data, including text, image, video, and movement, to generate videos. They understand the physical qualities of real-world environments by learning to represent and predict dynamics like motion, force, and spatial relationships from sensory data.

Foundation models are AI neural networks trained on massive unlabeled datasets to generate new data based on input data. Due to their generalizability, they can significantly accelerate the development of a wide range of generative AI applications. Developers can fine-tune these pretrained models on smaller, task-specific datasets for a custom domain-specific model.

Developers can leverage the power of foundation models to generate high-quality data for training AI models in industrial and robotics applications, such as factory robots, warehouse automation, and autonomous vehicles operating on highways or in challenging terrains. Physical AI systems require large-scale, visually, spatially, and physically accurate data for learning through realistic simulations. World foundation models generate this data efficiently at scale.

There can be different types of WFMs:

• Prediction Models – These models predict world generation and synthesize continuous motion based on a text prompt, input video, or by interpolating between two images. They enable realistic, temporally coherent scene generation, making them valuable for applications like video synthesis, animation, and robotic motion planning.
• Style Transfer Models – These models guide outputs based on specific inputs using ControlNet, a model network that conditions a model’s generation based on structured guidance such as segmentation maps, lidar scans, depth maps, or edge detection. By visually mirroring input instructions, these models can control layout and motion while producing diverse, photorealistic results grounded in a text prompt. This makes them useful for applications that require structured image or video synthesis, such as digital twin simulations and environmental reconstruction.
• Reasoning Models – These models take multimodal inputs and analyze them over time and space. They use a chain-of-thought reasoning approach based on reinforcement learning to understand what’s happening and decide on the best actions. These models enable AI to tackle complex tasks, such as distinguishing between real and synthetic data, selecting useful training data for robots or games, predicting robotic actions, and optimizing logistics for autonomous systems.

World models, when used with 3D simulators, serve as virtual environments to safely streamline and scale training for autonomous machines. With the ability to generate, curate, and encode video data, developers can better train autonomous machines to sense, perceive, and interact with dynamic surroundings.

WFMs bring significant benefits to every stage of the autonomous vehicle (AV) pipeline. With pre-labeled, encoded video data, developers can curate and train the AV stack to recognize the behavior of vehicles, pedestrians, and objects more accurately. These models can also generate new scenarios, such as different traffic patterns, road conditions, weather, and lighting, to fill training gaps and expand testing coverage. They can also create predictive video simulations based on text and visual inputs, accelerating virtual training and testing.

WFMs generate photorealistic synthetic data and predictive world states to help robots develop spatial intelligence. Using virtual simulations powered by physical simulators, these models let robots practice tasks safely and efficiently, accelerating learning through rapid testing and training. They help robots adapt to new situations by learning from diverse data and experiences.

Modified world models enhance planning by simulating object interactions, predicting human behavior, and guiding robots to reach goals accurately. They also enhance decision-making by conducting multiple simulations and learning from the feedback. With virtual simulations, developers can reduce real-world testing risks, cutting time, costs, and resources.

Trained with rich, multimodal data and advanced reasoning capabilities, WFMs can perform complex video analytics on massive amounts of recorded and live videos. These models enable natural language Q&A, automated summarization, object detection, event localization, and richer contextual understanding of visual content in videos—capabilities that surpass traditional computer vision methods.

Common applications of WFMs for video analytics are found in both industrial and smart-city settings to improve safety and operational efficiency.  Examples include identifying injury risks and unsafe behaviors for industrial safety, providing a detailed cause-and-effect understanding for rapid incident investigation, monitoring traffic, crowd flows, public safety incidents, and environmental hazards in smart cities, and identifying defects and irregularities on manufacturing lines through visual inspection for quality control.

Building a world model for a physical AI system, like a self-driving car, is resource- and time-intensive. First, gathering real-world datasets from driving around the globe in various terrains and conditions requires petabytes of data and millions of hours of simulation footage. Next, filtering and preparing this data demands thousands of hours of human effort. Finally, training these large models costs millions of dollars in GPU compute and requires many GPUs.

WFMs aim to capture the underlying structure and dynamics of the world, enabling more sophisticated reasoning and planning capabilities. Trained on vast amounts of curated, high-quality, real-world data, these neural networks serve as visually, spatially, and physically aware synthetic data generators for physical AI systems.

WFMs allow developers to extend generative AI beyond the confines of 2D software and bring its capabilities into the real world while reducing the need for real-world trials. While AI’s power has traditionally been harnessed in digital domains, world models will unlock AI for tangible, real-world experiences.

World models can create more realistic and physically accurate visual content by understanding the underlying principles of how objects move and interact. These models can generate realistic 3D worlds on demand for many uses, including video games and interactive experiences. In certain cases, outputs from highly accurate world models can take the form of synthetic data, which can be leveraged for training perception AI.

Current AI video generation can struggle with complex scenes and has a limited understanding of cause-and-effect relationships. However, world models paired with 3D simulation platforms and software are showing potential to demonstrate a deeper understanding of cause and effect in visual scenarios, such as simulating a painter leaving brushstrokes on a canvas.

WFMs help physical AI systems learn, adapt, and make better decisions by simulating real-world actions and predicting outcomes. They enable systems to “imagine” different scenarios, test actions, and learn from virtual feedback—much like a self-driving car practicing in a simulator to handle sudden obstacles or adverse weather conditions. By predicting possible outcomes, an autonomous machine can plan smarter actions without needing real-world trials, saving time and reducing risk.

When combined with large language models (LLMs), world models help AI understand instructions in natural language and interact more effectively. For example, a delivery robot could interpret a spoken request to "find the fastest route" and simulate different paths to determine the best one.

This predictive intelligence makes physical AI models more efficient, adaptable, and safer—helping robots, autonomous vehicles, and industrial machines operate smarter in complex, real-world environments.

Policy learning involves exploring strategies to determine the most effective actions. A policy model helps a system, such as a robot, determine the best action to take based on its current state and the broader state of the world. It links the system’s state (e.g., position) to an action (e.g., movement) to achieve a goal or improve performance. A policy model can be derived from fine-tuning a model. Policy models are commonly used in reinforcement learning, where they learn through interaction and feedback.

Use a reasoning WFM to filter and critique synthetic data, improving quality and relevance at speed.

World models enable strategy exploration, rewarding the most effective outcomes. Add a reward module to run simulations and build cost models that track resource use—boosting both performance and efficiency for real-world tasks.

World models require extensive real-world data, particularly video and images, to learn dynamic behaviors in 3D environments. Neural networks with billions of parameters analyze this data to create and update a hidden state or an internal representation of the environment. This enables robots to understand and predict changes, such as perceiving motion and depth from videos, predicting hidden objects, and preparing to react to potential events. Continuous improvement of the hidden state through deep learning allows world models to adapt to new scenarios.

Here are some of the key components for building world models:

Data curation is a crucial step for pretraining and continuous training of world models, especially when working with large-scale multimodal data. It involves processing steps like filtering, annotation, classification, and deduplication of image or video data to ensure high quality when training or fine-tuning highly accurate models.

In video processing, data curation starts with splitting and transcoding the video into smaller segments, followed by quality filtering to retain the high-quality data. State-of-the-art vision language models are used to annotate key objects or actions, while video embeddings help semantic deduplication to remove redundant data.

The data is then organized and cleaned for training. Throughout this process, efficient data orchestration ensures a smooth data flow among the GPUs, enabling them to handle large-scale data and achieve high throughput.

Tokenization converts high-dimensional visual data into smaller units called tokens, facilitating machine learning processing. Tokenizers transform pixel redundancies in images and video into compact, semantic tokens, enabling efficient training of large-scale generative models and inference on limited resources. There are two main methods:

• Discrete Tokenization: Represents images and videos as integers.
• Continuous Tokenization: Represents images and videos as continuous vectors.

This approach enhances model learning speed and performance.

Foundation models are AI neural networks trained on vast unlabeled datasets to perform various generative tasks. Developers can train a model architecture from scratch or fine-tune a pretrained foundation model for downstream tasks using additional data.

WFMs serve as generalist models, trained on extensive visual datasets to simulate physical environments. Using fine-tuning frameworks, these models can be specialized for precise applications in robotics, autonomous systems, and other physical AI domains. There can be multiple approaches to fine-tune a model:

• Unsupervised Fine-Tuning – Involves adapting a model using unlabeled data, allowing it to learn representations and patterns from new datasets without explicit labels. This method is useful for broad generalization and domain adaptation.
• Supervised Fine-Tuning – Uses labeled datasets where the model is explicitly guided to learn task-specific features. This approach enhances decision-making, improves structured pattern recognition, and ultimately develops reasoning capabilities for more complex AI-driven applications.

To get started easily and streamline the end-to-end development process, developers can leverage training frameworks, which include libraries, SDKs, and tools for data preparation, model training, optimization, and performance evaluation and deployment.