Research guide
What Data Do World Models Need?
A practical guide to observations, actions, state, outcomes, time alignment, and coverage for world-model training datasets.
- For
- Research leads planning a new world-model data program
- Reviewed
- July 24, 2026
Direct answer
World models need temporally ordered evidence of how an environment changes. The strongest datasets connect observations with the actions that caused change, the latent or measured state, and the resulting outcome.
A world model learns transition structure
A passive video corpus can teach appearance, motion priors, and common visual sequences. It cannot reliably identify which control input caused a transition. An action-conditioned dataset closes that gap by recording what the agent observed, which action was issued, what state followed, and whether the attempt progressed or failed.
The correct unit is usually an episode rather than a loose clip. Episode boundaries preserve task context, initial conditions, intervention history, and terminal state. A training loader should be able to sample complete trajectories or valid temporal windows without guessing where one attempt ends and another begins.
- Observations: pixels, audio, depth, language, or sensor measurements
- Actions: keyboard, mouse, gamepad, policy command, or robot control
- State: engine telemetry, pose, inventory, physics, or proprioception
- Outcomes: rewards, task progress, events, failure, and terminal condition
Timing is part of the label
Streams that share a folder are not necessarily synchronized. A small offset between video and control input can teach the model that an action follows its consequence. The release should declare clock source, sampling frequency, resampling rules, dropped-frame behavior, and uncertainty.
Keep commanded and measured actions distinct. In a game, a key press can be blocked by menu state or game logic. In robotics, a controller can clip, delay, or override a command. Preserving both intent and execution makes failures diagnosable.
Coverage beats raw hours
Recorded time is a weak proxy for useful transition coverage. A thousand hours of repetitive forward motion may teach less than a smaller release that deliberately spans environments, camera states, action combinations, failures, recoveries, and rare events.
Define the coverage matrix before collection. Each cell should connect an environment condition, task state, action family, outcome, and minimum accepted count. This turns collection into a measurable program instead of an upload contest.
Rights and provenance travel with the sample
A downloadable clip is not automatically lawful training data. Every release needs a traceable source, rights holder, allowed use, redistribution status, attribution requirement, modification record, and withdrawal path. Game footage also carries publisher, trademark, platform, and player-data considerations.
The practical test is simple: can a buyer trace each sample back to a documented collection and rights decision without relying on a sales claim? If not, the dataset is not ready for enterprise use.