Research guide
Action-Conditioned Data vs. Passive Video
How action labels change what a world model can learn, and when large passive-video corpora still provide value.
- For
- Teams deciding whether video-only pretraining is enough
- Reviewed
- July 24, 2026
Direct answer
Passive video shows what changed. Action-conditioned data also records what the agent did before the change. That causal handle is essential for controllable prediction, planning, and interactive simulation.
Passive video is useful but ambiguous
Video contains rich evidence about objects, scenes, motion, and temporal continuity. It can support representation learning and teach a generator what plausible futures look like. The ambiguity appears when several unobserved actions could explain the same visual transition.
Editing, camera cuts, overlays, replays, and spectator views add further ambiguity. Without action or state, the model may learn presentation conventions rather than environment dynamics.
Actions provide a controllable axis
An action-conditioned sequence lets a model estimate a distribution over future observations given both history and control. That makes counterfactual questions possible: what changes if the agent turns left, waits, accelerates, or selects another tool?
The action representation must match the data-generating system. A keyboard state, analog stick vector, high-level instruction, and motor command are not interchangeable labels. Normalize them only when the mapping is explicit and reversible.
Use a layered training strategy
Many practical systems can use passive video for broad visual pretraining and a smaller action-labeled corpus for controllability. The valuable question is not which dataset type wins. It is which learning objective each layer supports and how the evaluation isolates its contribution.
Maintain the boundary in metadata. Mark whether an action was observed, inferred, generated, or unavailable. Synthetic action labels can be useful, but they should never silently become ground truth.
Evaluate interventions, not only reconstruction
A model can reconstruct familiar frames while ignoring action. Evaluation should compare futures under different controls, measure event timing, test long-horizon consistency, and inspect rare action combinations.
Hold out environments, tasks, and action sequences that expose memorization. Report uncertainty where several futures remain valid rather than scoring one arbitrary frame as the only correct answer.