The action is not just another label
A control input changes which future should be predicted. The same scene can evolve differently if the agent accelerates, waits, turns, selects a tool, or issues a high-level command.
Representations must preserve semantics. Keyboard state, pointer delta, analog axis, discrete policy token, and robot motor command should not be flattened into one ambiguous field.
Alignment determines causality
If an action timestamp follows its visual consequence, the training sequence teaches the wrong order. Releases should declare the clock, capture latency, resampling method, execution delay, and uncertainty.
Commanded and executed actions can differ. Keep both when the environment can block, clip, delay, or override an input.
- Raw input and normalized action
- Issue time and effective time
- Validity and missing-data masks
- Environment and control-map version
Evaluate whether the model uses action
Reconstruction quality can look strong even when the model ignores control. Compare futures under alternative actions, measure event timing, and test long-horizon consistency.
Hold out action sequences and environments that expose memorization. Report uncertainty when several futures remain valid.