A VLA chooses actions; a world model predicts consequences
Language expresses intent and visual observations describe the current scene. The action target still needs a declared interface: pose delta, joint target, end-effector command, gripper state, discrete skill, or another embodiment-specific control.
Recording the state and outcome after execution makes the same episode useful for policy learning, world-model training, rollout evaluation, and failure analysis. One caption over an entire session cannot provide that causal resolution.
- Instruction, goal, and optional subgoal text
- Camera and sensor observations with calibration
- Action tokens plus physical units and control rate
- Embodiment, task, success, failure, and intervention metadata
Action tokenization must remain reversible
A VLA may predict discretized tokens, continuous action chunks, or higher-level skills. The release should preserve the original command beside any quantized or normalized representation.
Token vocabulary, bin boundaries, chunk horizon, control frequency, coordinate frame, and clipping behavior belong in the schema. Otherwise identical token IDs can imply different motion across robots or datasets.
Cross-embodiment mixtures need explicit normalization
Large VLA programs often mix tasks and robot types. Positive transfer is possible, but morphology, sensors, action spaces, control rates, and success definitions do not become equivalent because they share one loader.
Keep source-dataset identity and embodiment metadata available during training and evaluation. Hold out robots, scenes, objects, and instruction families to measure generalization rather than memorization.