A VLA chooses actions; the trajectory records consequences
Language expresses intent and visual observations describe the current scene. The action target still needs a declared interface: pose delta, joint target, end-effector command, gripper state, discrete skill, or another embodiment-specific control.
Recording the state and outcome after execution makes the same episode useful for policy learning, world-model training, rollout evaluation, and failure analysis. One caption over an entire session cannot provide that causal resolution.
- Instruction, goal, and optional subgoal text
- Camera and sensor observations with calibration
- Action tokens plus physical units and control rate
- Embodiment, task, success, failure, and intervention metadata
Source classes do different jobs
Robot-native trajectories contain executable controls and embodiment-specific dynamics. Teleoperation can produce demonstrations in the robot action space. Simulation can expose exact state and controlled counterfactuals. Human egocentric video can broaden task, object, language, and visual coverage, but it is not robot control data by default.
If human demonstrations contribute to VLA training, the release should name the transfer path: representation learning, pose or action retargeting, paired human and robot views, inverse dynamics, simulation reconstruction, or robot-native post-training. Derived actions remain estimates unless validated against the target embodiment.
Action tokenization must remain reversible
A VLA may predict discretized tokens, continuous action chunks, or higher-level skills. The release should preserve the original command beside any quantized or normalized representation.
Token vocabulary, bin boundaries, chunk horizon, control frequency, coordinate frame, and clipping behavior belong in the schema. Otherwise identical token IDs can imply different motion across robots or datasets.
Cross-embodiment mixtures need explicit normalization
Large VLA programs often mix tasks and robot types. Positive transfer is possible, but morphology, sensors, action spaces, control rates, and success definitions do not become equivalent because they share one loader.
Keep source-dataset identity and embodiment metadata available during training and evaluation. Hold out robots, scenes, objects, and instruction families to measure generalization rather than memorization.
Acceptance tests the action interface
A delivery should be accepted against the model input window and target action horizon, not by checking that files exist. Loader tests should reconstruct synchronized examples with calibration, masks, episode boundaries, and source identity intact.
Evaluation should separate action-prediction error, intervention sensitivity, task success, failure recovery, and generalization across held-out objects, scenes, instructions, and embodiments. Offline prediction is useful evidence, but it is not a substitute for closed-loop evaluation when deployment claims depend on control.
- Declared observation window, action horizon, and control frequency
- Original commands beside normalized, retargeted, or tokenized actions
- Episode-level provenance, rights, quality status, and split lineage
- Held-out task, scene, object, instruction, and embodiment tests