Research guide
How to Evaluate a World-Model Dataset
A buyer-side evaluation framework covering alignment, coverage, state validity, leakage, rights, and loader-level QA.
- For
- Data buyers, research engineers, and procurement reviewers
- Reviewed
- July 24, 2026
Direct answer
Evaluate a world-model dataset as a temporal system, not a pile of files. Test synchronized windows, action validity, state consistency, coverage, split integrity, rights, and end-to-end loading.
Start with the learning contract
Write down what the model should predict, condition on, and control. The answer determines which streams are required and which omissions are acceptable. A video predictor, behavior-cloning policy, physics model, and interactive simulator do not need the same data contract.
Request a representative sample before reviewing scale claims. Inspect complete episodes, not only polished highlight clips.
- Required conditioning history and prediction horizon
- Action granularity and control frequency
- State fields available at training and evaluation time
- Target environments, tasks, failures, and rare events
Measure temporal integrity
Decode samples through the intended loader and verify that timestamps are monotonic, frame counts match declarations, action windows align, and missing data is surfaced through masks. Spot checks should include rapid motion, transitions, and failure events.
Test clock drift over long episodes. A dataset can appear aligned for the first few seconds and become unusable later.
Audit coverage and leakage
Summaries should be queryable by environment, task, action, outcome, duration, and collection version. Aggregate hours can hide severe concentration in a few easy conditions.
Split by the factor the model must generalize across. Random clip splits often leak the same match, map, session, or near-duplicate sequence into both training and evaluation.
Test rights as data
Rights should resolve per release and, where necessary, per sample. Verify source, permitted use, attribution, restrictions, consent or player-data handling, and withdrawal procedure.
A license string without provenance is not a rights system. Require the evidence supporting the label and record the review date because platform and publisher terms can change.