What does WorldModelBench measure (instruction following, five physics laws, commonsense), which model versions were scored, how reliable are its human labels and VLM judge, and what limits do the authors state?

WorldModelBench (NVIDIA, Berkeley, UCSD, MIT; Feb 2025) has 350 image-plus-text conditions in 7 application domains and 56 subdomains, graded 0-10 by humans on instruction following (0-3), five binary physics laws (Newton's first law, mass conservation/solid mechanics, fluid mechanics, impenetrability, gravitation) and two commonsense quality checks. 14 model configurations were labelled with 67K human labels; the best, kling-v1.5, scored 8.82, and the authors report that VBench correlates only 0.28 with physics adherence versus 0.69 with frame quality. A fine-tuned 2B judge predicts totals within about 4.1% average error. The authors acknowledge the small prompt set (350) and that answers are withheld, so scoring a new model needs their judge. Curator qualifications: Counts: '350' 7, '56 subdomains' 3, '67K' 7, '8.82' 3, '0.28' 4, '0.69' 4; text: 'the correlation diminishes significantly to merely 0.28 ... VBench does not effectively distinguish between videos based on their adherence to physical laws'; 'correlation (0.69) between frame-wise quality win rates of WorldModelBench and VBench'. Judge: abstract 'achieving 8.6% higher avereage accuracy'; intro 'learns to predict human preference with 9.9% lower error rate than GPT-4o'; Table 5 caption text 'the find-tuned world model judger achieves higher accuracy than GPT-4o model'. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2502.20694v1",
      "note": "WorldModelBench arXiv 2502.20694v1 PDF"
    },
    {
      "url": "https://github.com/WorldModelBench-Team/WorldModelBench/blob/00b7aa17a05f9fd1ab5c8f66bcf476d04c9c33bf/README.md",
      "note": "WorldModelBench arXiv 2502.20694v1 PDF"
    },
    {
      "url": "https://huggingface.co/api/datasets/Efficient-Large-Model/worldmodelbench",
      "note": "WorldModelBench arXiv 2502.20694v1 PDF"
    }
  ]
}