What is VideoPhy-2, how are videos scored, how do systems score, and how far can its automatic evaluator be trusted?

VideoPhy-2 (arXiv v1, Mar 2025) tests physical commonsense on 197 real-world actions with 3,940 LLM-written prompts. Three human annotators rate each video for semantic adherence and physical commonsense on 1-5, and the joint score counts videos with both at least 4. At v1 the best of seven models, Wan2.1-14B, reaches 32.6 percent overall and 21.9 on the hard split; momentum and mass conservation are the most violated laws (about 40 percent). Its 7B auto-rater improves on Gemini-2.0-Flash-Exp but physical-commonsense correlation is still only 0.37. Curator qualifications: All figures verified. The flagged 22% vs 21.9 is rounding, not a conflict. The sample-size caveat for closed models (Sora 60, Ray2 394 vs 590) is confirmed from the setup text. Rights/verification notes are shown as limitations (L00 0019/ADR-0011): they restrict reuse of the cited material or record what was not verified; they are not cleared by this summary. Limitations: [component_reuse_restriction] arXiv non-exclusive licence on paper text; short excerpts only. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2503.06800v1",
      "note": "work/2503.06800v1.html VideoPhy-2 arXiv v1"
    }
  ]
}