How does Physics-IQ test physical understanding in video models, which models and versions were scored, what were the results, and what do the authors say they do and do not show?

Physics-IQ (Motamed et al., Google DeepMind) gives a model the first 3 seconds (or the last frame) of a real, static-camera physics event and asks it to continue for 5 seconds, then compares to the real continuation with four motion-based metrics, combined and normalised so two real takes of the same scene score 100. At v3 the best of eight configurations (VideoPoet, multiframe) scores 29.5 and Sora (image-to-video) 10.0; visual realism judged by Gemini 1.5 Pro was not significantly correlated with physics score (r = -0.46, p = .249). The authors call the metrics proxies, say they are on the conservative side (they penalise camera moves and shot cuts), and say results do not show that future models cannot learn physics. Curator qualifications: Physics-IQ v3 PDF: best model 'VideoPoet (multiframe), only achieves a score of 29.5' (line 1219); realism 'uncorrelated with physical understanding (Pearson's r = - 0.46, p=.249 not significant)' (line 944); paint smearing on glass handled (line 1059). README: '3-second conditioning segment', 'exactly 5 seconds'. LICENSE: software Apache 2.0; 'All other materials' CC-BY 4.0. Rights/verification notes are shown as limitations (L00 0019/ADR-0011): they restrict reuse of the cited material or record what was not verified; they are not cleared by this summary. Limitations: [component_reuse_restriction] LICENSE covers the repository; video assets hosted elsewhere were not checked for separate terms. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2501.09038v3",
      "note": "see curator disposition"
    },
    {
      "url": "https://github.com/google-deepmind/physics-IQ-benchmark/blob/d3f3de16a9c66b4d3bfd6bdb5518570e8b72ad48/LICENSE",
      "note": "see curator disposition"
    },
    {
      "url": "https://github.com/google-deepmind/physics-IQ-benchmark/blob/d3f3de16a9c66b4d3bfd6bdb5518570e8b72ad48/README.md",
      "note": "see curator disposition"
    }
  ]
}