How reliable are Physics-IQ scores: what did the 2026 Physics-IQ Verified audit change, and what is the repository leaderboard evidence worth?

A June 2026 audit (Physics-IQ Verified, with two original authors advising) re-scored six image-to-video models and reports that ambiguous prompts affect 69 of 198 videos and spurious motion artefacts 59, that clean prompts raise scores and artefact removal lowers them, and that model ranks move (Kendall tau 0.46). It also notes that Sora 2 scored notably worse in April 2026 than in October 2025. The repository README now carries two self-submitted leaderboards with scores far above the 2025 paper's, many using best-of-N selection or prompt rewriters, so those rows measure pipelines rather than models. Curator qualifications: Verified v1 (17 Jun 2026) PDF: 'Of the 198 videos, 69 contain unclear prompts and 59 contain artifacts, with 20 videos belonging to both groups'; 'The Sora 2 performance is notably worse in April 2026 than in October 2025. We confirm this in App. Tables 5&6'; 'P.J. and R.G. contributed in an advisory capacity' (P. Jaini and R. Geirhos are Physics-IQ v3 authors). Abstract: refines 57.6% of samples, improves over 34.8% of prompts, Kendall tau 0.46 over six I2V models. README: 'self-reported' 1, 'pull request' 3, 'best-of' 8. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2606.18943v1",
      "note": "see curator disposition"
    },
    {
      "url": "https://github.com/google-deepmind/physics-IQ-benchmark/blob/d3f3de16a9c66b4d3bfd6bdb5518570e8b72ad48/README.md",
      "note": "see curator disposition"
    },
    {
      "url": "https://arxiv.org/abs/2501.09038v3",
      "note": "see curator disposition"
    }
  ]
}