How well have these benchmarks validated their automatic physics judges against humans, and can their validation numbers be compared?

Partially verified: the curator re-read none of the cited sources; S1, S2, S4, S9, S12 rest on the producer's pinned reading. Five papers validate a judge or automatic scorer, each with a different statistic: VideoPhy-2-Autoeval Pearson 0.37 on physical commonsense (Gemini-2.0-Flash-Exp 0.11); PhyWorldBench's CAP ROC-AUC 75.1 for physical commonsense (plain GPT-o1 61.6); PhyGround's PhyJudge-9B aggregate relative bias 3.3 percent (Gemini-3.1-Pro 16.6) on held-out Veo-3.1 videos only; Last Exam's measurement evaluator 51.6 percent within-task pair agreement against 43.4 for a direct Qwen3.6-27B scorer and 56.6 between humans; Principia's VLMs 0.50-0.66 agreement with binary truth. The statistics are not comparable, none reports the same data for two judges, and all are the benchmark authors' own checks; the consistent message is that prompted general VLMs are weak physics judges and that fine-tuning or measurement helps modestly. Curator qualifications: Partially verified: the curator re-read none of the cited sources; S1, S2, S4, S9, S12 rest on the producer's pinned reading. Curator GET 2026-10-23 of the sources not re-read (mechanical source evidence, not semantic verification): S1: HTTP 200, bytes differ from the producer pin; S2: HTTP 200, bytes differ from the producer pin; S4: HTTP 200, bytes differ from the producer pin; S9: HTTP 200, bytes differ from the producer pin; S12: HTTP 200, bytes differ from the producer pin. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2503.06800v1",
      "note": "CAP"
    },
    {
      "url": "https://arxiv.org/abs/2507.13428v3",
      "note": "CAP"
    },
    {
      "url": "https://arxiv.org/abs/2605.10806v1",
      "note": "CAP"
    },
    {
      "url": "https://arxiv.org/abs/2610.08791v1",
      "note": "CAP"
    },
    {
      "url": "https://arxiv.org/abs/2609.04200v1",
      "note": "CAP"
    }
  ]
}