What did each of these open interactive world models actually evaluate (metrics, baselines, raters, test sets), and what do the evaluations not support?

The evaluations are author-run and use different protocols. Matrix-Game 1.0 uses its own GameWorld Score on Minecraft plus a double-blind human comparison against Oasis and MineWorld. Matrix-Game 2.0 reports the same score against YUME on wild scenes and is lower on motion smoothness, object consistency and scenario consistency. Matrix-Game 4.0's report and card give latency and throughput but no image-quality metric, and the card says it claims no benchmark. Hunyuan-GameCraft reports FVD, relative pose error, image quality and a 30-evaluator ranking on 150 images and 12 action signals, compared with camera-control baselines and Matrix-Game 1.0. LingBot-World 1 uses VBench on 100 generated videos over 30 s; LingBot 2.0 relies on comparison figures and an hour-long session. None measures persistence or revisit consistency quantitatively. Curator qualifications: The Matrix-Game 2.0 claim-vs-table tension is confirmed: on wild scenes it is below YUME on 3 of 6 metrics while the abstract says it 'sets new benchmarks'. That claim may rest on Minecraft or other comparisons, so record it as overclaiming relative to the wild-scene table, not a fabricated number. The MG4 card's explicit no-benchmark statement verifies. Scope: the abstract claim is general; the counter-evidence is the wild-scene Table 2 only. Minecraft-domain comparisons were not re-read. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2506.18701v1",
      "note": "work/mg2-arxiv-v2.html Table 2 (wild scenes) and abstract; Matrix-Game 4.0 card Limitations; GameCraft Table 2"
    },
    {
      "url": "https://huggingface.co/Skywork/Matrix-Game",
      "note": "work/mg2-arxiv-v2.html Table 2 (wild scenes) and abstract; Matrix-Game 4.0 card Limitations; GameCraft Table 2"
    },
    {
      "url": "https://arxiv.org/abs/2508.13009v2",
      "note": "work/mg2-arxiv-v2.html Table 2 (wild scenes) and abstract; Matrix-Game 4.0 card Limitations; GameCraft Table 2"
    },
    {
      "url": "https://huggingface.co/Skywork/Matrix-Game-4.0",
      "note": "work/mg2-arxiv-v2.html Table 2 (wild scenes) and abstract; Matrix-Game 4.0 card Limitations; GameCraft Table 2"
    },
    {
      "url": "https://arxiv.org/abs/2506.17201v1",
      "note": "work/mg2-arxiv-v2.html Table 2 (wild scenes) and abstract; Matrix-Game 4.0 card Limitations; GameCraft Table 2"
    },
    {
      "url": "https://arxiv.org/abs/2601.20540v1",
      "note": "work/mg2-arxiv-v2.html Table 2 (wild scenes) and abstract; Matrix-Game 4.0 card Limitations; GameCraft Table 2"
    },
    {
      "url": "https://arxiv.org/abs/2607.07534v1",
      "note": "work/mg2-arxiv-v2.html Table 2 (wild scenes) and abstract; Matrix-Game 4.0 card Limitations; GameCraft Table 2"
    }
  ]
}