When one model is scored under several action-following protocols (Oasis and Matrix-Game across MineWorld, Matrix-Game, GameCraft and Yume papers), do the scores agree, and are the protocols comparable?

They do not agree. Oasis keyboard accuracy is 0.77 in Matrix-Game 1.0, 0.73 in Matrix-Game 2.0, and F1 0.41 (precision 0.49, recall 0.44) in MineWorld's own table. Matrix-Game 1.0 scores 0.95 keyboard/0.95 mouse in its own paper, yet in GameCraft's tests (150 mixed images, 12 actions) it has the highest translation error and the second-highest rotation error of five methods (RPE trans 0.18, rot 0.35 versus 0.08 and 0.20 for GameCraft) and the lowest user-study action-accuracy rank (1.63 of 5), and in Yume-Bench its human-rated instruction following is 0.271 against Yume's 0.657. The sources differ in test set, domain, estimator and rater, so these are four separate protocols, not four measurements of one quantity. The differences are consistent with domain shift (Yume says Matrix-Game struggles to generalise to real-world scenes) but no source isolates the cause. Curator qualifications: With the lead's locators (FB45) the curator verified every cell. Oasis keyboard is 0.77 (Matrix-Game 1.0 Table 2) vs 0.73 (Matrix-Game 2.0 Table 1) under the same benchmark name, and Matrix-Game's own 0.95/0.95 contrasts with the highest RPE translation error in GameCraft's tests (0.18 vs 0.08). The protocols disagree. Also GameCraft: 0.25 to 6.6 FPS is a 26.4x table ratio vs the text's '10-20x' / 'up to 20x'. Resolves the C3R-0294 partial hold. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2506.18701v1",
      "note": "Table 2 rows 860-862; table S5.T1 (tags stripped); table S5.T2 (tags stripped)"
    },
    {
      "url": "https://arxiv.org/abs/2508.13009v2",
      "note": "Table 2 rows 860-862; table S5.T1 (tags stripped); table S5.T2 (tags stripped)"
    },
    {
      "url": "https://arxiv.org/abs/2504.08388v1",
      "note": "Table 2 rows 860-862; table S5.T1 (tags stripped); table S5.T2 (tags stripped)"
    },
    {
      "url": "https://arxiv.org/abs/2506.17201v1",
      "note": "Table 2 rows 860-862; table S5.T1 (tags stripped); table S5.T2 (tags stripped)"
    },
    {
      "url": "https://arxiv.org/abs/2507.17744v1",
      "note": "Table 2 rows 860-862; table S5.T1 (tags stripped); table S5.T2 (tags stripped)"
    }
  ]
}