How does EWMBench score models, which systems lead and by how much, and what bias, scale or protocol risks do the paper and repo leave open?

EWMBench reports a domain-adapted lead (EnerVerse_FT 4.70, LTX_FT 4.55) over Kling (3.87), Hailuo (3.41) and open models (2.97-3.29) on seven models. The 'Avg.' and 'Overall' columns are plain sums (I verified to 4 decimals), so the 'overall' mixes components of very different range: the semantic Diversity term spans 0.06 while Dyn spans 0.57. Semantic scores barely separate EnerVerse_FT (2.09), Kling (2.04) and Hailuo (2.02). The human study ranks four models with no annotator count or correlation statistic. Motion best-of-three is selected using the ground-truth trajectory, and the judge prompt is robot-specific (declares human hands abnormal). Weights and data are CC BY-NC-SA 4.0 per cards and README, but the repo has no LICENSE file. Curator qualifications: Curator re-summed Table 2: motion Avg = HSD+Dyn+nDTW, semantics Avg = Diversity+BLEU+CLIP+Logics, Overall = SceneC+both Avgs, equal to 4 decimals for all seven models (OpenSora 0.0001 rounding). The 'normalised 0-1' wording versus Overall up to 4.70 is kept as a conflict. Repository licence text is CC BY-NC-SA 4.0 per README with no LICENSE file (producer E11). Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/html/2505.09694v2",
      "note": "Table 2"
    },
    {
      "url": "https://github.com/AgibotTech/EWMBench/tree/3a5531c0e6b6be51431251ed5617b9e48c852098",
      "note": "Table 2"
    },
    {
      "url": "https://huggingface.co/agibot-world/EWMBench-model",
      "note": "Table 2"
    },
    {
      "url": "https://arxiv.org/html/2505.09694v1",
      "note": "Table 2"
    }
  ]
}