How are WorldSimBench scores produced and validated, which systems lead and by how much, and which of the paper's own claims do its tables fail to support?

WorldSimBench validates its learned evaluator against GPT-4o with PLCC (0.60 AD, 0.43 RM in-distribution; zero-shot 0.34-0.49 AD, 0.44-0.47 RM) and accuracy for OE, but gives no evaluation-set sizes, annotator counts or agreement. The explicit and implicit modes disagree: by my computation OE rank agreement across seven models is Spearman 0.04 (0.21 on the five text-conditioned models). The authors admit inflated scores in robot manipulation and Velocity, and several statements in the text do not match the paper's own tables (GPT-4o negative-correlation attribution, 'AD variation is minimal', Open-Sora-Plan's trajectory lead). All results are for eight older open models fine-tuned on scenario data, so 'who leads' says nothing about current commercial systems. Curator qualifications: Table 3 in the work copy shows AD PLCC GPT-4o@OpenSora 0.03 and GPT-4o@Lavie -0.04, RM GPT-4o@OpenSora -0.06 and GPT-4o@Lavie 0.17, while Section 5.2 says GPT-4o is negative for OpenSora in AD and Lavie in RM: names appear swapped, as the producer says. PLCC values 0.60/0.43 and zero-shot ranges located. No evaluation-set sizes found. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/html/2410.18072v1",
      "note": "Table 3 and Section 5.2"
    }
  ]
}