How did GenAI-Arena adapt arena voting to text-to-image, image editing and text-to-video, and what pitfalls and vote-quality numbers did it report?

Problem. Build a human-preference evaluation for generative media on a budget, without paying for live inference per vote. Conditions. GenAI-Arena (arXiv 2406.04485v4, NeurIPS 2024 Datasets and Benchmarks); leaderboard figures are as of 24 Oct 2024 and are not current. Method. (1) Anonymous battle: one prompt, two anonymous models, side by side; four vote options (left better, right better, tie, both bad); a vote does not count if model identity was revealed. (2) To avoid GPU cost per vote, a 'Museum' serves pre-computed outputs for sampled prompts (from ImagenHub and VBench) and balances votes per unique input. (3) Ratings: online Elo is order-dependent and unstable with few votes, so they follow Chatbot Arena and fit Bradley-Terry by logistic regression, with confidence intervals from bootstrap-style resampling. Reported pitfalls. Random pair selection gave some models far more matches; the authors note Elo can be biased by an imbalance between 'easy games' (against weak models) and 'hard games', which they used to explain an anomalously high rank for one video model; reliable ratings need a lot of votes. Vote quality (measured). 6,300 image-generation, 1,154 image-editing and 2,024 text-to-video votes; an author review of 350 sampled votes found 303 valid, of which 76.24% were 'clearly reasonable' and 93.07% clearly or vaguely reasonable. Failure modes. Votes come from a self-selected audience; author review is not independent. Limitations. Small vote counts, 2024-era models, rankings must not be quoted as current. Version and date. v4, 11 Nov 2024. Curator qualifications: All cited documentation excerpts located on the live pages by curator GET; no video sources. Limitations: Review was done by the authors on a small sample; leaderboards are dated Oct 2024. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/pdf/2406.04485",
      "note": "3.1 Voting rules; 3.3 Elo; 3.3 Elo; 4.2 Discussion; Table 6"
    }
  ]
}