Qwen-Image-Bench scoring: match prompt IDs, map 0/1/2 to 0/60/100 and average active pillars

Qwen-Image-Bench evaluates text-to-image outputs with a diagnostic judge. This guide follows dataset README revision d2493deb153b020cf169c7e3f57d15e4dd697038, captured on 2026-10-28. The full text was inspected; the paper, scripts, dataset rows, generated images and judge weights were not. The README retains image-to-text in YAML while describing text-to-image evaluation in its body. It does not explain that metadata choice; it does not establish a broken generation interface. The documented benchmark has 1,000 bilingual prompt records, balanced between 500 long and 500 short prompts. Those counts describe prompt length, not a split of 500 Chinese versus 500 English records. It organizes evaluation into five pillars, 23 intermediate capabilities and 56 facets. Quality, aesthetics and alignment are joined by real-world fidelity and creative generation. Text rendering and visual storytelling are named parts of the taxonomy; this does not establish a tested metric for subject preservation across edits or shots. The dataset configuration names a test split in qwen_image_bench_hf_v0518.jsonl. Its described rows contain bilingual prompt and dimension fields, a relative generated-image path for each model, and per-model raw judge responses. Those fields were read in the README, not checked against actual rows. A separate ground-truth reference-image schema and the generators' exact call settings are not established by this document. For your own images, the documented input is CSV, JSON or JSONL with ID, prompt and image_path. ID must match the benchmark's 1–1000 metadata IDs, which select the dimensions applicable to that prompt. The default metadata file is metadata/bench_metadata.json; --local-metadata can override it. The README's example calls python judge.py --input your_data.jsonl --model Qwen/Qwen-Image-Bench. --model is required; the example model ID is not a default or a pinned weight revision. The model namespace Qwen/Qwen-Image-Bench and dataset namespace datasets/Qwen/Qwen-Image-Bench are separate resources. This contract does not justify attaching an existing ID to an unrelated production prompt; inspect the metadata and checklists before extending the task. Scoring maps raw 0 to 0, raw 1 to 60 and raw 2 to 100. N/A is excluded. It is not a linear rescaling in which raw 1 becomes 50, and an inapplicable facet is not a failed facet. The README describes bottom-up aggregation from facets through capabilities to pillars. Each sample's overall score is the unweighted mean of its active pillars, three to five per prompt; the model score averages the 1,000 prompt-level scores. A flat average over every facet, a zero-filled inactive pillar or an average over a different prompt subset is therefore not the documented overall protocol. Exact lower-level weighting, parsing and correction behavior still require the implementation. There are two documented paths. To recompute scores from already stored responses, the README shows python compute_scores.py --input qwen_image_bench_hf_v0518.jsonl and reports scores_result.xlsx plus scores_detail.json. To judge new images, judge.py writes per-row judged CSV/JSONL with the original fields, combined judge_model_output and per-pillar raw outputs, plus aggregate bench_scores JSON and Excel files. Recomputing stored responses is distinct from rerunning the judge or regenerating images. These commands and outputs are documentation, not a successful run here. The documented judge parameter set is seed42, temperature0, top_k1, top_p1.0, repetition_penalty1.05, max_new_tokens4096, enable_thinking true and max_batch_size24. These are judge inference settings, not the source image generators' settings. The CLI exposes batch size and token controls, so the table does not prove a hard batch limit or runtime equivalence after overrides. The README describes a Qwen3.6-27B-based judge and reports expert annotation and Spearman rho0.92 agreement. That is an author-reported result; this text does not supply enough held-out validation detail, uncertainty or release pins to establish independent production reliability. The shown leaderboard evaluates images made from Chinese prompts. English leaderboard results are described as a future release in this snapshot. Bilingual prompt fields do not make those scores bilingual, and the named model labels do not identify immutable API/model versions or generation settings. Treat its rankings as source-reported snapshot results rather than current model recommendations. A rubric score also does not by itself certify client acceptance, legal safety, bias absence or temporal consistency. Before a reproducible evaluation, inspect and pin judge.py, score_utils.py, checklists.py, bench_metadata.json, requirements and the backend, then bind the judge weights, prompt language, exact prompt IDs, generated-image hashes and generation conditions. Preserve raw judge responses and parse failures; compare only a matched protocol. This is a proposed verification sequence, not executed work. The README names these files but their contents and enforcement of ID/rubric matching remain uninspected. The card declares Apache-2.0. Full license documents and per-component terms were not inspected; the declaration does not clear all generated images, source data, model weights or service outputs. This record is original method analysis and citations, with no source-media or dataset-row republication. Evidence is source_inspected / not_run / unreviewed. Local Qwen extraction and its support checks are provisional same-operator work; the frontier text review does not constitute an outside independent attestation.

{
  "citations": [
    {
      "url": "https://huggingface.co/datasets/Qwen/Qwen-Image-Bench/raw/d2493deb153b020cf169c7e3f57d15e4dd697038/README.md",
      "note": "Full pinned README text; exact line/byte/excerpt hashes in source pins. Linked assets and implementation not inspected."
    }
  ]
}