How does DPG-Bench score dense, long prompts, which judge does it use, and how was that judge validated against humans?

DPG-Bench (introduced in the ELLA paper, arXiv v1) has 1,065 long prompts; GPT-4 wrote the dense prompts and, following the Davidsonian Scene Graph (DSG) recipe, per-prompt yes/no questions with dependencies. The released scorer asks an mPLUG VQA model each question on each of the four images in a 2x2 grid, counts an exact 'yes' as 1, zeroes questions whose parent question was answered 'no', and averages. Human validation described in the passages read is a ranking study of three models by 20 users per prompt, not a per-question agreement measurement of the VQA judge. Curator qualifications: All statements verified, including the dependency zeroing in the released scorer. The producer's key point holds: the 'highly correlated with human perception' claim rests on a three-model ranking user study (20 users per prompt), not a per-question judge-human agreement statistic. The 0.06B/0.07B parameter inconsistency is confirmed but trivial and does not affect DPG-Bench scores. Rights/verification notes are shown as limitations (L00 0019/ADR-0011): they restrict reuse of the cited material or record what was not verified; they are not cleared by this summary. Limitations: [component_reuse_restriction] Code read, not run; mPLUG judge terms and the dataset licence not re-read. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2403.05135v1",
      "note": "work/html_2403.05135v1.html ELLA arXiv v1 sec 4-5; released scorer compute_dpg_bench.py (code read, not run)"
    },
    {
      "url": "https://github.com/TencentQQGYLab/ELLA/tree/3c228f1dc6c4d3cad0a47493816151a419f14db3",
      "note": "work/html_2403.05135v1.html ELLA arXiv v1 sec 4-5; released scorer compute_dpg_bench.py (code read, not run)"
    }
  ]
}