sourceactiveImage generation Local models and model tooling Evaluation and media provenance

djghosh13/geneval (GenEval)

by @veyrul-import · operator Veyrul founder operations (declared) · founder · revision 1 · updated 2026-10-18 · src_01M569CC5G3YAB8GRJPBT3JZXA

Evidence: review revision unknown · inspection metadata_checked · execution not_run · review unreviewed · origin research_agent

Source: https://github.com/djghosh13/geneval

THUDM/ImageReward (https://github.com/THUDM/ImageReward; human-preference reward model, Apache-2.0, NeurIPS 2023, VisionReward successor noted in Dec 2024 news) was fetched and is a recommended complementary evaluation record for L07. L07 2026-10-17: merged duplicate records from lane-d:djghosh13--geneval (same canonical URL).

Structured fields

{
  "archived": false,
  "checked_at": "2026-10-17T00:00:00.000Z",
  "usefulness": "Object-focused automatic evaluation of text-to-image alignment: uses an object detector to score single object, two objects, counting, colors, position and color attribution (paper arXiv 2310.11513). Widely referenced for prompt-following comparisons, but it does not measure identity consistency, camera coverage or aesthetics.",
  "source_type": "repository",
  "availability": "available",
  "dependencies": [
    "mmdetection 2.x",
    "Mask2Former detector weights",
    "conda"
  ],
  "source_owner": "Dhruba Ghosh, Hanna Hajishirzi, Ludwig Schmidt",
  "next_check_at": "2026-10-31T00:00:00.000Z",
  "known_limitations": [
    "Depends on legacy mmdetection 2.x; reproducibility on modern stacks unverified.",
    "Measures prompt-object alignment only.",
    "Limited to countable/detectable objects; says nothing about aesthetics or identity consistency."
  ],
  "upstream_updated_at": "2025-03-03T21:01:42.000Z"
}

Evidence claims

[sourced_fact] README: object-detection-based framework scoring single-object and two-object generation, counting, colors, position and color attribution; MIT license. source
[sourced_fact] Setup requires a conda env, the mmdetection 2.x branch and a downloaded Mask2Former detector; page shows 14 commits and no update date. source
[inference] Scope limits are an inference from the listed task types: no identity, multi-shot continuity or aesthetic measurement. source
[upstream_claim] README: object-focused framework evaluating compositional image properties (object co-occurrence, position, count, color) via existing detection models, claiming strong human agreement. · locator: README abstract source
[sourced_fact] Repo metadata: MIT (Copyright 2023 Dhruba Ghosh), not archived, last push 2025-03-03. · locator: GitHub API fields; LICENSE header source
[upstream_claim] Research review verdict at export: reviewed (as of 2026-10-17). Recorded by the research pipeline, not a hub review outcome; evidence.review stays unreviewed until an independent reviewer records one.

Rights

component	license	hosting	redistribution	modification	training	Evidence and notes
code	MIT	permitted	permitted	permitted	unknown	https://github.com/djghosh13/geneval Repo page states MIT; LICENSE file not separately fetched. Detector weights (Mask2Former) have separate terms not checked.

Linked records

Machine-readable: JSON · Markdown · revisions · changes