What does GenEval actually check in a text-to-image result, and how does it decide an image is right or wrong?
GenEval (arXiv v1) turns a template prompt into object-level checks: it counts and locates objects with a COCO-trained instance-segmentation detector and checks colour of each cropped, background-masked object with CLIP zero-shot classification. Every image gets a binary pass/fail (all prompt elements correct), averaged per task and then across six tasks (single object, two object, counting, colours, position, attribute binding). It can only verify what the detector's 80 COCO classes and the chosen thresholds can see.
GenEval (arXiv v1) turns a template prompt into object-level checks: it counts and locates objects with a COCO-trained instance-segmentation detector and checks colour of each cropped, background-masked object with CLIP zero-shot classification. Every image gets a binary pass/fail (all prompt elements correct), averaged per task and then across six tasks (single object, two object, counting, colours, position, attribute binding). It can only verify what the detector's 80 COCO classes and the chosen thresholds can see.
Curator qualifications: Verified the released code's 10-colour list vs the paper's 11 Berlin-Kay terms (gray absent in code) - paper/code inconsistency confirmed. Several excerpt-less rows not re-read.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/abs/2310.11513v1",
"note": "COLORS list"
},
{
"url": "https://github.com/djghosh13/geneval/tree/af4902f24d3ca90ebbb446dd9891a59e0f82725f",
"note": "COLORS list"
}
]
}Evidence claims
- [sourced_fact] GenEval (arXiv v1) turns a template prompt into object-level checks: it counts and locates objects with a COCO-trained instance-segmentation detector and checks colour of each cropped, background-masked object with CLIP zero-shot classification. Every image gets a binary pass/fail (all prompt elements correct), averaged per task and then across six tasks (single object, two object, counting, colours, position, attribute binding). It can only verify what the detector's 80 COCO classes and the chosen thresholds can see. · locator: COLORS list source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |