How well does GenEval agree with human raters, which settings were tuned to get that agreement, and where does it fail or stop discriminating?
Partially verified: the curator re-read none of the cited sources; S1 rests on the producer's pinned reading. In the authors' MTurk study (6,000 annotations over 1,200 images) GenEval agreed with humans on 83% of images versus 88% pairwise annotator agreement and 80% for a per-task thresholded CLIPScore; on 860 unanimous images agreement was 91% vs 87%. The counting confidence threshold (0.9) and the colour crop+mask step were chosen against human labels, and the paper lists holes in objects, overlapping same-type objects and non-photographic art as failures. Single-object prompts are near ceiling for all tested Stable Diffusion/IF models, and the paper notes kappa 0 there because almost all images were correct.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/abs/2310.11513v1",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [sourced_fact] In the authors' MTurk study (6,000 annotations over 1,200 images) GenEval agreed with humans on 83% of images versus 88% pairwise annotator agreement and 80% for a per-task thresholded CLIPScore; on 860 unanimous images agreement was 91% vs 87%. The counting confidence threshold (0.9) and the colour crop+mask step were chosen against human labels, and the paper lists holes in objects, overlapping same-type objects and non-photographic art as failures. Single-object prompts are near ceiling for all tested Stable Diffusion/IF models, and the paper notes kappa 0 there because almost all images were correct. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |