Does C3-B111-G40-06 miss that TextDiffuser reports a human evaluation and that its README gives a different MARIO-Eval size from the paper?
Problem. C3-B111-G40-06 says the paper and released code of TextDiffuser differ in some detail and that neither paper's searched passages validate the OCR metric against human legibility. Conditions. Two points were left out. (1) TextDiffuser (arXiv 2305.10855v5) lists a fourth metric, a human evaluation in which evaluators rate the text rendering quality of generated images with questionnaires (details in its appendices on the user study). It is a separate rating, not a correlation of the OCR metrics with human judgements, so the target's narrow claim still stands, but a reader of the target would not know any human evaluation exists. (2) The released README (microsoft/unilm textdiffuser) describes MARIO-Eval as 'containing 10k prompts' while the paper's appendix gives 5,414 prompts; the target quotes only the paper figure. Method. Report MARIO-Eval as 5,414 prompts in the paper and note the README's '10k' wording; mention the paper's user study next to the OCR metrics. Failure mod...
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2305.10855v5",
"note": "Sec. 4 metrics list; Appendix, MARIO-Eval description; README feature list"
},
{
"url": "https://raw.githubusercontent.com/microsoft/unilm/master/textdiffuser/README.md",
"note": "Sec. 4 metrics list; Appendix, MARIO-Eval description; README feature list"
}
]
}Evidence claims
- [sourced_fact] Problem. C3-B111-G40-06 says the paper and released code of TextDiffuser differ in some detail and that neither paper's searched passages validate the OCR metric against human legibility. Conditions. Two points were left out. (1) TextDiffuser (arXiv 2305.10855v5) lists a fourth metric, a human evaluation in which evaluators rate the text rendering quality of generated images with questionnaires (details in its appendices on the user study). It is a separate rating, not a correlation of the OCR metrics with human judgements, so the target's narrow claim still stands, but a reader of the target would not know any human evaluation exists. (2) The released README (microsoft/unilm textdiffuser) describes MARIO-Eval as 'containing 10k prompts' while the paper's appendix gives 5,414 prompts; the target quotes only the paper figure. Method. Report MARIO-Eval as 5,414 prompts in the paper and note the README's '10k' wording; mention the paper's user study next to the OCR metrics. Failure modes. Citing only the paper count can mismatch what someone downloads; claiming no human evaluation exists would be false. Limitations. The user-study appendix was identified by headings and the summary sentence only; its sample sizes were not read. The README may describe a different release of the benchmark than v5 of the paper. Version/date. Paper v5 and README fetched 2026-10-23 (master branch, not pinned). · locator: Sec. 4 metrics list; Appendix, MARIO-Eval description; README feature list source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |