Which English-language role-play or persona benchmarks exist, and what does each one actually score (is character-to-character voice distinctiveness among the scored things)?
Four benchmark-style resources were read. RoleBench scores reference overlap (ROUGE-L) and judge/human win rates for single-turn answers in a role; PersonaGym scores five rubric tasks per demographic persona with an LLM-judge ensemble; PersonaEval scores whether a judge can name the speaker in human-written dialogue; the Speaker Verification work is the only one read that defines a cross-character measure (Distinction Score) on generated lines. CharacterEval is Chinese-only on its abstract page. None of the four reports a human-rated test of whether generated characters sound different from each other.
Four benchmark-style resources were read. RoleBench scores reference overlap (ROUGE-L) and judge/human win rates for single-turn answers in a role; PersonaGym scores five rubric tasks per demographic persona with an LLM-judge ensemble; PersonaEval scores whether a judge can name the speaker in human-written dialogue; the Speaker Verification work is the only one read that defines a cross-character measure (Distinction Score) on generated lines. CharacterEval is Chinese-only on its abstract page. None of the four reports a human-rated test of whether generated characters sound different from each other.
Curator qualifications: Four resources measure different things: RoleBench (ROUGE-L and win rates, single turn), PersonaGym (rubric tasks with LLM-judge ensemble), PersonaEval (judge names the speaker in human dialogue), Speaker Verification (only cross-character Distinction Score).
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2310.00746v3",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/html/2407.18416v5",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/html/2508.10014v1",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/abs/2401.01275",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/abs/2606.22454",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/abs/2405.10150",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/html/2405.10150v2",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [sourced_fact] Four benchmark-style resources were read. RoleBench scores reference overlap (ROUGE-L) and judge/human win rates for single-turn answers in a role; PersonaGym scores five rubric tasks per demographic persona with an LLM-judge ensemble; PersonaEval scores whether a judge can name the speaker in human-written dialogue; the Speaker Verification work is the only one read that defines a cross-character measure (Distinction Score) on generated lines. CharacterEval is Chinese-only on its abstract page. None of the four reports a human-rated test of whether generated characters sound different from each other. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |