Which automatic voice/persona measures have been checked against human judgement, how well, and what protocol can an agent run to test whether generated characters stay distinguishable and consistent?
Across the sources read, validation of automatic persona/voice measures is thin: PersonaScore had five experts and 1,500 ratings (kappa 0.71; mean Spearman 75.1%); the consistency judge of Abdulhai et al. agreed with humans more (kappa 0.552) than humans agreed with each other (0.303); the persistent-persona judge was checked by one author (kappa 0.65, 250 ratings); RoleBench reports GPT-4 and human win rates without an agreement statistic; the Distinction Score has no human check on generated lines. An evaluation protocol that combines blind attribution, cast-level distance, drift probes and a validated judge is proposed; no source documents it as a whole.
Across the sources read, validation of automatic persona/voice measures is thin: PersonaScore had five experts and 1,500 ratings (kappa 0.71; mean Spearman 75.1%); the consistency judge of Abdulhai et al. agreed with humans more (kappa 0.552) than humans agreed with each other (0.303); the persistent-persona judge was checked by one author (kappa 0.65, 250 ratings); RoleBench reports GPT-4 and human win rates without an agreement statistic; the Distinction Score has no human check on generated lines. An evaluation protocol that combines blind attribution, cast-level distance, drift probes and a validated judge is proposed; no source documents it as a whole.
Curator qualifications: Validation of automatic persona measures is thin (small expert panels; judge-human agreement exceeding human-human in one study); the proposed evaluation protocol has not been run.
Evidence status: labelled proposal derived from the cited sources; untested; correlated desk review (Claude); not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2407.18416v5",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/html/2511.00222v1",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/html/2512.12775v2",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/html/2508.10014v1",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/html/2310.00746v3",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/html/2405.10150v2",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [proposal] Across the sources read, validation of automatic persona/voice measures is thin: PersonaScore had five experts and 1,500 ratings (kappa 0.71; mean Spearman 75.1%); the consistency judge of Abdulhai et al. agreed with humans more (kappa 0.552) than humans agreed with each other (0.303); the persistent-persona judge was checked by one author (kappa 0.65, 250 ratings); RoleBench reports GPT-4 and human win rates without an agreement statistic; the Distinction Score has no human check on generated lines. An evaluation protocol that combines blind attribution, cast-level distance, drift probes and a validated judge is proposed; no source documents it as a whole. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |