Can an LLM judge reliably say which character is speaking, and what does that imply for using an LLM to check character voice?
PersonaEval (arXiv 2508.10014v1, 6 Aug 2025) gives LLM evaluators dialogue context plus four or five described candidate roles and asks who the second speaker is. The paper reports the best LLM at around 69% top-1 accuracy against 90.8% for 20 human volunteers on a hard sample, and says models lean on speaking style while humans use addressing and intent. Items are human-authored (English novels, Chinese scripts, English explainer videos), so the result says how judges treat existing text, not how they treat generated dialogue.
PersonaEval (arXiv 2508.10014v1, 6 Aug 2025) gives LLM evaluators dialogue context plus four or five described candidate roles and asks who the second speaker is. The paper reports the best LLM at around 69% top-1 accuracy against 90.8% for 20 human volunteers on a hard sample, and says models lean on speaking style while humans use addressing and intent. Items are human-authored (English novels, Chinese scripts, English explainer videos), so the result says how judges treat existing text, not how they treat generated dialogue.
Curator qualifications: PersonaEval: best LLM ~69% top-1 vs 90.8% humans on a hard sample; models lean on style, humans on addressing and intent.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2508.10014v1",
"note": "candidate evidence excerpts"
},
{
"url": "https://raw.githubusercontent.com/maple-zhou/PersonaEval/19f7568a84f52e206dccdf67146235c8376cec9c/README.md",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [sourced_fact] PersonaEval (arXiv 2508.10014v1, 6 Aug 2025) gives LLM evaluators dialogue context plus four or five described candidate roles and asks who the second speaker is. The paper reports the best LLM at around 69% top-1 accuracy against 90.8% for 20 human volunteers on a hard sample, and says models lean on speaking style while humans use addressing and intent. Items are human-authored (English novels, Chinese scripts, English explainer videos), so the result says how judges treat existing text, not how they treat generated dialogue. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |