Does the Deezer Research paper "Distinguishing Fictional Voices" include any human evaluation of LLM-generated dialogue, and what does it actually measure about character voice?
No. The paper (arXiv 2401.16968v1, 30 Jan 2024, Deezer Research and Loria) is fully automatic and uses only human-written novel text. It encodes each character's quotes with pretrained authorship-verification models and tests whether characters can be told apart (character vs character) and whether unseen quotes can be attributed to characters. Chapterwise: LUAR reached AUC 81.6 for character-vs-character but 53.6 for quote attribution, only slightly above chance. The authors guess the attribution task is hard "even for humans" but ran no human test. S52's negative is confirmed for this paper.
No. The paper (arXiv 2401.16968v1, 30 Jan 2024, Deezer Research and Loria) is fully automatic and uses only human-written novel text. It encodes each character's quotes with pretrained authorship-verification models and tests whether characters can be told apart (character vs character) and whether unseen quotes can be attributed to characters. Chapterwise: LUAR reached AUC 81.6 for character-vs-character but 53.6 for quote attribution, only slightly above chance. The authors guess the attribution task is hard "even for humans" but ran no human test. S52's negative is confirmed for this paper.
Curator qualifications: Deezer/Loria 2401.16968: fully automatic authorship-verification of human-written novel character quotes (no humans, no generated text).
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/abs/2401.16968",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/html/2401.16968v1",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [sourced_fact] No. The paper (arXiv 2401.16968v1, 30 Jan 2024, Deezer Research and Loria) is fully automatic and uses only human-written novel text. It encodes each character's quotes with pretrained authorship-verification models and tests whether characters can be told apart (character vs character) and whether unseen quotes can be attributed to characters. Chapterwise: LUAR reached AUC 81.6 for character-vs-character but 53.6 for quote attribution, only slightly above chance. The authors guess the attribution task is hard "even for humans" but ran no human test. S52's negative is confirmed for this paper. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |