Is there an automatic measure of whether a model makes different characters sound different from each other, and what did it show?
Speaker Verification in Agent-Generated Conversations (arXiv 2405.10150v2) defines a Distinction Score: embed lines the same model generated for two different roles and take one minus their cosine similarity. In self-play with RoleBench roles, every role-play system scored below the real-utterance value (72.91): from 63.57 (LLaMA2-Chat-7B) to 38.98 (CharacterLLM); the authors conclude that generated lines are similar across role settings and suggest pretraining style persists. The measure rests on verifiers validated on human-written text; the generated-line scores were not compared with human ratings in the passages read.
Speaker Verification in Agent-Generated Conversations (arXiv 2405.10150v2) defines a Distinction Score: embed lines the same model generated for two different roles and take one minus their cosine similarity. In self-play with RoleBench roles, every role-play system scored below the real-utterance value (72.91): from 63.57 (LLaMA2-Chat-7B) to 38.98 (CharacterLLM); the authors conclude that generated lines are similar across role settings and suggest pretraining style persists. The measure rests on verifiers validated on human-written text; the generated-line scores were not compared with human ratings in the passages read.
Curator qualifications: Every tested role-play system scored below real utterances on Distinction (72.91), from 63.57 down to 38.98: generated lines for different roles are more alike than real ones. Verifier validated on human text only.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2405.10150v2",
"note": "arXiv 2405.10150v2 HTML Table 3 and speaker-verification table"
},
{
"url": "https://raw.githubusercontent.com/IzzetYoung/AGCSpeakerVerification/9215f365fffd6697244e7e15d4548d60dc6cde51/README.md",
"note": "arXiv 2405.10150v2 HTML Table 3 and speaker-verification table"
}
]
}Evidence claims
- [sourced_fact] Speaker Verification in Agent-Generated Conversations (arXiv 2405.10150v2) defines a Distinction Score: embed lines the same model generated for two different roles and take one minus their cosine similarity. In self-play with RoleBench roles, every role-play system scored below the real-utterance value (72.91): from 63.57 (LLaMA2-Chat-7B) to 38.98 (CharacterLLM); the authors conclude that generated lines are similar across role settings and suggest pretraining style persists. The measure rests on verifiers validated on human-written text; the generated-line scores were not compared with human ratings in the passages read. · locator: arXiv 2405.10150v2 HTML Table 3 and speaker-verification table source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |