What did Park et al. measure about LLM characters (believability ablation, rater counts, failure modes), what changed between arXiv v1 and v2, and what is only proposed?
The controlled evaluation had 100 raters rank interview responses from the full architecture, three ablations (no reflection; no reflection or planning; no memory, planning or reflection) and a crowdworker-authored baseline; the full architecture ranked highest (TrueSkill mu 29.89) and the memoryless ablation lowest. An end-to-end two-day run measured information diffusion and hallucination. Reported failure modes are embellishment of memories, retrieval failures, norm misclassification and overly formal or cooperative speech. A number differs between v1 (12 agents, 48%) and v2 (13 agents, 52%). The evaluation is of a simulation sandbox: production-game use (consistency over long play, lore constraints, cost) is proposed, not measured.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/abs/2304.03442v2",
"note": "cited at https://arxiv.org/pdf/2304.03442v2; work/ga_v2.pdf arXiv 2304.03442v2 (local pdftotext); work/ga_v1.pdf arXiv 2304.03442v1"
},
{
"url": "https://arxiv.org/pdf/2304.03442v1",
"note": "work/ga_v2.pdf arXiv 2304.03442v2 (local pdftotext); work/ga_v1.pdf arXiv 2304.03442v1"
},
{
"url": "https://arxiv.org/pdf/2304.03442v2",
"note": "observed URL (dated/alias form of the cited catalogue source; canonical identity retained)"
}
]
}Evidence claims
- [sourced_fact] The controlled evaluation had 100 raters rank interview responses from the full architecture, three ablations (no reflection; no reflection or planning; no memory, planning or reflection) and a crowdworker-authored baseline; the full architecture ranked highest (TrueSkill mu 29.89) and the memoryless ablation lowest. An end-to-end two-day run measured information diffusion and hallucination. Reported failure modes are embellishment of memories, retrieval failures, norm misclassification and overly formal or cooperative speech. A number differs between v1 (12 agents, 48%) and v2 (13 agents, 52%). The evaluation is of a simulation sandbox: production-game use (consistency over long play, lore constraints, cost) is proposed, not measured. · locator: cited at https://arxiv.org/pdf/2304.03442v2; work/ga_v2.pdf arXiv 2304.03442v2 (local pdftotext); work/ga_v1.pdf arXiv 2304.03442v1 source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |