When an LLM reads a conversation where a character was absent for part of it, how often does it credit that character with the missed information, and how consistent are its answers across ways of asking?
FANToM (Kim et al., arXiv 2310.15421v3) builds 256 multi-party conversations in which a character leaves and rejoins, then asks the same underlying question six ways (free-form belief, multiple-choice belief, answerability list and yes/no, information-access list and yes/no). The model reads everything, so an answer containing what the absent character missed is the failure. In the 2023 models tested, success on one format did not carry to the others: the best chain-of-thought GPT-4 run scored 26.6 on 'all question types' against 87.5 for humans, and the dominant list error was including characters who were unaware. Results are for 2023 models.
FANToM (Kim et al., arXiv 2310.15421v3) builds 256 multi-party conversations in which a character leaves and rejoins, then asks the same underlying question six ways (free-form belief, multiple-choice belief, answerability list and yes/no, information-access list and yes/no). The model reads everything, so an answer containing what the absent character missed is the failure. In the 2023 models tested, success on one format did not carry to the others: the best chain-of-thought GPT-4 run scored 26.6 on 'all question types' against 87.5 for humans, and the dominant list error was including characters who were unaware. Results are for 2023 models.
Curator qualifications: FANToM: 256 multi-party conversations where a character leaves and rejoins; same question asked six ways; models leak what the absent character missed; consistency across formats low (2023 models).
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2310.15421v3",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [sourced_fact] FANToM (Kim et al., arXiv 2310.15421v3) builds 256 multi-party conversations in which a character leaves and rejoins, then asks the same underlying question six ways (free-form belief, multiple-choice belief, answerability list and yes/no, information-access list and yes/no). The model reads everything, so an answer containing what the absent character missed is the failure. In the 2023 models tested, success on one format did not carry to the others: the best chain-of-thought GPT-4 run scored 26.6 on 'all question types' against 87.5 for humans, and the dominant list error was including characters who were unaware. Results are for 2023 models. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |