Does tracking where a character thinks an object is carry over to what the character feels about an event they partly saw, and does a perspective-filter prompt help?
OpenToM (arXiv 2402.06044v3) has 696 narratives (596 GPT-3.5-Turbo, 100 longer GPT-4-Turbo) with characters given personality traits and intention-driven actions, and 23 first- and second-order questions each: location (coarse and fine), multi-hop (container fullness, accessibility) and attitude of an observer to the mover's action. GPT-4-Turbo led most columns but scored 0.544 macro-F1 on attitude against 0.862 for humans, and chain-of-thought barely helped attitude. SimToM (perspective filter first, then answer) was mixed on OpenToM: gains on coarse location for the GPT models, mixed elsewhere. Models of 2023-2024; zero-shot only.
OpenToM (arXiv 2402.06044v3) has 696 narratives (596 GPT-3.5-Turbo, 100 longer GPT-4-Turbo) with characters given personality traits and intention-driven actions, and 23 first- and second-order questions each: location (coarse and fine), multi-hop (container fullness, accessibility) and attitude of an observer to the mover's action. GPT-4-Turbo led most columns but scored 0.544 macro-F1 on attitude against 0.862 for humans, and chain-of-thought barely helped attitude. SimToM (perspective filter first, then answer) was mixed on OpenToM: gains on coarse location for the GPT models, mixed elsewhere. Models of 2023-2024; zero-shot only.
Curator qualifications: OpenToM: 696 narratives with personality-driven actions; GPT-4-Turbo weak on attitude questions (0.544 macro-F1).
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2402.06044v3",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [sourced_fact] OpenToM (arXiv 2402.06044v3) has 696 narratives (596 GPT-3.5-Turbo, 100 longer GPT-4-Turbo) with characters given personality traits and intention-driven actions, and 23 first- and second-order questions each: location (coarse and fine), multi-hop (container fullness, accessibility) and attitude of an observer to the mover's action. GPT-4-Turbo led most columns but scored 0.544 macro-F1 on attitude against 0.862 for humans, and chain-of-thought barely helped attitude. SimToM (perspective filter first, then answer) was mixed on OpenToM: gains on coarse location for the GPT models, mixed elsewhere. Models of 2023-2024; zero-shot only. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |