If an LLM correctly says a character does not know a secret, will it then predict that the character acts as if they do not know?
SimpleToM (arXiv 2410.13648v2) uses two-sentence everyday stories (a hidden fact; a person acting without knowing it) and asks three questions: is X aware (mental state), what will X do (behaviour), was X's action reasonable (judgment). Models were near ceiling on awareness (GPT-4o 95.6, Claude-3.5-Sonnet 97.9) but far lower on behaviour (49.5, 67.0) and judgment (15.3, 24.9) where chance is 50; o1-preview reached 84.1 on behaviour. The failure in the authors' example is predicting the observer acts on the hidden fact. Reminders of the awareness answer lifted behaviour above 80 for three models but judgment stayed low.
SimpleToM (arXiv 2410.13648v2) uses two-sentence everyday stories (a hidden fact; a person acting without knowing it) and asks three questions: is X aware (mental state), what will X do (behaviour), was X's action reasonable (judgment). Models were near ceiling on awareness (GPT-4o 95.6, Claude-3.5-Sonnet 97.9) but far lower on behaviour (49.5, 67.0) and judgment (15.3, 24.9) where chance is 50; o1-preview reached 84.1 on behaviour. The failure in the authors' example is predicting the observer acts on the hidden fact. Reminders of the awareness answer lifted behaviour above 80 for three models but judgment stayed low.
Curator qualifications: Models that correctly say a character is unaware of a hidden fact still predict that character acting on it, and judge the action unreasonable; awareness near ceiling, behaviour and judgment far lower.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2410.13648v2",
"note": "arXiv 2410.13648v2 HTML results table"
}
]
}Evidence claims
- [sourced_fact] SimpleToM (arXiv 2410.13648v2) uses two-sentence everyday stories (a hidden fact; a person acting without knowing it) and asks three questions: is X aware (mental state), what will X do (behaviour), was X's action reasonable (judgment). Models were near ceiling on awareness (GPT-4o 95.6, Claude-3.5-Sonnet 97.9) but far lower on behaviour (49.5, 67.0) and judgment (15.3, 24.9) where chance is 50; o1-preview reached 84.1 on behaviour. The failure in the authors' example is predicting the observer acts on the hidden fact. Reminders of the awareness answer lifted behaviour above 80 for three models but judgment stayed low. · locator: arXiv 2410.13648v2 HTML results table source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |