How can an agent measure whether a mystery's reveal is both surprising and fair (set up so the clues add up in hindsight), and what do the open metrics and their human check actually show?
Wagner, Keydar and Abend (arXiv v2, May 2026) define surprise as how badly a "gullible" reader (an LLM told to take events at face value) predicts the culprit paragraph by paragraph, and coherence as how well clue-competent readers predict it, and argue theoretically that fair play needs a hindsight gap. On whodunit stories their metrics rank Christie above Conan Doyle for surprise and fair play, and LLM-generated mysteries rarely reach both; a 20-person human check gives small correlations (r about 0.1 to 0.25). It is a usable check design for detective outlines and a research result, not a validated test, and code and data were not yet released.
Wagner, Keydar and Abend (arXiv v2, May 2026) define surprise as how badly a "gullible" reader (an LLM told to take events at face value) predicts the culprit paragraph by paragraph, and coherence as how well clue-competent readers predict it, and argue theoretically that fair play needs a hindsight gap. On whodunit stories their metrics rank Christie above Conan Doyle for surprise and fair play, and LLM-generated mysteries rarely reach both; a 20-person human check gives small correlations (r about 0.1 to 0.25). It is a usable check design for detective outlines and a research result, not a validated test, and code and data were not yet released.
Curator qualifications: Verified r = -0.13. The 'validates the metrics' vs small correlations tension is kept. AI connection: an LLM-reader surprise/coherence metric is a documented research application, with weak human validation.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/html/2507.13841v2",
"note": "results"
}
]
}Evidence claims
- [sourced_fact] Wagner, Keydar and Abend (arXiv v2, May 2026) define surprise as how badly a "gullible" reader (an LLM told to take events at face value) predicts the culprit paragraph by paragraph, and coherence as how well clue-competent readers predict it, and argue theoretically that fair play needs a hindsight gap. On whodunit stories their metrics rank Christie above Conan Doyle for surprise and fair play, and LLM-generated mysteries rarely reach both; a 20-person human check gives small correlations (r about 0.1 to 0.25). It is a usable check design for detective outlines and a research result, not a validated test, and code and data were not yet released. · locator: results source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |