What does ConStory-Bench measure about consistency errors in long LLM-written stories, how does its checker work, and how far can its checker be trusted compared with human readers?

ConStory-Bench (arXiv v1, March 2026) gives 2,000 prompts for 8,000-10,000-word stories, a 5-category / 19-subtype taxonomy and an o4-mini based four-stage checker that outputs quoted contradictory spans with character offsets. The authors report that errors cluster in factual and timeline dimensions and near 40-60% of the narrative, and that the checker found 55% of 1,000 injected errors against 17.1% for two professional novel writers. That human comparison used LLM-injected errors and two annotators paid $1.00 per story, so it supports checker-as-triage, not checker-as-ground-truth. Curator qualifications: All figures verified. The flagged tension is confirmed as author overreach: the paper concludes automated evaluation is 'more comprehensive and reliable' than human judgement on the basis of two web novel writers annotating 200 stories in two days at $1.00 per story, on LLM-injected errors. The producer's triage-not-ground-truth reading is correct. Precision: the annotators are 'professional web novel writers' (not 'novel writers'), who completed 200 stories within two days; include the time constraint as context for their low recall. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2603.05890v1",
      "note": "work/cs_v1.html ConStory-Bench arXiv v1 HTML"
    },
    {
      "url": "https://github.com/Picrew/ConStory-Bench/tree/3f4195adda94d977ad9bd56f58ec51eaf1a1cd93",
      "note": "work/cs_v1.html ConStory-Bench arXiv v1 HTML"
    },
    {
      "url": "https://github.com/Picrew/ConStory-Bench/blob/3f4195adda94d977ad9bd56f58ec51eaf1a1cd93/README.md",
      "note": "work/cs_v1.html ConStory-Bench arXiv v1 HTML"
    }
  ]
}