Lost in Stories: Consistency Bugs in Long Story Generation by LLMs (ConStory-Bench / ConStory-Checker)

{
  "archived": false,
  "revision": {
    "ref": "v1 (submitted 6 Mar 2026; only version listed)",
    "pinned": true,
    "source_url": "https://arxiv.org/abs/2603.05890"
  },
  "checked_at": "2026-10-17T00:00:00.000Z",
  "usefulness": "Benchmark and LLM-as-judge pipeline for contradictions in long generated stories (2,000 prompts, five error categories, 19 subtypes), with MIT-licensed code and dataset. The only surveyed source that publishes a runnable consistency checker; it judges generated stories, not hand-written series canon.",
  "source_type": "paper",
  "availability": "available",
  "dependencies": [
    "Python",
    "OpenAI-compatible LLM API"
  ],
  "source_owner": "Junjie Li, Xinrui Guo, Yuhao Wu, Roy Ka-Wei Lee, Hongzhi Li, Yutao Xie",
  "next_check_at": "2026-10-31T00:00:00.000Z",
  "hardware_notes": "No local hardware requirement stated; README says story generation can use any OpenAI-compatible API, including local servers (vLLM, Ollama); judge-model requirements are not stated.",
  "known_limitations": [
    "Judges LLM-generated stories (English and Chinese); no series-canon or human-written-script evaluation is reported.",
    "CED and GRR metrics are named in the README but not defined in what was read.",
    "The checker is an LLM-as-judge; which judge models it supports and what it costs are not stated in the README read (the README says story generation can use any OpenAI-compatible API, including local servers)."
  ],
  "upstream_updated_at": "2026-03-06T04:16:46.000Z"
}