FlawedFictions tests whether an LLM spots a plot hole in a short human-written story, using an algorithm that injects plot holes (reported: state-of-the-art LLMs struggle regardless of reasoning effort and degrade as stories get longer; LLM summaries and generated stories introduce plot holes). Con…

FlawedFictions tests whether an LLM spots a plot hole in a short human-written story, using an algorithm that injects plot holes (reported: state-of-the-art LLMs struggle regardless of reasoning effort and degrade as stories get longer; LLM summaries and generated stories introduce plot holes). ConStory-Bench tests consistency inside very long LLM-generated stories with an evidence-grounded LLM judge and a taxonomy of five categories and 19 subtypes (reported: errors are most common in factual and temporal dimensions). SCORE proposes tracking key item statuses and episode summaries with retrieval to find related episodes (reported: improvement over baseline GPT models; abstract only). FABLES and SummScreen show that faithfulness of LLM summaries to long narratives is itself unreliable. Only SCORE is framed around episodes, and it has no linked code. Rationale: Statements are the authors' abstract claims, each quoted in evidence/quotes.jsonl for the inspected records. Counterevidence: No source reports results on human-written multi-episode canon, and the benchmarks use different error definitions, so their numbers are not comparable. Uncertainty: Transfer from these benchmarks to series continuity is untested (see w2-t01-Q02). Sources: - https://arxiv.org/abs/2504.11900 - https://arxiv.org/abs/2603.05890 - https://arxiv.org/abs/2503.23512 - https://arxiv.org/abs/2404.01261 - https://arxiv.org/abs/2104.07091 Checked: 2026-10-17

{
  "context": "How do the available continuity and plot-hole detectors differ in what they test?"
}