Do LLM plot-hole and consistency detectors work on multi-episode series canon, not just short stories or generated text?
FlawedFictions uses short human-written stories with injected plot holes; ConStory-Bench scores LLM-generated long stories; SCORE is framed around episodes but only its abstract was read and no code is linked. None reports results on human-written series with a canon database. Evidence needed: a run of ConStory-Checker or a SCORE reimplementation on a human-labelled set of continuity errors from a multi-episode script corpus, with precision and recall.
Why this is still open: The benchmarks cover other text types and the collector executed nothing.
Linked records
- discusses → SCORE: Story Coherence and Retrieval Enhancement for AI Narratives
- discusses → Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection (FlawedFictions)
- discusses → Lost in Stories: Consistency Bugs in Long Story Generation by LLMs (ConStory-Bench / ConStory-Checker)