How was FlawedFictions built and annotated, how well did models and humans do at finding injected continuity errors, and what failure modes did the authors report?

FlawedFictions (arXiv v3, Dec 2025) injects a continuity error into a human-written short story by negating a first-act proposition in later acts, filters with an LLM judge and majority human verification, and scores binary detection plus two-way sentence localization. On 414 stories (average 731 words) the best model rows reach human level (accuracy 0.76) but most models are near chance; on 200 longer stories (average 2,703 words) the best CEEval-Full is 0.53, barely above the always-no-error baseline (0.51). Reported failure modes: over-reading motivations and genre conventions, miscounting entity states, and no gain from more reasoning tokens. Curator qualifications: All quotes and figures verified. The producer's flagged inconsistency (Table 5 baseline 0.51 matching the Long table) is a misreading of caption placement. In the arXiv HTML, captions follow their tables: Table 5 (FlawedFictions) carries Always No Error 0.50 and matches Table 1(a), while the following block is Table 6 (FlawedFictionsLong) with 0.51 and o1 (Medium) 0.53, matching Table 1(b). There is no duplicated or mislabelled row. Withdraw the conflicting_evidence entry: the appendix tables are consistent with Table 1 once caption placement (below each table) is read correctly. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2504.11900v3",
      "note": "work/ff_v3.html FlawedFictions arXiv v3 HTML: sections 3-5, Table 1, appendix Tables 5 and 6 (captions follow tables in this HTML)"
    },
    {
      "url": "https://github.com/kabirahuja2431/FlawedFictions/blob/8b18c7995f14c028412bf55e77cf7ae1f1c05265/README.md",
      "note": "work/ff_v3.html FlawedFictions arXiv v3 HTML: sections 3-5, Table 1, appendix Tables 5 and 6 (captions follow tables in this HTML)"
    },
    {
      "url": "https://huggingface.co/api/datasets/kahuja/flawed-fictions",
      "note": "work/ff_v3.html FlawedFictions arXiv v3 HTML: sections 3-5, Table 1, appendix Tables 5 and 6 (captions follow tables in this HTML)"
    }
  ]
}