What expert-designed tests can check creativity in a short story, how reliable are they, and how did LLM stories score?

Partially verified: the curator re-read none of the cited sources; S6 rests on the producer's pinned reading. TTCW (Chakrabarty et al.) turns the Torrance dimensions (fluency, flexibility, originality, elaboration) into 14 binary yes/no tests proposed by 8 creative-writing experts, e.g. narrative pacing, scene vs summary, ending, originality in thought, character development. Ten new experts applied them to 48 stories (12 New Yorker stories plus 36 from GPT-3.5, GPT-4 and Claude v1.3 on the same plot, length-matched): New Yorker stories passed 84.7% of tests vs 8.7% (GPT-3.5), 27.9% (GPT-4), 30.0% (Claude v1.3). Per-test Fleiss kappa 0.27-0.66 (mean 0.41); aggregate pass counts correlate 0.69 between experts. Curator qualifications: Partially verified: the curator re-read none of the cited sources; S6 rests on the producer's pinned reading. Curator GET 2026-10-23 of the sources not re-read (mechanical source evidence, not semantic verification): S6: HTTP 200, bytes differ from the producer pin. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2309.14556v3",
      "note": "candidate evidence excerpts"
    }
  ]
}