Is there open evidence that giving an LLM an event-causality graph improves its judgment of story quality, and does it test coincidence or unearned rescue?

Sun, Chao and Li (arXiv:2311.09648v1) prompt ChatGPT (gpt3.5-turbo) to extract event-causality graphs from stories, then append the graph to a scoring prompt that tells the model to 'reward stories with rich causal chains and penalize those that lack or have confusing causal chains'. On OpenMEVA (200 prompts, 1,000 generated five-sentence stories) they report higher correlation with human ratings: about 1.4% from adding the word 'causal' and an average 6.7% dataset-level increase with graphs (the abstract states 5.7%, an unreconciled difference). The paper does not test coincidence, deus ex machina or unearned resolutions, and its stories are five-sentence outputs of older models, so it supports 'causal structure helps an LLM judge' only for that setting. Curator qualifications: All 9 excerpts located. Curator clarification: the paper says the extracted causal relations 'improve Kendall correlation by 5.7% on story evaluation' (abstract/intro), while the 6.7% is 'on the dataset level, on average' across three correlation metrics. Different measures, so the producer's 'unreconciled difference' is plausibly not a contradiction. Five-sentence stories from older models only. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/html/2311.09648v1",
      "note": "abstract/intro and Section 4"
    }
  ]
}