Do automatic metrics (BLEU, ROUGE, BERTScore, BARTScore...) track human story judgments, and which should be used?

Partially verified: the curator re-read none of the cited sources; S4 rests on the producer's pinned reading. On HANNA, 72 automatic metrics correlate mostly weakly (<30%) to moderately (30-50%) with human criteria at story level; system-level correlations are higher (40-70% for most) so metrics can rank systems but not individual stories. Best story-level metrics (|r| about 33-59%) include chrF, BERTScore, S3, BARTScore, SUPERT for relevance, and simple Repetition-3 and Novelty-1 statistics topped coherence and surprise; the authors advise relying on human annotation for ASG evaluation. Curator qualifications: Partially verified: the curator re-read none of the cited sources; S4 rests on the producer's pinned reading. Curator GET 2026-10-23 of the sources not re-read (mechanical source evidence, not semantic verification): S4: HTTP 200, bytes differ from the producer pin. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2208.11646v4",
      "note": "candidate evidence excerpts"
    }
  ]
}