Can a language model's next-word prediction error stand in for where human listeners perceive event boundaries in a story?

Kumar et al. (2023) used GPT-2 XL to compute four per-word measures (surprisal, entropy, Bayesian surprise as KL divergence between successive next-word distributions, and embedding cosine distance) on three audio stories and tested them against human button-press segmentation with leave-one-story-out cross-validation. Only transient Bayesian surprise reliably predicted boundaries; correlations were small (about 0.05 to 0.12 in Table 1) though above a circular-shift null. This is a text-only audio-story study with an older model and no film, so it does not show that a language model finds scene boundaries in screenplays or video. Curator qualifications: Licence element: CC BY-NC 4.0. Text: 'transient Bayesian surprise robustly predicted event segmentation across all of the hyperparameters'; 'None of the other three disfluency measures reliably predicted human event boundaries'. Table 1 Bayesian surprise correlations: 0.05-0.07 at window 20, 0.06-0.07 at 50, 0.09-0.10 at 100, 0.10-0.11 at 150, 0.10-0.12 at 200, with percentiles mostly >= 95; Surprisal at window 20: 0.0 (47.6th percentile) and 0.01 (53.2). Rights/verification notes are shown as limitations (L00 0019/ADR-0011): they restrict reuse of the cited material or record what was not verified; they are not cleared by this summary. Limitations: [component_reuse_restriction] Article licence CC BY-NC 4.0 (quote limits for combined text). Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://www.ebi.ac.uk/europepmc/webservices/rest/PMC11654724/fullTextXML",
      "note": "Kumar et al. 2023, Europe PMC PMC11654724 XML, Table 1"
    }
  ]
}