What evaluations of AI directors and emergent-narrative systems do the designers and open papers actually report, and how strong is the evidence that they improve player experience?

The designer sources report outcomes, not measurements: Valve cites sales and awards for Left 4 Dead and asserts that a crude intensity estimate yields good pacing; no pacing data appear in the slides read, and RimWorld and Dwarf Fortress sources report none. A 2024 open paper says directly that no comparative data exist for the L4D director. Its own human-subject study of three directors in a small farming game (108 started, 32 usable) found that non-random directors beat random on time played and quests presented, with enjoyment lower only for random in the second session, yet its conclusion says players failed to perceive a difference. Valve's playtesting deck lists physiological arousal measures but contains no director-specific results. The evidence base for pacing controllers is therefore thin and mixed. Curator qualifications: Counts consistent: 'In total, 108 players started the study. Of these, only thirty-nine players fully completed the study'; seven refreshed and were excluded; 'We are left with thirty-two complete responses that have usable data'. Abstract: 'Our results show that a non-random AI Director provides a better player experience than a random AI Director'; conclusion: 'a curated AID leads to a longer time played and less quests presented than a random AID, but players fail to perceive a difference' (wording differs as the candidate says). SQ6: 'Players indicated that they did not enjoy playing random during the second session' (statistically significant per the text). Rights: abs page links CC BY 4.0; PDF footer 'Copyright © 2024, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.' Valve slides (S1, S2, S15) not re-read in this pass. Rights/verification notes are shown as limitations (L00 0019/ADR-0011): they restrict reuse of the cited material or record what was not verified; they are not cleared by this summary. Limitations: [rights_conflict_record] S14 licence conflict (CC BY 4.0 link vs AAAI all-rights-reserved footer) preserved; short quotation only. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://steamcdn-a.akamaihd.net/apps/valve/2009/GDC2009_ReplayableCooperativeGameDesign_Left4Dead.pdf",
      "note": "arXiv 2410.03733v1 PDF (equals producer pin): abstract, Sec. 1, 5.1, 6.2, 8; page-1 footer; arXiv abs page v1 licence link creativecommons.org/licenses/by/4.0/"
    },
    {
      "url": "https://arxiv.org/pdf/2410.03733v1",
      "note": "arXiv 2410.03733v1 PDF (equals producer pin): abstract, Sec. 1, 5.1, 6.2, 8; page-1 footer; arXiv abs page v1 licence link creativecommons.org/licenses/by/4.0/"
    },
    {
      "url": "https://cdn.fastly.steamstatic.com/apps/valve/2009/GDC2009_ValvesApproachToPlaytesting.pdf",
      "note": "arXiv 2410.03733v1 PDF (equals producer pin): abstract, Sec. 1, 5.1, 6.2, 8; page-1 footer; arXiv abs page v1 licence link creativecommons.org/licenses/by/4.0/"
    },
    {
      "url": "https://steamcdn-a.akamaihd.net/apps/valve/2009/ai_systems_of_l4d_mike_booth.pdf",
      "note": "arXiv 2410.03733v1 PDF (equals producer pin): abstract, Sec. 1, 5.1, 6.2, 8; page-1 footer; arXiv abs page v1 licence link creativecommons.org/licenses/by/4.0/"
    }
  ]
}