What do peer-reviewed critiques say about how TTS MOS tests are reported and how test design changes the scores?

A bare TTS score like 'MOS 4.2' is underspecified. Kirkland et al. (SSW 2023) found most 2021-22 Interspeech/SSW papers omit the scale labels, increments and listener instructions. In their own test, asking about 'quality' rather than 'naturalness' raised MOS (3.52 vs 3.21). The VoiceMOS 2022 organisers state MOS values can't be compared across listening tests, because each test has its own listeners and systems. ITU-T P.800 also warns that results on the older 0-4 scale must be raised by one to compare. Report the question, scale, listener pool and stimuli with any MOS. Curator qualifications: Papers and code read 2026-10-22; no listening test or ASR scored. P.800 (08/96) verified on producer copy p800.pdf (sha256 562fd1fdb52761e4, equal to pin): 'the mean scores must all be increased by one to be comparable'. Limitations: Papers and code read 2026-10-22; no listening test or ASR scored. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://www.itu.int/rec/T-REC-P.800-199608-I/en",
      "note": "Kirkland et al., SSW 2023: abstract, results, discussion; VoiceMOS Challenge 2022 paper (arXiv 2203.11389), Section 1"
    },
    {
      "url": "https://arxiv.org/pdf/2203.11389",
      "note": "Kirkland et al., SSW 2023: abstract, results, discussion; VoiceMOS Challenge 2022 paper (arXiv 2203.11389), Section 1"
    },
    {
      "url": "https://www.isca-archive.org/ssw_2023/kirkland23_ssw.pdf",
      "note": "Kirkland et al., SSW 2023: abstract, results, discussion; VoiceMOS Challenge 2022 paper (arXiv 2203.11389), Section 1"
    }
  ]
}