How is a CMOS (comparative MOS) test set up in a recent TTS paper, and what threshold is treated as no difference?

Problem: decide whether two voice outputs differ audibly. Conditions: listeners compare a synthesized clip against a human or other-system clip for the same text. Method (as described by Seed-TTS): for each test sample evaluators first hear a reference clip of the target speaker; then hear the synthesized output and the ground-truth human recording in random order; they rate which has higher speaker similarity and expressiveness on a scale from -2 to +2 (the extremes mean least and strongest preference for the first sample); results are rearranged into one direction (system versus human) and averaged over evaluators and sentences. The authors state that an absolute CMOS below 0.1 is empirically considered insignificant between systems, and report CMOS of -0.07 (English) and -0.08 (Mandarin) versus human. Use (inference): randomise order and sample-to-listener assignment, hide system identity, include the human reference as a control, fix the reference clip, and report n listeners, n items and a confidence interval rather than relying on the 0.1 rule. Failure modes: small panels, unreported rater counts and no uncertainty, rater fatigue, asking about 'overall quality' when the question was speaker similarity. Limitations: the 0.1 figure is described as empirical and is not a standard; the paper reports no per-listener variance in the text read here; developer-run tests have conflict of interest. Curator qualifications: All cited documentation excerpts located on the live pages by curator GET; no video sources. Limitations: Listener counts for this test were not found in the text read; the threshold's basis is only 'empirical'. The test is run by the model's developer. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/pdf/2406.02430",
      "note": "Evaluation metrics; Evaluation metrics; In-context learning results"
    }
  ]
}