How do two open zero-shot TTS papers run their listening tests (CMOS and SMOS), how many raters do they say they used, and what is missing for judging whether a score difference is real?
Seed-TTS (arXiv v1) uses a comparative MOS where raters hear a reference clip then the system and a human recording in random order, scores -2 to +2, and calls differences under 0.1 insignificant, but states no rater count or interval for it and does not release the subjective set. F5-TTS (arXiv v3) states 20 native raters per language over 30 rounds on a -3 to +3 CMOS plus a 1-5 SMOS, and itself lists other papers' rater counts (6, 12). In the extracted text of both papers there is no confidence interval, standard error or inter-rater statistic, so differences such as Seed-TTS CMOS -0.07 versus -0.08 cannot be assessed from the reports.
Seed-TTS (arXiv v1) uses a comparative MOS where raters hear a reference clip then the system and a human recording in random order, scores -2 to +2, and calls differences under 0.1 insignificant, but states no rater count or interval for it and does not release the subjective set. F5-TTS (arXiv v3) states 20 native raters per language over 30 rounds on a -3 to +3 CMOS plus a 1-5 SMOS, and itself lists other papers' rater counts (6, 12). In the extracted text of both papers there is no confidence interval, standard error or inter-rater statistic, so differences such as Seed-TTS CMOS -0.07 versus -0.08 cannot be assessed from the reports.
Curator qualifications: All excerpts located; CMOS/SMOS listening test designs and rater counts as stated.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/abs/2406.02430v1",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/abs/2410.06885v3",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [sourced_fact] Seed-TTS (arXiv v1) uses a comparative MOS where raters hear a reference clip then the system and a human recording in random order, scores -2 to +2, and calls differences under 0.1 insignificant, but states no rater count or interval for it and does not release the subjective set. F5-TTS (arXiv v3) states 20 native raters per language over 30 rounds on a -3 to +3 CMOS plus a 1-5 SMOS, and itself lists other papers' rater counts (6, 12). In the extracted text of both papers there is no confidence interval, standard error or inter-rater statistic, so differences such as Seed-TTS CMOS -0.07 versus -0.08 cannot be assessed from the reports. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |