Which WER variants and text normalisations change ASR subtitle scores, and is COMET usable for subtitle translation evaluation?
WER on connected speech is asymmetric and can exceed 1 (bound max(N1,N2)/N1), which motivated MER and WIL; Whisper's authors report text normalisation cutting WER by up to 50 percent on some sets and warn of overfitting to Whisper's style; SubER's authors found case-sensitive WER with punctuation did not consistently improve correlation with post-editing. IWSLT 2024 ranked offline speech translation by COMET but subtitling by SubER, with BLEU/chrF/BLEURT for translation and CPS/CPL/LPB compliance rates.
WER on connected speech is asymmetric and can exceed 1 (bound max(N1,N2)/N1), which motivated MER and WIL; Whisper's authors report text normalisation cutting WER by up to 50 percent on some sets and warn of overfitting to Whisper's style; SubER's authors found case-sensitive WER with punctuation did not consistently improve correlation with post-editing. IWSLT 2024 ranked offline speech translation by COMET but subtitling by SubER, with BLEU/chrF/BLEURT for translation and CPS/CPL/LPB compliance rates.
Curator qualifications: Papers read 2026-10-22; no metric computed. Break-token WER row and COMET paper body not re-read beyond the abstract page.
Limitations: Papers read 2026-10-22; no metric computed.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/abs/2205.05805",
"note": "Morris, Maier, Green, Interspeech 2004 PDF, abstract and eq. 2; Radford et al., Whisper (arXiv 2212.04356) PDF, text normalisation passage; Wilken et al., SubER (arXiv 2205.05805) PDF, Table 1 discussion; IWSLT 2024 Findings PDF, offline ST metrics and subtitling metrics"
},
{
"url": "https://github.com/apptek/SubER/blob/3395bfec76c8c9bbc220a6ee48c8566f16c99ac5/README.md",
"note": "Morris, Maier, Green, Interspeech 2004 PDF, abstract and eq. 2; Radford et al., Whisper (arXiv 2212.04356) PDF, text normalisation passage; Wilken et al., SubER (arXiv 2205.05805) PDF, Table 1 discussion; IWSLT 2024 Findings PDF, offline ST metrics and subtitling metrics"
},
{
"url": "https://aclanthology.org/2024.iwslt-1.1.pdf",
"note": "Morris, Maier, Green, Interspeech 2004 PDF, abstract and eq. 2; Radford et al., Whisper (arXiv 2212.04356) PDF, text normalisation passage; Wilken et al., SubER (arXiv 2205.05805) PDF, Table 1 discussion; IWSLT 2024 Findings PDF, offline ST metrics and subtitling metrics"
},
{
"url": "https://arxiv.org/abs/2009.09025",
"note": "Morris, Maier, Green, Interspeech 2004 PDF, abstract and eq. 2; Radford et al., Whisper (arXiv 2212.04356) PDF, text normalisation passage; Wilken et al., SubER (arXiv 2205.05805) PDF, Table 1 discussion; IWSLT 2024 Findings PDF, offline ST metrics and subtitling metrics"
},
{
"url": "https://www.isca-archive.org/interspeech_2004/morris04_interspeech.pdf",
"note": "Morris, Maier, Green, Interspeech 2004 PDF, abstract and eq. 2; Radford et al., Whisper (arXiv 2212.04356) PDF, text normalisation passage; Wilken et al., SubER (arXiv 2205.05805) PDF, Table 1 discussion; IWSLT 2024 Findings PDF, offline ST metrics and subtitling metrics"
},
{
"url": "https://arxiv.org/abs/2212.04356",
"note": "Morris, Maier, Green, Interspeech 2004 PDF, abstract and eq. 2; Radford et al., Whisper (arXiv 2212.04356) PDF, text normalisation passage; Wilken et al., SubER (arXiv 2205.05805) PDF, Table 1 discussion; IWSLT 2024 Findings PDF, offline ST metrics and subtitling metrics"
}
]
}Evidence claims
- [sourced_fact] WER on connected speech is asymmetric and can exceed 1 (bound max(N1,N2)/N1), which motivated MER and WIL; Whisper's authors report text normalisation cutting WER by up to 50 percent on some sets and warn of overfitting to Whisper's style; SubER's authors found case-sensitive WER with punctuation did not consistently improve correlation with post-editing. IWSLT 2024 ranked offline speech translation by COMET but subtitling by SubER, with BLEU/chrF/BLEURT for translation and CPS/CPL/LPB compliance rates. · locator: Morris, Maier, Green, Interspeech 2004 PDF, abstract and eq. 2; Radford et al., Whisper (arXiv 2212.04356) PDF, text normalisation passage; Wilken et al., SubER (arXiv 2205.05805) PDF, Table 1 discussion; IWSLT 2024 Findings PDF, offline ST metrics and subtitling metrics source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |
Linked records
- cites → https://aclanthology.org/2024.iwslt-1.1.pdf
- cites → https://arxiv.org/abs/2009.09025
- cites → https://arxiv.org/abs/2205.05805
- cites → https://arxiv.org/abs/2212.04356
- cites → https://github.com/apptek/SubER/blob/3395bfec76c8c9bbc220a6ee48c8566f16c99ac5/README.md
- cites → https://www.isca-archive.org/interspeech_2004/morris04_interspeech.pdf