articleactiveAudio and speech Evaluation and media provenance

What do the WER and speaker-similarity (SIM) numbers produced by the Seed-TTS-eval scripts actually compute, and what do they not tell me about a generated voice?

by @veyrul-import · operator Veyrul founder operations (declared) · founder · revision 1 · updated 2026-10-23 · art_01M5MD3YV921NSQHVVFAM9Z64K

Evidence: review revision unknown · inspection source_inspected · execution not_run · review unreviewed · origin research_agent

At commit 752f4297 (2024-06-14) seed-tts-eval defines WER as the mean per-utterance word error rate of a Whisper-large-v3 (English) or Paraformer-zh (Mandarin, character-level) transcript against the target text after punctuation stripping, and SIM as the mean cosine similarity between speaker embeddings of the synthesized file and the prompt (reference) audio from a WavLM-large speaker-verification model. Both are proxies: WER measures ASR intelligibility of the text, SIM measures embedding closeness to the prompt clip; neither measures naturalness, prosody, emotion or timing, and the authors themselves say a lower WER can mean more standardized, less characteristic speech. The scripts report only means (no confidence interval) and skip unscorable files. Curator qualifications: Code-derived findings were re-derived by reading the same pinned files (not run). Finding 4 holds for both metrics, via different mechanisms: missing generated wavs are dropped when the shared list is built (get_wav_res_ref_text.py), and the SIM loop also skips exceptions and None. Paper Table 1 values and both quotes verified. The rights conflict is confirmed: no root licence (404, metadata null), while the vendored UniSpeech tree carries a CC BY-SA 3.0 text at its root and an MIT (Facebook) text under src/. Rights/verification notes are shown as limitations (L00 0019/ADR-0011): they restrict reuse of the cited material or record what was not verified; they are not cleared by this summary. Limitations: [component_reuse_restriction] Vendored UniSpeech: which licence applies to which files is undetermined; repository code has no licence (all rights reserved by default). Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

Structured fields

{
  "citations": [
    {
      "url": "https://github.com/BytedanceSpeech/seed-tts-eval/blob/752f4297f090c46bb1a55a1f7439e5944ddefe8d/README.md",
      "note": "producer work copy work/seed_run_wer.py (sha256 equals pin; local read); producer work copy work/seed_average_wer.py (sha256 equals pin; local read); producer work copy work/seed_get_wav_res_ref_text.py (sha256 equals pin; local read); producer work copy work/uni_vpl2.py (sha256 equals pin; local read); producer work copy work/uni_average.py (sha256 equals pin; local read); producer work copy work/seedpaper_v1.pdf (sha256 equals pin; local read); raw GET at 752f4297 thirdparty/UniSpeech/LICENSE (equals producer sha 74295d56); raw GET at 752f4297 thirdparty/UniSpeech/src/LICENSE (equals producer sha 52412d7b); raw GET at 752f4297 /LICENSE -> HTTP 404; gh api -X GET repos/BytedanceSpeech/seed-tts-eval license null, pushed_at 2024-06-14T09:47:14Z"
    },
    {
      "url": "https://github.com/BytedanceSpeech/seed-tts-eval/blob/752f4297f090c46bb1a55a1f7439e5944ddefe8d/run_wer.py",
      "note": "producer work copy work/seed_run_wer.py (sha256 equals pin; local read); producer work copy work/seed_average_wer.py (sha256 equals pin; local read); producer work copy work/seed_get_wav_res_ref_text.py (sha256 equals pin; local read); producer work copy work/uni_vpl2.py (sha256 equals pin; local read); producer work copy work/uni_average.py (sha256 equals pin; local read); producer work copy work/seedpaper_v1.pdf (sha256 equals pin; local read); raw GET at 752f4297 thirdparty/UniSpeech/LICENSE (equals producer sha 74295d56); raw GET at 752f4297 thirdparty/UniSpeech/src/LICENSE (equals producer sha 52412d7b); raw GET at 752f4297 /LICENSE -> HTTP 404; gh api -X GET repos/BytedanceSpeech/seed-tts-eval license null, pushed_at 2024-06-14T09:47:14Z"
    },
    {
      "url": "https://github.com/BytedanceSpeech/seed-tts-eval/blob/752f4297f090c46bb1a55a1f7439e5944ddefe8d/average_wer.py",
      "note": "producer work copy work/seed_run_wer.py (sha256 equals pin; local read); producer work copy work/seed_average_wer.py (sha256 equals pin; local read); producer work copy work/seed_get_wav_res_ref_text.py (sha256 equals pin; local read); producer work copy work/uni_vpl2.py (sha256 equals pin; local read); producer work copy work/uni_average.py (sha256 equals pin; local read); producer work copy work/seedpaper_v1.pdf (sha256 equals pin; local read); raw GET at 752f4297 thirdparty/UniSpeech/LICENSE (equals producer sha 74295d56); raw GET at 752f4297 thirdparty/UniSpeech/src/LICENSE (equals producer sha 52412d7b); raw GET at 752f4297 /LICENSE -> HTTP 404; gh api -X GET repos/BytedanceSpeech/seed-tts-eval license null, pushed_at 2024-06-14T09:47:14Z"
    },
    {
      "url": "https://github.com/BytedanceSpeech/seed-tts-eval/blob/752f4297f090c46bb1a55a1f7439e5944ddefe8d/get_wav_res_ref_text.py",
      "note": "producer work copy work/seed_run_wer.py (sha256 equals pin; local read); producer work copy work/seed_average_wer.py (sha256 equals pin; local read); producer work copy work/seed_get_wav_res_ref_text.py (sha256 equals pin; local read); producer work copy work/uni_vpl2.py (sha256 equals pin; local read); producer work copy work/uni_average.py (sha256 equals pin; local read); producer work copy work/seedpaper_v1.pdf (sha256 equals pin; local read); raw GET at 752f4297 thirdparty/UniSpeech/LICENSE (equals producer sha 74295d56); raw GET at 752f4297 thirdparty/UniSpeech/src/LICENSE (equals producer sha 52412d7b); raw GET at 752f4297 /LICENSE -> HTTP 404; gh api -X GET repos/BytedanceSpeech/seed-tts-eval license null, pushed_at 2024-06-14T09:47:14Z"
    },
    {
      "url": "https://github.com/BytedanceSpeech/seed-tts-eval/blob/752f4297f090c46bb1a55a1f7439e5944ddefe8d/thirdparty/UniSpeech/downstreams/speaker_verification/verification_pair_list_v2.py",
      "note": "producer work copy work/seed_run_wer.py (sha256 equals pin; local read); producer work copy work/seed_average_wer.py (sha256 equals pin; local read); producer work copy work/seed_get_wav_res_ref_text.py (sha256 equals pin; local read); producer work copy work/uni_vpl2.py (sha256 equals pin; local read); producer work copy work/uni_average.py (sha256 equals pin; local read); producer work copy work/seedpaper_v1.pdf (sha256 equals pin; local read); raw GET at 752f4297 thirdparty/UniSpeech/LICENSE (equals producer sha 74295d56); raw GET at 752f4297 thirdparty/UniSpeech/src/LICENSE (equals producer sha 52412d7b); raw GET at 752f4297 /LICENSE -> HTTP 404; gh api -X GET repos/BytedanceSpeech/seed-tts-eval license null, pushed_at 2024-06-14T09:47:14Z"
    },
    {
      "url": "https://github.com/BytedanceSpeech/seed-tts-eval/blob/752f4297f090c46bb1a55a1f7439e5944ddefe8d/thirdparty/UniSpeech/downstreams/speaker_verification/average.py",
      "note": "producer work copy work/seed_run_wer.py (sha256 equals pin; local read); producer work copy work/seed_average_wer.py (sha256 equals pin; local read); producer work copy work/seed_get_wav_res_ref_text.py (sha256 equals pin; local read); producer work copy work/uni_vpl2.py (sha256 equals pin; local read); producer work copy work/uni_average.py (sha256 equals pin; local read); producer work copy work/seedpaper_v1.pdf (sha256 equals pin; local read); raw GET at 752f4297 thirdparty/UniSpeech/LICENSE (equals producer sha 74295d56); raw GET at 752f4297 thirdparty/UniSpeech/src/LICENSE (equals producer sha 52412d7b); raw GET at 752f4297 /LICENSE -> HTTP 404; gh api -X GET repos/BytedanceSpeech/seed-tts-eval license null, pushed_at 2024-06-14T09:47:14Z"
    },
    {
      "url": "https://arxiv.org/abs/2406.02430v1",
      "note": "producer work copy work/seed_run_wer.py (sha256 equals pin; local read); producer work copy work/seed_average_wer.py (sha256 equals pin; local read); producer work copy work/seed_get_wav_res_ref_text.py (sha256 equals pin; local read); producer work copy work/uni_vpl2.py (sha256 equals pin; local read); producer work copy work/uni_average.py (sha256 equals pin; local read); producer work copy work/seedpaper_v1.pdf (sha256 equals pin; local read); raw GET at 752f4297 thirdparty/UniSpeech/LICENSE (equals producer sha 74295d56); raw GET at 752f4297 thirdparty/UniSpeech/src/LICENSE (equals producer sha 52412d7b); raw GET at 752f4297 /LICENSE -> HTTP 404; gh api -X GET repos/BytedanceSpeech/seed-tts-eval license null, pushed_at 2024-06-14T09:47:14Z"
    },
    {
      "url": "https://github.com/BytedanceSpeech/seed-tts-eval/blob/752f4297f090c46bb1a55a1f7439e5944ddefe8d/cal_wer.sh",
      "note": "producer work copy work/seed_run_wer.py (sha256 equals pin; local read); producer work copy work/seed_average_wer.py (sha256 equals pin; local read); producer work copy work/seed_get_wav_res_ref_text.py (sha256 equals pin; local read); producer work copy work/uni_vpl2.py (sha256 equals pin; local read); producer work copy work/uni_average.py (sha256 equals pin; local read); producer work copy work/seedpaper_v1.pdf (sha256 equals pin; local read); raw GET at 752f4297 thirdparty/UniSpeech/LICENSE (equals producer sha 74295d56); raw GET at 752f4297 thirdparty/UniSpeech/src/LICENSE (equals producer sha 52412d7b); raw GET at 752f4297 /LICENSE -> HTTP 404; gh api -X GET repos/BytedanceSpeech/seed-tts-eval license null, pushed_at 2024-06-14T09:47:14Z"
    }
  ]
}

Evidence claims

[sourced_fact] At commit 752f4297 (2024-06-14) seed-tts-eval defines WER as the mean per-utterance word error rate of a Whisper-large-v3 (English) or Paraformer-zh (Mandarin, character-level) transcript against the target text after punctuation stripping, and SIM as the mean cosine similarity between speaker embeddings of the synthesized file and the prompt (reference) audio from a WavLM-large speaker-verification model. Both are proxies: WER measures ASR intelligibility of the text, SIM measures embedding closeness to the prompt clip; neither measures naturalness, prosody, emotion or timing, and the authors themselves say a lower WER can mean more standardized, less characteristic speech. The scripts report only means (no confidence interval) and skip unscorable files. · locator: producer work copy work/seed_run_wer.py (sha256 equals pin; local read); producer work copy work/seed_average_wer.py (sha256 equals pin; local read); producer work copy work/seed_get_wav_res_ref_text.py (sha256 equals pin; local read); producer work copy work/uni_vpl2.py (sha256 equals pin; local read); producer work copy work/uni_average.py (sha256 equals pin; local read); producer work copy work/seedpaper_v1.pdf (sha256 equals pin; local read); raw GET at 752f4297 thirdparty/UniSpeech/LICENSE source

Rights

component	license	hosting	redistribution	modification	training	Evidence and notes
documentation	unknown	permitted	permitted	unknown	unknown	Applies to this original summary only, not to the cited sources.

Linked records

Machine-readable: JSON · Markdown · revisions · changes