What do the authors of video-to-audio models say the models get wrong, so I know what not to ask of them?
Documented failure modes cluster: speech and lip sync (MMAudio 'unintelligible mumbles'; Google V2A 'uncanny lip-syncing'; Movie Gen does not support voice), dense, small or occluded motion (Movie Gen: tap dance, footsteps), unfamiliar sound concepts (MMAudio: 'RPG firing'), unrequested music (MMAudio), dependence on clean input video (Google V2A), and long-form or precise-timestamp reasoning (ThinkSound's MLLM). These come from the model authors' own statements; none was reproduced.
Documented failure modes cluster: speech and lip sync (MMAudio 'unintelligible mumbles'; Google V2A 'uncanny lip-syncing'; Movie Gen does not support voice), dense, small or occluded motion (Movie Gen: tap dance, footsteps), unfamiliar sound concepts (MMAudio: 'RPG firing'), unrequested music (MMAudio), dependence on clean input video (Google V2A), and long-form or precise-timestamp reasoning (ThinkSound's MLLM). These come from the model authors' own statements; none was reproduced.
Curator qualifications: Excerpted evidence located (two unexcerpted rows); author-stated limitations.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/abs/2412.15322v2",
"note": "candidate evidence excerpts"
},
{
"url": "https://github.com/hkchengrex/MMAudio/blob/974010a026c731054592d8f777218bd9d85a6c24/README.md",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/abs/2410.13720v2",
"note": "candidate evidence excerpts"
},
{
"url": "https://deepmind.google/blog/generating-audio-for-video/",
"note": "candidate evidence excerpts"
},
{
"url": "https://arxiv.org/abs/2506.21448v3",
"note": "candidate evidence excerpts"
},
{
"url": "https://storage.googleapis.com/deepmind-media/Model-Cards/Veo-3-Model-Card.pdf",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [sourced_fact] Documented failure modes cluster: speech and lip sync (MMAudio 'unintelligible mumbles'; Google V2A 'uncanny lip-syncing'; Movie Gen does not support voice), dense, small or occluded motion (Movie Gen: tap dance, footsteps), unfamiliar sound concepts (MMAudio: 'RPG firing'), unrequested music (MMAudio), dependence on clean input video (Google V2A), and long-form or precise-timestamp reasoning (ThinkSound's MLLM). These come from the model authors' own statements; none was reproduced. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |
Linked records
- cites → https://arxiv.org/abs/2410.13720v2
- cites → https://arxiv.org/abs/2412.15322v2
- cites → https://arxiv.org/abs/2506.21448v3
- cites → https://deepmind.google/blog/generating-audio-for-video
- cites → https://github.com/hkchengrex/MMAudio/blob/974010a026c731054592d8f777218bd9d85a6c24/README.md
- cites → https://storage.googleapis.com/deepmind-media/Model-Cards/Veo-3-Model-Card.pdf