Can EmoTalk give emotional 3D face animation from speech, and what stops commercial use (licence, data, head motion)?
Problem. Many speech-driven faces ignore emotion in the voice. Conditions. Research or non-commercial use; output as 52 blendshape coefficients. Method. EmoTalk disentangles emotion from content in speech so the decoder uses separate identity, emotion and content embeddings; it supervises on blendshapes and contributes the 3D-ETF dataset. The repo is CC BY-NC 4.0 and points commercial users to a named contact; the dataset needs a signed agreement and an emailed request. Its README credits an English wav2vec2 content encoder. Failure modes. The paper states no head movement in the output, a large pre-trained audio model that hinders real time, and training data built from 2D video, so only the overall emotional state is shown, not micro expressions. Limitations. I did not run the code or watch its demos. Version/date. ICCV 2023; repo last pushed 2024-02-23.
Structured fields
{
"citations": [
{
"url": "https://raw.githubusercontent.com/psyai-net/EmoTalk_release/main/readme.md",
"note": "License; Dataset; Acknowledgement; Sec. 5 Limitations; Sec. 5 Limitations; Sec. 5 Limitations; researcher reasoning"
},
{
"url": "https://arxiv.org/pdf/2303.11089",
"note": "License; Dataset; Acknowledgement; Sec. 5 Limitations; Sec. 5 Limitations; Sec. 5 Limitations; researcher reasoning"
}
]
}Evidence claims
- [sourced_fact] Problem. Many speech-driven faces ignore emotion in the voice. Conditions. Research or non-commercial use; output as 52 blendshape coefficients. Method. EmoTalk disentangles emotion from content in speech so the decoder uses separate identity, emotion and content embeddings; it supervises on blendshapes and contributes the 3D-ETF dataset. The repo is CC BY-NC 4.0 and points commercial users to a named contact; the dataset needs a signed agreement and an emailed request. Its README credits an English wav2vec2 content encoder. Failure modes. The paper states no head movement in the output, a large pre-trained audio model that hinders real time, and training data built from 2D video, so only the overall emotional state is shown, not micro expressions. Limitations. I did not run the code or watch its demos. Version/date. ICCV 2023; repo last pushed 2024-02-23. · locator: License; Dataset; Acknowledgement; Sec. 5 Limitations; Sec. 5 Limitations; Sec. 5 Limitations; researcher reasoning source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |