What is a latent action model, and how is one trained from video that has no action labels?
A latent action model (LAM) is trained on pairs of consecutive frames with no action labels. An inverse dynamics model compresses what changed into a small latent; a forward model must predict the next frame from the past frame plus that latent, so the latent has to carry the action-like change. LAPO (2023, v2 pinned) fixed this template; Genie (2024) used it with 8 discrete codes so a person could pick one per step; AdaWorld (2025) used a continuous latent. LAPO tested on synthetic game data with known actions; web video was the stated motivation, not the tested case.
A latent action model (LAM) is trained on pairs of consecutive frames with no action labels. An inverse dynamics model compresses what changed into a small latent; a forward model must predict the next frame from the past frame plus that latent, so the latent has to carry the action-like change. LAPO (2023, v2 pinned) fixed this template; Genie (2024) used it with 8 discrete codes so a person could pick one per step; AdaWorld (2025) used a continuous latent. LAPO tested on synthetic game data with known actions; web video was the stated motivation, not the tested case.
Curator qualifications: All excerpts located in the work texts. Genie's codebook of 8 is a stated design choice; the discrete-vs-continuous disagreement is preserved.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/abs/2312.10812v2",
"note": "method; conclusion; Sec. on decoders"
},
{
"url": "https://arxiv.org/abs/2402.15391v1",
"note": "method; conclusion; Sec. on decoders"
},
{
"url": "https://arxiv.org/abs/2503.18938v4",
"note": "method; conclusion; Sec. on decoders"
}
]
}Evidence claims
- [sourced_fact] A latent action model (LAM) is trained on pairs of consecutive frames with no action labels. An inverse dynamics model compresses what changed into a small latent; a forward model must predict the next frame from the past frame plus that latent, so the latent has to carry the action-like change. LAPO (2023, v2 pinned) fixed this template; Genie (2024) used it with 8 discrete codes so a person could pick one per step; AdaWorld (2025) used a continuous latent. LAPO tested on synthetic game data with known actions; web video was the stated motivation, not the tested case. · locator: method; conclusion; Sec. on decoders source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |