In Kang et al.'s controlled 2D testbed, how do in-distribution, out-of-distribution and combinatorial generalisation differ for video diffusion models, and what can not be concluded about commercial video models?

Kang et al. (ICML 2025) train diffusion-transformer video models from scratch on Box2D/PHYRE simulations so the training set is fully controlled. Within the training range the models track the simulator almost exactly; outside it (new speeds or radii) error is about 35x higher and does not shrink with 100x more data or 14x more parameters. Where every object type is seen but not every combination, scaling the number of combination templates (6 to 60) cut human-labelled 'abnormal' out-of-template videos from 67% to 10%. Attribute-pair tests show the models copy the nearest training example, preferring colour over size over velocity over shape. The setting is a 2D toy world, so the paper does not measure Sora-class or other commercial models. Curator correction 2026-10-24 (discovery audit record C3-B680-01, curator-reviewed C3R-0683; excerpts re-located by independent curator GET 2026-10-24): Problem. C3-B006-W3-02 says outside the training range error is 'about 35x higher'. Conditions. Anyone reusing that number. Method. The paper's general statement is that OOD velocity errors are 'an order of magnitude higher' than ID errors in all settings; the only worked ratio is one cell (DiT-L, uniform motion, 3M data: OOD 0.427 vs ID 0.012, about 35.6x). Correction (overstated): cite 'an order of magnitude or more' or give the single-cell ratio with its conditions, not a general 35x. The 100x data (30K to 3M), 14x parameters (22M to 310M) and 67% to 10% abnormal-rate statements in the candidate are supported. Limitations. Other scenarios in Figure 3 were not read numerically. Version. arXiv v2, ICML 2025. Curator qualifications: Key figures and quotes verify. The producer's conflicting-evidence entry is confirmed and strengthened. Physics-IQ's reference 40 is Kang et al. by its reference list (not only via the PhyWorld project name), so the two texts do summarise the same study differently: Physics-IQ stresses that models learn specific laws given enough data (the in-distribution half), while Kang et al.'s headline is out-of-distribution failure. This is selective citation, not contradictory data. Identification upgrade: Physics-IQ v3 reference 40 is explicitly Kang et al. (How far is video generation from world model); the same work also appears as reference 8 (duplicate reference entry in Physics-IQ). Reclassify the conflict as divergent secondary characterisation of one result (ID success vs OOD failure); no data disagreement. The ~35x OOD error ratio was not re-read in this pass. Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2411.02385v2",
      "note": "work/2411.02385v2.pdf Kang et al. v2 (pdftotext); work/2501.09038v3.pdf Physics-IQ v3 results discussion and references"
    },
    {
      "url": "https://github.com/phyworld/phyworld/blob/ee0e473d9fc4be605c6b9d919da07b0bfaecbb3e/README.md",
      "note": "work/2411.02385v2.pdf Kang et al. v2 (pdftotext); work/2501.09038v3.pdf Physics-IQ v3 results discussion and references"
    },
    {
      "url": "https://arxiv.org/abs/2501.09038v3",
      "note": "work/2411.02385v2.pdf Kang et al. v2 (pdftotext); work/2501.09038v3.pdf Physics-IQ v3 results discussion and references"
    }
  ]
}