Which failure modes of text world models did arXiv 2604.13824 actually measure (over-optimistic simulation, time-outs, room-connectivity errors), and how large were they?
S1 measures one failure mode quantitatively: the "too easy simulator" in which a world model lets a weak agent succeed that fails in the real environment (TextWorld, Qwen3-0.6B: 85/200 false positives with the supervised model, 19/200 after BehR; Qwen3-1.7B: 69 and 34). It also reports, descriptively, that 82-97% of WebShop world-model episodes hit the 50-step cap because the model does not reproduce purchase completion, and one worked TextWorld trajectory in which broken room connectivity traps the agent. It reports no per-category error rates for inventory, location or object state.
S1 measures one failure mode quantitatively: the "too easy simulator" in which a world model lets a weak agent succeed that fails in the real environment (TextWorld, Qwen3-0.6B: 85/200 false positives with the supervised model, 19/200 after BehR; Qwen3-1.7B: 69 and 34). It also reports, descriptively, that 82-97% of WebShop world-model episodes hit the 50-step cap because the model does not reproduce purchase completion, and one worked TextWorld trajectory in which broken room connectivity traps the agent. It reports no per-category error rates for inventory, location or object state.
Curator qualifications: 'Too easy simulator' failure measured: TextWorld weak-agent false successes 85/200 to 19/200 (Qwen3-0.6B) after BehR; 82-97% of WebShop world-model episodes hit the 50-step cap.
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/abs/2604.13824v2",
"note": "candidate evidence excerpts"
}
]
}Evidence claims
- [sourced_fact] S1 measures one failure mode quantitatively: the "too easy simulator" in which a world model lets a weak agent succeed that fails in the real environment (TextWorld, Qwen3-0.6B: 85/200 false positives with the supervised model, 19/200 after BehR; Qwen3-1.7B: 69 and 34). It also reports, descriptively, that 82-97% of WebShop world-model episodes hit the 50-step cap because the model does not reproduce purchase completion, and one worked TextWorld trajectory in which broken room connectivity traps the agent. It reports no per-category error rates for inventory, location or object state. · locator: candidate evidence excerpts source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |