RAP reports large planning gains from an LLM world model while Wang et al. find LLM state prediction unreliable; can the two be reconciled from the papers alone?

The two papers do not contradict each other because they measure different things on different tasks with different models, but the papers alone cannot say why RAP works. RAP (LLaMA-33B, frozen) reports plan success of 0.88 at 4 steps and 0.42 at 6 steps with 20 MCTS iterations on Blocksworld. LLM-Sim (GPT-4) reports 49.7-77.1% best-case on changing transitions of a different set of games. Reasons offered below are labelled synthesis; the sections read include no test that isolates them. Curator qualifications: Excerpts present (E1, E3-E5 text; E2 and E6 in layout). Curator arithmetic: RAP(20) case-weighted (30, 57, 114 cases) = 0.637, CoT GPT-4 = 0.480, ratio 1.327, matching the paper's 'average success rate of 64%' and '33% relative improvement' (the weighting is inferred, not stated). Table 5 layout: (R1,R2) 0.88; (R1,R2,R3) 0.91; R1 0.46; R2 0.21; R3 0.14; none 0.02; caption names no subset or iteration count; its 0.88 equals Table 1's 4-step RAP(20) value, suggesting (not establishing) a 4-step subset, which would reconcile the CE with the 0.64 average. LLM-Sim: 'the LLM is provided with the ground truth previous state ... the LLM always performs a single-step prediction'; 'At best, GPT-4 is able to simulate 77.1% of dynamic action-driven transitions correctly. In contrast, GPT-4 simulates at most 49.7% of dynamic environment-driven transitions correctly.' Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.

{
  "citations": [
    {
      "url": "https://arxiv.org/abs/2305.14992v2",
      "note": "RAP v2 PDF: Sec. 4.1 case counts, Table 1, abstract/intro (64%, 33%), Table 5 (layout lines 1148-1155); LLM-Sim arXiv 2406.06485v1 PDF (equals pin): Sec. 2.3, Sec. 4 results; case-weighted averages"
    },
    {
      "url": "https://arxiv.org/abs/2406.06485v1",
      "note": "RAP v2 PDF: Sec. 4.1 case counts, Table 1, abstract/intro (64%, 33%), Table 5 (layout lines 1148-1155); LLM-Sim arXiv 2406.06485v1 PDF (equals pin): Sec. 2.3, Sec. 4 results; case-weighted averages"
    },
    {
      "url": "https://arxiv.org/abs/2305.14909v2",
      "note": "RAP v2 PDF: Sec. 4.1 case counts, Table 1, abstract/intro (64%, 33%), Table 5 (layout lines 1148-1155); LLM-Sim arXiv 2406.06485v1 PDF (equals pin): Sec. 2.3, Sec. 4 results; case-weighted averages"
    }
  ]
}