If an LLM updates the numbers (state variables) of a game or story each turn, how often did it get the update wrong in published tests?
RPGBench (arXiv 2502.00595 v1) had LLMs act as an RPG engine for 10 rounds: they planned events, narrated, and wrote updated state variables. Scripts, not an LLM judge, checked event conditions and variable updates. The best model, Gemini 2.0 Flash Exp, had 76.5% of rounds free of mechanical errors; GPT-4o 69.3%; the lowest were 11.3-20.4% (Claude 3.5 Sonnet, GPT-4o mini, Llama 3.1/3.3 70B). Variable-update error rates per variable ran from 3.4% to 30.8%. Accuracy fell as rounds increased for GPT-4o (69.3% at 10 rounds to 66.8% at 25). Combined with Wang et al. (-02), the sources support expecting frequent state errors unless a script checks them.
RPGBench (arXiv 2502.00595 v1) had LLMs act as an RPG engine for 10 rounds: they planned events, narrated, and wrote updated state variables. Scripts, not an LLM judge, checked event conditions and variable updates. The best model, Gemini 2.0 Flash Exp, had 76.5% of rounds free of mechanical errors; GPT-4o 69.3%; the lowest were 11.3-20.4% (Claude 3.5 Sonnet, GPT-4o mini, Llama 3.1/3.3 70B). Variable-update error rates per variable ran from 3.4% to 30.8%. Accuracy fell as rounds increased for GPT-4o (69.3% at 10 rounds to 66.8% at 25). Combined with Wang et al. (-02), the sources support expecting frequent state errors unless a script checks them.
Curator qualifications: Verified: the text's 'perform comparably well' for Gemini 2.0 Flash Exp, GPT-4o and Gemini 1.5 Pro is not borne out by Table 3 (0.765, 0.693, 0.554).
Evidence status: statements located in the cited sources by a correlated desk review (Claude); not executed; not independently reviewed.
Structured fields
{
"citations": [
{
"url": "https://arxiv.org/abs/2502.00595v1",
"note": "RPGBench text vs Table 3"
},
{
"url": "https://arxiv.org/abs/2406.06485v1",
"note": "RPGBench text vs Table 3"
}
]
}Evidence claims
- [sourced_fact] RPGBench (arXiv 2502.00595 v1) had LLMs act as an RPG engine for 10 rounds: they planned events, narrated, and wrote updated state variables. Scripts, not an LLM judge, checked event conditions and variable updates. The best model, Gemini 2.0 Flash Exp, had 76.5% of rounds free of mechanical errors; GPT-4o 69.3%; the lowest were 11.3-20.4% (Claude 3.5 Sonnet, GPT-4o mini, Llama 3.1/3.3 70B). Variable-update error rates per variable ran from 3.4% to 30.8%. Accuracy fell as rounds increased for GPT-4o (69.3% at 10 rounds to 66.8% at 25). Combined with Wang et al. (-02), the sources support expecting frequent state errors unless a script checks them. · locator: RPGBench text vs Table 3 source
Rights
| component | license | hosting | redistribution | modification | training | Evidence and notes |
|---|---|---|---|---|---|---|
| documentation | unknown | permitted | permitted | unknown | unknown | Applies to this original summary only, not to the cited sources. |