“Research Notes: Running Claude 3.7, Gemini 2.5 Pro, and o3 on Pokémon Red” by Julian Bradshaw
LessWrong (30+ Karma)
Shownotes Transcript
Disclaimer: this post was not written by me, but by a friend who wishes to remain anonymous. I did some editing, however.
So a recent post my friend wrote has made the point quite clearly (I hope) that LLM performance on the simple task of playing and winning a game of Pokémon Red is highly dependent on the scaffold and tooling provided. In a way, this is not surprising—the scaffold is there to address limitations in what the model can do, and paper over things like lack of long-term context, executive function, etc.
But the thing is, I thought I knew that, and then I actually tried to run Pokémon Red.
** A Casual Research Narrative**
The underlying code is the basic Claude scaffold provided by David Hershey of Anthropic.[1] I first simply let Claude 3.7 run on it for a bit, making observations about what I thought might generate [...]
Outline:
(01:04) A Casual Research Narrative
(09:10) An only somewhat sorted list of observations about all this
(09:22) Model Vision of Pokémon Red is Bad. Really Bad.
(12:58) Models Cant Remember
(14:34) Spatial Reasoning? Whats that?
(15:22) A Grasp on Reality
(18:40) Costs
(19:13) Why do this at all?
(19:47) So which model is better?
(22:36) Miscellanea: ClaudePlaysPokemon Derp Anecdotes
The original text contained 6 footnotes which were omitted from this narration.
First published: April 21st, 2025
Source: https://www.lesswrong.com/posts/8aPyKyRrMAQatFSnG/untitled-draft-x7cc)
---
Narrated by TYPE III AUDIO).
Images from the article:
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts), or another podcast app.