#1 The AI learned to walk into walls, then unlearned it

2026-08-28 · training

Every night an AI plays Pokémon Red on this channel, unattended, while I sleep. It runs in Peter Whidden's PokemonRedExperiments environment, trained with Stable-Baselines3 — a PPO policy, a neural network that learns by trying things and being rewarded for the ones that work out. This week I changed how it is rewarded, retrained it from scratch, and it got dramatically worse. Here is the whole thing, with the numbers.

The change

The environment counts about 2,558 "event flags" — bits the game sets when something happens. Talking to your mum sets one. Beating the first gym leader sets one. The original reward treated every single one identically, which means beating Brock scored the same as opening a menu.

Decomposing the reward on the best checkpoint made that concrete:

Beating the first gym was worth about two and a half generic flags. So I hand-authored a 17-stage critical path through the game — get the starter, beat the rival, beat Brock, reach Bill, get HM01 — and rewarded those at roughly ten times a generic event, which is what the project that actually completed the game does.

What happened

The retrained policy could not leave the first room. Not in a 60-minute evaluation, not in 625,000 steps. So I compared it against a control I should have had from the beginning: a policy that presses buttons completely at random.

Random walks out of the house, crosses Pallet Town, and reaches Oak's Lab. The trained policy sat in the bedroom. It was two and a half times worse than doing nothing.

How it failed, exactly

The action distribution gave it away. With seven buttons, uniform is about 14% each:

DOWN=9%   LEFT=3%   RIGHT=25%   UP=24%   A=17%   B=13%   START=5%

It had learned to press into the bedroom walls. Leaving requires walking down the stairs — DOWN was the action it had suppressed hardest, along with LEFT.

This is not an undertrained policy. Undertrained looks like random. This is a policy that learned something confidently wrong.

The likely mechanism is a trap in my own reward change. I cut the generic event reward by 62% to make room for the new critical-path rewards — but every one of those new rewards is unreachable from inside the starting house. A fresh policy lives entirely in the part of the game where I had just made the signal weaker. It optimised what was left, and what was left did not include leaving.

It climbed out

The tempting move was to change the reward again immediately. I did not, because the run was still going — and at 18 million steps it looked like this:

And the action distribution had repaired itself:

@ 1.47M   DOWN=9%    LEFT=3%    RIGHT=25%   UP=24%
@ 18M     DOWN=18%   LEFT=15%   RIGHT=15%   UP=15%

DOWN went from 9% to 18%. It stopped mashing itself into the wall and started preferring the action that leaves the room. It got out of the house and reached Oak's Lab, and it is now half again better than random per step where it had been two and a half times worse.

Why it could escape

PPO carries an entropy bonus — a small, constant pressure against the action distribution collapsing to certainty. Even while the policy was badly skewed it kept occasionally sampling DOWN, and every time it stumbled out of the house it collected exploration reward that pulled the gradient back the right way. The collapse was a local optimum with a leak in it, not a dead end.

Lessons

• A learning curve is not monotonic, and "worse" is not the same as "broken." Had I retrained at 1.47M steps I would have thrown away a run that was already recovering.
• Reward changes have a phase. Strengthening late-game signal at the cost of early-game signal punishes exactly the part of the game a fresh policy is stuck in. If I do it again I will add the new rewards on top rather than rebalancing to keep a constant total.
• Behaviour is more legible than score. "2 maps" reads as undertrained. The action histogram — DOWN at 3% — says precisely what went wrong, and it took ten seconds to produce.
• Wait for the data you are already collecting. The run that answered the question was running the entire time I was theorising about it.

It is still well behind the reference checkpoint — about seven times fewer tiles per step, at 18M steps against 26.2M. Whether the critical-path reward is a net improvement is genuinely undecided. But it is no longer walking into walls, and I can measure the difference.

The code is not public yet — it will be, and this post will link it when it is.