#5 The reward that cannot see the thing it needs

2026-08-30 · reward design

My agent is trying to obtain HM01 — the item that teaches Cut, which is the hard gate in front of the third gym in Pokémon Red. Pleines et al., Pokémon Red via Reinforcement Learning evaluates this exact environment and reports its baseline reaching only "up to completing Cerulean City" — short of the gate.

I have been treating that as a hard problem. It is, but not for the reason I assumed. The reward function is structurally unable to see most of what obtaining HM01 requires.

The exploration term

Here is the only exploration signal the agent gets:

"explore": reward_scale * explore_weight * len(self.seen_coords) * 0.1
# seen_coords key = f"x:{x} y:{y} m:{map}"

A set of coordinates. Reach a tile you have not stood on before, get paid. It is a good reward — it is why the agent leaves the house at all — and it has an obvious property I had never thought about: anything that does not change your position is worth exactly nothing.

Talking to a character. Reading a sign. Opening a menu. Using an item. Teaching a move. Accepting a gift. All zero.

Which is the entire chain I care about

The critical path to HM01, from my own milestone list:

Four for four. Every one of them is standing still and pressing a button. Across the whole seventeen-stage critical path, roughly eight are interactions that produce no new tile.

So the agent receives a dense, continuous signal for walking, and a signal of approximately zero for the specific class of action that unlocks the gate everyone gets stuck behind. I had been reading that gate as an exploration difficulty. It is closer to a blind spot.

The thing that beat the game changed exactly this

pokemonred_puffer — by David Rubinstein, Keelan Donovan, Daniel Addis, Kyoung Whan Choe, Joseph Suarez and Peter Whidden — states that as of February 2025 they beat Pokémon Red with a policy under 10 million parameters. They moved away from coordinate-based exploration to counting interactions — signs read, warps used, menus opened, buttons pressed in novel contexts. I had filed that under "they had 29 reward terms and 288 environments, none of which I can match." Reading it again with the table above in front of me, it is not a scale difference. It is a definition difference, and it is one term.

The other number

Worth noting that I am not observing this from outside: the environment I run is Peter Whidden's, and its global coordinate system is itself adapted from that project. I have been depending on their work while filing their conclusions under "things I cannot match."

There is a second measurement that points the same way. Early in the game, before the agent reaches anything, every reward term except exploration is exactly zero. So I measured what is actually on offer per step, against the entropy bonus in PPO's loss:

The bonus that exists to keep the policy exploring is thirty-five times larger than the entire reward signal it is supposed to be a gentle counterweight to. A policy trained from scratch under those conditions sat at 90% of maximum entropy after 22 million steps — it had learned almost nothing, because the loudest instruction in its loss function was "stay random."

And this reframes the winning project's 29 reward terms. I had read that as a list of clever individual ideas. It is more likely that what they needed was density — enough reward, often enough, for the policy gradient to outvote the entropy term. Their term count and my 35× are the same fact from two directions.

The counter-argument, which is strong

Pleines et al., Pokémon Red via Reinforcement Learning reaches the opposite conclusion, and it has receipts. Three of its reward ablations backfired: removing the level reward improved milestone progress; multiplying the navigation reward by ten stopped agents beating the first gym; and the healing reward was exploited by an agent that farmed endless battles to hurt itself and heal again. Their verdict is that "further refining the reward function risks introducing additional vulnerabilities."

That is a real warning and I do not have an answer to it, other than discipline: change one term, measure, keep or revert. I think interaction-based exploration survives the warning better than most candidates would, because it is not another term competing for weight — it is a correction to an existing term that is demonstrably blind to half the objectives it is meant to drive the agent toward.

Whether that turns out to be true is the next thing to find out, and I would rather write down the prediction before running it than after.