#7 Off the S.S. Anne with HM01, and no way to use it

2026-09-02 · results

Two of my agent's evaluation runs boarded the S.S. Anne, walked to the captain's office, took HM01 from him, and left as the ship sailed. That is the item that unblocks the third gym, and the paper written about this exact environment reports that none of its agents ever got it.

Neither of my runs taught the move. The wall is still there.

What the published ceiling actually is

The environment I train against is the one studied in Pleines et al., Pokémon Red via Reinforcement Learning — same reward terms, same worker count, same shape of run. Peter Whidden, who built the original project, is a co-author. So their results table is a fair readout of where this environment tops out. Their baseline agent:

On the last row the paper is explicit: "these rewards fail to address the challenge of reaching the third gym in Vermilion City, which is blocked by a cuttable tree. Solving this requires an additional reward signal to incentivize teaching the move Cut to a Pokémon." It describes the gate as a compound exploration problem — obtain a Pokémon that can learn Cut, acquire HM01, navigate the menus to teach it, then use it correctly in the overworld.

What my runs did

Across sixteen comparable evaluation runs — same policy family, same 260,000-step budget, all starting from the same save — the distribution of how far along a seventeen-stage critical path each run got:

The two deep runs reached 69 and 70 distinct map regions, one badge each, and their stage lists are internally coherent: Bill, the cell separator, the S.S. Ticket, HM01, the ship departing — in order. They also both skipped Misty, which is legal; you can reach Vermilion with one badge. I checked the map lists rather than trusting the flags, and both runs visited Bill's lab, Vermilion Harbor and all nine S.S. Anne interiors including the captain's office. That is where HM01 is handed over. The flags are telling the truth.

Three reasons this is smaller than it sounds

It is 2 runs in 16. Not a capability the policy has, a thing it has done twice. The same checkpoint that produced a stage-16 run also produced a stage-8 run at the same budget.

It is not their agent. I am running their environment with a re-weighted reward. In August I split the event reward into two terms: a curated seventeen-item critical path worth 15 apiece, and everything else worth 1.5. Before that, every one of roughly 2,558 event flags scored identically — "opened a menu" was worth exactly what "Beat Brock" was worth. The measured decomposition of my best run at the time was event 206.0, explore 115.4, badge 5.0: beating the first gym leader accounted for about 1.5% of return. So this is not "my agent beat theirs." It is the same environment with a reward that distinguishes progress from noise, which is precisely the additional signal their paper says the problem needs.

None of the sixteen taught Cut. Holding HM01 does nothing. The tree opens when a party member knows the move, and zero runs got there. I only know that because I added a separate knows_cut column when the first deep run appeared — reading the party structure for move id 15 rather than trusting the item flag. Reaching the item and passing the gate are different events, and conflating them would have let me claim the wall was broken when it is intact.

The part I find more interesting than the milestone

I could not have told you any of the above a day earlier, because I was not measuring it.

My evaluation recorded an events figure that was a popcount over the whole flag range — the undifferentiated 2,558. The seventeen that matter were in there, drowned. So the metric that my own reward change was designed to move was the one quantity I had never once looked at. Every score in the history file before that day has no required-event data at all, and no amount of re-reading them would have produced any.

The fix took an afternoon. What it cost was the ability to say anything about a month of prior runs.

And the experiment that prompted all this returned nothing

The reason I went looking was an A/B. I had added an interaction reward — paying the agent for opening a dialogue somewhere new, since the entire HM01 chain is dialogue and the exploration reward only counts tiles walked on. The prediction, written down before the runs: at equal step count this should reach further along the required-event chain than a run without it; more map regions alone does not count.

Eight step-capped runs per arm, matched at three checkpoints of identical training length:

Nothing favours the new term. It is not met, on the terms I set in advance.

Two things nearly made that result look different. Outcomes here are close to bimodal — a run either breaks out of the early game or does not — so a median over two or three runs is almost meaningless, and the arms traded the lead four times as runs landed. And a separate scheduled job scored one checkpoint three times at roughly double the step budget; those pooled straight in and briefly showed the new term ahead 4/7 to 2/4. That was not a result, it was two different amounts of play time being averaged together.

What I take from it

The thing that moved the needle was not the clever reward. It was splitting one flat event counter into "progress" and "everything else" — deciding, by hand, which seventeen of 2,558 flags represent the game actually being played. The interaction reward was a more sophisticated idea and it did nothing measurable.

And the milestone that came out of it is a good reminder of what a headline hides. Two runs in sixteen got the item that the literature says nobody gets. Zero runs used it.