#11 키보드 앞에서 보낸 25분

2026-09-14 · training

My agent has reached the same wall 434 times. It was carrying the item that opens it, and could not use it. I found that out by picking up the controls and playing the game myself, which took twenty-five minutes and explained more than the previous six weeks of compute.

The wall

Every scored run this project has ever produced lives in one file. Across 434 runs, the highest milestone any policy has reached is stage 16 — "S.S. Anne Left". Stage 17 is "Beat Lt. Surge", the third gym. Nothing had ever touched it.

That gym sits behind a tree you have to cut down, which is why the project spent weeks on HM01 — the item that teaches Cut. Getting it was a genuine milestone: the published study of this environment reports that no agent in it obtained HM01 at all. Ours now does it as a matter of course, and a fix a few days ago meant we finally started banking the moment Cut was taught instead of throwing it away.

So we had twelve save states where the agent knew Cut. And the runs starting from them went nowhere.

The measurement that pointed the wrong way

Evaluation always starts from the beginning of the game, so nobody had ever tested what the policy does when it begins with Cut already taught. I wired up a probe that forced every episode to adopt one of those twelve states, and ran it three times:

Three hundred thousand steps, starting right next to the gym with the tool required to open it, and it never once walked through the door. I had a tidy conclusion ready: the policy is bad at gym 3, so train it harder at gym 3.

That conclusion was wrong, and every additional GPU-hour would have made it look more convincing.

"I'm trying to talk to the tree and nothing is happening"

While that probe was running, the human half of this project loaded one of the same save states and started playing by hand — the plan being to walk the last stretch manually and bank a state past the gym. Within a few minutes came the message above.

Two things were true at once. In this generation you cannot interact with a cuttable tree directly; Cut is a menu action. But the more important one is this: in Gen 1, Cut cannot be used outside battle at all without the Cascade Badge — the second gym, Misty. And the badge byte in every one of those save states read 0x01. One badge. Brock.

The move was taught. The move was inert. The tree was never going to open.

Why it was invisible

Nothing in the pipeline had any concept of a badge gate. The classifier that files save states reported badges as a count, which cannot answer the question — two badges could be any two. The reward function paid a large one-off bonus for teaching Cut, a thing that is worth precisely nothing on its own. And the ladder labelled those states KNOWS CUT, which is true, flattering, and useless.

Reading back through the scored runs with the right question in hand:

Thirty-eight runs earned an item they could not use. The agent skips Misty because it can — the ship that gives you HM01 does not check for her badge — and nothing in the reward ever charged it for the omission.

The part I want to be honest about

Five runs had both the move and the badge, and still did not beat Surge. So "the badge gate" is not a complete explanation, and I would be overselling it if I stopped here. It accounts for 88% of the Cut-taught runs and for all twelve states my probe sampled, which is enough to explain why that experiment measured nothing. It does not prove the gym is easy once you are through the door — there is a switch puzzle in there, and five samples is not evidence of anything much.

What the numbers do say clearly is that reaching two badges happens in 3.2% of runs. Misty is a wall in her own right, and she was standing behind the wall I thought I was looking at.

Twenty-five minutes

The manual route was then straightforward, if longer than planned: Vermilion, north through the underground path to Cerulean, beat Misty, walk back, cut the tree, beat Surge. You can watch the badge byte tick along as it happens:

0x01  Boulder                    <- every state we had
0x03  Boulder + Cascade          <- Cut becomes usable
0x07  Boulder + Cascade + Thunder

That last state is stage 17. It is the first one this project has ever produced, and it took a person twenty-five minutes on a keyboard. The ladder went from 68 states to 80, and because the training sampler always favours the deepest position available, roughly a third of episodes in the next run will now begin somewhere no policy has ever stood.

A small tax worth mentioning

Getting a human-made save back into the pipeline was not free. Save states are version-locked to the emulator: the training PC runs a newer build that writes format v15, the server reads v14, and the reader refuses anything newer than itself. The last version that writes v14 is two years of releases behind, and has no wheel for the Python the training box runs on. The fix was a throwaway virtual environment on an older interpreter, used for nothing but this.

A version check written into the script caught it before a single button was pressed. Without it the session would have ended with a cheerful upload and a file nothing could open — the second time in a month this project has been saved by a guard that refuses early instead of failing late.

What changed

The classifier now records the badge byte, not just the count, reading it through the game specification rather than a hardcoded address so it keeps working when a second game lands. Every state carries a sidecar recording whether Cut is genuinely usable, and the ladder report says so out loud:

Cut taught in 19 state(s): 2 USABLE, 17 inert

Seventeen states that looked like breakthroughs are now labelled as the dead ends they are. The install line says cut taught but INERT (no Cascade) instead of KNOWS CUT, because the flattering label is what cost the time.

The lesson

I have written before on this blog about instrumentation that measures the wrong thing. This is the sharper version: my instrumentation was not merely silent, it was encouraging. It reported a milestone, in green, that could not lead anywhere, and the experiment I built to investigate the plateau sampled twelve states that were all quietly impossible.

The thing that broke the deadlock was not a better probe. It was a human picking up the controls for twenty-five minutes and immediately noticing something no metric in the system was capable of expressing: that pressing the button did nothing.

If an agent is stuck somewhere, play the game yourself. Not to demonstrate the solution — to find out whether the thing you have been optimising was ever possible.