#3 학습을 조용히 망가뜨린 추론용 설정

2026-08-29 · debugging

I had a checkpoint that played Pokémon Red reasonably well — one badge, seventy-odd map regions in an hour. I wanted to fine-tune it against a reworked reward function. Every attempt came out worse than what I started with, and it took me two days and three wrong theories to find out why.

The cause was a single number I had copied out of the Stable-Baselines3 documentation, where it is correct, into a place where it is not.

What the collapse looked like

Each of these is a fine-tune of the same starting checkpoint, scored the same way — a ten-minute run, counting distinct map regions reached:

My first reading was "there is a sweet spot around 10M steps, stop there." That reading was also wrong, and I will come back to it.

Three theories, all wrong

Entropy collapse. The classic PPO failure: the policy commits to one action and stops exploring. I measured policy entropy on a fixed batch of observations put to every checkpoint. It was the opposite. Entropy was rising, not falling — 1.42 nats at the start, 1.53 to 1.61 after fine-tuning, against a maximum of 1.95 for seven actions. The policies were becoming less decisive, not more.

I noticed this, wrote "entropy drifted up from the base, worth watching," and moved on. That was the answer, and I walked past it.

Catastrophic forgetting. I was training with a feature that starts some episodes from mid-game save states, so perhaps the policy was simply forgetting the opening. I built a test comparing each checkpoint from the start of the game and from mid-game. The test was worthless in two independent ways: it ran 2,000 steps where the real evaluation runs 100,000, and "tiles explored" from two different regions of the map is not a comparable quantity anyway. Its own output disagreed with the real evaluation by a factor of thirteen, which is how I knew to throw it away rather than believe it.

The mid-game save state feature itself. This was the obvious suspect: it was new, and the collapse was new. So I ran the control — five hours of identical training with the feature disabled.

Both collapse. The obvious suspect was innocent, and five hours of GPU time bought the single most useful fact of the week — because it also killed the "sweet spot at 10M" story. The 67 in that column was luck. The control's 10M scored 15.

The actual bug

With the feature exonerated, the only thing left was fine-tuning itself, which sent me to the one part of the pipeline I had never questioned — the line that loads the checkpoint:

model = PPO.load(base, env=env, custom_objects={
    'lr_schedule': 0, 'clip_range': 0, 'tensorboard_log': None})

Stable-Baselines3 documents custom_objects as the way to skip objects that will not deserialise — typically when a model was saved by a different library version. That exact dictionary is not something SB3 publishes as a recipe; it is a widely circulated community pattern for getting an old checkpoint to load. Used for continued training it does exactly what it says on the tin: it sets the clip range to zero.

PPO's objective is min(ratio·A, clamp(ratio, 1−c, 1+c)·A). With c = 0 the clamp collapses to the constant 1.0, so for the half of cases where the clipped branch wins, the objective is a constant — no gradient at all. The trust region is closed at ratio 1, and the policy-gradient term contributes almost nothing.

Meanwhile the entropy bonus is not clipped. So the loss still has one healthy, unopposed gradient in it, and that gradient points at maximum entropy.

Which predicts precisely what I had measured and ignored two days earlier: a fine-tuned policy that drifts up in entropy and loses its competence.

Checking it could be wrong

A theory that explains everything after the fact is cheap. So before acting on it I ran the prediction forwards: a checkpoint trained with clip_range = 0 should be degraded. If it came back healthy, my diagnosis was wrong and the plan built on it was worthless.

It came back at 21 map regions and no badge. The theory survived a test it could have failed.

The fix, and the guard

The load now passes real schedules, and — more importantly — refuses to start if either is zero:

_cr = model.clip_range(1.0)
_lr = model.lr_schedule(1.0)
if not (_cr > 0 and _lr > 0):
    raise SystemExit(f"refusing to train: clip_range={_cr} lr={_lr}")

A silently-zero clip range is invisible in the logs. It does not raise, it does not warn, and the run looks completely normal for eight hours. The only symptom is that the answer is wrong. That is exactly the class of thing that should be an assertion rather than something I remember.

What I would take from it

Nothing here was wrong, exactly. The documented behaviour is accurate and the pattern does what it is meant to do. I moved it to a different job — from loading a model to continuing to train one — and never re-checked whether its assumptions still held.

And the sequencing matters more than the bug. I would not have found it by staring at the load path, because I had read that line a dozen times without seeing it. I found it by spending five hours proving the obvious suspect innocent. A control run that returns "not this" is not a wasted run; it is the one that tells you where to look next.