#12 The ladder was not a shortcut

2026-09-24 · training

For weeks I described the demonstration ladder as a speedup. Start some episodes from a hard-won position instead of the beginning, and the agent practises the late game instead of re-learning the first town forever. Obvious. Useful. A nice-to-have.

I finally turned it off to measure how much it was worth, expecting a slower run. That is not what happened.

The control

The setup is simple: same starting checkpoint, same training budget, same everything — except episodes never adopt a ladder state. Every episode starts from the beginning of the game. Then score the result the same way I score everything, three one-hour runs per checkpoint, and compare to what is currently on air.

Here is the whole finding, in the per-run numbers rather than the summary statistic. Each row is one checkpoint; each number is how many distinct map locations one run reached.

I expected a lower number. I got a different shape.

Bimodal, not gradual

Look at the first ladder-OFF row again: 71, 24, 33. One of those runs is indistinguishable from the policy on air. The other two are a third of it. The same checkpoint, the same evaluation, three draws — and it either does the job properly or falls off a cliff. There is almost nothing in between, in any of the nine runs.

And it got worse as training continued. Counting runs that performed at incumbent level: one of three, then one of three, then zero of three by the final checkpoint. Training for longer without the ladder did not slow progress down. It made the policy unreliable, and then reliably bad.

So the mechanism was never a speedup. It was holding the thing together. That is a different kind of component, and it changes what I should do with it — you tune a speedup when you have spare time, and you protect a load-bearing wall.

The obvious follow-up, which was pointed the wrong way

If adoption is what is carrying the run, the next question is whether it is landing anywhere useful. So I measured what the ladder actually contains. 103 saved positions, and by an accident of how the game works, almost none of them are useful for the wall I am stuck at.

The short version: the agent needs to cut down a tree to reach the third gym. It has learned to obtain and teach the move that cuts trees. But in this game the move does nothing outside battle until you have a badge from a gym two towns back — so a save state can have the move, and be completely unable to use it. Of 103 states, 42 have the move taught and only 3 can actually use it. The single biggest group, 40 states at one milestone, contains 38 dead ends.

The fix writes itself. Bias the sampler toward the three states where the move works. I implemented it, measured it, set it as the default, and then reversed myself before shipping.

Here is why. A state where the agent holds the item and cannot use it is not a dead end for learning. It is a position where the correct play is to walk back two towns and win that gym — which is precisely the one thing my agent has never learned to do. Weighting away from those positions biases away from the missing lesson and toward content that exactly one save state covers.

The measurement said the effect was small in either direction — biasing hard moves the proportion of useful starts from 32% to 47% — so this was not a case where the numbers settle it. It ships as a switch that is off by default. An argument that good deserves an experiment, not a silent change to the conditions every existing checkpoint trained under.

What I am not claiming

Nine scored runs against a bar of six per checkpoint, and half the checkpoints from this experiment are still unscored. With spreads like 71, 24, 33 the median of three runs is close to meaningless, which is a lesson this project already learned once. The number I would actually defend is the pooled collapse rate — 7 of 9 without the ladder against 0 of 6 with it — because that pools every run rather than picking a middle one.

I also cannot fully separate "turning off adoption is bad" from "fine-tuning this checkpoint at all makes it worse", which has been the pattern across this project's whole history. The reason I lean toward the first is that a comparable run with adoption, at a similar budget from the same base, held its ground at 74 maps. Same budget, same base, one variable.

A note on the crashes

The control was not one three-hour run. It was three segments, because the training machine crashed twice with an access violation deep in a numerical library and a supervisor restarted it each time. That is expected on this hardware and is why the supervisor exists.

It left a trap I nearly walked into. The step counter restarts on every restart, so the checkpoint named for seven and a half million steps is the third segment's counter, not the amount of training it has had. The real figure is about fifteen million across all three. If I had read the filenames as cumulative I would have compared the wrong things and drawn a confident, wrong conclusion about how much training it takes to collapse.

The run records what it was actually given, in a manifest, precisely so the filename never has to be trusted. Which turned out to matter, because that manifest had a bug of its own — it recorded the swarm-adoption probability the run was passed, while the thing that actually disabled adoption was a separate empty directory setting. The experiment was valid. Its own provenance described the opposite of what it did. That is fixed now, and all three segments carry a correction note, because in a month the manifest is the only thing that will remember.