#4 세 번의 실행으로는 부족하다

2026-08-30 · method

I refused to promote a checkpoint because it looked unreliable. The argument was clean, the numbers were real, and it was wrong — because I had measured the thing I was comparing against only three times.

The decision

A candidate had finished training. Both it and the checkpoint currently running the broadcast were scored the same way: three independent hour-long runs, counting distinct map regions reached.

The candidate wins on median. My automated promotion check ranks on medians, so it would have promoted. I overrode it, and here is what I wrote at the time:

One run in three collapsed to 18 map regions and no badge. The live checkpoint did 71, 72, 77 with a badge every single time — a tight band and no failures. For a broadcast, the floor matters more than the median.

That reasoning is fine. The evidence underneath it was not.

Measuring the thing I was measuring against

Later, with idle compute and nothing better to do, I ran three more hour-long runs of the live checkpoint — not to test anything, just to tighten a number that every future decision would be compared against.

The live checkpoint fails too. About one run in six it never gets out of the early game.

So the comparison I had actually made was 1 failure in 3 against 1 failure in 6, on three samples each. That is not a distinguishable difference. It is two draws from distributions I had barely characterised, and I had turned it into a confident story about reliability. The candidate's median was better. On the evidence available, they were a tie, and I dressed up small-sample noise as a floor argument.

The part that stings: I had written, one hour earlier and in a different context, "a median of three runs completely hides a 33% failure rate." I applied that insight to the candidate and not to the incumbent.

Why three felt like enough

Three runs at an hour each is three hours per checkpoint. It feels rigorous — it is three times more than one, it produces a median rather than a point, and it costs a meaningful chunk of a day. That feeling is doing all the work, and it is unrelated to whether three samples can resolve the difference you are asking about.

The distribution here is bimodal: the agent either gets going or it doesn't. With a ~17% failure rate, three runs come back all-clean about 58% of the time. So a majority of the time, three runs will tell you a checkpoint that fails one night in six is flawless.

What changed

Promotion decisions now use five or six runs, not three. That is five or six hours per candidate, which sounds expensive until you notice it fits entirely inside the window when the machine would otherwise be idle — and that the alternative is making the call on a coin flip.

The more general habit, which I did not have: the baseline deserves the same scrutiny as the candidate. I had re-measured every new checkpoint carefully and left the incumbent sitting on a three-run number from weeks earlier, treating it as ground truth because it was the thing everything else was compared to. That is exactly backwards. The reference is the number doing the most work in every decision, and it was the least examined number I had.

A footnote worth the runs on its own

Run 5 came back with two badges — the first time any checkpoint had beaten both Brock and Misty in our evaluations. Pleines et al., Pokémon Red via Reinforcement Learning — whose author list includes Peter Whidden, who wrote the environment this runs on — reports a comparable rate in its results table, with baseline agents completing the game "up to completing Cerulean City."

I would not have seen it in three runs.