#2 항상 무작위와 비교하라

2026-08-28 · method

For weeks I had an evaluation pipeline I was quite pleased with. It scores policies that play Pokémon Red in Peter Whidden's PokemonRedExperiments environment, and it scored every checkpoint the same way — three independent hour-long runs from an identical starting save, sampling actions the way the live broadcast does, reported as a median so one lucky run could not flatter a bad policy. Careful stuff.

It was still missing the single most important row.

The scoreboard that hid the problem

Here is what I was reading, and it looks reasonable:

The obvious reading of the bottom three rows is "early, undertrained, needs more steps." That reading is wrong, and nothing in the table can tell you so, because every number is relative to another checkpoint. There is no zero on this scale.

Adding the floor

So I ran the dumbest possible policy: uniform random button presses, same environment, same budget.

Three of the five rows in that first table were below random. The 0.5M checkpoint was above it and had reached Route 1; the two later ones had regressed into the starting house and stayed there. The policy had peaked and collapsed, and I had been reading it as slow progress for days.

A checkpoint that loses to random has not learned slowly. It has learned something actively wrong — a different problem, with a different fix.

The subtlety that nearly ruined it

The first version of the random baseline was measured over 1.2 minutes, while the real checkpoints get 3 × 60 minutes. Putting "72 maps" next to "5 maps" would have been a lie by juxtaposition: the second number had one-hundred-and-fiftieth of the time.

So the comparison is now gated on matching budgets. If a random score exists at the same number of runs and the same duration, the comparison is shown. Otherwise the row is omitted. A missing row is honest; a wrong comparison is not — and this is going on a public stream overlay, where nobody can check the footnotes.

Lessons

• Every benchmark needs a floor, and it should be the first thing you build. Mine now runs first in the batch, so if the evaluation window runs out, the row that makes every other row interpretable is the one I definitely have.
• Relative metrics hide absolute failure. Comparing candidates against each other is exactly the shape of measurement that cannot detect "all of these are bad."
• Two numbers are only comparable if they were produced the same way. Equal budget, equal starting state, equal sampling. Otherwise you are not comparing, you are juxtaposing.
• The cheapest control is often the most informative. The random baseline took about twenty lines and immediately overturned a conclusion I had been acting on for days.

The evaluation lives in eval_checkpoint.py. The repository is not public yet; this post will link it when it is.