#8 We are not running a BOINC server

2026-09-06 · architecture

My training machine is producing about thirty checkpoints a night. My server can score four or five. That gap is the whole reason I spent a weekend reading BOINC's source and documentation, and it is also the reason I decided not to run a BOINC server.

This is the write-up of that decision, because "we looked at the obvious tool and did not use it" is a more useful thing to publish than an announcement would have been.

The arithmetic that started it

Scoring one checkpoint honestly costs three hours: three independent hour-long runs, because a single run of a stochastic policy in a game this long is close to meaningless — the same checkpoint has scored 62 map regions in one run and 31 in the next. My server is two shared cores that also broadcast for eight hours a night, so it has roughly fourteen usable hours a day. Four checkpoints, maybe five.

Last night my training run uploaded fifteen checkpoints in under twelve hours and kept going. The backlog of candidates waiting to be scored went from 28 to 39 overnight. At three hours each that is 117 hours of work queued against a machine that can offer fourteen a day.

There is a mature answer to "I need more CPU than I own, and the work is embarrassingly parallel": volunteer computing. The reference implementation is BOINC, the Berkeley infrastructure behind SETI@home, Einstein@home and Rosetta@home. So the obvious move is to stand up a BOINC project and let people attach to it.

Three reasons I am not doing that

The server does not fit. The BOINC server introduction is blunt about what it needs: "Your server should have 64-bit processors, at least 8 GB of RAM, and at least 40 GB of free disk space." That is on top of a LAMP stack, MySQL, and a handful of daemons. My box has 11 GB total and about 7 GB free while the broadcast is encoding. It would fit only by evicting the thing the machine exists to do.

BOINC distributes the inputs, and my input is a copyrighted ROM. The model is that the project ships the application and its data files to volunteers. Even the newer container path works this way — the docker_wrapper documentation describes mounting the slot directory and optionally the project directory into the container, with no notion of an input the volunteer supplies from their own shelf. I cannot ship a Game Boy ROM to strangers. Every contributor has to bring their own dumped cartridge, which is a requirement BOINC has no shape for.

The audience signed up for science. People attach to BOINC projects because they are folding proteins or searching for pulsars. A project that plays Pokémon is not that, and I would rather not find out how that conversation goes on a volunteer forum. It is also worth noting my own server runs a BOINC client donating idle cycles to Asteroids@home, and that is tolerated on a free cloud tier precisely because it is philanthropic. I would rather not muddy that.

What I took instead

Rejecting the implementation is not the same as rejecting the design. BOINC has had twenty years to discover what a volunteer system needs, and the parts map cleanly onto what I already had:

The client-side half comes more from Stanford's Folding@home than from Berkeley. Its client is configured through a local Web Control page served on the contributor's own machine, with a slider for how much of the computer to give up. That is a better fit for "let people control how much they contribute" than anything I would have designed, and it needs no account and no inbound connection.

The one place BOINC's design does not transfer

BOINC validates by sending the same work unit to several hosts and comparing what comes back. The Debian project's BOINC server guide puts the principle plainly: validation rests on "replication: two independent runs by different individuals should give the same results."

Mine did not. Two independent runs of my evaluation gave different results by design, and that turned out to be true in two separate ways.

The first is deliberate. My evaluation samples from the policy rather than taking its most-likely action, because that is how the broadcast actually calls it; measuring a deterministic policy would measure something that never runs. Fine — but it means results vary.

The second was an oversight I only found by looking for it. There was no seeding anywhere in the evaluation. No seed passed to the environment, none set on the numerical libraries. Worse, the environment's reset() accepted a seed argument, assigned it to self.seed, and never read that value again anywhere in the file. It looked seeded. It was not.

Fixing that was necessary and not sufficient, which is the part worth passing on. My run loop was bounded by wall clock — run for sixty minutes. Two machines of different speeds given the same seed therefore execute a different number of steps and diverge anyway. A unit of work that two strangers can be asked to compute and be checked against each other has to fix the amount of work too.

So a work unit here is not a checkpoint. It is (checkpoint, seed, step count), and with all three pinned the same unit run twice produces byte-identical metrics. I checked: same seed, 1,034 tiles and 15 map regions both times; different seed, 929 and 12.

One caveat I have not solved and am not pretending to: this does not survive a change of CPU architecture, because floating-point operations reorder. BOINC hit that long before I did and has a name for the answer — homogeneous redundancy, meaning you only compare replicas from hosts of the same class. I will compare x86 against x86.

Why only one kind of work is on offer

There are three jobs a contributor could plausibly do: evaluate a checkpoint, contribute a save state, or train and send back a new checkpoint. Only the first two are open, and the reason is not politeness.

A Stable-Baselines3 checkpoint is not data. Loading one runs pickle over its metadata — the policy class and the observation and action spaces are stored that way — and unpickling attacker-controlled bytes executes attacker-controlled code. The Python documentation is unambiguous about this: "it is possible to construct malicious pickle data which will execute arbitrary code during unpickling." My evaluation runs as a user with unrestricted sudo. Accepting a checkpoint from a stranger and scoring it would be remote root.

Evaluation results are numbers in a JSON object. Save states are Game Boy memory dumps, with no code path that interprets them as anything else, and I verify each one by loading it and reading the game's own flags rather than trusting its filename. Those two are safe in a way that does not depend on me being careful. Training is not, and it waits until there is a sandbox worth the name.

Advisory by plumbing, not by policy

The last piece is what a contributed number is allowed to do. It would be easy to say "contributed results are advisory" and then, six months later, quietly let them into the promotion decision because it was convenient.

So contributed results land in a different file from measured ones. They are read at exactly one place: breaking ties between checkpoints that are equally short of runs, so a promising one gets my server's own hours sooner. They never become a stored score, and the routine that decides what goes on the live broadcast cannot see that file at all. Somebody who fabricates a result can make me look at a checkpoint earlier. They cannot make me believe anything about it, because my machine re-scores it and only that measurement counts.

That distinction is deliberate: it is a property of where the data lives, not a rule someone has to keep remembering.

Where this actually is

Honestly: the endpoints are live and nobody has used them. Registration, work handout, checkpoint download and result submission all work end to end, and I have tested them against my own server with a throwaway account. The client that turns that into somebody else's idle CPU is half written. No volunteer has contributed anything, because I have not asked anyone yet.

The reason I am building it at all is not really the backlog — I could fix that by uploading fewer checkpoints. It is that the published study of this environment reports that no agent obtained HM01, the item gating the third gym, and my own runs have now reached it six times and taught the move three times. Those are the best of 129 runs; the median still stops around the first gym. Rare success is exactly the thing more machines produce more of.

The client is public and auditable at github.com/HughMungis/poke-at-home — the worker, the container, the environment and the design document. The server is not, and that is deliberate rather than defensive: it is one process that also serves this site's authenticated admin surface, so publishing it would expose a great deal that has nothing to do with volunteer computing. None of the security rests on that, though. The endpoints are internet-facing and the client documents every one of them, so anything that only worked while the server source stayed secret would not be worth relying on.