One AI service, made from a house full of ordinary computers
A main workstation, three mini PCs, two handheld gaming PCs and an older laptop with a small graphics card. Separately, each is a computer. Together they answer chat, vision, speech and image requests, run each other's test suites, share a graphics card politely and report their own vitals. These notes cover how, and the mistakes that taught most of it.
01Why bother
Local AI on one PC hits a wall quickly: one graphics card, one model at a time, and every application wired to its own address. Meanwhile the rest of the house was full of capable machines doing nothing in particular — and one of them spends its evenings showing films.
A fleet only helps if nobody has to think about it. An app should ask for "a transcription" or "a quick answer", not for a specific machine; a long job should find an idle computer by itself; and a person sitting at a machine should never notice that it is also working for the others. Everything below exists to make that true, and most of it was learned by measuring an assumption and watching it fall over.
02The map
03One front door
Every application in the house talks to a single OpenAI-compatible address on the main workstation. Behind it sit lanes, one per kind of work:
Each reply says which machine served it, so a slow answer is a fact about a machine rather than a mystery. When every member of a lane is busy, the door says so plainly — "busy, try again in three seconds" — instead of letting a request queue forever. Speech is the exception, because people are waiting on it: a clip waits up to thirty seconds for a free machine, and if the first machine is slow, the same clip goes to a second and the first good answer wins.
04Who gets the work
The router has a few rules, and every one of them came from a measurement:
- Quick requests go to the fastest machine that clears a quality bar; deep requests go to the most capable model, wherever it is. The big model answers hard questions; a small one on a handheld answers "what's the capital of Peru".
- A machine with the model already loaded beats one that would have to load it. Loading is the slowest thing in the building.
- The home network comes first. The same machine answered in 4.5 ms at home and 574 ms through the tunnel.
- Speeds are measured, never assumed. Every machine is re-benchmarked when its number is missing or more than an hour old, and a job's cost is fitted as a fixed start-up cost plus a cost per unit of work.
- The person at the keyboard wins. When the workstation is in use, its own engines drop to the back of the queue.
05Sharing a graphics card
The workstation's graphics cards serve the chat model, image generation and a voice studio, and they all want the same memory. The arbiter is the adult in the room: applications claim memory before they use it, a measured amount is always kept free, and a load that would not fit is refused rather than allowed to overflow.
Why refuse instead of squeezing in? Because on this platform running out of video memory does not fail — it quietly moves the overflow into system memory and carries on at a fraction of the speed. That story is lesson three.
06Models that sleep
A model that nobody has asked for in fifteen minutes is stopped, and its memory goes back to the machine for whatever its owner is doing. The next request that needs it wakes it — about seven seconds for chat, fourteen for a voice — and a busy lane wakes more than one. A model that crashes is restarted without anyone being paged. On one handheld, parking took free memory from about 2.5 GB to 24.6 GB: a gaming machine again, until it is needed.
07The compute lane
Any script, test suite or AI agent can say "run this somewhere" and get the best idle machine. The job receives a snapshot of the code exactly as it is on disk, streams its log back as it runs, and exits with the job's own exit code — so to the caller it behaves like a command run locally, only on someone else's processor.
- Manners first. Jobs run at the lowest priority. When a person sits down at a machine, its jobs drop to half the processor; if that person runs short of memory, jobs pause, idle models are parked, and after a minute the job moves to another machine and carries on from its last checkpoint.
- House rules. The machine that doubles as the living-room TV takes no work during movie night, and runs quietly overnight.
- Batches. Many items fan out across machines at once; each machine's time for each kind of job is recorded, and a straggler gets a backup copy elsewhere.
08Moving files
A small file service on each machine moves data in several parallel streams, resumes after an interruption, and hash-checks every file before it counts as delivered. The same file between the same two machines went from 0.6 MB/s by the old route to 112.3 MB/s — a 19.1 GB film from about nine hours to about three minutes. Video can also be streamed straight from whichever machine has it, rather than copied first.
09Watching it all
Every machine reports its processors, memory, graphics card, network and temperatures every thirty seconds, into a history that is kept rather than rolled over. A live status page sits beside a history page that can answer "what happened last Tuesday" for any of it.
The same history feeds a "should I buy hardware?" advisor with one strict rule: it recommends a purchase only when a limit was actually hit, never because headroom merely looks low. Recording all of this costs 0.007% of one core per machine, and a year-long query comes back in 5 to 21 milliseconds.
10What runs on it
- A province in 3D. NBLidar turns open laser-survey data into a walkable world, one square kilometre per fleet job — 396 tiles for one river corridor, each machine downloading, building and deleting its own piece.
- Games with test suites. VoxelStrike spreads its test battery across whichever machines are free, and VoxelBlox bakes its browser racing game on the fleet.
- Long recordings. A meeting is cut at sentence gaps, spread across the speech machines and stitched back together in order.
- AI coding agents. Dozens of sessions ask the same front door for answers and the same compute lane for test runs.
11Eighteen lessons
Each of these is something that looked fine until it was measured. Most of the system is the residue of them.
01The router trusted a guess
Machines that had never been measured were given a polite fallback speed, and the one real measurement was taken at start-up, while the fleet was asleep. So all quick traffic went to a machine doing 10.7 tokens a second instead of one doing 14.2 — and the others never got any traffic, so they were never measured either. A guess that feeds itself.
Now: keep measuring until every machine has a real number. After the fix the three candidates read 18.6, 14.0 and 16.7. No measurement is not an average measurement.
02Three days without a single job
One speech machine ran at 11.4× real time, but a stale figure said 1.67×, so it lost every race for three days. Another had been learned at 4.69× and actually ran at 16.9×. Calibrating on silence made it worse: silence read 2.8–9.8× where real speech read 5.4–14.2×.
Now: hourly recalibration, with real speech. Silence is not a test clip.
03Slow is the new error
When a graphics card runs out of memory here, nothing fails. The overflow slides into system memory and everything carries on — chat at about 20 tokens a second instead of 70, and a 13-second image in 119. No error, no warning, just a building that got slower.
Now: a free-memory floor measured over 41 samples, and a load that would cross it is refused.
04"Empty" described the software, not the card
The restore logic saw no model loaded and put a 17 GB model back on a card the voice studio was using. It kept answering correctly — with 5,988 MB of its 16.7 GB actually on the card and the rest wherever the system could find room.
Now: check free memory before every restore, honour reservations, and allow one model operation at a time.
05Swap thrash
One machine took both chat and speech traffic, and its graphics card could hold one model at a time. Every request evicted the other model: 9.3 seconds a request, against 1.9 with the right model already resident.
Now: a machine with the model loaded always outranks one that would have to load it, and no machine is sent work that would evict what it is good at.
06Fluent, plausible, and mostly missing
The speech engine returned "OK" and a perfectly fluent transcript for 390 seconds of audio — containing 23% of what was said. A ten-minute clip came back 13.9% complete. Nothing about the output looked wrong.
Now: no machine is sent more than five minutes of audio. Split across six machines, that ten-minute clip took 18.8 seconds instead of 138, and came back 95.6% complete.
07Commas pause longer than full stops
Cutting a recording at the nearest pause lost 59 words; at the longest pause, 79. It turned out comma pauses ran 1.16 seconds — longer than the 1.12-second gap between sentences. What did work was the speech before the pause: a sentence gap follows about 2.8 seconds of talking, a comma 1.0–1.8. Cutting by that kept 550 of 550 words, once half a second of padding rescued the last five.
Now: transcription bugs are found by counting words, not by reading transcripts.
08A new file is not a new server
All five workers ran stale code at the same time while every check passed. The old process kept serving, and an update tool reported "ok" after writing a second copy of the file beside the one in use.
Now: one update tool that checks what is actually serving, and every worker publishes a fingerprint of the code it really loaded.
09Same machine, 127 times further away
The same worker answered in 4.5 ms over the home network and 574 ms over the tunnel. Both were "up", and both worked.
Now: the home network always comes first, and latency is learned per path, not per machine.
10187× for the price of a better route
The same file between the same two machines moved at 0.6 MB/s by the old route and 112.3 MB/s with the new tool. One promising 1.4 MB/s result was thrown out on the spot: another copy had been running at the same time, so it measured two things at once.
Now: the home path is chosen even when the tunnel happens to test faster, and a measurement taken under someone else's load does not count.
11Many hands, honestly counted
Eight processor-heavy items took 131 seconds one after another and 32 seconds across four machines. A second, flashier result — 18 seconds against 27 with learned job times — was not claimed, because another job had been running during the comparison.
Now: a speed-up counts only when both sides ran under the same conditions.
12396 for 396, all failed
A batch of map tiles failed every single item at setup, twice, inside a minute — each claiming a file was missing from its snapshot when the file was right there. Small batches with identical settings passed. The likely cause: a race when many jobs land on one machine at once.
Now: one job per tile, twelve at a time, after one warm-up job per machine. Zero failures since.
13The packer rewrote the luggage
The first snapshot tool helpfully converted line endings on the way out, so a remote test failed on a file that was perfectly correct on disk.
Now: snapshot bytes are read straight from disk and identified by their hash. A copy is either identical or it is not a copy.
14Right environment, wrong Python
The first remote test run used the job agent's own interpreter instead of the project's, because Windows looks in the parent program's folder before the job's own search path. Result: a missing module that was not missing.
Now: the program is resolved against the job's own search path, every time.
15Asleep for fifty-five years
On the first live run, models with no history counted as idle since the beginning of computer time — about 55 years — and ten were parked at once. Separately, memory measured per process said stopping a model freed 14 MB; the machine's own free memory said about 22 GB.
Now: memory freed is measured as the change in the machine's free memory, not as a process's opinion of itself.
16Don't bother the person at the keyboard
With the workstation busy, a low-priority service froze for one to ten seconds at a time, and a 2.4-second dictation took 39.
Now: background jobs sit below the fleet's own services, and the owner's own work pushes jobs off the workstation entirely.
17Never wait forever, never refuse for no reason
Twenty clips sent at once to six machines got thirteen instant "busy" replies. Not wrong, exactly — just unhelpful.
Now: speech queues briefly and sends a duplicate to a second machine when the first is slow. All twenty succeed in 16 seconds, and ten of ten survived a machine being switched off mid-request.
18Pixel-perfect is a myth
Spreading a game's test battery across the fleet found seven real failures on the main branch on its first runs — and one more surprise: one family of graphics drivers drew the same frame anywhere from 1 to 178 pixels differently, which broke every check that assumed identical output.
Now: picture checks allow for the driver, and a test that only passes on one machine is a test of that machine.