One AI service, made from a house full of ordinary computers
A main workstation, two mini PCs, two handheld gaming PCs and two laptops: an older one with a small graphics card, and the one I actually carry around. On their own, each is just a computer. Together they answer chat, vision, speech and image requests, run each other's test suites, take turns on a graphics card without squabbling, and report their own vitals. These are my notes on how — and on the mistakes that taught me most of it.
- I turned a workstation and a drawer's worth of spare machines into one AI and compute service that every app in the house reaches through a single front door.
- Work goes wherever a measurement says it should: the machine with the model already loaded, on the home network, at a speed that was tested rather than guessed.
- It has manners: idle models go to sleep, jobs back off when a person sits down, and the living-room TV gets movie night off.
- Most of the design is scar tissue — eighteen times something looked fine until I measured it, and each one is written up below with its moral.
- The biggest lesson: slow and incomplete are failures too, so count what comes back instead of admiring how fluent it sounds.
01Why bother
Local AI on one PC hits a wall fast: one graphics card, one model at a time, and every app wired to its own address. Meanwhile the rest of the house was full of capable machines doing nothing in particular — and one of them spends its evenings showing films.
A fleet only helps if nobody has to think about it. An app should ask for "a transcription" or "a quick answer", not for a particular machine; a long job should find an idle computer on its own; and whoever is sitting at a machine should never notice it is also working for the others. Everything below exists to make that true. Most of it I learned the humbling way: by measuring an assumption and watching it fall over.
02The map
03Architecture at a glance
The map shows the machines; this is the software on top of them. It is hub and spokes: the workstation holds the front door, the biggest models and the history, and every other machine does work and reports back. That keeps every app simple — one address, always — and the trade-off is the obvious one for a hub: a lot of eggs in one well-watched basket.
04How it grew
None of this was designed up front. It started life as a terminal for keeping an eye on dozens of AI coding agents at once, and every part below arrived when something measurable went wrong without it. 428 commits since 2026-05-31, week by week, with the day each part first landed:
- 2026-06-24One front doora single AI endpoint for every app in the house
- 2026-08-08Measured routingwork goes where it is fastest or already loaded
- 2026-08-17The GPU arbitermemory claimed before use; silent spills refused
- 2026-08-28Split transcriptionlong recordings cut at sentences, spread out
- 2026-09-05Verified file movesparallel, resumable, hash-checked
- 2026-09-05Stale code caughtevery worker proves what it is running
- 2026-09-15Card sharingchat, images and voices on one card, politely
- 2026-10-08The compute laneany job, any idle machine
- 2026-10-08Sleeping modelsidle models parked, woken on demand
- 2026-10-12History and dashboardsevery machine, every 30 seconds, kept
- 2026-10-18Local AI platformchat, vision and embeddings by name
05One front door
Every app in the house talks to a single OpenAI-compatible address on the main workstation. Behind it sit lanes, one per kind of work:
Every reply says which machine served it, so a slow answer is a fact about a machine rather than a mystery novel. When every member of a lane is busy, the door says so plainly — "busy, try again in three seconds" — instead of letting a request queue forever. Speech is the exception, because a person is waiting on the other end: a clip waits up to thirty seconds for a free machine, and if the first machine dawdles, the same clip goes to a second and the first good answer wins.
06Who gets the work
The router has only a few rules, and every one of them came from a measurement — usually one that proved me wrong:
- Quick requests go to the fastest machine that clears a quality bar; deep requests go to the most capable model, wherever it is. The big model gets the hard questions; a small one on a handheld can handle "what's the capital of Peru".
- A machine with the model already loaded beats one that would have to load it. Loading is the slowest thing in the building.
- The home network comes first. The same machine answered in 4.5 ms at home and 574 ms through the tunnel.
- Speeds are measured, never assumed. Every machine is re-benchmarked when its number is missing or more than an hour old, and a job's cost is fitted as a fixed start-up cost plus a cost per unit of work.
- The person at the keyboard wins. When the workstation is in use, its own engines drop to the back of the queue.
07Sharing a graphics card
The workstation's graphics cards serve the chat model, image generation and a voice studio, and all of them want the same memory. So there is an arbiter, the adult in the room: apps claim memory before they use it, a measured amount is always kept free, and a load that would not fit is refused rather than allowed to overflow.
Why refuse instead of squeezing in? Because on this platform, running out of video memory doesn't fail. It quietly moves the overflow into system memory and carries on at a fraction of the speed — like a car that answers an empty tank by switching to pedals. That story is lesson three.
08Models that sleep
A model nobody has asked for in fifteen minutes is put to bed, and its memory goes back to the machine for whatever its owner is doing. The next request that needs it wakes it — about seven seconds for chat, fourteen for a voice — and a busy lane wakes more than one. A model that crashes is restarted without anyone being paged. On one handheld, parking took free memory from about 2.5 GB to 24.6 GB: a gaming machine again, until it is needed.
09The compute lane
Any script, test suite or AI agent can say "run this somewhere" and get the best idle machine. The job receives a snapshot of the code exactly as it is on disk, streams its log back as it runs, and exits with its own exit code — so to the caller it behaves like a command run locally, just on someone else's processor.
- Manners first. Jobs run at the lowest priority. When a person sits down at a machine, its jobs drop to half the processor; if that person runs short of memory, jobs pause, idle models are parked, and after a minute the job moves to another machine and carries on from its last checkpoint.
- House rules. The machine that doubles as the living-room TV takes no work during movie night, and runs quietly overnight. Some priorities are not negotiable.
- Batches. Many items fan out across machines at once; each machine's time for each kind of job is recorded, and a straggler gets a backup copy elsewhere.
10Moving files
A small file service on each machine moves data in several parallel streams, picks up where it left off after an interruption, and hash-checks every file before it counts as delivered. The same file between the same two machines went from 0.6 MB/s by the old route to 112.3 MB/s — a 19.1 GB film from about nine hours to about three minutes, which is the difference between "start it before bed" and "start it before the kettle boils". Video can also be streamed straight from whichever machine has it, rather than copied first.
11Watching it all
Every machine reports its processors, memory, graphics card, network and temperatures every thirty seconds, into a history that is kept rather than rolled over. A live status page sits beside a history page that can answer "what happened last Tuesday?" for any of it.
The same history feeds a "should I buy hardware?" advisor with one strict rule, which is as much a rule for me as for the software: it recommends a purchase only when a limit was actually hit, never because headroom merely looks low. Recording all of this costs 0.007% of one core per machine, and a year-long query comes back in 5 to 21 milliseconds.
12What runs on it
- A province in 3D. NBLidar turns open laser-survey data into a walkable world, one square kilometre per fleet job — 396 tiles for one river corridor, each machine downloading, building and deleting its own piece.
- Games with test suites. VoxelStrike spreads its test battery across whichever machines are free, and VoxelBlox bakes its browser racing game on the fleet.
- Long recordings. A meeting is cut at sentence gaps, spread across the speech machines and stitched back together in order.
- AI coding agents. Dozens of sessions ask the same front door for answers and the same compute lane for test runs.
13Eighteen lessons
Each of these looked fine until I measured it, and most of the system is the scar tissue they left behind. Every one ends with the rule I adopted and a moral you can take back to your own projects.
01The router trusted a guess
Machines I had never measured got a polite fallback speed, and the one real measurement was taken at start-up, while the fleet was still asleep. So all quick traffic went to a machine doing 10.7 tokens a second instead of one doing 14.2 — and the others never got any traffic, so they were never measured either. A guess that feeds itself.
Now: keep measuring until every machine has a real number. After the fix the three candidates read 18.6, 14.0 and 16.7. No measurement is not an average measurement.
A default value is a decision in disguise; when you don't know, say so and go find out.
02Three days without a single job
One speech machine ran at 11.4× real time, but a stale figure said 1.67×, so it lost every race for three days. Another had been learned at 4.69× and actually ran at 16.9×. Calibrating on silence made it worse: silence read 2.8–9.8× where real speech read 5.4–14.2×.
Now: hourly recalibration, with real speech. Silence is not a test clip.
Benchmark with the work you actually do, and keep benchmarking — a stale number loses races without telling anyone.
03Slow is the new error
When a graphics card runs out of memory here, nothing fails. The overflow slides into system memory and everything carries on — chat at about 20 tokens a second instead of 70, and a 13-second image in 119. No error, no warning, just a building that got slower.
Now: a free-memory floor measured over 41 samples, and a load that would cross it is refused.
If something can fail by getting slow instead of loud, give it a threshold and the authority to say no.
04"Empty" described the software, not the card
The restore logic saw no model loaded and put a 17 GB model back on a card the voice studio was using. It kept answering correctly — with 5,988 MB of its 16.7 GB actually on the card and the rest wherever the system could find room.
Now: check free memory before every restore, honour reservations, and allow one model operation at a time.
Ask the hardware what is free, not the software what it believes it loaded.
05Swap thrash
One machine took both chat and speech traffic, and its graphics card could hold only one model at a time. Every request evicted the other model: 9.3 seconds a request, against 1.9 with the right model already resident.
Now: a machine with the model loaded always outranks one that would have to load it, and no machine is sent work that would evict what it is good at.
Send work to where its state already lives; a warm machine beats a fast one.
06Fluent, plausible, and mostly missing
The speech engine said "OK" and handed back a perfectly fluent transcript of 390 seconds of audio — containing 23% of what was said. A ten-minute clip came back 13.9% complete. Nothing about the output looked wrong, which was exactly the problem.
Now: no machine is sent more than five minutes of audio. Split across six machines, that ten-minute clip took 18.8 seconds instead of 138, and came back 95.6% complete.
Confident output is not evidence of complete output; check how much came back, not how good it sounds.
07Commas pause longer than full stops
Cutting a recording at the nearest pause lost 59 words; at the longest pause, 79. It turned out comma pauses ran 1.16 seconds — longer than the 1.12-second gap between sentences. Who knew? What did work was the speech before the pause: a sentence gap follows about 2.8 seconds of talking, a comma 1.0–1.8. Cutting by that kept 550 of 550 words, once half a second of padding rescued the last five.
Now: transcription bugs are found by counting words, not by reading transcripts.
Check your intuitions about data against the data; the obvious signal is often the wrong one.
08A new file is not a new server
All five workers ran stale code at the same time while every check passed. The old process kept serving, and an update tool reported "ok" after writing a second copy of the file beside the one in use.
Now: one update tool that checks what is actually serving, and every worker publishes a fingerprint of the code it really loaded.
"Deployed" means running, so verify against the running process, not the file you just wrote.
09Same machine, 127 times further away
The same worker answered in 4.5 ms over the home network and 574 ms over the tunnel. Both were "up", and both worked.
Now: the home network always comes first, and latency is learned per path, not per machine.
"Up" is not a speed; measure the route, not just the destination.
10187× for the price of a better route
The same file between the same two machines moved at 0.6 MB/s by the old route and 112.3 MB/s with the new tool. One promising 1.4 MB/s result was thrown out on the spot: another copy had been running at the same time, so it measured two things at once.
Now: the home path is chosen even when the tunnel happens to test faster, and a measurement taken under someone else's load does not count.
A benchmark that shared the machine with something else is a rumour, however good it looks.
11Many hands, honestly counted
Eight processor-heavy items took 131 seconds one after another and 32 seconds across four machines. A second, flashier result — 18 seconds against 27 with learned job times — went unclaimed, because another job had been running during the comparison. It hurt a little.
Now: a speed-up counts only when both sides ran under the same conditions.
Claim the result you could defend if someone reran it, not the best one you happened to see.
12396 for 396, all failed
A batch of map tiles failed every single item at setup, twice, inside a minute — each claiming a file was missing from its snapshot when the file was right there. Small batches with identical settings passed. The likely cause: a race when many jobs land on one machine at once.
Now: one job per tile, twelve at a time, after one warm-up job per machine. Zero failures since.
When everything fails at once, suspect the stampede before the items; warm up, then ramp up.
13The packer rewrote the luggage
The first snapshot tool helpfully converted line endings on the way out, so a remote test failed on a file that was perfectly correct on disk. Thanks, I hate it.
Now: snapshot bytes are read straight from disk and identified by their hash. A copy is either identical or it is not a copy.
A tool that copies data should never try to improve it.
14Right environment, wrong Python
The first remote test run used the job agent's own interpreter instead of the project's, because Windows looks in the parent program's folder before the job's own search path. Result: a missing module that was not missing.
Now: the program is resolved against the job's own search path, every time.
Resolve programs the way the job will see the world, not the way your launcher happens to.
15Asleep for fifty-five years
On the first live run, models with no history counted as idle since the beginning of computer time — about 55 years — and ten were parked at once. Quite the nap. Separately, memory measured per process said stopping a model freed 14 MB; the machine's own free memory said about 22 GB.
Now: memory freed is measured as the change in the machine's free memory, not as a process's opinion of itself.
Give "no history yet" its own explicit meaning, and measure an effect from the outside rather than asking the thing that caused it.
16Don't bother the person at the keyboard
With the workstation busy, a low-priority service froze for one to ten seconds at a time, and a 2.4-second dictation took 39.
Now: background jobs sit below the fleet's own services, and my own work pushes jobs off the workstation entirely.
Background work the foreground can feel is not background work; rank it below everything a person is waiting on.
17Never wait forever, never refuse for no reason
Twenty clips sent at once to six machines got thirteen instant "busy" replies. Not wrong, exactly — just unhelpful.
Now: speech queues briefly and sends a duplicate to a second machine when the first is slow. All twenty succeed in 16 seconds, and ten of ten survived a machine being switched off mid-request.
Between waiting forever and refusing instantly lies a useful middle: wait a little, and hedge your bets.
18Pixel-perfect is a myth
Spreading a game's test battery across the fleet found seven real failures on the main branch on its first runs — plus one more surprise: one family of graphics drivers drew the same frame anywhere from 1 to 178 pixels differently, which broke every check that assumed identical output.
Now: picture checks allow for the driver, and a test that only passes on one machine is a test of that machine.
Run your tests on more than one machine; it is the cheapest way to learn which of your assumptions are about the code and which are about the box.
14The rules that came out of it
If the lessons are the stories, these are what they boil down to.
15What I'd do differently
Plenty, with hindsight. These are the ones I would act on from day one:
- Measure before routing, not after. A polite fallback guess and a stale speech figure sent traffic to the wrong machines — for three days, in one case. I would ship calibration, with real speech rather than silence, before the router ever picked a winner.
- Make "what is actually serving?" the first deploy check. All five workers once ran stale code while every check passed. The fingerprint each worker now publishes belonged in the very first update tool.
- Treat slow as broken from the start. Running out of video memory never raised an error; it just crawled. The memory floor and the right to refuse came only after I had watched chat and images slow to a fraction of their speed.
- Count words from the first transcript. I trusted fluent text before I counted it, and a ten-minute clip came back 13.9% complete. A word count would have caught it on day one.
- Test against an empty history. "No history" read as about 55 years idle, and ten models were parked at once on the first live run. A fresh, empty fleet is a test case, not an edge case.
The one thing I would keep: growing it from real problems instead of a grand plan. The grand plan would have been wrong too, just less instructively.