Community GPU Inference: Notes From Building the Control Plane
There is a large installed base of consumer GPUs, and there is a growing bill for LLM inference. The obvious trade, renting the former to pay the latter, has been attempted many times and mostly produces either a token with no callers or a box rental with no product. InferSpine is a project I am building to study the missing middle: the software control plane that turns untrusted, heterogeneous, residential GPUs into an OpenAI-compatible endpoint with routing, metering, and payout. This is a note on what that system has taught me so far, including the parts that don’t work yet. ...