Community GPU Inference: Notes From Building the Control Plane

There is a large installed base of consumer GPUs, and there is a growing bill for LLM inference. The obvious trade, renting the former to pay the latter, has been attempted many times and mostly produces either a token with no callers or a box rental with no product. InferSpine is a project I am building to study the missing middle: the software control plane that turns untrusted, heterogeneous, residential GPUs into an OpenAI-compatible endpoint with routing, metering, and payout. This is a note on what that system has taught me so far, including the parts that don’t work yet.

Disclaimers up front. This is a work in progress, not an MVP announcement.

The observation

Inference is not a one-time capital expense. It’s the recurring cost that grows with a product’s success, which makes it the cost that doesn’t go away. Two facts follow from that.

The first is that builders don’t want GPUs. They want a base_url, an API key, a usage number that matches the invoice, and latency that doesn’t ruin the UX. The OpenAI SDK surface became the POSIX of this industry: whatever your backend is, if you speak that shape, adoption cost is one config line. Everything else, catalogs, dashboards, playgrounds, is secondary.

The second is that consumer GPU supply is large and cheap. A gaming machine with a 16–24 GB card (32 GB on a 5090; many Steam cards are 8–12 GB and too small for this workload) sits unused much of the day. Its marginal cost of compute is electricity, and its owner has no realistic path to selling that capacity into the inference market.

Between these two facts sit several existing answers, and each one takes only a corner of the problem:

DePIN here means Decentralized Physical Infrastructure Networks: crypto-native networks that try to coordinate physical supply with tokens.

The gap is the intersection: an OpenAI-compatible API, on community supply, with routing, metering, and payout you can check. SaladCloud is the closest attempt I can point at. Nobody has convincingly closed all three, and InferSpine is how I’m learning why: none of the hard parts are the parts that look hard from the outside.

The thesis

My claim is specific enough to be wrong:

The durable layer in community-GPU inference is the software control plane — routing, metering, reputation, and session affinity — not the silicon and not the serving engine.

That claim has two halves. The first half, “not the silicon,” is a statement about where a small team can compete. The second half, “the control plane is where the difficulty lives,” is a statement about what actually breaks when you build it. The rest of this post is evidence for both, from InferSpine.

The chip layer versus the software layer

It’s worth being precise about the stack, because “AI infrastructure” gets used for five very different layers:

1. Silicon. GPUs, TPUs, LPUs, HBM, interconnect. Capital-intensive, multi-year cycles, brutal margins for everyone except the leader.
2. Kernels and runtime. CUDA, ROCm, Metal, Triton, attention kernels. Owned by vendors and a handful of research groups.
3. Serving engine. vLLM, llama.cpp, SGLang, TensorRT-LLM. Paged attention, continuous batching, speculative decoding, quantization. This layer is excellent and largely open source.
4. Control plane. Who serves this request, on what hardware, at what price, with what guarantee, and how do you know the tokens you billed were actually produced.
5. Product surface. Keys, credits, usage, playground, docs.

Layers 1 and 2 are out of reach for a small team, and layer 3 is already the best deal in the stack: you can get strong batching and KV-cache management for free, on a consumer card, today. That is exactly what makes layers 4 and 5 interesting. The engine got good enough that the bottleneck moved.

Here is the strongest case against my thesis, because it’s strong. Most of the value in this industry has accrued to the chip layer, and it isn’t an accident: inference is a latency-and-throughput problem, and a lot of that latency is a hardware property. Purpose-built silicon wins deterministic low-latency serving in a way no amount of routing can match. Community GPUs are structurally worse: typically no NVLink (the 3090 was an exception), GDDR rather than HBM, residential upstream bandwidth, 16–24 GB of VRAM instead of 80, and a human who might launch a game at any moment. If the market only rewards frontier-model, low-latency, high-SLA serving, then a community network is a worse product at every point on the curve and the thesis is dead.

My answer isn’t that this is wrong. It’s that it defines a segment rather than the whole market. The workloads where community supply is plausible share a shape: mid-sized open-weight models in the roughly 8B–32B class, quantized, where “good enough quality at a fraction of the cost” beats “best possible token.” Classification, extraction, summarization, embeddings, agent sub-steps, drafting passes, batch pipelines, and chat for cost-sensitive products. What community supply cannot credibly sell is frontier-model serving with a latency SLA. That boundary is a design constraint. Pretending otherwise is how this category loses credibility.

So: no chip. The bet is that on top of free serving engines and hardware whose capital cost is already sunk, what’s scarce is trust and coordination, and both are software.

What has to be true

Before writing code it’s worth writing down the conditions under which the thesis survives. I have four.

Interactive latency has to be achievable, not just average throughput. Time-to-first-token is the number users feel. A network that averages well but has a long tail of cold-start prefills on a random node feels broken.

Metering has to be exactly right, including streams. If a builder’s invoice doesn’t match their own token count, nothing else matters. Same on the supply side: an operator who suspects underpayment leaves.

Supply quality has to be measurable and routable. Untrusted nodes will be slow, will disappear mid-request, and, given an incentive, will claim to serve a model they aren’t serving. The control plane needs a signal that moves traffic away from them in minutes, not in weeks.

The developer path has to be genuinely one line. Any friction beyond changing base_url and a key competes against a vendor with a working dashboard and years of operational history.

Notice that none of these is a machine learning problem. They’re all distributed-systems and accounting problems.

InferSpine

InferSpine has three hops, plus a console. Requests enter an OpenAI-compatible gateway, which owns keys, pricing, rate limits, and the credit ledger. The gateway forwards to an orchestration service that owns the registry of nodes and picks one. That service pushes the job down a WebSocket to a node running a local engine, and tokens stream back up the same path: WebSocket to the manager, then SSE to the caller.

  flowchart TB
    sdk["App / SDK<br/>OpenAI-compatible call"]
    console["Developer console<br/>keys · usage · playground"]
    gw["Gateway (Go)<br/>API keys · pricing · rate limit · credit ledger"]
    mgr["Node manager (Go)<br/>registry · health · routing · WS hub"]
    node["Node agent (Python)<br/>on the GPU owner's machine"]
    engine["Local engine<br/>vLLM (CUDA) / llama.cpp"]

    sdk -->|"Bearer api key · POST /v1/chat/completions"| gw
    console --> gw
    gw -->|"internal token · HTTP"| mgr
    node -->|"dials OUT: WSS + owner JWT<br/>register · heartbeat"| mgr
    mgr -->|"pushes load / execute / execute_stream"| node
    node --> engine
    mgr -.->|"tokens back: WS, then SSE"| gw

Three decisions in that diagram carried more weight than I expected.

Nodes dial out; the manager pushes down. Residential GPUs sit behind NAT with no stable inbound address, so a node opens an outbound WebSocket, authenticates as its owner, and the manager pushes commands into that socket. This makes supply onboarding trivial, no port forwarding, no tunnel, and moves all the complexity into the manager, which now holds stateful connections. That single choice is responsible for most of the scaling work described later.

The gateway is a proxy for prompts, not a source of truth for usage. It peeks at the model name and whether the request is streamed, and it may inject stream_options.include_usage so the engine actually emits a usage chunk. It does not rewrite prompts or reshape responses. Keeping prompt bytes intact is what keeps OpenAI compatibility from rotting every time a new parameter appears upstream.

That split matters for trust. Compatibility can stay dumb. The ledger cannot. An untrusted node can lie about usage. The debit has to match an independent tokenizer count at the gateway, not the number the node volunteered.

Routing is boring on purpose, for now. Today’s selection is a weighted random pick over online nodes that have the model loaded, weighted by VRAM:

func weightedPick(targets []RoutingTarget, rng *rand.Rand) int {
	if len(targets) == 0 {
		return -1
	}
	total := 0
	for _, t := range targets {
		total += t.WeightVRAMMB
	}
	if total <= 0 {
		return rng.Intn(len(targets))
	}
	pick := rng.Intn(total)
	cumulative := 0
	for i, t := range targets {
		cumulative += t.WeightVRAMMB
		if pick < cumulative {
			return i
		}
	}
	return len(targets) - 1
}

That’s the whole algorithm. It’s a placeholder standing where a reliability score belongs, and I left it deliberately simple until I had real signals to score on. Which brings me to the lessons.

What I’ve learned so far

1. Streaming is where metering quietly breaks

This is my favorite failure in the whole system because it is invisible in every demo.

Non-streamed completions return a usage object with prompt and completion token counts. The gateway reads it, computes cost, debits the ledger. It works, and it worked from very early on.

Streamed completions don’t work that way unless you ask. OpenAI and vLLM emit a final usage-only server-sent event — a chunk with a usage object and an empty choices array — only when stream_options.include_usage is true. Without that flag, there is no usage event to capture, and streamed requests are unmetered by construction. llama.cpp has also put usage on the stop chunk rather than on a separate empty-choices chunk, so the node agent has to accept both shapes.

InferSpine’s stream loop was written to forward chunks that contain content, and a usage-only chunk contains no content, so it hit a continue and was dropped on the floor. Everything downstream behaved perfectly: tokens streamed to the caller, the request logged as successful, and the meter recorded zero.

Every streamed request in the system was free.

Nothing errored. No alert fired. The demo looked flawless, because streaming is the demo. I found it reading the billing path rather than by observing a symptom, which is the part worth generalizing: a metering bug does not present as a bug. It presents as revenue that never existed and an invoice a builder can’t reconcile. If you’re building anything usage-metered, the test you need isn’t “does the stream work,” it’s “does a streamed request produce a nonzero debit that matches an independent tokenizer count.”

The fix is small, request usage, capture the usage chunk instead of skipping it, forward it to match the upstream contract, and also attach counts to the terminal internal frame, but the lesson is a testing lesson, not a code lesson. Metering needs assertions on the ledger, not on the response body, and those assertions cannot trust the node.

2. Sessions are physics, not state management

A multi-turn chat sends the whole conversation on every turn. If turn two lands on a different node than turn one, that node has a cold KV cache and re-prefills the entire history. The user sees a time-to-first-token that grows with conversation length, on a random schedule, for no visible reason.

The tempting fix is to migrate the KV cache between nodes. Don’t. For a 7B–8B model the cache is hundreds of megabytes at a few thousand tokens, and it grows into gigabytes for longer context or a ~30B model. Prefill of a multi-thousand-token history on a consumer card is already hundreds of milliseconds to around a second. Shipping that state across a residential uplink at tens of megabits per second takes minutes. The arithmetic never closes.

The workable fix is affinity: the caller passes a conversation ID, the manager remembers which node served it, and prefers that node if it’s still online and still holds the model. If it’s gone, you fall back to a normal pick and eat the full re-prefill. Soft, best-effort, TTL’d:

candidates = online nodes with the model loaded
if conversation_id present:
    preferred = affinity[conversation_id]
    if preferred in candidates:
        return preferred
pick = weightedPick(candidates, weight = f(vram, score))
affinity[conversation_id] = pick   # TTL ~30m, control-plane hint only

The TTL on that map is not a promise that the KV cache is still resident. A busy consumer card can evict it much sooner. A miss is a slower turn, not an error.

It says “we’ll try to keep you warm” instead of promising a session that lives on a GPU somewhere. The metric, sticky hit rate against TTFT, tells you exactly what the optimization is worth.

The same reasoning kills mid-stream failover. If a node dies halfway through generating, you cannot transparently resume on another node: the output is non-deterministic across a node boundary — even the same weights can diverge across GPUs, kernels, and batching — and you’d have to decide who pays for the partial tokens. Every retry policy on a streamed body is a double-charge risk. I chose to fail the request instead. A visible error beats a silent inconsistency in a system whose point is trustworthy accounting.

3. Trust on untrusted hardware is a ladder, not a breakthrough

The verifiability question comes up in every conversation about this category, usually phrased as “how do you know the node ran the model it claims?” The honest answer is that high integrity, low latency, and low cost don’t combine freely on untrusted consumer hardware. ZK proofs of LLM inference are still orders of magnitude too expensive at this scale. Optimistic schemes are too slow for interactive chat. Proof-of-quality sampling is gameable. Trusted execution needs specific hardware — NVIDIA confidential computing is H100/Blackwell-class, not GeForce — that community operators don’t have.

So instead of a breakthrough, a ladder, ordered by cost:

Rung 0 is the one that pays rent, and it’s just careful engineering. Take the signals you already have, success rate, TTFT, mid-request disconnects, heartbeat freshness, and roll them into a score:

score = 0.5 * success_rate_ema
      + 0.3 * latency_score(ttft_ema)
      + 0.2 * uptime_score(last_seen)

on mid_request_disconnect: score *= 0.5   (floor 0.05)

weight = max(vram_mb, 1) * (0.2 + 0.8 * score)

Folding the score into the existing routing weight means detection and response are the same mechanism: a node that starts misbehaving loses traffic share continuously, rather than sitting online until a human ejects it. The floor term matters too: it keeps new and recovering nodes reachable instead of leaving them unable to earn traffic back.

Rung 1 is worth describing because it’s cheap and underrated: keep a set of golden (model, prompt) pairs, periodically challenge nodes, and hash the first K tokens or, more stably, a logit fingerprint across several prompts. Greedy decode is not bit-stable across hardware, so this is statistical on purpose. It doesn’t prove anything about any individual user request. It raises the cost of naively serving a smaller model under a larger model’s name from “free” to “you will be caught statistically.” That’s a real deterrent, and describing it as statistical rather than verified is the difference between a claim that holds up and one that doesn’t.

Which leads to the rung that isn’t on the ladder. The node executing your request sees your prompt in plaintext, because inference requires it. You can encrypt transit, you can isolate the host so the agent cannot read the GPU owner’s files, you can avoid persisting message bodies in logs. You cannot make the GPU blind without trusted execution hardware. I’ve found it useful to write down which promises are keepable and which aren’t:

Every project in this category is tempted to blur that line. The temptation is worth resisting, because anyone who reads the diagram will ask, and the answer “our nodes can’t see your data” is one architecture diagram away from being caught.

4. Stateful connections make scaling a routing problem

The outbound-WebSocket decision comes due when you run more than one manager instance. A request can land on instance 1 while the node it needs holds its socket on instance 2.

I stood up a four-instance InferSpine topology against shared Postgres and Redis and drove traffic through it. The numbers were more encouraging than the failure modes.

The connection layer is a non-issue on the scale I tested. One instance held roughly 20,000 simulated node connections on a laptop, about 40 KB of resident memory per connection, linear, no cliff. For a 10,000-node fleet across four to six instances, the WebSocket layer needs a file-descriptor limit raised and nothing else. Worth knowing, because “you’d need a huge fleet to hold that many connections” is a common and wrong intuition.

Cross-instance routing costs a Redis pub/sub hop: the receiving instance looks up which instance owns the node’s socket, subscribes to a per-request response channel, publishes the command, and the owning instance relays it down the real socket and publishes the response back.

  sequenceDiagram
    participant GW as gateway
    participant I1 as inst-1 (received the request)
    participant PG as Postgres
    participant R as Redis
    participant I2 as inst-2 (owns the socket)
    participant N as node-8

    GW->>I1: POST /v1/chat/completions
    I1->>PG: candidates: online + model loaded
    PG-->>I1: includes node-8
    I1->>I1: local hub miss
    I1->>R: GET ws:owner:node-8 -> inst-2
    I1->>R: SUBSCRIBE ws:resp:<request_id>
    I1->>R: PUBLISH ws:cmd:inst-2
    R->>I2: command delivered
    I2->>N: forward over real WebSocket
    N-->>I2: response
    I2->>R: PUBLISH ws:resp:<request_id>
    R->>I1: delivered
    I1-->>GW: response

Measured on that topology (same-region Redis), the hop cost about 60% more latency than the same-instance path, 8.8 ms versus 5.4 ms mean, which is cheap enough that replacing Redis with a broker or building a gossip-based directory would be extra moving parts without a matching bottleneck.

The real finding was what happened when I killed the instance that owned a node’s socket mid-connection. Requests didn’t fail fast; they hung for the full caller timeout, five seconds for a status call, sixty for inference, three hundred for model management, because the publish-into-a-dead-channel path had no liveness check and simply blocked until the context expired. Redis pub/sub is fire-and-forget, so “the owner is dead,” “the message was lost during a reconnect,” and “the owner is just slow” are indistinguishable: all three look like silence.

The fix isn’t a different transport, it’s a protocol that distinguishes those cases. Check a liveness key before publishing, so a dead owner fails in milliseconds instead of minutes. Split “received” from “completed” with a short uniform acknowledgment window, two to three seconds, before switching to the operation’s real timeout. Tighten the heartbeat TTL. That decouples “is the recipient alive” from “how long does its work take,” which the original design conflated: any request/response layer built over pub/sub needs an explicit liveness signal, or every failure becomes a timeout.

Two adjacent findings from the same exercise, both the kind of thing you only see by actually running it. Unguarded CREATE TABLE on boot crashes the second instance in a synchronized deploy with a duplicate-key error on a system catalog index, which means schema-on-boot silently blocks you from ever going multi-instance. And per-request Redis Subscribe calls pin a connection from the go-redis pool until Close, so concurrent in-flight streams can exhaust the PoolSize you sized for ordinary commands.

5. Write down what you cannot promise

The pattern across all of the above is that the failure modes weren’t exotic. They were the specific bugs you get when a system’s demo path and its trust path are different paths. Streaming worked and streaming billing didn’t. Chat continuity worked at the database layer and not at the GPU layer. Nodes connected fine and node death hung callers. Local development ran on cleartext WebSockets, which is fine until “the default” becomes “production.”

Writing down what works, what’s broken, and what’s impossible turns gaps into tickets, or into claims you stop making.

Pros, cons, and when I’d call it dead

And the conditions under which I’d stop, written down in advance so they’re harder to talk away later:

• Interactive latency stays outside a usable band after routing, affinity, and scoring are all shipped. That would mean the physics, not the implementation, is the problem.
• Streamed metering can’t be made accurate against an independent token count. Without that, there is no trustworthy product to sell.
• Demand only ever materializes as speculative interest rather than builders paying for tokens they use. That would mean I built supply-side infrastructure for a market that doesn’t exist.

Writing the stop conditions down first is the part of this I’d recommend regardless of the domain. It’s much easier to define failure before you’re invested in the codebase.

What’s next

The near-term InferSpine queue follows directly from the gaps above: request and relay streaming usage end to end, then assert on the ledger against an independent tokenizer, not the node’s usage field; force encrypted transport by default and scrub prompts from node-side logs; ship conversation affinity behind a flag and measure sticky hit rate against TTFT; replace the VRAM-only weight with the reliability score; then a fingerprint challenge spike to see how fast a swapped model can be detected.

None of that is a breakthrough. All of it is the difference between a demo and something a builder would put a key into. Restated: in this category the hard part is the control plane, and the control plane is mostly careful accounting, honest failure modes, and measured decisions about what you refuse to promise.

The open question I keep circling is the one I can’t answer from the code. There’s a real segment where “good enough quality, much cheaper, best-effort latency” wins, and a real segment where it never will. I don’t yet know the exact shape of the boundary between them, and I suspect nobody in this category does. That boundary, not the verification math, is what decides whether community GPU inference is a business or a nice piece of distributed systems engineering.

If you’re working on something in this space, particularly on metering accuracy or reputation-based routing on untrusted nodes, I’d like to compare notes.