When Pooling Your Own GPUs Beats the API

The Default Is Renting

If your product depends on an LLM, you probably reach for an API key. OpenAI, Anthropic, maybe a Groq or Together endpoint for speed. You send text in, get text out, and pay per token. The model is big and lives somewhere else. That is the default because it works and because the alternative used to be impossible.

But the default is a specific economic bet: that the convenience of renting is worth the variable cost and the loss of control. For a lot of teams, that bet is becoming harder to justify as usage scales up.

Stripe's Link data shows the top 10% of AI spenders nearly doubled their monthly outlay from December 2025 to March 2026. Per-token pricing means your inference bill grows in lockstep with every user you acquire. There is no volume discount architecture. No way to say "I have this GPU sitting idle, use it instead."

The Pooling Alternative

Mesh LLM, released this week by the iroh team, is a concrete example of a different approach. It pools the GPUs and memory from any number of machines and exposes the whole thing as a single OpenAI-compatible endpoint at localhost:9337. You start a node on your laptop, add a machine in your office, add a cloud instance, and the mesh figures out where to route each request.

Three modes of operation:

• Local -- run the model on whatever machine the request hits.
• Peer-routed -- forward the request to a peer that already has the model loaded.
• Split ("Skippy") -- partition a model by layer ranges across several machines. Layers 0 through 15 on one node, 16 through 31 on another. Activations flow through the pipeline. The client never knows.

This last mode is the interesting one. A 235B parameter mixture-of-experts model normally requires a single machine with enormous memory. With Skippy, three modest machines can each hold a third of the layers and pass activations between them over QUIC. The latency is higher than a monolithic GPU, but the hardware cost is dramatically lower.

The Infrastructure Underneath

The mesh protocol runs on iroh, a peer-to-peer networking library. Every node's identity is a public key. There is no central server. iroh handles NAT traversal, hole-punching, and relay fallback to connect any two nodes wherever they sit. The transport is QUIC, with three ALPN protocols: one for the main mesh (gossip, routing, HTTP tunnels), one for the control plane (config sync, ownership), and one for latency-sensitive activation transport on split models.

The whole design is built around a simple insight: most teams already have GPU hardware sitting in closets, under desks, or on cloud instances they are paying for anyway. What they lack is a way to make those machines act as one.

The Cost Math

This is where it gets concrete. Consider a team running 10 million inference calls per month on a mid-size model like Llama 3.1 70B.

API route: At roughly $0.90 per million input tokens and $0.90 per million output tokens (assuming a typical provider), 10 million calls with an average of 500 input and 100 output tokens costs about $5,400 a month. Every month. No cap.

Pooling route: One node with an RTX 4090 (24GB VRAM, about $1,600) can run 70B models with 4-bit quantization at roughly 10-15 tokens/second. If you need faster throughput, add a second card or a peer. The upfront is $3,000 to $5,000 in hardware. After that, the marginal cost of each inference call is electricity and cooling.

The breakeven is somewhere around 6 to 9 months of sustained API usage, depending on your throughput requirements. After that, the pooled hardware is cheaper by a wide margin.

But the cost argument is only half of it.

The Control Argument

API-based inference bundles the model, the hardware, the scheduling, and the pricing into one opaque box. You cannot control when the model is updated. You cannot inspect or audit the inference path. You cannot guarantee that your data does not leave a specific region. And if the provider changes their pricing or terms, your cost structure shifts overnight.

For products handling sensitive data, the data residency question alone can justify self-hosting. The compliance paperwork for routing user data through third-party inference providers is not free. Some categories (health, finance, legal) simply cannot use public API endpoints without extensive contractual coverage.

Mesh LLM lets you run the whole thing on hardware you control, in a network you control. The data never leaves your mesh unless you explicitly add a public peer. That is a structural advantage, not a nice-to-have.

Where It Falls Down

None of this is free. The trade-offs matter:

• Latency. A split model running across three machines over the open internet will always be slower than a monolithic GPU in a datacenter. Mesh LLM is built for throughput, not single-request speed. If your product needs sub-100ms response times, API providers still win.
• Operational burden. You now manage GPU drivers, model quantization, peer discovery, and node health. One person on the team needs to understand this stack. If you have zero ops bandwidth, renting is cheaper in total cost of ownership.
• Model availability. API providers give you access to the latest models the day they launch. Running them on your own hardware takes time: quantization, optimization, testing. You will always trail the frontier by some margin.
• Elasticity. API scaling is instant. Your own GPUs have a fixed capacity. A traffic spike that exceeds your pool means degraded performance or dropped requests. You need headroom or a hybrid failover strategy.

The Decision Framework

Pooling your own GPUs makes sense when:

• Your monthly inference bill exceeds the amortized cost of the hardware you would need (roughly $1,000 to $2,000 per month per high-end GPU-equivalent workload)
• You already own underutilized GPUs (workstations, dev machines, leftover cloud instances)
• Your product has data residency or privacy requirements that make third-party APIs complicated
• Your inference volume is predictable enough to plan capacity
• You have at least one person who can maintain the stack

Stick with APIs when:

• Your latency requirements are tight (below 200ms)
• Your traffic is spiky and unpredictable
• You need to rapidly switch between frontier models as they launch
• Your team does not have GPU operations experience
• Your inference volume is still low enough that the API bill is tolerable

The smartest setup for most teams is hybrid. Route high-throughput, latency-tolerant, sensitive workloads to your mesh. Use API providers for burst, frontier models, and latency-sensitive paths. Treat the API as your elastic overflow, not your only option.

Builders Should Pay Attention

Mesh LLM is one project in a growing category. Ollama proved that local inference is practical. Llama.cpp proved that quantization makes big models run on consumer hardware. Mesh LLM adds the distributed layer: multiple machines working together. The pattern is clear. The cost of AI inference is trending toward hardware ownership for anyone with sustained usage, and the tools to make that practical are arriving in rapid succession.

The question is not whether you can run your own inference. It is whether you have done the math to know when you should.