Decide your AI bill,
layer by layer

Why does multi-agent inference cost so much more than it should?

The coordination tax turns out to be a cache miss. And at 150 GPUs it comes back.

The short answer

Ten agents reading the same case file prefill it ten times. On a metered API you pay for all ten and none of it looks like waste, because it is billed as input tokens, which is exactly what it is.

Removing that duplication means reusing key-value state across agents. That only works if you operate the inference server. No frontier API exposes it.

And here is the part I did not expect. The fix works beautifully on one node and returns almost nothing at 150, unless routing knows which node already holds your prefix. At one node the cache hit rate is 96%. At 150 nodes with ordinary load balancing it is 7%. Same code, same model, same hardware.

That number is not a measurement. It falls out of arithmetic, which is why I trust it more than most of the numbers in this piece.

Where does the duplication come from?

Model APIs have no memory. Every call carries the whole history, so a twenty-step agent sends its opening context twenty times. Put ten agents on one document and you have paid to read it a hundred times.

The provider caches help — when they can. Prefix caching only resumes if the new request is byte-identical up to the cached point. One changed character and everything after it is recomputed at full price.

Agents differ by construction. That is what makes them different agents. So their prefixes diverge, and how much they diverge depends entirely on where in the prompt the divergence starts.

Which deployment tiers can actually fix it?

The line that matters is not cloud versus on-premise. It is tenant versus operator.

Rented GPU gives you every cache capability that owned hardware does. Renting a bare instance and running your own vLLM or SGLang puts you on the operator side of that line, in someone else's building.

On-premise adds three separate things: utilisation economics, fabric choice, and data residency. None of them is a caching argument. Keep them apart. An architect who hears "you need on-prem for KV caching" will correctly reject it, and you lose the room.

Does prompt order really matter more than hardware?

Sometimes, yes. This is the cheapest finding in the whole piece.

If role instructions sit at the top of the prompt, prefixes diverge at position zero and no cache hits anywhere — including on frontier APIs, where it would otherwise have worked. Move the shared corpus to the front and role instructions to the end, and a large fraction of the benefit comes back without changing a single thing about your infrastructure.

Most agent frameworks put role instructions first. It reads more naturally. It costs a fortune.

What happens to cache hit rate at 150 GPUs?

This is the section that changed how I think about the whole problem.

A request only hits a warm cache if it lands on the node that already holds the prefix. With random load balancing over N nodes and K calls sharing that prefix, the number of distinct nodes touched — and therefore the number of cold prefills — is a balls-in-bins expectation:

cold(N, K) = N · (1 − (1 − 1/N)^K)
hit_rate   = 1 − cold(N, K) / K

N =   1, K = 24  →   1 cold prefill   →  96% hit rate
N =  16, K = 24  →  ~13 cold prefills →  46% hit rate
N = 150, K = 24  →  ~22 cold prefills →   7% hit rate

No hardware term appears anywhere in that expression. It does not care which GPU you bought, which model you serve, or which framework you run. It is a property of routing.

Which means the mechanism everyone is optimising at single-node scale quietly stops paying somewhere between sixteen and forty nodes, and the fix is not a better kernel. It is prefix affinity — pinning a task's calls to the pipeline that already holds its state.

What does the hardware actually do?

You need real throughput numbers to turn any of this into money. I used InferenceX, which publishes its data under Apache 2.0 — rare, and the reason I could build a public calculator on top of it at all.

Peak prefill throughput per GPU, DeepSeek-V4-Pro at 8192 in / 1024 out, best disaggregated configuration per platform:

But the number that actually surprised me is not in that table. Across the whole dataset, median prefill throughput is 1,287 tok/s/GPU colocated and 6,025 disaggregated — roughly 4.7×. Separating prefill from decode is a larger lever than most of what I set out to measure.

It also collapses two questions into one. Disaggregated serving means KV state has to move from the prefill GPUs to the decode GPUs. So "where does the cache live" and "how do you route for affinity" stop being separate problems at fleet scale. They become a bandwidth problem, and the fabric becomes a hardware conversation rather than a software one.

Where does the electricity come in?

This is why the whole thing sits under UnitStack rather than standing alone.

Follow the chain down. Duplicated prefill becomes prefill tokens. Prefill tokens become GPU-seconds. GPU-seconds become GPU-hours once you divide by utilisation. And then:

kWh = GPU-hours × kW per GPU × PUE

Which means the prompt-order switch two sections up is not only a billing decision. Reorder a prompt, and the annual megawatt-hours move. A software choice made in a config file lands on somebody's cooling plant, and neither person can currently see the other.

That conversion is one line of arithmetic and I have not seen anyone publish it against a real agent workload. It is the next thing I am building.

What this doesn't tell you

Four things, and I would rather say them than have you find them.

The goodput correction cuts against my own conclusion. GPUs fail, jobs restart, and some of the hours you paid for produce nothing. SemiAnalysis put total goodput loss between roughly 13.5% and 28.4% depending on provider tier. That loss is asymmetric: on a frontier API you do not pay for the provider's failures, and on hardware you own you pay for every one. It is a real argument against the case I am making, and any TCO model that leaves it out is selling something.

Sharing needs architectural identity. Same base model across agents. Specialise by system prompt or LoRA adapters, not by running different models — different weights mean different cache formats and the whole mechanism evaporates.

Gains concentrate in prefill-heavy work. Long shared context, short outputs. A long-generation workload is decode-bound and will barely notice any of this.

Nobody knows whose joules those are. Under continuous batching, forty requests share one forward pass. There is no physical fact about how many joules belong to request seventeen. You have to pick an allocation rule — by tokens, by sequence length, by KV footprint, by marginal cost — and different defensible rules give different answers. I would rather publish the rule and the disagreement than quietly pick one and print a number.

Metrics

Coordination tax was the name before I understood the mechanism. It turns out to be, quite literally, a cache miss.

Reproduce it

Every figure here should regenerate from the published configuration on your own hardware. Where a number is an assumption, the calculator takes yours instead of mine.

Results that only hold on the author's cluster are marketing.

Methods note

Hardware throughput data from InferenceX (SemiAnalysis), Apache 2.0, attribution required. Goodput loss ranges cited from ClusterMAX (SemiAnalysis); their methodology is theirs and linked rather than reproduced. Model pricing via the CloudPrice API, fetched 20 July 2026, 12:58 AM UTC.

One caveat for anyone else using the InferenceX interactivity export: the TTFT, TPOT, ITL and E2E Latency columns are labelled in milliseconds but the values appear to be in seconds. 1/TPOT reproduces the interactivity column exactly, and TTFT + OSL × TPOT reconciles with end-to-end latency — both only hold if the unit is seconds. I lost an evening to it. Worth checking against the current release before you rely on it either way.