Decide at any layer,
optimise every layer

Why does multi-agent inference cost so much more than it should?

The coordination tax turns out to be a cache miss. And at 150 GPUs it comes back.

The short answer

Ten agents reading the same case file prefill it ten times. On a metered API you pay for all ten and none of it looks like waste, because it is billed as input tokens, which is exactly what it is.

Removing that duplication means reusing key-value state across agents. That only works if you operate the inference server. No frontier API exposes it.

And here is the counterintuitive part. The fix works beautifully on one node and returns almost nothing at 150, unless routing knows which node already holds your prefix. At one node the cache hit rate is 96%. At 150 nodes with ordinary load balancing it is 7%. Same code, same model, same hardware.

That number is not a measurement. It falls out of arithmetic, which is why it is more trustworthy than most of the numbers in this piece.

Where does the duplication come from?

Model APIs have no memory. Every call carries the whole history, so a twenty-step agent sends its opening context twenty times. Put ten agents on one document and you have paid to read it a hundred times.

The provider caches help — when they can. Prefix caching only resumes if the new request is byte-identical up to the cached point. One changed character and everything after it is recomputed at full price.

Agents differ by construction. That is what makes them different agents. So their prefixes diverge, and how much they diverge depends entirely on where in the prompt the divergence starts.

Which deployment tiers can actually fix it?

The line that matters is not cloud versus on-premise. It is tenant versus operator.

Rented GPU gives you every cache capability that owned hardware does. Renting a bare instance and running your own vLLM or SGLang puts you on the operator side of that line, in someone else's building.

On-premise adds three separate things: utilisation economics, fabric choice, and data residency. None of them is a caching argument. Keep them apart. An architect who hears "you need on-prem for KV caching" will correctly reject it, and you lose the room.

Does prompt order really matter more than hardware?

Sometimes, yes. This is the cheapest finding in the whole piece.

If role instructions sit at the top of the prompt, prefixes diverge at position zero and no cache hits anywhere — including on frontier APIs, where it would otherwise have worked. Move the shared corpus to the front and role instructions to the end, and a large fraction of the benefit comes back without changing a single thing about your infrastructure.

Most agent frameworks put role instructions first. It reads more naturally. It costs a fortune.

What happens to cache hit rate at 150 GPUs?

This is the section that reframes the whole problem.

A request only hits a warm cache if it lands on the node that already holds the prefix. With random load balancing over N nodes and K calls sharing that prefix, the number of distinct nodes touched — and therefore the number of cold prefills — is a balls-in-bins expectation:

cold(N, K) = N · (1 − (1 − 1/N)^K)
hit_rate   = 1 − cold(N, K) / K

N =   1, K = 24  →   1 cold prefill   →  96% hit rate
N =  16, K = 24  →  ~13 cold prefills →  46% hit rate
N = 150, K = 24  →  ~22 cold prefills →   7% hit rate

No hardware term appears anywhere in that expression. It does not care which GPU you bought, which model you serve, or which framework you run. It is a property of routing.

Which means the mechanism everyone is optimising at single-node scale quietly stops paying somewhere between sixteen and forty nodes, and the fix is not a better kernel. It is prefix affinity — pinning a task's calls to the pipeline that already holds its state.

What does the hardware actually do?

You need real throughput numbers to turn any of this into money. These come from InferenceX, which publishes its data under Apache 2.0 — rare, and the reason a public calculator can be built on top of it at all.

Peak prefill throughput per GPU, DeepSeek-V4-Pro at 8192 in / 1024 out, best disaggregated configuration per platform:

But the most striking number is not in that table. Across the whole dataset, median prefill throughput is 1,287 tok/s/GPU colocated and 6,025 disaggregated — roughly 4.7×. Separating prefill from decode is a larger lever than most of what this benchmark set out to measure.

It also collapses two questions into one. Disaggregated serving means KV state has to move from the prefill GPUs to the decode GPUs. So "where does the cache live" and "how do you route for affinity" stop being separate problems at fleet scale. They become a bandwidth problem, and the fabric becomes a hardware conversation rather than a software one.

Where does the electricity come in?

This is why the whole thing sits under UnitStack rather than standing alone.

Follow the chain down. Duplicated prefill becomes prefill tokens. Prefill tokens become GPU-seconds. GPU-seconds become GPU-hours once you divide by utilisation. And then:

kWh = GPU-hours × kW per GPU × PUE

Which means the prompt-order switch two sections up is not only a billing decision. Reorder a prompt, and the annual megawatt-hours move. A software choice made in a config file lands on somebody's cooling plant, and neither person can currently see the other.

That conversion is one line of arithmetic, and no published version of it exists against a real agent workload yet. It is the next thing UnitStack is building.

What this doesn't tell you

Four things, stated here rather than left for you to find.

The goodput correction cuts against the conclusion here. GPUs fail, jobs restart, and some of the hours you paid for produce nothing. SemiAnalysis put total goodput loss between roughly 13.5% and 28.4% depending on provider tier. That loss is asymmetric: on a frontier API you do not pay for the provider's failures, and on hardware you own you pay for every one. It is a real argument against the case made here, and any TCO model that leaves it out is selling something.

Sharing needs architectural identity. Same base model across agents. Specialise by system prompt or LoRA adapters, not by running different models — different weights mean different cache formats and the whole mechanism evaporates.

Gains concentrate in prefill-heavy work. Long shared context, short outputs. A long-generation workload is decode-bound and will barely notice any of this.

Nobody knows whose joules those are. Under continuous batching, forty requests share one forward pass. There is no physical fact about how many joules belong to request seventeen. You have to pick an allocation rule — by tokens, by sequence length, by KV footprint, by marginal cost — and different defensible rules give different answers. Better to publish the rule and the disagreement than quietly pick one and print a number.

Metrics

Coordination tax was the name before the mechanism was understood. It turns out to be, quite literally, a cache miss.

Reproduce it

Every figure here should regenerate from the published configuration on your own hardware. Where a number is an assumption, the calculator takes yours instead of mine.

Results that only hold on the author's cluster are marketing.

Methods note

Hardware throughput data from InferenceX (SemiAnalysis), Apache 2.0, attribution required. Goodput loss ranges cited from ClusterMAX (SemiAnalysis); their methodology is theirs and linked rather than reproduced. Model pricing via the CloudPrice API, fetched 20 July 2026, 12:58 AM UTC.

One caveat for anyone else using the InferenceX interactivity export: the TTFT, TPOT, ITL and E2E Latency columns are labelled in milliseconds but the values appear to be in seconds. 1/TPOT reproduces the interactivity column exactly, and TTFT + OSL × TPOT reconciles with end-to-end latency — both only hold if the unit is seconds. It is an easy evening to lose. Worth checking against the current release before you rely on it either way.