Blog · November 4, 2026
Your GPU bill has two halves. Only one of them serves tokens.
A GPU invoice tells you how many GPU-hours you paid for. It doesn’t tell you how many of those hours actually served a request. For LLM inference that gap is where most of the waste hides: a replica kept warm overnight, a node sized for a peak that comes twice a week, a spare model nobody calls anymore.
Paid hours vs served hours
Every GPU-hour you pay for lands in one of two buckets:
- Served: the GPU was doing work for a real request.
- Idle: the GPU was allocated and billed, but waiting.
Utilization is the ratio between them:
utilization = served GPU-hours / paid GPU-hours
idle share = 1 − utilization
You don’t need perfect telemetry to start. Average GPU utilization per endpoint and day is enough to see the shape of the problem.
What idle time does to your cost per token
The price you see is per paid GPU-hour. The price you actually pay is per served GPU-hour:
effective $/GPU-hour = list $/GPU-hour ÷ utilization
At 50% utilization every useful hour costs twice the list price. At 25%, four times. That multiplier carries straight through to cost per token:
cost per 1M tokens = (paid GPU-hours × $/GPU-hour) ÷ tokens served × 1,000,000
That’s why two teams renting the same GPU at the same price can see very different unit costs. The rate is identical; the utilization isn’t.
Where idle time usually comes from
- Warm replicas for latency. Keeping a minimum replica count avoids cold starts, but off-peak those replicas mostly wait.
- Sizing for the peak. A fleet sized for the busiest hour runs underused the rest of the day unless it scales down.
- Oversized GPUs. A model that fits on one GPU spread across a bigger node, or a large GPU serving a small model at low concurrency.
- Forgotten endpoints. Experiments, old model versions and staging copies that still bill every hour.
- Low batching. Serving requests one at a time leaves most of the GPU’s throughput unused even when it looks “busy”.
What to do about it
- Measure per endpoint, per day: cost, GPU-hours, utilization and tokens. Rank endpoints by idle dollars, not by idle percentage; a small endpoint at 10% utilization matters less than a big one at 60%.
- Scale with the traffic curve: lower minimum replicas off-peak, or scale to zero where cold starts are acceptable.
- Right-size: check whether the model fits on fewer or smaller GPUs at the precision you serve. Where to run does the memory math for you.
- Batch more: continuous batching in the serving engine raises tokens per GPU-hour without touching the hardware.
- Pay less for the hours you keep: compare on-demand and spot for the same GPU on the GPU Radar.
Put a number on it
The idle cost calculator takes your GPU, count, utilization and $/GPU-hour and gives you monthly and yearly idle spend. It’s an estimate, but it’s usually enough to decide whether the problem is worth an afternoon.
If you’d rather have someone look at it with you, we run a read-only GPU cost audit.