Blog · November 4, 2026
How many GPUs does your model need? A back-of-the-envelope method
Before comparing prices you need to know what you’re pricing: how many GPUs, of which type, does the model actually need? You can get a good first answer with four numbers and no benchmarks.
Step 1: memory for the weights
Each parameter takes a fixed number of bytes, set by the precision you serve at:
| Precision | Bytes per parameter | 70B model |
|---|---|---|
| BF16 / FP16 | 2 | 140 GB |
| FP8 | 1 | 70 GB |
| INT4 | 0.5 | 35 GB |
weights (GB) = parameters (billions) × bytes per parameter
Step 2: headroom for the KV cache and runtime
Weights alone won’t run. The serving engine also needs memory for the KV cache (which grows with context length and concurrent requests), activations and its own runtime. A reasonable starting point is 20% on top of the weights:
memory needed = weights × 1.2
Long contexts or high concurrency need more. Treat 20% as the floor, not the ceiling.
Step 3: divide by usable VRAM
Don’t plan on 100% of a GPU’s memory; leave about 10% for fragmentation and the CUDA context. Then round up to a power of two, because tensor parallelism splits a model across 1, 2, 4 or 8 GPUs:
GPUs = next power of two ≥ memory needed ÷ (VRAM × 0.9)
Beyond 8 GPUs you leave a single node and need multi-node serving, which is a different (and pricier) conversation.
Worked example: a 70B model
Llama 3.3 70B has about 70.6 billion parameters. With the method above:
| GPU | VRAM | BF16 (≈169 GB) | FP8 (≈85 GB) | INT4 (≈42 GB) |
|---|---|---|---|---|
| B300 | 288 GB | 1 | 1 | 1 |
| H200 | 141 GB | 2 | 1 | 1 |
| H100 | 80 GB | 4 | 2 | 1 |
| A100 80GB | 80 GB | 4 | 2* | 1 |
| RTX PRO 6000 | 96 GB | 2 | 1 | 1 |
| RTX 5090 | 32 GB | 8 | 4 | 2 |
| RTX 4090 | 24 GB | 8 | 4 | 2 |
* The A100 has no native FP8 support, so FP8 weights run through slower kernels or get upcast. In practice, plan an A100 deployment in BF16 or INT4.
A few things stand out:
- Precision moves you down a tier. Going from BF16 to FP8 halves the GPU count on almost every row.
- Bigger memory can be cheaper. One H200 at FP8 can replace two H100s. Whether that saves money depends on the two prices, which change weekly; compare them on the GPU Radar.
- Consumer cards fit, but mind the interconnect. Eight RTX 4090s technically hold a 70B model in BF16, but PCIe-only cards without NVLink pay a real throughput cost when a model is split that many ways.
At the large end the same math applies: a 405B model at FP8 needs about 486 GB, so 8× H100, 4× H200 or 2× B300.
What this method leaves out
It sizes for fitting the model, not for your traffic. Throughput depends on memory bandwidth, batch size and output length; once you know your tokens per month, compare renting GPUs against paying an API with Tokens vs GPUs.
Skip the arithmetic
Where to run does all of the above for a catalog of open models (or a custom size) and attaches the cheapest live offer for each GPU, billed in whole instances. Results are estimates, with the price timestamp next to every number.