Models
What it takes to self-host an open model
Memory, GPUs per replica and cost per 1M output tokens from live GPU prices, for each model idle’s planners know.
| Model | Parameters | Memory BF16 | Memory FP8 | Memory INT4 |
|---|---|---|---|---|
| Llama 3.1 8BMeta | 8B | 19 GB | 10 GB | 5 GB |
| Mistral 7BMistral | 7.2B | 17 GB | 9 GB | 4 GB |
| Qwen 2.5 14BQwen | 14.7B | 35 GB | 18 GB | 9 GB |
| Gemma 2 27BGoogle | 27.2B | 65 GB | 33 GB | 16 GB |
| Qwen 2.5 32BQwen | 32.5B | 78 GB | 39 GB | 20 GB |
| Mixtral 8x7BMistral | 46.7B12.9B active | 112 GB | 56 GB | 28 GB |
| Llama 3.3 70BMeta | 70.6B | 169 GB | 85 GB | 42 GB |
| Qwen 2.5 72BQwen | 72.7B | 174 GB | 87 GB | 44 GB |
| Mixtral 8x22BMistral | 141B39B active | 338 GB | 169 GB | 85 GB |
| Llama 3.1 405BMeta | 405B | 972 GB | 486 GB | 243 GB |
| DeepSeek V3 (671B MoE)DeepSeek | 671B37B active | 1610 GB | 805 GB | 403 GB |
Weights plus ~20% for KV cache and runtime. Your model isn’t here? Enter its size in Where to run.