Notes on GPU costs
Blog
- How many GPUs does your model need? A back-of-the-envelope method Weights times bytes per parameter, plus headroom for the KV cache, divided by usable VRAM. Then round to a power of two.
- Your GPU bill has two halves. Only one of them serves tokens. Split paid GPU-hours into served and idle, and the real cost of every token falls out of one division.