How many LLM agents fit on your GPU?
The number of agents a GPU can serve at once is set by memory, not compute: model weights take a fixed slice of VRAM, and every active agent needs its own KV cache that grows with context length. This calculator subtracts the weights from your VRAM and divides what is left by the per-agent KV cache for the model and context you pick.
How to use it
- Pick a model and a weight quantization.
- Set the context length each agent needs.
- Pick your GPU's VRAM to see how many agents fit and how much memory each one uses.
Methodology and sources
Per-agent KV cache = 2 (keys and values) x full-attention layers x KV heads x head dimension x context tokens x bytes per element. Layer, head and dimension counts come from each model's published config.json on Hugging Face. Weight sizes come from published GGUF files. Compute buffers and allocator overhead add a little on top, and real servers add paging and batching effects, so treat the result as a planning estimate.
FAQ
Why does context length matter so much?
KV cache grows linearly with context. Doubling each agent's context roughly halves how many agents fit in the memory left after weights.
Does quantizing the KV cache help?
Yes. An 8-bit KV cache uses about half the memory of 16-bit, so roughly twice as many agents fit, with a small quality cost.
Why do mixture-of-experts models fit more agents?
MoE models often have fewer KV heads per layer, so each agent's cache is smaller even when total weights are large.
Is this exact?
No. It is a planning estimate. Runtime overhead, paging and batching vary by server (llama.cpp, vLLM, SGLang).
Can you set up private LLM infrastructure for us?
Yes. We design and run private LLM gateways with model routing and cost control. Start a custom build from the contact page.
Embed this tool
Paste this on your own page or blog post.
<iframe src="https://cyberautomations.com/apps/gpu-agent-sizing/index.html" width="100%" height="640" loading="lazy" style="border:0"></iframe>
<p><a href="https://cyberautomations.com/tools/gpu-agent-sizing">Tool by CyberAutomations</a></p>Related service
LLM gateways, model routing and cost control, run on infrastructure you own.
Need this tuned to your data?
This tool runs on public data. We build the same kind of thing on your own data, wired into your systems.