How Much GPU VRAM Does a Local LLM Need?
VRAM decides which local models fit; GPU speed decides how quickly those models run. Buying a fast 8 GB card solves the second problem while leaving the first one untouched.
Start with the weight floor
Use this approximation before shopping:
weight memory in GB ≈ parameters in billions × bits per weight ÷ 8
| Model size | 4-bit weight floor | 8-bit weight floor | FP16 weight floor |
|---|---|---|---|
| 8B | ~4 GB | ~8 GB | ~16 GB |
| 14B | ~7 GB | ~14 GB | ~28 GB |
| 32B | ~16 GB | ~32 GB | ~64 GB |
| 70B | ~35 GB | ~70 GB | ~140 GB |
The table is only the starting line. Model metadata, runtime buffers, and the key-value cache consume additional memory. Different quantization formats with the same nominal bit count can also differ in size.
Context is not free
Long context increases the cache used to remember previous tokens. The exact cost depends on the model architecture, cache precision, batch size, and runtime. A model that starts comfortably at a short context can run out of memory during a long coding session. Leave meaningful headroom instead of planning to fill every advertised gigabyte with weights.
Concurrency multiplies the problem. Two agents with separate contexts are not one agent taking turns; both may need active cache and runtime memory.
What the common capacities mean
- 8 GB: compact 4-bit models and short contexts. Useful entry point, tight for serious agent work.
- 12 GB: more comfortable for 7B–14B-class quantized models. The Arc B580 is a capacity-value option when runtime support is confirmed.
- 16 GB: the practical single-GPU sweet spot for larger compact models and some 30B-class 4-bit workloads. See the RTX 5080 and RX 9070 XT.
- 24 GB: roomier long-context and multimodal work, but still below the raw 4-bit weight floor of many 70B models.
- 32 GB: excellent single-card capacity; enough for many serious workflows, but not a blanket 70B-fit promise. The RTX 5090 sits here.
- 48 GB+: the range where full-GPU 70B-class 4-bit work becomes realistic, subject to runtime overhead and context.
Multi-GPU is capacity with conditions
Two cards can provide aggregate memory only when the runtime supports splitting the model across them. The workload also pays communication overhead, and the motherboard must provide suitable slots and spacing while the case and PSU handle the heat and power. VRAM does not automatically pool into one transparent bucket merely because a second GPU is installed.
The buying rule
Pick the model, quantization, maximum context, and concurrent-session count first. Confirm them against the runtime’s current documentation, then buy the smallest GPU configuration with comfortable headroom. If the requirement lands between consumer capacities, system-RAM offload is the economical compromise; it is slower, but much cheaper than pretending a smaller model is the same job.