Meta · 70.6B parameters · Llama 3.3 Community License
What GPU do you need to run Llama 3.3 70B?Memory, cards and speed.
Even at its smallest (4-bit), Llama 3.3 70B needs about 44 GB of GPU memory: more than any single consumer card has. At its largest (16-bit) it needs about 143 GB.
GPU memory
Llama 3.3 70B, by version.
Includes about 1.5 GB for the program and a short conversation (about 4,000 tokens). Long conversations need more. Speeds are ceilings: real speeds are usually half to two-thirds of them. How we work it out
| Version | Memory needed | Card size | Cards that fit | From | Top speed |
|---|---|---|---|---|---|
| 4-bit | about 44 GB | None | More than any consumer card holds | — | — |
| 8-bit | about 76 GB | None | More than any consumer card holds | — | — |
| 16-bit | about 143 GB | None | More than any consumer card holds | — | — |
Meta's strongest model that's practical to run yourself. Too big for one consumer card, even at 4-bit: it needs a 48 GB card or two 24 GB cards.
Run it
Too big for one card here, for now.
Llama 3.3 70B needs about 44 GB even at its smallest, and the biggest card on KernelMachine has 32 GB. Ollama would run most of it from the computer's memory instead, many times slower, so a smaller model is the better pick here. On a machine with enough GPU memory, one command downloads and runs it. Models that fit
$ ollama run llama3.3:70b
# Model files: huggingface.co/meta-llama/Llama-3.3-70B-Instruct
Common questions
Questions, answered plainly.
How much VRAM does Llama 3.3 70B need?
4-bit: about 44 GB; 8-bit: about 76 GB; 16-bit: about 143 GB. That includes about 1.5 GB for the program and a short conversation; longer conversations need more.
Can an RTX 4090 run Llama 3.3 70B?
Not on its own. Even at its smallest, Llama 3.3 70B needs about 44 GB, more than the RTX 4090's 24 GB.
Can I run Llama 3.3 70B on KernelMachine?
Not yet. KernelMachine rents one card at a time, and the biggest has 32 GB; Llama 3.3 70B needs about 44 GB even at its smallest.
Other models
Compare with other chat models.
Preparing an invite-only beta
Get an email when access opens.
No launch date is set. We'll email you once when KernelMachine opens. Nothing else.