For developers · Updated October 13, 2026

Why memory bandwidth decides how fast an LLM answers, card by card

KernelMachine is preparing an invite-only beta; no launch date is set. Where this guide describes KernelMachine, it's how it works at launch. Join the waitlist

Short answer: when one person chats with a model, each new word (token) means reading all of the model's weights from the card's memory once. So the top speed is roughly the card's memory bandwidth divided by the size of the weights. An RTX 4090 (1,008 GB/s) running an 8B model at 4-bit (about 4.8 GB) tops out near 210 tokens a second. Real speeds are usually half to two-thirds of that ceiling.

This rule lets you compare cards for chat and coding models without a benchmark for every combination. It also explains why compressed (quantized) models are faster, and why a newer card with more compute isn't always faster at answering.

Why memory, not compute, sets the speed

To produce one token, the model multiplies every weight once. For a single conversation that's very little arithmetic per byte read, so the card spends most of its time waiting for memory. Doubling the compute barely helps; doubling the bandwidth nearly doubles the speed. Halving the size of the weights, by using 4-bit instead of 8-bit, does the same.

The ceiling, card by card

Top speed for an 8B chat model at 4-bit (about 4.8 GB of weights) on the cards we list:

Bandwidth from NVIDIA's published specs. The ceiling is bandwidth divided by the weights' size. Data-center cards go much higher: an H100 (SXM) reads about 3,350 GB/s.

Bigger models are slower in proportion. A 32B model at 4-bit (about 19.7 GB) tops out around 48 tokens/s on an RTX 3090, 51 on an RTX 4090 and 91 on an RTX 5090. Check any model on our model pages.

Why real speeds are lower

• Overheads. Software never reads memory at the full published rate. Well-tuned engines get a little over half to about two-thirds of the ceiling for one conversation.
• The conversation itself. The model also re-reads its working memory of the conversation (the KV cache) for every token, so answers slow down as a chat grows long.
• Reading the prompt. Before the first word appears, the model reads your whole prompt. That part is limited by compute, not memory, so very long prompts take a while to start, especially on older cards.

When the rule changes

• Mixture-of-experts models (such as Qwen3 30B-A3B or gpt-oss) read only the experts they use for each token: about 3 to 5 billion parameters instead of all of them. They answer far faster than their size suggests, but still need memory for everything.
• Many users at once. A server such as vLLM answers many conversations in one pass over the weights, so total throughput grows almost in step with the number of users, while each user's speed drops only a little. See serving a model with vLLM.
• Split across two cards. Splitting a model's layers across two cards lets a bigger model fit, but each token still passes through every layer, so it isn't faster than one card with the same bandwidth.

Measure it yourself

Ollama prints the speed after each answer when you add --verbose; look for "eval rate":

llama.cpp's llama-bench tool measures prompt reading and answering separately, which is handy for comparing cards.

Choosing a card on KernelMachine

On KernelMachine, Chat suggests 16 GB, and a model you pick sets the memory it needs, with room for the conversation. Take the cheapest card with that much, or one by name. Among cards with the same memory, more bandwidth means faster answers: of the 24 GB cards, an RTX 3090 (936 GB/s) answers almost as fast as an RTX 4090 (1,008 GB/s) at about a third of the price. The speed the rent page shows next to each card (a 4090 is 1.6× a 3090) counts compute, which matters more for images and training than for chat. Each card page shows its bandwidth and top speed.

Read next

• PyTorch and CUDA versions for every NVIDIA GPU, and the RTX 50 "sm_120" fix
• Serve an open model as an OpenAI-compatible API with vLLM
• Fine-tune an open LLM on one GPU with QLoRA: a working example