For developers · Updated
Serve an open model as an OpenAI-compatible API with vLLM
KernelMachine is preparing an invite-only beta; no launch date is set. Where this guide describes KernelMachine, it's how it works at launch. Join the waitlist
Short answer: install vLLM with pip install vllm, start a model with vllm serve Qwen/Qwen3-8B --max-model-len 8192, and point any OpenAI client at http://localhost:8000/v1. An 8B model at full 16-bit quality needs a 24 GB card; use a 4-bit (AWQ or GPTQ) version on a 16 GB card.
vLLM is an open-source server for running language models fast. It speaks the same API as OpenAI's, so code written for OpenAI works by changing one address. Its strength is serving many requests at once: it batches them together, which gets far more total throughput from one card than answering them one by one.
1. Pick a model that fits
vLLM usually runs models at full 16-bit quality, about 2 GB per billion parameters, plus room for conversations. It reserves 90% of the card's memory at start and uses whatever the weights don't need as conversation memory.
| Model | Weights | Card |
|---|---|---|
| Qwen3 8B, 16-bit | about 16.4 GB | 24 GB (RTX 3090, 4090) |
| Qwen3 8B, 4-bit AWQ | about 6 GB | 12 to 16 GB |
| Qwen3 14B, 4-bit AWQ | about 10 GB | 16 to 24 GB |
| Qwen3 32B, 4-bit AWQ | about 19 GB | 32 GB (RTX 5090) |
Many model makers publish 4-bit versions (look for "AWQ" or "GPTQ" in the name on Hugging Face). Check any model's needs on our model pages.
2. Install and start it
$ pip install vllm
$ vllm serve Qwen/Qwen3-8B --max-model-len 8192 --host 127.0.0.1The first start downloads the model, which takes a few minutes for 16 GB. When it prints that the server is running, it's listening on port 8000.
--max-model-lencaps the conversation length. Qwen3 allows about 40,000 tokens, more than a 24 GB card can hold next to the 16-bit weights, so without the cap vLLM stops with "larger than the maximum number of tokens that can be stored in KV cache".--host 127.0.0.1keeps it off the network. Reach it through an SSH tunnel, or add--api-keywith a long secret if it must be reachable.--gpu-memory-utilization 0.8leaves more room if something else shares the card.
RTX 50-series cards need a vLLM release built for CUDA 12.8 or newer; older ones fail with "no kernel image is available". See PyTorch and CUDA versions.
3. Call it
From the command line:
$ curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Hello"}]}'From Python, with OpenAI's own library:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
reply = client.chat.completions.create(
model="Qwen/Qwen3-8B",
messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
)
print(reply.choices[0].message.content)Qwen3 thinks step by step before answering by default. To skip that for quick replies, add extra_body={"chat_template_kwargs": {"enable_thinking": False}} to the call.
vLLM or Ollama?
| vLLM | Ollama | |
|---|---|---|
| Best for | Serving an app or many users at once | One person chatting, quick experiments |
| Models | Hugging Face models, 16-bit or AWQ/GPTQ 4-bit | Its own library, mostly 4-bit GGUF |
| Memory | Takes 90% of the card up front | Takes what the model needs |
| API | OpenAI-compatible, port 8000 | Its own plus OpenAI-compatible, port 11434 |
For one conversation at a time, both are limited by the card's memory bandwidth, so they answer at similar speeds. The difference shows when requests arrive together. Why bandwidth sets the speed
On KernelMachine
Pick the Fine-tuning template: vLLM comes installed. Open its terminal in Jupyter and run vllm serve with your model. Notebook cells reach it at localhost:8000. From your own computer, run km open: it prints a local address such as http://localhost:8888, and the API's base address is then http://localhost:8888/proxy/8000/v1. Only you can reach it, the same as the rental's Jupyter. For chatting with a model in your browser, the Chat model template runs Ollama with Open WebUI. What's inside each image
For a batch job (a file of prompts in, a file of answers out), use vLLM's offline mode in a km run job: the card stops when the batch is done. Put the steps in a short script; words after -- are its arguments, here a presigned link to upload the answers to. Installing vLLM takes a few minutes, and that time is billed.
$ cat batch.sh
pip install -q vllm
curl -fsSL https://example.com/batch.py -o batch.py # your script, from a link
python batch.py # writes answers.jsonl
curl --fail -T answers.jsonl "$1"
$ km run --size 24GB --script batch.sh -- "$UPLOAD_URL"