On August 5, OpenAI released gpt-oss-120b and gpt-oss-20b, its first open-weight language models since GPT-2, under the Apache 2.0 license. The headline numbers were about hardware, not benchmarks: the 120b model fits on a single 80 GB GPU and the 20b model runs in 16 GB of memory. On the same day, Ollama, llama.cpp, vLLM and Hugging Face shipped support.

That makes it a good moment to write down how local inference actually works. Whether a model "fits" depends on three numbers you can calculate in advance: the parameter count, the bits per weight after quantization, and the memory for the context (the KV cache). Which tool you should use depends on whether you are one person on a laptop or a team serving many requests.

This cheatsheet covers the formats, the memory math, the three main tools, and the OpenAI-compatible endpoints that let you swap a local model into existing code.

1. Vocabulary

Term Meaning
Open-weight The trained weights are downloadable and can be run anywhere. Says nothing about training data or code being open
Parameters Number of weights. "20b" means about 20 billion
Precision Bits used per weight: FP32 (32), BF16/FP16 (16), INT8/FP8 (8), 4-bit formats
Quantization Storing weights in fewer bits than they were trained in, trading a little quality for a lot of memory and speed
bpw (bits per weight) Average bits per weight including scales and metadata. Q4_K_M is about 4.9, not 4
KV cache The attention keys and values stored for every token in the context. Grows linearly with context length
Offloading Keeping some layers in system RAM (CPU) and the rest in GPU VRAM. Works, but the CPU part is much slower
MoE (mixture of experts) Only some expert sub-networks run per token. All weights must be in memory; active parameters set the speed
Prompt processing / generation Reading the input (compute-bound, fast) vs. producing tokens one at a time (memory-bandwidth-bound, slower)

2. Quantization formats

Format Used by What it is Notes
Safetensors (BF16/FP16) Transformers, vLLM Unquantized reference weights 2 bytes per parameter. The source for every other format
GGUF llama.cpp, Ollama, LM Studio Single binary file with tensors and metadata (architecture, tokenizer, chat template). Introduced by llama.cpp in August 2023, replacing GGML One file to download and run. Contains many quantization types
GPTQ vLLM, Transformers Post-training quantization using approximate second-order information, layer by layer (Frantar et al., 2022) Common for 4-bit GPU serving. Needs a calibration dataset
AWQ vLLM, Transformers Activation-aware weight quantization: protects the small share of weights that matter most, judged by activation magnitudes (Lin et al., 2023) Similar use as GPTQ; often comparable quality at 4 bits
FP8 vLLM on recent NVIDIA and AMD GPUs 8-bit floating point weights (and optionally activations) Near-lossless, about half the memory of BF16
MXFP4 gpt-oss; llama.cpp, Ollama, vLLM OCP Microscaling format (2023): blocks of 32 FP4 (E2M1) values sharing one 8-bit (E8M0) scale, so 4 + 8/32 = 4.25 bits per weight New in practice with gpt-oss, which ships its MoE weights in MXFP4

GGUF quantization types

Names follow a pattern: Q<bits>_<scheme>_<size>. K means the "k-quant" scheme (super-blocks with their own scales), and S/M/L are small, medium and large mixes, where more sensitive tensors get more bits. IQ types use an importance matrix for better quality at very low bit rates. Q4_0, Q5_0 and Q8_0 are older, simpler block formats.

Measured by llama.cpp on Llama 3.1 8B (16.0 bpw, 14.96 GiB in F16):

Type bpw Size (GiB) Use it when
Q8_0 8.50 7.95 Memory is not tight and you want results very close to F16
Q6_K 6.56 6.14 A safe high-quality choice
Q5_K_M 5.70 5.33 Good balance if Q4 shows quality problems
Q4_K_M 4.89 4.58 The default. Also what llama-server -hf picks when a repo has it
IQ4_XS 4.46 4.17 Slightly smaller than Q4_K_M at similar quality
Q3_K_M 4.00 3.74 Squeezing a bigger model into limited memory
Q2_K 3.16 2.95 Last resort; quality loss is noticeable

Rules of thumb:

  • A bigger model at Q4 usually beats a smaller model at Q8 for the same memory, as long as you stay at roughly 4 bits or more.
  • Small models suffer more from aggressive quantization than large ones. Be more conservative below about 8B parameters.
  • Test on your own tasks. Quantization damage shows up unevenly: long reasoning chains, code and non-English text tend to show it first.

3. VRAM math

Memory needed is roughly:

total = weights + KV cache + overhead

  • Weights = parameters x bpw / 8 bytes. An 8B model at Q4_K_M: 8.03e9 x 4.89 / 8 = about 4.9 GB (4.6 GiB).
  • KV cache = 2 (K and V) x layers x KV heads x head dimension x bytes per element x tokens. Models with grouped-query attention (few KV heads) need much less.
  • Overhead: compute buffers, CUDA context, runtime. Budget about 0.5-2 GB, more for large batches.

Quick mental math for weights:

Precision GB per billion parameters 8B 32B 70B
BF16/FP16 2.0 16 64 140
Q8_0 / FP8 about 1.06 / 1.0 8.5 34 75
Q4_K_M about 0.61 4.9 20 43
MXFP4 / 4.25 bpw about 0.53 4.3 17 37

The KV cache is what surprises people. For Llama 3.1 8B (32 layers, 8 KV heads, head dimension 128), FP16 KV cache costs 128 KiB per token: 1 GiB at 8K context, 16 GiB at the full 128K. The context, not the weights, is often what stops a model from fitting.

A small estimator:

GIB = 1024**3

# Bits per weight measured by llama.cpp on Llama 3.1 8B (tools/quantize README)
BPW = {"F16": 16.0, "Q8_0": 8.50, "Q6_K": 6.56, "Q5_K_M": 5.70,
       "Q4_K_M": 4.89, "Q3_K_M": 4.00, "IQ4_XS": 4.46}


def weights_gib(params_b: float, quant: str) -> float:
    """Memory for the weights: parameters x bits per weight / 8."""
    return params_b * 1e9 * BPW[quant] / 8 / GIB


def kv_cache_gib(layers: int, kv_heads: int, head_dim: int, ctx: int,
                 bytes_per_elem: float = 2, batch: int = 1) -> float:
    """K and V for every layer and every token in the context (FP16 by default)."""
    return 2 * layers * kv_heads * head_dim * bytes_per_elem * ctx * batch / GIB


def estimate(name, params_b, quant, layers, kv_heads, head_dim, ctx, overhead_gib=1.0):
    w = weights_gib(params_b, quant)
    kv = kv_cache_gib(layers, kv_heads, head_dim, ctx)
    total = w + kv + overhead_gib
    print(f"{name:<22} {quant:<7} ctx={ctx:>6}: weights {w:5.1f} + KV {kv:5.1f} "
          f"+ overhead {overhead_gib:.1f} = {total:5.1f} GiB")


# Llama 3.1 8B: 32 layers, 8 KV heads, head_dim 128 (from config.json)
estimate("Llama 3.1 8B", 8.03, "Q4_K_M", 32, 8, 128, 8_192)
estimate("Llama 3.1 8B", 8.03, "Q4_K_M", 32, 8, 128, 131_072)
estimate("Llama 3.1 8B", 8.03, "Q8_0", 32, 8, 128, 32_768)
# Llama 3.1 70B: 80 layers, 8 KV heads, head_dim 128
estimate("Llama 3.1 70B", 70.6, "Q4_K_M", 80, 8, 128, 8_192)
estimate("Llama 3.1 70B", 70.6, "F16", 80, 8, 128, 8_192)

# Llama 3.1 8B           Q4_K_M  ctx=  8192: weights   4.6 + KV   1.0 + overhead 1.0 =   6.6 GiB
# Llama 3.1 8B           Q4_K_M  ctx=131072: weights   4.6 + KV  16.0 + overhead 1.0 =  21.6 GiB
# Llama 3.1 8B           Q8_0    ctx= 32768: weights   7.9 + KV   4.0 + overhead 1.0 =  12.9 GiB
# Llama 3.1 70B          Q4_K_M  ctx=  8192: weights  40.2 + KV   2.5 + overhead 1.0 =  43.7 GiB
# Llama 3.1 70B          F16     ctx=  8192: weights 131.5 + KV   2.5 + overhead 1.0 = 135.0 GiB

Ways to cut memory when it does not fit:

Lever Effect Cost
Lower quant (Q5 to Q4) Weights shrink about 15-20% Some quality
Shorter context KV cache shrinks linearly Less input per request
Quantized KV cache (llama.cpp --cache-type-k q8_0, and -ctv for V) KV cache roughly halved at 8 bits Small quality cost; check quality on long inputs
Partial offload (-ngl / --n-gpu-layers in llama.cpp) Fits models larger than VRAM Generation slows sharply with each layer on the CPU
Apple Silicon unified memory GPU can use most of system RAM Lower bandwidth than high-end discrete GPUs

Speed in one sentence

Token generation is limited by memory bandwidth: every generated token reads all active weights once. That is why a quantized model is faster as well as smaller, and why MoE models like gpt-oss (5.1B active parameters for the 120b model) generate far faster than their total size suggests.

4. The gpt-oss models

gpt-oss-120b gpt-oss-20b
Total parameters 117B 21B
Active per token 5.1B 3.6B
Architecture MoE Transformer MoE Transformer
Context 128K 128K
Memory target Single 80 GB GPU (e.g. H100, MI300X) 16 GB of memory (consumer hardware)
License Apache 2.0 Apache 2.0

What is specific about them:

  • MXFP4 out of the box. OpenAI post-trained the models with the MoE weights quantized to MXFP4 (4.25 bits per parameter). MoE weights are over 90% of all parameters, so the released checkpoint is already "quantized" and small: vLLM cites about 63 GB for the 120b model. Do not expect a big win from re-quantizing it to Q4_K_M; Ollama and llama.cpp run MXFP4 natively.
  • Reasoning effort. Both models support low, medium and high reasoning effort, set in the system prompt (e.g. "Reasoning: high"). Higher effort means more tokens and latency.
  • Harmony format. They were trained on OpenAI's "harmony" response format, and the model card says they will not work correctly without it. Use a recent version of your runtime and its built-in chat template (in llama.cpp, pass --jinja) rather than a hand-written prompt format.

5. Choosing a tool

Ollama llama.cpp vLLM
Best for One user, laptop or workstation, fastest start Full control, any hardware, CPU and GPU mix Serving many concurrent users on GPUs
Formats GGUF (its own registry, or imports), MXFP4 for gpt-oss GGUF Safetensors: BF16, FP8, GPTQ, AWQ, MXFP4 (gpt-oss)
Hardware macOS, Windows, Linux; NVIDIA, AMD, Apple Almost anything: CPU, CUDA, Metal, Vulkan, ROCm NVIDIA and AMD datacenter and recent consumer GPUs, Linux
Throughput Good for one user Good for one or a few users Highest under concurrency: PagedAttention and continuous batching
Default port 11434 8080 8000
Effort Lowest Medium: you pick files and flags Highest: Python, CUDA versions, GPU memory tuning

A simple decision rule:

  1. Just want to try models locally? Ollama.
  2. Need a specific GGUF quant, CPU offload, an unusual GPU, or maximum control? llama.cpp.
  3. Serving an app with several concurrent users on real GPUs? vLLM.

6. Commands

Ollama

ollama run gpt-oss:20b              # pulls the model on first use, then opens a chat
ollama pull gpt-oss:120b            # download only
ollama ps                           # what is loaded, and how much is on GPU vs CPU
ollama list                         # installed models

OLLAMA_CONTEXT_LENGTH=16384 ollama serve   # raise the default context window
# inside a chat:  /set parameter num_ctx 16384

Ollama's default context window is 4,096 tokens for most models (8,192 for gpt-oss). That is a common cause of "the model ignores the start of my document": raise it, and remember the KV cache grows with it.

llama.cpp

# Download from Hugging Face and serve (default quant: Q4_K_M if the repo has it)
llama-server -hf ggml-org/gpt-oss-20b-GGUF --jinja -c 16384 -ngl 99 --port 8080

# A local GGUF, with flash attention and an 8-bit KV cache
llama-server -m ./models/qwen-7b-instruct-Q4_K_M.gguf -c 32768 -ngl 99 -fa \
  --cache-type-k q8_0 --cache-type-v q8_0

# Make your own quant from an F16 GGUF
llama-quantize ./model-f16.gguf ./model-Q4_K_M.gguf Q4_K_M

-ngl 99 means "put all layers on the GPU"; lower it until the model fits if you run out of VRAM. -c sets the context size (default 4,096; 0 loads the model's value).

vLLM

# OpenAI-compatible server on port 8000
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ --max-model-len 16384 --gpu-memory-utilization 0.90

# gpt-oss-120b on a single 80 GB GPU (needs a vLLM build with gpt-oss support)
vllm serve openai/gpt-oss-120b

# Split a large model across 2 GPUs
vllm serve <model> --tensor-parallel-size 2

vLLM pre-allocates a fixed share of GPU memory (--gpu-memory-utilization) and fills what is left after the weights with KV cache blocks. --max-model-len caps the context, which is often the fix for "not enough KV cache" errors at startup.

7. OpenAI-compatible endpoints

All three tools expose an OpenAI-style /v1/chat/completions endpoint, so the official openai Python client works by changing the base URL. Ollama added this in February 2024; llama.cpp's llama-server and vLLM's server have it built in, and vLLM also serves the Responses API for gpt-oss.

from openai import OpenAI

BACKENDS = {
    "ollama": ("http://localhost:11434/v1", "gpt-oss:20b"),
    "llama.cpp": ("http://localhost:8080/v1", "gpt-oss-20b"),  # llama-server serves one model; the name is informational
    "vllm": ("http://localhost:8000/v1", "openai/gpt-oss-120b"),
}

base_url, model = BACKENDS["ollama"]
client = OpenAI(base_url=base_url, api_key="not-needed-locally")  # required by the client, ignored by default

resp = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": "Reasoning: low. Answer in two sentences."},
        {"role": "user", "content": "Why does the KV cache grow with context length?"},
    ],
    temperature=0.2,
)
print(resp.choices[0].message.content)

Compatibility notes:

  • Chat completions are reliable; the rest varies. Tool calling, structured output (JSON schema), logprobs and embeddings differ between tools and versions. Test what you need.
  • Model names differ. Ollama uses tags (gpt-oss:20b), vLLM the Hugging Face repo ID, llama.cpp whatever is loaded.
  • Secure it before exposing it. Ollama binds to 127.0.0.1 by default. vLLM supports --api-key; llama-server has --api-key too. Do not put an unauthenticated model server on a public interface.
  • Many tools accept a custom base URL (coding agents, IDE extensions, LangChain and similar), so a local model can often be swapped in with configuration only.

8. A practical checklist

  • [ ] Calculate weights + KV cache + overhead before downloading 60 GB.
  • [ ] Start with Q4_K_M (GGUF) or AWQ/GPTQ 4-bit (vLLM); move up a level if quality is not enough.
  • [ ] Set the context length deliberately: big enough for your inputs, no bigger.
  • [ ] Check that all layers are on the GPU (ollama ps, llama.cpp startup logs, vLLM memory report).
  • [ ] Use the model's own chat template; for gpt-oss, a runtime version with harmony support.
  • [ ] Evaluate on 20-50 of your real prompts against the hosted model you would otherwise use.
  • [ ] Put authentication in front of any server reachable from the network.

Why this matters

  • Memory is predictable. Parameters x bpw plus KV cache tells you in advance whether a model fits. No trial and error with 60 GB downloads.
  • 4-bit is the practical default. Q4_K_M and MXFP4 make 20B-class models laptop-friendly and 120B-class MoE models single-GPU.
  • Context costs memory. Long-context use cases are often limited by the KV cache, not the weights.
  • The tool follows the workload. Ollama for one person, llama.cpp for control and odd hardware, vLLM for concurrency.
  • OpenAI-compatible APIs make local models a configuration change, which makes privacy, cost and offline use real options rather than rewrites.

Final Thoughts

gpt-oss made local inference mainstream news, but the mechanics are the same for every open-weight model: pick a precision, do the memory math, choose a runtime that matches the workload, and point your existing client at a local URL.

Start small: an Ollama install and a 20B-class model on the hardware you have. Measure quality on your own prompts and memory with ollama ps. When you need more control or more users, the same model and the same client code move to llama.cpp or vLLM with little more than a change of base URL.

Sources