Inference Optimization and Performance

Running a 7B-parameter model naively on a single GPU gives you about 10 tokens per second. With the right combination of quantization, batching, and KV cache management, you can push that to 100+ tokens/second and serve dozens of concurrent users. The techniques are not magic - they are engineering tradeoffs you need to understand before choosing one.

This article covers the practical optimization stack: quantization, batching strategies, KV cache, and when to use vLLM versus HuggingFace.

Quantization: INT8 and 4-bit

Quantization reduces the precision of model weights, shrinking memory footprint and increasing throughput at some cost to quality.

pythonCopy
# INT8 with bitsandbytes (minimal quality loss for most tasks)
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

int8_config = BitsAndBytesConfig(load_in_8bit=True)
model_int8 = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=int8_config,
    device_map="auto",
)

# 4-bit NF4 (significant memory reduction, slightly more quality loss)
nf4_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model_4bit = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=nf4_config,
    device_map="auto",
)

Memory comparison for Llama 3.1 8B: FP16 = ~16GB, INT8 = ~8GB, 4-bit NF4 = ~4.5GB. For most tasks, INT8 is indistinguishable from FP16 in quality. 4-bit is fine for classification and summarization; avoid it for tasks requiring precise reasoning or math.

Batching: Latency vs Throughput

Serving a single request at a time maximizes latency. Batching maximizes throughput. The tension is real.

pythonCopy
from transformers import pipeline
import time

generator = pipeline("text-generation", model=model_int8, tokenizer=tokenizer,
                      batch_size=8)

def benchmark_batching(prompts: list[str], batch_sizes: list[int]):
    for bs in batch_sizes:
        start = time.perf_counter()
        results = generator(prompts[:bs], max_new_tokens=100, do_sample=False)
        elapsed = time.perf_counter() - start
        throughput = bs / elapsed
        latency = elapsed / bs
        print(f"batch={bs}: {throughput:.1f} req/s, {latency*1000:.0f}ms/req")

benchmark_batching(test_prompts, [1, 4, 8, 16])

At batch_size=1: ~200ms latency, 5 req/s. At batch_size=8: ~800ms latency, 10 req/s. The tradeoff depends on your SLA. For interactive features, cap batch size at 4. For async workloads, push it higher.

KV Cache

The KV cache stores key-value tensors from previous tokens so you do not recompute attention for the full sequence at every step. It is enabled by default in HuggingFace and is the main reason generation is fast for short sequences but slow (and memory-intensive) for very long ones.

pythonCopy
# KV cache grows with sequence length
# For Llama 3.1 8B: ~0.5MB per token per layer at FP16
# 32 layers * 0.5MB * 4096 tokens = ~65GB for max context at FP16
# KV cache quantization helps:
from transformers import GenerationConfig

gen_config = GenerationConfig(
    max_new_tokens=512,
    cache_implementation="quantized",  # transformers >= 4.38
    cache_config={"nbits": 4, "backend": "quanto"},
)

KV cache quantization to 4-bit reduces memory by ~4x with minimal quality impact for most tasks.

vLLM vs HuggingFace

Use HuggingFace Transformers when you are experimenting, fine-tuning, or need full control. Use vLLM when you are serving.

bashCopy
# Start vLLM server (replaces your custom serving code)
pip install vllm
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --quantization awq \
    --max-model-len 8192 \
    --tensor-parallel-size 1

vLLM's key advantage is PagedAttention: it manages the KV cache as fixed-size pages, eliminating fragmentation. This allows 2–4x more concurrent users at the same latency compared to a naive HuggingFace server. The OpenAI-compatible API means dropping it into any existing client.

Benchmark before committing: for your specific model and hardware, run python -m vllm.benchmark_throughput to measure requests/second at your target latency percentile.

Speculative Decoding and Prompt Caching

Autoregressive decoding generates one token per forward pass, and each pass is memory-bound: the GPU spends most of its time reading weights, not computing. Speculative decoding attacks that directly. A small draft model proposes several tokens cheaply; the large target model verifies the whole batch in a single forward pass and accepts the longest prefix that matches what it would have produced. Output is identical to the target model's distribution; you just get two to three tokens per expensive pass instead of one. Most serving stacks (vLLM, TensorRT-LLM, SGLang) support it with a flag and a draft model, and it is the first thing to try when latency matters and you cannot shrink the model.

Prompt caching is the input-side equivalent. If many requests share a long prefix (a system prompt, a tool list, a reference document), the KV cache for that prefix can be computed once and reused. Serving frameworks do this automatically for exact prefix matches; hosted APIs expose it explicitly and bill cached tokens at a discount. Structure prompts so the stable part comes first, and monitor cache hit rate as a first-class metric.

Small Models and On-Device Inference

The frontier moved, and so did the floor. Small language models (roughly 1B to 8B parameters) now handle classification, extraction, routing, summarization, and narrow tool-calling tasks at quality that would have required a large API model two years ago. At 4-bit quantization a 7B model fits in about 4 GB and runs at interactive speed on a laptop GPU, a modern phone, or a single cheap cloud GPU.

That changes the default architecture for high-volume paths. Instead of "call the big model for everything", the pattern is: small model on the hot path, verifier on its output, escalate to a large or reasoning model only when the verifier rejects. Costs drop by an order of magnitude and p50 latency drops with them. On-device inference also removes data-residency and privacy questions entirely for the requests it handles.

The catch is that small models are less forgiving of sloppy prompts and more sensitive to distribution shift, so this is exactly where an eval suite pays for itself: you need to know the small model's accuracy on your inputs, not on a leaderboard.

Common Mistakes

Benchmarking on a single request. Throughput and latency behave differently under load. Always test with concurrent users at your expected p95 load.

Using 4-bit quantization for math-heavy tasks. Quality degrades noticeably for arithmetic and structured reasoning at 4-bit. Start with INT8 and only drop to 4-bit if memory forces it.

Not tracking VRAM during a load test. The KV cache grows with sequence length and concurrent users. A model that fits at idle will OOM under real traffic if you do not cap max_model_len.

Confusing latency and throughput optimization. Batching helps throughput. Speculative decoding helps latency. They are different problems with different solutions.

Where to Go Next

See also: [transformer-inference-context-engineering], [transformers-attention-in-practice], [ai-system-design-reliability]