Open-Weight and Small Models in 2026: When to Self-Host

Last reviewed: September 2026. Model families and their capabilities change quickly; the decision framework below does not.

Two things changed the serving decision. Open-weight model families (Llama, Qwen, DeepSeek, Gemma, Mistral, among others) closed most of the gap with API models on production tasks. And small models, in the 1B to 8B range, got good enough for a large share of real workloads while running on a laptop, a phone, or a single cheap GPU. "Call the big API model for everything" is now a choice with alternatives, and the alternatives are often cheaper by an order of magnitude.

The Decision

Self-host when at least one of these is true:

• Data cannot leave your environment. Regulatory, contractual, or trust reasons. This alone decides it for many teams.
• Volume makes per-token cost dominant. At high request volume, the engineering cost of running inference is smaller than the API bill. Do the arithmetic with real numbers; the crossover is lower than most people guess for narrow, high-volume tasks.
• You need to fine-tune. Adapters on open-weight models are cheap and swappable. Fine-tuning through an API is possible but limited.
• You need deterministic latency. No shared queue, no rate limits, no surprise degradations.
• You need a model that will not change under you. A pinned open-weight model behaves the same next year.

Use an API when:

• You need the frontier. The best models are still API-only, and for hard tasks the gap is real.
• Volume is low. The engineering cost of serving dominates.
• Nobody can own the serving stack. GPUs, drivers, serving frameworks, upgrades, on-call. It is a real job.
• You need breadth. One API model that handles everything acceptably versus several small models that each handle one thing well.

Where Small Models Win

Small models are not worse large models. They are a different tool. They win on:

• Classification and routing. Intent, category, "does this need the big model".
• Extraction. Structured fields from semi-structured text, especially fine-tuned.
• Narrow tool calling. A fixed set of tools with clear descriptions.
• Summarization at volume. Where "good enough, fast, cheap" is the requirement.
• On-device. Privacy, offline, latency. A 7B-class model at 4-bit runs at interactive speed on a modern phone.

They lose on open-ended reasoning, long multi-step tasks, unfamiliar domains, and anything where the prompt is sloppy: small models are less forgiving. Which means the eval suite matters more, not less, when you go small. You need to know the small model's accuracy on your inputs.

Mixture-of-Experts and the Parameter Count

Most large open-weight models are now mixture-of-experts: a large total parameter count, but only a fraction active per token. A "400B" MoE model might activate 20B parameters per token. Consequences:

• Compute per token tracks the active count. Inference is cheaper than the total suggests.
• Memory tracks the total count. You need to hold all the experts, which decides how many GPUs.
• Benchmark comparisons by parameter count are meaningless across MoE and dense models.

When someone quotes a parameter count, ask "active or total".

The Hybrid

Most teams end up with a tiered stack, and the tiers are a routing decision:

1. Small open-weight model on the hot path: classification, extraction, routing, simple tool calls. Self-hosted or on-device.
2. Mid-size open-weight model for the bulk of generation, self-hosted if volume justifies it.
3. Frontier API model (standard or reasoning) for escalations: the verifier rejected the cheap answer, the task is hard, or the input is unusual.

Log the route per request. Route distribution is a cost and quality signal, and drift in it means the inputs changed.

What Self-Hosting Actually Involves

Honest list: GPU procurement or cloud GPU pricing, a serving framework (vLLM, TensorRT-LLM, SGLang, or a hosted open-weight provider as a middle path), quantization choices, prefix caching, batching configuration, model updates, an eval to re-run on every update, monitoring, and someone on call. The "hosted open-weight" providers remove most of that list at a per-token price between self-hosting and the frontier APIs, and they are the right first step for many teams.

What to Practice Next

Take your highest-volume LLM call. Measure its accuracy on a 200-request eval with a small open-weight model (via a hosted provider, no infrastructure), the mid-size model, and your current API model. Put cost per 1,000 requests next to accuracy. If the small model is within a couple of points, you have found your hot path. The module serving-models-and-llm-systems-in-production covers routing, cascades, and the serving stack.