Fine-Tuning, Adaptation, and When Not to Fine-Tune

Fine-tuning is often the first solution engineers reach for when a foundation model does not behave exactly as wanted. It is frequently the wrong first solution. Prompting, RAG, and output validation solve most problems faster, cheaper, and with less operational overhead. This module is as much about when not to fine-tune as about how to do it correctly.

The Decision Framework

Before fine-tuning, systematically eliminate cheaper alternatives:

Problem: model doesn't do X well
↓
Step 1: Is the problem prompt-solvable?
  → Add examples (few-shot), explicit instructions, chain-of-thought
  → Use a more capable model temporarily to measure ceiling
  → If prompt engineering gets you 90% of the way: STOP

Step 2: Is the problem a knowledge/retrieval problem?
  → Use RAG with relevant documents in context
  → If adding context fixes it: build a retrieval system, not a fine-tune

Step 3: Is the problem format/structure?
  → Use structured output / function calling
  → If forced format solves it: STOP

Step 4: Is the problem style/tone/domain-specific vocabulary?
  → Fine-tuning is genuinely appropriate here
  → The model needs to internalize patterns that cannot be expressed in a prompt

Step 5: Is the problem task-specific performance on a measurable benchmark?
  → Fine-tune if and only if you have enough labeled data (500+ examples minimum)

What Fine-Tuning Actually Changes

Continued training on a curated dataset adjusts the model's weights to favor certain output patterns. It does not add new knowledge (the model does not "memorize" your documents the way RAG does). What it does change:

• Style and tone: the model learns your domain's voice
• Format adherence: the model reliably follows structural conventions your prompts could not enforce
• Task-specific quality: narrow domain performance when you have thousands of examples
• Instruction following: the model learns your organization's specific interpretations of common instructions

Parameter-Efficient Fine-Tuning with LoRA

Full fine-tuning updates all model weights - expensive and often unnecessary. Low-Rank Adaptation (LoRA) adds small trainable rank-decomposition matrices alongside the frozen original weights:

$W' = W + \Delta W = W + BA$

Where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ with rank $r \ll \min(d, k)$. Typically $r = 8$ or $r = 16$. Training $BA$ instead of $W$ reduces trainable parameters by ~100x.

pythonCopy
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from trl import SFTTrainer
import datasets

model_name = "mistralai/Mistral-7B-Instruct-v0.2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")

# LoRA config: target the attention projections (most impactful)
lora_config = LoraConfig(
    r=16,                        # Rank - higher = more capacity, more VRAM
    lora_alpha=32,               # Scaling: effective LR = lora_alpha / r
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 20,971,520 || all params: 7,262,015,488 || trainable%: 0.2888

LoRA trains < 0.3% of the model's parameters while achieving fine-tuning quality close to full parameter updates on most tasks. The small adapter can be merged back into the base model for inference (no overhead) or kept separate for easy rollback.

Supervised Fine-Tuning (SFT) Data Format

pythonCopy
def format_for_sft(instruction: str, input_text: str, output: str) -> str:
    """Mistral/LLaMA instruction format"""
    if input_text:
        return f"[INST] {instruction}\n\n{input_text} [/INST] {output}"
    return f"[INST] {instruction} [/INST] {output}"

# Build dataset
data = [
    {
        "text": format_for_sft(
            instruction="Extract the named entities from this text.",
            input_text="Apple reported record earnings in Q3 2024.",
            output='{"organizations": ["Apple"], "dates": ["Q3 2024"]}',
        )
    },
    # ... 500+ more examples
]

dataset = datasets.Dataset.from_list(data)

Data quality matters far more than quantity. 500 high-quality, diverse examples typically outperforms 5,000 noisy ones. Filter for:

• Correct outputs (human-reviewed, not LLM-generated without verification)
• Diversity across all cases you care about
• Coverage of edge cases, not just the easy majority

Training Configuration

pythonCopy
training_args = TrainingArguments(
    output_dir="./fine_tuned_model",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,   # Effective batch = 4 * 4 = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    bf16=True,                        # Use bfloat16 on A100/H100
    logging_steps=10,
    save_strategy="epoch",
    evaluation_strategy="epoch",
    load_best_model_at_end=True,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
    max_seq_length=2048,
)
trainer.train()

DPO: Fine-Tuning from Preference Data

Supervised fine-tuning teaches the model to mimic good examples. Direct Preference Optimization (DPO) teaches the model to prefer one response over another, using pairs of (chosen, rejected) responses:

pythonCopy
from trl import DPOTrainer, DPOConfig

# Dataset format: {prompt, chosen, rejected}
dpo_data = [
    {
        "prompt": "[INST] Explain overfitting [/INST]",
        "chosen": "Overfitting occurs when a model learns the training data too well, capturing noise rather than the underlying signal...",
        "rejected": "Overfitting is when the model is too big.",
    },
]

dpo_config = DPOConfig(
    beta=0.1,          # KL constraint strength - higher = stay closer to base model
    learning_rate=1e-5,
    num_train_epochs=1,
    bf16=True,
)

dpo_trainer = DPOTrainer(
    model=model,
    ref_model=None,    # Use LoRA ref model automatically
    args=dpo_config,
    train_dataset=datasets.Dataset.from_list(dpo_data),
    tokenizer=tokenizer,
)
dpo_trainer.train()

DPO is increasingly preferred over RLHF (Reward Model + PPO) because it is simpler, more stable, and does not require a separate reward model. Use it when you have human preference data or can generate it (e.g., via a stronger LLM evaluating outputs).

Beyond SFT and DPO: RL Post-Training and Reasoning Models

SFT teaches a model to imitate examples. DPO teaches it to prefer one response over another. Neither explains where the reasoning models that dominate benchmarks come from. Those come from a third stage: reinforcement learning against a reward you can compute, usually called RLVR (reinforcement learning with verifiable rewards).

The idea is simple. For tasks with a checkable answer (math with a numeric result, code with unit tests, structured extraction with a schema), you do not need a human preference label. You sample several responses per prompt, check each one, and push the model toward the ones that were correct. The model is free to develop long chains of intermediate reasoning because nothing constrains how it gets to the answer, only whether it does.

The algorithm most teams use for this is GRPO (Group Relative Policy Optimization). Compared to classic PPO it drops the separate value model. For each prompt you sample a group of G responses, score them, and use each response's advantage relative to the group mean as the training signal:

pythonCopy
# One GRPO step, stripped to the idea
for prompt in batch:
    responses = [policy.sample(prompt) for _ in range(G)]
    rewards = [verify(prompt, r) for r in responses]        # 1.0 if correct else 0.0
    mean, std = statistics.mean(rewards), statistics.pstdev(rewards) + 1e-6
    advantages = [(r - mean) / std for r in rewards]
    # increase log-prob of tokens in above-average responses, decrease below-average,
    # with a KL penalty toward the reference model so the policy does not drift
    loss = grpo_loss(policy, ref_policy, prompt, responses, advantages, kl_coef=0.04)
    loss.backward(); optimizer.step()

What you should take from this as an application engineer:

• Reasoning is a training outcome, not a prompt trick. Models that "think" were trained to spend tokens on intermediate steps because it raised their verifiable reward. You can ask a non-reasoning model to "think step by step", but you are not getting the same thing.
• Verifiable reward is the constraint. RLVR works spectacularly for math and code and poorly for "write a good essay". If your task has no cheap verifier, you are back to preference data, and DPO or a reward model.
• Test-time compute is a dial you pay for. Reasoning models let you trade tokens (latency and cost) for accuracy. The gains are real and they diminish; a task that needs 2,000 thinking tokens rarely gets better with 20,000. Measure the curve for your task before you turn the dial up in production.
• Synthetic data is how post-training scales. Most SFT and preference datasets today are generated by a stronger model and filtered by a verifier or a judge. The risk is model collapse and inherited quirks; the mitigation is verification (tests, schema checks, rejection sampling) before anything enters the training set.

Should you run GRPO? Almost never for a product team. Fine-tuning a small open-weight model with RLVR on a narrow verifiable task (your own SQL dialect, your own API's call patterns) is feasible with open tooling and a few hundred GPU-hours, and it is a legitimate portfolio project. But the decision framework at the top of this module still holds: prompt first, RAG second, SFT third, RL last, and only when you can write the verifier.

Serving a LoRA-Fine-Tuned Model

pythonCopy
from peft import AutoPeftModelForCausalLM

# Option 1: Load with adapter (separate from base model - easy rollback)
model = AutoPeftModelForCausalLM.from_pretrained(
    "./fine_tuned_model",
    torch_dtype="auto",
    device_map="auto",
)

# Option 2: Merge adapter into base model (no inference overhead)
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged_model")

# Serve with vLLM for production throughput
# vllm serve ./merged_model --tensor-parallel-size 1 --max-model-len 4096

For production inference, vLLM (PagedAttention) provides 5-30x higher throughput than naive HuggingFace inference. Use it for any model serving more than a few QPS.

When Fine-Tuning Fails

Symptom	Likely cause	Fix
Model forgets general capabilities	Too many epochs / LR too high	Reduce epochs; add eval on general benchmarks
Model outputs training examples verbatim	Dataset too small, overfitting	Add more data; increase LoRA dropout
No improvement over base model	Wrong target modules, data quality	Try r=32, target more layers; clean data
Catastrophic output degradation	LR too high	Use 1e-4 max for LoRA

Common Mistakes and Bad Instincts

Fine-tuning to fix factual hallucination. The model learns style and format from SFT data, but it does not reliably learn to stop making factual claims it cannot verify. RAG is the right solution for factual grounding - not fine-tuning.

Using a dataset of 50 examples. Below ~200 examples, LoRA does not have enough signal to converge meaningfully. Below 500, quality is highly sensitive to individual examples. For a real fine-tuning job, plan for at least 500 high-quality examples.

Not tracking validation loss separately from training loss. A decrease in training loss with flat or increasing validation loss is overfitting. The optimal checkpoint is at the validation loss minimum - always use load_best_model_at_end=True.

Fine-tuning on API-generated outputs without review. Using a strong model (GPT-4o) to generate training examples for a smaller model (Mistral 7B) can work, but the outputs must be reviewed and filtered. Errors in training data compound - the fine-tuned model learns to replicate the mistakes.

Where to Go Next

• reasoning-models-post-training-and-test-time-compute: the full SFT to DPO to GRPO pipeline and when to route to a reasoning model

• agents-tools-and-workflow-graphs: fine-tune models specifically on tool-calling examples to improve agentic reliability

• serving-models-and-llm-systems-in-production: deploy fine-tuned models with vLLM for production throughput

• observability-drift-feedback-loops-and-llm-evals: evaluate whether fine-tuning actually improved the target metric in production