Fine-Tuning and Post-Training: LoRA, SFT, DPO, and Reasoning RL

A base model predicts the next token. Everything that makes it follow instructions, use tools, refuse harmful requests, or reason through a math problem is added afterward, in a set of stages collectively called post-training. This module explains those stages well enough that you can read a model card, decide when fine-tuning is the right tool for a product problem (rarely, but decisively when it is), and run a real fine-tuning project on a single GPU.

What Fine-Tuning Changes and What It Does Not

Fine-tuning continues training a pretrained model on your data. It changes the model's behavior: format, style, domain vocabulary, the way it handles a specific kind of input. It is bad at adding knowledge: a model fine-tuned on your documentation will still hallucinate facts from it, because a few thousand examples do not overwrite what billions of pretraining tokens established. Knowledge goes in the context (retrieval); behavior goes in the weights (fine-tuning). That one sentence prevents most bad fine-tuning projects.

The decision order, every time:

1. Prompting. Can a better instruction and a few examples get you there? Usually yes.
2. Retrieval. Is the problem missing information? Put it in the context.
3. Fine-tuning. Is the problem consistent behavior on a narrow task, at volume, where prompting is expensive or unreliable? Now fine-tune.
4. Reinforcement learning. Do you have a verifier and a task where the model needs to get better than its examples? Only then.

Parameter-Efficient Fine-Tuning: LoRA

Full fine-tuning updates every weight, which for a 7B model means tens of gigabytes of optimizer state and a multi-GPU setup. LoRA (low-rank adaptation) freezes the original weights and trains small low-rank matrices that are added to certain layers. The intuition: the change you need to make to a pretrained model is low-dimensional, so you can represent it with far fewer parameters than the model has.

A LoRA adapter for a 7B model is typically tens of megabytes. You can train it on one consumer GPU, keep several adapters for different tasks, and swap them at serving time. QLoRA goes further by quantizing the frozen base model to 4-bit during training, which fits a 7B model into a single 16 GB GPU.

pythonCopy
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("<small-open-weight-model>", load_in_4bit=True)
config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"],
                    lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(model, config)
model.print_trainable_parameters()   # a fraction of a percent of the total

Supervised Fine-Tuning: The Data Is the Model

SFT trains the model to produce a target response given a prompt. The recipe is simple; the data is everything.

jsonCopy
{"messages": [
  {"role": "system", "content": "You convert support tickets into structured JSON."},
  {"role": "user", "content": "Customer says the app crashes on login since yesterday's update, iPhone 14."},
  {"role": "assistant", "content": "{\"issue\": \"crash\", \"area\": \"login\", \"since\": \"last update\", \"device\": \"iPhone 14\", \"severity\": \"high\"}"}
]}

Rules that separate a working SFT project from a failed one:

• A few hundred excellent examples beat ten thousand mediocre ones. Every example is a vote for a behavior; wrong examples are votes for wrong behavior.
• Cover the distribution. If 10% of real inputs are ambiguous, 10% of your examples should be ambiguous, with the response you want in that case.
• Hold out a test set before you look at anything. Your eval must be untouched by training.
• Most SFT data today is partly synthetic. A large model generates candidates, a verifier or a human filters them. This is fine and normal. The filter is what matters.

Preference Optimization: DPO

SFT teaches "produce this". Preference data teaches "prefer this over that", which is how you tune style, helpfulness, and safety where there is no single right answer. Direct Preference Optimization trains directly on (prompt, preferred, rejected) triples without a separate reward model, which made it the default over the older RLHF-with-PPO pipeline for most teams.

jsonCopy
{"prompt": "Explain overfitting to a product manager.",
 "chosen": "Overfitting is when a model memorizes the training examples instead of learning the pattern...",
 "rejected": "Overfitting occurs when the empirical risk minimizer exhibits high variance..."}

DPO needs pairs, and pairs are cheap to generate: sample two responses, have a human (or a validated judge) pick. A few thousand pairs meaningfully shift a model's style.

Reasoning RL: Where Thinking Models Come From

Reasoning models, the ones that spend tokens thinking before they answer, come from a third stage: reinforcement learning with verifiable rewards (RLVR). For tasks where you can check the answer (math with a numeric result, code with tests, output that must parse), sample several responses, check each, and push the model toward the ones that were right. Nothing tells the model how to get there, so it learns to reason, backtrack, and self-check, because those raise its reward.

The algorithm most used is GRPO (Group Relative Policy Optimization). For each prompt, sample a group of responses, score them, and use each one's score relative to the group average as its training signal. No separate value model, which keeps it cheap and stable. A KL penalty keeps the model near its starting point so it does not find degenerate ways to satisfy the checker.

What matters for you:

• Reasoning is a training outcome. "Think step by step" in a prompt borrows the form, not the training.
• RLVR needs a verifier. It works for math and code and not for "write a good email". No verifier, no RLVR; use preference data.
• Reasoning models cost more per request. The gain is real on verifiable tasks and small elsewhere. Measure on your task.

A Project You Can Run

Pick a narrow, verifiable task: converting natural-language questions into SQL for one schema, extracting structured fields from a type of document, classifying tickets into your categories. Then:

1. Generate 500 candidate examples with a large model. Filter with a verifier (the SQL runs and returns the expected rows; the JSON validates against the schema) and keep the ones that pass. Hold out 100 as a test set.
2. LoRA-fine-tune a small open-weight model (1B to 8B) on the 400.
3. Evaluate on the 100 against the base model with a few-shot prompt and against the large model. Report accuracy, latency, and cost per request.
4. Optional: run a short GRPO phase with the same verifier and report whether it improves on SFT.

This is a complete post-training pipeline on one GPU in a weekend, and it is a strong portfolio project because it demonstrates the decision, the data, the training, and the evaluation end to end.

Common Mistakes and Bad Instincts

• Fine-tuning for knowledge. Use retrieval.
• Skipping prompting and retrieval. Fine-tuning is step three.
• Large, sloppy datasets. Fewer, better, filtered.
• No held-out test set. You will not know if it worked.
• RL without a verifier. Use DPO.
• Believing benchmark deltas transfer. Measure on your task.

Where to Go Next

• llm-application-engineering: prompting and context first, always
• embeddings-retrieval-and-rag-systems: where knowledge goes
• model-serving-apis-inference-and-performance-tradeoffs: serving an adapter, and deciding when a small fine-tuned model beats an API call

What to Practice Next

Run the project above on a task you choose. Deliver a repository with the data generation and filtering script, the training script, the eval script, and a README with the accuracy, latency, and cost table comparing base model, fine-tuned model, and large model. Write one paragraph on whether you would ship the fine-tuned model and why.