Deep Learning for Product Engineers

Most engineers who use deep learning models do not train them from scratch. They fine-tune, adapt, or call via API. But the engineers who ship the best ML-powered products are the ones who understand enough about how these models work to make good architecture and debugging decisions. This article covers what matters - not for research, but for building.

Architecture Choices That Affect Your Product

When you choose a model, you are making implicit engineering decisions. The architecture affects speed, memory, and what the model is good at.

Encoder-only (BERT, RoBERTa): Best for tasks where you need a single vector representation of a text - classification, semantic search, embeddings. Faster and cheaper per inference than generative models. No generation capability.

Decoder-only (GPT-4, Llama, Mistral): Best for generation tasks. Autoregressive: each token depends on all previous tokens. Slow per token. KV cache makes longer contexts more expensive.

Encoder-decoder (T5, BART): Best for transformation tasks - translation, summarization, structured extraction where input and output are different lengths.

pythonCopy
from transformers import pipeline

# Classifier: encoder-only (fast, cheap)
classifier = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
result = classifier("This product is excellent")
# [{'label': 'POSITIVE', 'score': 0.9998}]

# Embedder: encoder-only (returns vectors, not tokens)
from sentence_transformers import SentenceTransformer
embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
vector = embedder.encode("This product is excellent")
# numpy array of shape (384,)

# Generative: decoder-only (flexible, expensive)
generator = pipeline("text-generation", model="Qwen/Qwen2.5-1.5B-Instruct")

Use the simplest model that works. A distilbert classifier runs in 5ms; GPT-4o runs in 2,000ms. For binary classification, the encoder wins on every dimension except flexibility.

Batch Size Effects

Batch size affects training stability and throughput. For inference, it affects GPU utilization.

pythonCopy
import torch
import time

def benchmark_batch_size(model, tokenizer, texts, batch_sizes):
    model.eval()
    results = {}
    with torch.no_grad():
        for bs in batch_sizes:
            batch = texts[:bs]
            inputs = tokenizer(batch, padding=True, truncation=True,
                               max_length=128, return_tensors="pt").to(model.device)
            start = time.perf_counter()
            for _ in range(20):
                _ = model(**inputs)
            elapsed = (time.perf_counter() - start) / 20
            results[bs] = {
                "latency_ms": elapsed * 1000,
                "throughput_rps": bs / elapsed,
            }
    return results

A batch size of 1 wastes GPU parallelism. A batch size of 32 on a 40ms workload adds 40ms of queuing time but quadruples throughput. For async workloads, always batch. For interactive requests, serve individually or with a short dynamic batching window (5–10ms).

Overfitting Signals

Overfitting is easy to see if you are looking. The training loss keeps going down; the validation loss levels off or starts rising.

pythonCopy
def plot_training_curves(train_losses, val_losses):
    import matplotlib.pyplot as plt
    epochs = range(1, len(train_losses) + 1)
    plt.figure(figsize=(8, 4))
    plt.plot(epochs, train_losses, 'b-', label='Train loss')
    plt.plot(epochs, val_losses, 'r-', label='Val loss')
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.title('Training curves - watch for val loss plateau or rise')
    plt.legend()
    plt.tight_layout()
    plt.show()
    # Warning signs:
    # val_loss rises while train_loss falls → overfitting
    # both plateau early → learning rate too low or insufficient data
    # both diverge → learning rate too high

For fine-tuning, typical remedies are: reduce epochs (early stopping), add dropout, use a smaller learning rate, or add more training data. LoRA fine-tuning is particularly resistant to overfitting because it trains very few parameters.

When to Use a Smaller Model

Bigger is not always better. Use a smaller model when:

1. Latency is your constraint. A 1.5B model is 10x faster than a 13B model. For classification or extraction where you have labeled data, fine-tune the small model.

2. You have domain-specific labeled data. A fine-tuned 7B model often outperforms GPT-4o on narrow tasks with a training set of 500+ examples.

3. Cost matters. Serving a local 1.5B model costs $0.001/1K tokens. GPT-4o costs$10/1M input tokens. At 10M daily tokens, that is $10/day vs$100/day.

pythonCopy
# Quick quality check: compare models on your task
from openai import OpenAI
import json

test_cases = [
    {"input": "Order #1234 was damaged", "expected": "complaint"},
    {"input": "Where is my package?", "expected": "tracking"},
    {"input": "I love your service!", "expected": "positive"},
]

def evaluate_classifier(model_name: str, test_cases: list) -> float:
    client = OpenAI()
    correct = 0
    for case in test_cases:
        resp = client.chat.completions.create(
            model=model_name,
            messages=[
                {"role": "system", "content": "Classify as complaint/tracking/positive. Return JSON: {\"label\": ...}"},
                {"role": "user", "content": case["input"]},
            ],
            response_format={"type": "json_object"},
        )
        label = json.loads(resp.choices[0].message.content).get("label")
        correct += (label == case["expected"])
    return correct / len(test_cases)

Benchmark your specific task. The cost of running this evaluation is $0.01. The cost of deploying the wrong model is weeks of latency debugging.

Common Mistakes

Using generative models for classification tasks. If your task has a fixed output space, a classifier is almost always faster, cheaper, and more accurate. Use generative models for open-ended tasks.

Not setting a max_length. Models with no truncation will silently handle long inputs differently across versions. Always set max_length and truncation=True.

Ignoring the tokenizer's behavior. Different tokenizers split text differently. "ChatGPT" might be 1 token in one tokenizer and 3 in another. This affects cost estimates, context utilization, and chunking strategies.

Where to Go Next

See also: [transformers-attention-in-practice], [inference-optimization-performance], [transformer-inference-context-engineering]