Neural Networks and Optimization for Practitioners

Neural networks are function approximators. That sentence is not a simplification - it is the correct mental model for an engineer who already understands composition, abstraction, and complexity. You stack parameterized linear transformations with non-linear activations, chain them into a differentiable graph, and minimize a scalar loss by following gradients backward through the chain rule. Everything else - CNNs, Transformers, diffusion models - is specialization on this foundation.

This module gives you the practitioner's picture: what is actually happening numerically, why training fails in predictable ways, and which optimization choices matter in practice.

What a Neural Network Is Mathematically

A network with one hidden layer computes:

$\hat{y} = W_2 \cdot \sigma(W_1 x + b_1) + b_2$

Where $\sigma$ is a non-linear activation (ReLU, GELU, SiLU), $W_1$ and $W_2$ are learnable weight matrices, and $x$ is the input vector. Stacking more layers composes more transformations - each layer learns a progressively more abstract representation of the input.

Training adjusts the $W$ matrices using the gradient of a loss $\mathcal{L}$ (e.g., cross-entropy for classification) with respect to every parameter, computed via backpropagation.

pythonCopy
import numpy as np

def relu(x):
    return np.maximum(0, x)

def relu_grad(x):
    return (x > 0).astype(float)

class TwoLayerNet:
    def __init__(self, input_dim, hidden_dim, output_dim, lr=0.01):
        self.W1 = np.random.randn(input_dim, hidden_dim) * 0.01
        self.b1 = np.zeros(hidden_dim)
        self.W2 = np.random.randn(hidden_dim, output_dim) * 0.01
        self.b2 = np.zeros(output_dim)
        self.lr = lr

    def forward(self, X):
        self.X = X
        self.z1 = X @ self.W1 + self.b1
        self.a1 = relu(self.z1)
        self.z2 = self.a1 @ self.W2 + self.b2
        return self.z2  # raw logits

    def backward(self, dL_dz2):
        dL_dW2 = self.a1.T @ dL_dz2
        dL_db2 = dL_dz2.sum(axis=0)
        dL_da1 = dL_dz2 @ self.W2.T
        dL_dz1 = dL_da1 * relu_grad(self.z1)
        dL_dW1 = self.X.T @ dL_dz1
        dL_db1 = dL_dz1.sum(axis=0)
        self.W2 -= self.lr * dL_dW2
        self.b2 -= self.lr * dL_db2
        self.W1 -= self.lr * dL_dW1
        self.b1 -= self.lr * dL_db1

Working through this once by hand is the best way to build intuition. Frameworks automate it, but knowing why gradients flow the way they do helps you debug vanishing gradients, exploding norms, and dead neurons.

Activation Functions: Why They Matter

The activation between layers determines the gradient signal that flows backward:

Activation	Range	Dead neuron risk	Common use
ReLU	[0, ∞)	High (negative inputs → 0 grad)	Cheap default
Leaky ReLU	(-∞, ∞)	Low	When ReLU dies
GELU	(-∞, ∞)	Very low	Transformers
SiLU/Swish	(-∞, ∞)	Low	Modern vision
Sigmoid	(0, 1)	High saturation	Output only

For new projects, use GELU in Transformers and SiLU in CNNs/MLPs. Never use sigmoid or tanh in hidden layers - they saturate and slow training.

Optimization Algorithms

SGD with momentum: Accumulates a velocity vector in directions of persistent gradient. Simple, but sensitive to learning rate.

Adam: Maintains per-parameter adaptive learning rates using first and second moment estimates. Almost always converges faster than vanilla SGD. The default choice.

AdamW: Adam with decoupled weight decay. Weight decay in Adam is mathematically incorrect (it couples with the second moment). AdamW fixes this. Use AdamW by default for Transformers.

pythonCopy
import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(128, 256),
    nn.GELU(),
    nn.Linear(256, 256),
    nn.GELU(),
    nn.Linear(256, 10),
)

# AdamW with weight decay - correct for most modern networks
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)

# Cosine annealing: reduce LR smoothly to near zero over training
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)

Learning Rate: The Most Important Hyperparameter

An LR too high causes loss to diverge or oscillate. Too low and training stalls or converges to a bad local minimum. Rules of thumb:

• Adam/AdamW: start at 3e-4 for most problems; 1e-4 for fine-tuning pretrained models
• SGD: start at 0.1 with momentum 0.9 for vision from scratch
• LR finder: run a quick sweep from 1e-7 to 1e-1, plot loss vs. LR, pick the point of steepest decrease minus one order of magnitude

pythonCopy
# Quick manual LR range test
lrs = [1e-5, 3e-5, 1e-4, 3e-4, 1e-3, 3e-3, 1e-2]
for lr in lrs:
    # Reset model, run N steps, record final loss
    pass

# In practice: use fastai's LRFinder or PyTorch Lightning's tune

Batch Normalization and Layer Normalization

Normalization layers stabilize training by controlling the distribution of activations between layers.

BatchNorm: normalizes across the batch dimension. Used in CNNs. Requires batch size ≥ 8 to be stable; behaves differently at train vs. inference.

LayerNorm: normalizes across the feature dimension of a single sample. Used in Transformers. No batch-size dependency; consistent at train and inference.

pythonCopy
# BatchNorm - after a conv or linear layer, before activation
nn.Sequential(
    nn.Linear(256, 256),
    nn.BatchNorm1d(256),
    nn.ReLU(),
)

# LayerNorm - in Transformer blocks
nn.Sequential(
    nn.Linear(512, 512),
    nn.LayerNorm(512),
    nn.GELU(),
)

Gradient Clipping

In deep networks, gradients can grow exponentially through many layers (exploding gradients). This is catastrophic: one large update destroys the parameter values you've carefully trained.

pythonCopy
# Clip gradient norm before every optimizer step
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

Always clip when training RNNs, LSTMs, or deep Transformers from scratch. AdamW + gradient clipping + warmup LR schedule is the canonical recipe for stable Transformer training.

Dropout and Regularization

Dropout randomly zeroes a fraction of activations during training, forcing the network to learn redundant representations and reducing overfitting.

pythonCopy
model = nn.Sequential(
    nn.Linear(512, 512),
    nn.GELU(),
    nn.Dropout(p=0.1),  # 10% dropout - common for Transformers
    nn.Linear(512, 10),
)

Rules of thumb: p=0.1–0.2 for Transformers and large MLPs; p=0.3–0.5 for small models prone to overfitting. Turn off at inference - PyTorch does this automatically when you call model.eval().

Diagnosing Common Training Failures

Symptom	Likely cause	Fix
Loss NaN immediately	LR too high or bad weight init	Reduce LR; check for NaN in data
Loss flat from epoch 1	LR too low, dead ReLU	Increase LR; switch to GELU
Loss drops then spikes	LR too high, no clipping	Reduce LR; add grad clipping
Train loss low, val high	Overfitting	Dropout, weight decay, more data
Train and val both high	Underfitting	Bigger model, more epochs, lower reg

Common Mistakes and Bad Instincts

Not calling model.eval() during inference. Dropout and BatchNorm behave differently in train vs. eval mode. Forgetting model.eval() means your inference includes random dropout, producing different outputs every call.

Using sigmoid hidden activations in 2025. Engineers who learned deep learning from older resources sometimes still use sigmoid/tanh in hidden layers. These saturate and produce near-zero gradients. Use ReLU, GELU, or SiLU instead.

Skipping gradient clipping for deep architectures. It is a one-liner that prevents a class of catastrophic failures. Add it by default when training from scratch.

Tuning architecture before diagnosing data issues. Most "bad model" problems are bad data problems. Plot your training data. Check class balance. Verify there is no label noise before adding more layers.

Where to Go Next

• pytorch-for-real-model-development: build a complete training loop using PyTorch idioms you will use in every project going forward
• transformers-and-modern-nlp-for-engineers: apply this optimization knowledge to the Transformer architecture that powers GPT, BERT, and every modern LLM