Math for ML III: Calculus, Optimization, and Gradients

Calculus Is About One Question

Every neural network trains by repeatedly asking: "if I change this weight by a tiny amount, does the loss go up or down - and by how much?" That is calculus. Specifically, it is derivatives and the chain rule.

You do not need to derive integrals or prove convergence theorems. You need to understand derivatives geometrically, follow the chain rule through a network, and reason about why training sometimes works badly.

Derivatives: Rate of Change

The derivative of a function f at a point x tells you: if x increases by a tiny amount ε, how much does f change?

pythonCopy
import numpy as np

# Numerical derivative (finite difference approximation)
def numerical_derivative(f, x, eps=1e-5):
    return (f(x + eps) - f(x - eps)) / (2 * eps)

# Example: derivative of x² is 2x
f = lambda x: x ** 2
df_dx = numerical_derivative(f, x=3.0)
print(f"Numerical: {df_dx:.6f}")   # ≈ 6.000000
print(f"Analytical: {2 * 3.0}")    # Exactly 6.0

In ML, x is a model parameter (weight or bias) and f is the loss function. A positive derivative means increasing this parameter increases loss - so you should decrease it. A negative derivative means increasing it decreases loss - so you should increase it.

Common Derivative Rules

d/dx [c]     = 0              (constant)
d/dx [xⁿ]   = n·xⁿ⁻¹        (power rule)
d/dx [eˣ]   = eˣ             (exponential)
d/dx [ln x] = 1/x            (natural log)
d/dx [f + g] = f' + g'       (sum rule)
d/dx [f · g] = f'g + fg'     (product rule)

Derivatives of activation functions appear constantly:

pythonCopy
# ReLU: f(x) = max(0, x)
# derivative: 1 if x > 0, 0 if x < 0, undefined at 0 (use 0 in practice)
def relu_grad(x):
    return (x > 0).astype(float)

# Sigmoid: σ(x) = 1 / (1 + e^{-x})
# derivative: σ(x) · (1 - σ(x))   -  peaks at 0.25 when x=0
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_grad(x):
    s = sigmoid(x)
    return s * (1 - s)

Partial Derivatives and Gradients

When the loss depends on many parameters (a neural network has millions), we compute a partial derivative for each parameter - how much the loss changes when that parameter changes, holding all others fixed.

The gradient is the vector of all partial derivatives:

∇L(w) = [∂L/∂w₁, ∂L/∂w₂, ..., ∂L/∂wₙ]

The gradient points in the direction of steepest increase in the loss. Gradient descent moves in the opposite direction - downhill.

pythonCopy
# Gradient of MSE loss for a linear model y_pred = w * x + b
def mse_gradients(X, y_true, w, b):
    n = len(y_true)
    y_pred = w * X + b
    dL_dw = (2 / n) * np.sum((y_pred - y_true) * X)
    dL_db = (2 / n) * np.sum(y_pred - y_true)
    return dL_dw, dL_db

# Gradient descent update
def update(w, b, dL_dw, dL_db, lr=0.01):
    return w - lr * dL_dw, b - lr * dL_db

The Chain Rule: How Backpropagation Works

The chain rule says: if y = f(g(x)), then dy/dx = f'(g(x)) × g'(x). For neural networks - which are compositions of many functions (layers) - the chain rule applied backward through all layers is backpropagation.

Loss = L(softmax(W₂ · ReLU(W₁ · x + b₁) + b₂))

∂L/∂W₁ = ∂L/∂output × ∂output/∂hidden × ∂hidden/∂W₁

Each factor in this product is a local gradient computed at that layer. PyTorch's autograd computes this automatically by tracking operations in the forward pass and reversing them in the backward pass.

pythonCopy
import torch

# PyTorch autograd: everything happens automatically
W1 = torch.randn(3, 2, requires_grad=True)
W2 = torch.randn(1, 3, requires_grad=True)
x = torch.tensor([1.0, 2.0])

# Forward pass
h = torch.relu(W1 @ x)
output = W2 @ h
loss = output.pow(2).mean()

# Backward pass - computes all gradients via chain rule
loss.backward()

print(W1.grad)   # ∂loss/∂W1  -  use this to update W1
print(W2.grad)   # ∂loss/∂W2  -  use this to update W2

loss.backward() does in one call what would require pages of chain-rule derivation by hand for a deep network.

Gradient Descent and Its Variants

The gradient descent update rule:

w ← w - α × ∇L(w)

Where α is the learning rate.

Batch gradient descent: Compute the gradient over the entire dataset per step. Most accurate gradient estimate, but one step requires a full data pass.

Stochastic gradient descent (SGD): Compute the gradient on one example. Fast but noisy - high variance in the gradient estimate.

Mini-batch gradient descent: Compute the gradient on a batch of 32–256 examples. The practical standard - balances speed and stability, vectorizes efficiently on GPUs.

pythonCopy
batch_size = 32
for epoch in range(num_epochs):
    idx = np.random.permutation(len(X))
    X_shuffled, y_shuffled = X[idx], y[idx]

    for start in range(0, len(X), batch_size):
        X_batch = X_shuffled[start:start + batch_size]
        y_batch = y_shuffled[start:start + batch_size]
        dL_dw, dL_db = mse_gradients(X_batch, y_batch, w, b)
        w, b = update(w, b, dL_dw, dL_db, lr=0.01)

Convexity and Why It Matters

A function is convex if the line segment between any two points on its graph lies above the graph. Convex functions have one global minimum - gradient descent is guaranteed to find it.

MSE loss for linear regression is convex: gradient descent always converges to the optimal solution. Neural network loss surfaces are not convex: they have many local minima and saddle points. For large, well-regularized networks, local minima tend to be nearly as good as the global minimum in practice, but there is no theoretical guarantee.

Saddle points are a more common obstacle than local minima in deep learning - points where the gradient is zero but the function is neither a minimum nor a maximum. SGD's noise often escapes them; Adam's adaptive learning rates handle them well.

Learning Rate: The Most Important Hyperparameter

The learning rate α controls step size in gradient descent. Too large: training overshoots and diverges. Too small: training is slow and may get stuck.

pythonCopy
import matplotlib.pyplot as plt

# Learning rate effect on convergence
lrs = [0.001, 0.01, 0.1, 0.5]
for lr in lrs:
    w, b = 0.0, 0.0
    losses = []
    for _ in range(50):
        dw, db = mse_gradients(X_train, y_train, w, b)
        w -= lr * dw
        b -= lr * db
        losses.append(np.mean((w * X_val + b - y_val) ** 2))
    plt.plot(losses, label=f"lr={lr}")
plt.legend()

Learning rate schedules improve over a fixed rate:

• Step decay: halve the LR every k epochs
• Cosine annealing: smoothly decay from max to min following a cosine curve
• Warmup + decay: start with a small LR, ramp up, then decay (common in transformers)

pythonCopy
# Cosine annealing schedule
def cosine_lr(epoch, total_epochs, lr_max=0.01, lr_min=1e-5):
    return lr_min + 0.5 * (lr_max - lr_min) * (1 + np.cos(np.pi * epoch / total_epochs))

Numerical vs. Analytical Gradients

When implementing a new loss function or layer, verify your analytical gradient against a numerical approximation:

pythonCopy
def gradient_check(f, params, eps=1e-5):
    """Compare numerical and analytical gradients for a function."""
    numerical = np.zeros_like(params)
    for i in range(len(params)):
        params_plus = params.copy(); params_plus[i] += eps
        params_minus = params.copy(); params_minus[i] -= eps
        numerical[i] = (f(params_plus) - f(params_minus)) / (2 * eps)
    return numerical

# If numerical ≈ analytical (relative error < 1e-5), the gradient is correct

PyTorch has torch.autograd.gradcheck built in for this purpose.

Common Mistakes and Bad Instincts

Forgetting to zero gradients before the backward pass. In PyTorch, gradients accumulate by default. Call optimizer.zero_grad() at the start of every training step or you will add gradients from the previous batch.

Choosing a learning rate without looking at the loss curve. The first thing to check when training does not improve: plot the training loss over time. A flat line means the LR is too small or the problem is unsolvable. An exploding line means the LR is too large.

Using sigmoid activations in deep networks. The sigmoid derivative peaks at 0.25 and approaches zero at large inputs. In a 10-layer network, multiplying 0.25 ten times = 0.00001. Gradients vanish and early layers do not learn. Use ReLU (gradient = 1 when active) for hidden layers.

Treating gradient descent as black-box magic. When training fails - loss spikes, does not decrease, or converges to a bad solution - the cause is almost always diagnosable from the loss curve, gradient norms, and learning rate. Understanding the mechanics means you can debug rather than just re-running with a different random seed.

Where to Go Next

With Modules 3, 4, and 5 complete, you have the mathematical foundation the rest of the curriculum builds on. Module 6 moves into applied data work: NumPy, Pandas, and the habits that turn raw data into reliable model inputs.