Linear Algebra for Gradient Descent

Gradient descent is the engine of modern machine learning. Every neural network, every logistic regression, every matrix factorization model learns by running gradient descent or one of its variants. Understanding why it works - not just that it works - gives you the intuition to debug training failures, tune learning rates, and reason about optimization at a level that tutorials skip.

This article builds the linear algebra foundation from scratch using Python. No hand-waving.

Vectors as Directions in Space

A vector is just an ordered list of numbers. But the geometric interpretation is what matters for optimization: a vector represents a direction (and a magnitude) in a multi-dimensional space.

pythonCopy
import numpy as np
import matplotlib.pyplot as plt

# A 2D vector: direction and magnitude
v = np.array([3.0, 4.0])
print(f"Vector: {v}")
print(f"Magnitude (L2 norm): {np.linalg.norm(v):.4f}")  # 5.0

# Unit vector: same direction, magnitude 1
v_unit = v / np.linalg.norm(v)
print(f"Unit vector: {v_unit}")  # [0.6, 0.8]

# Dot product: how much two vectors point in the same direction
w = np.array([1.0, 0.0])  # pointing right
print(f"Dot product v·w: {np.dot(v, w):.4f}")  # = 3.0 (projection onto x-axis)

The dot product is central to gradient descent. When two vectors point in the same direction, their dot product is large and positive. When they are perpendicular, it is zero. This is why moving in the direction of the gradient increases the loss - and moving against it decreases it.

Matrix Multiply as Transformation

A matrix transforms vectors from one space to another. In a linear layer of a neural network, the weight matrix W transforms an input vector x into an output vector y.

pythonCopy
# A 3x2 weight matrix transforms 2D inputs to 3D outputs
W = np.array([[1.0, 2.0],
              [3.0, 4.0],
              [5.0, 6.0]])

x = np.array([1.0, 1.0])  # input
y = W @ x                  # output: shape (3,)
print(f"Input shape: {x.shape}, Weight shape: {W.shape}, Output shape: {y.shape}")
print(f"Output: {y}")  # [3. 7. 11.]

Every forward pass in a neural network is a sequence of matrix multiplications and element-wise nonlinearities. The gradient of the loss flows back through these transformations via the chain rule - that is backpropagation.

The Gradient as a Vector of Partial Derivatives

The gradient of a scalar-valued function is a vector that points in the direction of steepest increase. Each component is the partial derivative with respect to one parameter.

pythonCopy
# Loss function: L(w1, w2) = (w1 - 3)^2 + (w2 - 5)^2
# Minimum at w1=3, w2=5
# Gradient: [2*(w1-3), 2*(w2-5)]

def loss(w: np.ndarray) -> float:
    return (w[0] - 3)**2 + (w[1] - 5)**2

def gradient(w: np.ndarray) -> np.ndarray:
    return np.array([2 * (w[0] - 3), 2 * (w[1] - 5)])

w = np.array([0.0, 0.0])
print(f"Loss at {w}: {loss(w):.4f}")   # 34.0
print(f"Gradient at {w}: {gradient(w)}")  # [-6, -10]  -  points away from minimum

The gradient at any point tells you which direction the loss increases fastest. To decrease the loss, move in the opposite direction.

Why Gradient Descent Works

pythonCopy
def gradient_descent(initial_w: np.ndarray, lr: float, n_steps: int) -> list:
    w = initial_w.copy()
    history = [{'step': 0, 'w': w.copy(), 'loss': loss(w)}]

    for step in range(1, n_steps + 1):
        grad = gradient(w)
        w = w - lr * grad  # move opposite to gradient
        history.append({'step': step, 'w': w.copy(), 'loss': loss(w)})

    return history

history = gradient_descent(np.array([0.0, 0.0]), lr=0.1, n_steps=20)
for entry in history[::5]:
    print(f"Step {entry['step']:2d}: w={entry['w']}, loss={entry['loss']:.6f}")

At each step, w = w - lr * gradient(w). The gradient points uphill; subtracting it points downhill. The learning rate lr controls the step size. The algorithm converges when the gradient becomes close to zero - meaning you are at (or near) a minimum.

Learning Rate Intuition

pythonCopy
# Too large: oscillates and may diverge
h_large = gradient_descent(np.array([0.0, 0.0]), lr=1.0, n_steps=20)
print(f"Large LR final loss: {h_large[-1]['loss']:.6f}")

# Too small: converges slowly
h_small = gradient_descent(np.array([0.0, 0.0]), lr=0.01, n_steps=20)
print(f"Small LR final loss: {h_small[-1]['loss']:.6f}")

# Just right
h_good = gradient_descent(np.array([0.0, 0.0]), lr=0.1, n_steps=20)
print(f"Good LR final loss: {h_good[-1]['loss']:.6f}")

If the learning rate is too large, the steps overshoot the minimum and the loss oscillates or diverges. If it is too small, the steps are tiny and training is slow. In practice, learning rate schedules (starting large and decaying) and adaptive optimizers like Adam (which maintain per-parameter learning rates) address this.

Numerical Gradient Check

You can verify your gradient implementation by comparing it to a numerical approximation:

pythonCopy
def numerical_gradient(w: np.ndarray, eps: float = 1e-5) -> np.ndarray:
    grad = np.zeros_like(w)
    for i in range(len(w)):
        w_plus = w.copy(); w_plus[i] += eps
        w_minus = w.copy(); w_minus[i] -= eps
        grad[i] = (loss(w_plus) - loss(w_minus)) / (2 * eps)
    return grad

w_test = np.array([1.0, 2.0])
analytical = gradient(w_test)
numerical = numerical_gradient(w_test)
print(f"Analytical gradient: {analytical}")
print(f"Numerical gradient:  {numerical}")
print(f"Max difference: {np.max(np.abs(analytical - numerical)):.2e}")
# Should be ~1e-10

Gradient checking is a standard debugging technique for any custom backward pass implementation.

Common Mistakes

Forgetting to zero gradients: In PyTorch, gradients accumulate by default. Call optimizer.zero_grad() at the start of each batch.

Learning rate set by feel: Use learning rate finders or start with 1e-3 for Adam and 0.01 for SGD, then tune.

Confusing gradient direction: The gradient points toward increasing loss. You subtract it.

Where to Go Next

• training-pipelines-experiment-strategy - structure the training loop where gradient descent lives
• milestone-gate-1-ml-core-transition - check whether you can apply this knowledge in a real training loop
• evaluation-metrics-error-analysis-systems - once you can train, learn how to evaluate correctly