Backpropagation: The Intuition You Need Before the Math

The Credit Assignment Problem

After a forward pass through a neural network, we know how wrong the final output was. But we have millions of weights spread across dozens of layers. Which weights caused the error? By how much? In which direction should each be adjusted?

Backpropagation solves this: it efficiently computes, for every weight in the network, how much the loss would change if we slightly increased that weight. This is the gradient. Gradient descent then uses it to adjust all weights simultaneously.

The Chain Rule: The Only Math You Need

Backpropagation is an application of the chain rule from calculus: if y = f(g(x)), then dy/dx = f'(g(x)) × g'(x).

In a neural network, the output is a composition of many functions (one per layer). To find how the loss changes with respect to a weight in layer 1, we need to propagate the gradient backward through all the layers between layer 1 and the output - multiplying gradients at each step.

Loss = L(output)
output = f_n(... f_2(f_1(input, W_1), W_2) ..., W_n)

dL/dW_1 = dL/d_output × d_output/d_f_{n-1} × ... × d_f_2/d_f_1 × d_f_1/dW_1

This chain of multiplications is backpropagation. It is applied simultaneously to all weights in the network, starting from the output layer and propagating backward.

A Two-Layer Network by Hand

pythonCopy
import numpy as np

# One training example
x = np.array([1.0, 2.0])
y_true = np.array([1.0])

# Initialize weights randomly
W1 = np.random.randn(2, 3) * 0.1   # Input (2) → Hidden (3)
W2 = np.random.randn(3, 1) * 0.1   # Hidden (3) → Output (1)

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def sigmoid_grad(z):
    s = sigmoid(z)
    return s * (1 - s)

# Forward pass
z1 = x @ W1          # (3,)
a1 = sigmoid(z1)     # (3,)  -  hidden activations

z2 = a1 @ W2         # (1,)
a2 = sigmoid(z2)     # (1,)  -  output prediction

loss = ((a2 - y_true) ** 2).mean()

# Backward pass  -  computing gradients layer by layer
dL_da2 = 2 * (a2 - y_true)
dL_dz2 = dL_da2 * sigmoid_grad(z2)
dL_dW2 = a1[:, None] * dL_dz2[None, :]  # Outer product

dL_da1 = dL_dz2 @ W2.T
dL_dz1 = dL_da1 * sigmoid_grad(z1)
dL_dW1 = x[:, None] * dL_dz1[None, :]

# Update weights
lr = 0.1
W1 -= lr * dL_dW1
W2 -= lr * dL_dW2

Each dL_dW is the gradient of the loss with respect to that weight matrix. The update subtracts this gradient scaled by the learning rate.

Why Automatic Differentiation Changed Everything

In practice, nobody implements backpropagation by hand for a 100-layer network. Deep learning frameworks (PyTorch, JAX, TensorFlow) implement automatic differentiation: they track every operation in the forward pass and automatically compute the gradients in the backward pass.

pythonCopy
import torch

x = torch.tensor([1.0, 2.0], requires_grad=False)
y_true = torch.tensor([1.0])

W1 = torch.randn(2, 3, requires_grad=True)
W2 = torch.randn(3, 1, requires_grad=True)

# Forward pass  -  PyTorch tracks operations
a1 = torch.sigmoid(x @ W1)
a2 = torch.sigmoid(a1 @ W2)
loss = ((a2 - y_true) ** 2).mean()

# Backward pass  -  PyTorch computes all gradients automatically
loss.backward()

# Gradients are now in W1.grad and W2.grad
print(W1.grad)  # Gradient of loss w.r.t. W1

loss.backward() does what we computed manually above - but for any network, any depth, any architecture. This is why PyTorch's autograd is one of the most important pieces of infrastructure in modern AI.

Why Vanishing Gradients Happen

In the chain of multiplications that forms the backward pass, if any derivative is close to zero, the entire gradient signal shrinks. For networks with many layers using sigmoid activations, the sigmoid derivative (which peaks at 0.25) multiplied many times → the gradient at early layers becomes vanishingly small.

ReLU (gradient = 1 when active, 0 when inactive) does not have this problem for active neurons. Batch normalization and residual connections (skip connections in ResNets) also help by providing alternative paths for gradients to flow.

Understanding vanishing gradients explains why ReLU replaced sigmoid for hidden layers, why deep networks require careful initialization, and why residual connections enabled training networks with 100+ layers.

Common Mistakes

Confusing the direction of gradient flow. Many learners assume gradients travel in the same direction as the forward pass, but backpropagation applies the chain rule in reverse - from the loss back through each layer to the inputs. Thinking of gradients as flowing "forward" leads to wrong intuitions about which weights are responsible for which errors.

Vanishing gradients through sigmoid activations. In deep networks with many sigmoid layers, gradients are multiplied by values between 0 and 0.25 at every layer. After just five or six layers, the gradient reaching early weights is effectively zero, so those layers stop learning entirely. This is why modern architectures prefer ReLU or normalization strategies rather than stacking sigmoids.

Believing backprop "learns" the features. Backpropagation only computes and propagates error signals - it does not update weights itself. The optimizer (SGD, Adam, etc.) reads those gradients and applies the actual weight change. Confusing the two makes it harder to diagnose whether a training problem is a gradient issue or an optimizer tuning issue.

Mapping Each Gradient to Behavior

In the two-layer example above, each gradient answers a local question:

Gradient	Question it answers	What changes if it is large
dL_dW2	Which hidden activations pushed the output the wrong way?	The final layer changes how it combines learned features
dL_da1	Which hidden units were responsible for the output error?	Credit or blame is assigned back to hidden features
dL_dW1	Which input-to-hidden weights created those hidden activations?	Early weights change the features the network detects

This is the practical intuition: later layers learn how to use features; earlier layers learn which features to produce. Backpropagation connects the final mistake to every earlier choice that contributed to it.

A Gradient-Check Exercise

When implementing backprop by hand, test one weight numerically:

pythonCopy
epsilon = 1e-5
original = W1[0, 0]

W1[0, 0] = original + epsilon
loss_plus = forward_loss(x, y_true, W1, W2)

W1[0, 0] = original - epsilon
loss_minus = forward_loss(x, y_true, W1, W2)

W1[0, 0] = original
numeric_grad = (loss_plus - loss_minus) / (2 * epsilon)
backprop_grad = dL_dW1[0, 0]

print(numeric_grad, backprop_grad)

If the two numbers are close, your chain rule implementation is probably right. If they differ wildly, the bug is usually a missing transpose, a shape mistake, or applying an activation derivative to the wrong variable.

How to Read Training Failures

If loss is flat from the first step, gradients may be zero, the learning rate may be too small, or the model may be disconnected from the loss. If loss explodes, gradients may be too large or the learning rate may be too high. If only early layers fail to change, inspect activation choice and normalization because gradients may be vanishing before they reach those weights.