Gradient Descent Explained From Scratch

The Hill-Walking Analogy

Imagine you are standing on a hilly landscape in dense fog. You cannot see far, but you can feel the ground slope under your feet. Your goal is to reach the lowest point in the valley. The strategy: at every step, feel which direction is downhill and take a step that way. Repeat until the ground feels flat.

This is gradient descent. The "landscape" is your model's loss function - a surface whose height represents how wrong the model is. The "lowest point" is the minimum loss. The "feel which direction is downhill" is computing the gradient. The "step" is the weight update.

The Loss Function: Measuring Wrongness

Before gradient descent can run, you need a loss function - a measure of how wrong the model's predictions are.

For regression: Mean Squared Error (MSE)

pythonCopy
import numpy as np

def mse_loss(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

For binary classification: Binary Cross-Entropy

pythonCopy
def binary_cross_entropy(y_true, y_pred):
    eps = 1e-8  # avoid log(0)
    return -np.mean(y_true * np.log(y_pred + eps) + (1 - y_true) * np.log(1 - y_pred + eps))

The loss is a single number. Our goal: adjust the model's weights to make this number as small as possible.

The Gradient: Which Way Is Uphill?

The gradient of the loss with respect to a parameter is a partial derivative: "if I increase this parameter by a tiny amount, how much does the loss change?"

A positive gradient means: increasing this weight increases loss → decrease the weight. A negative gradient means: increasing this weight decreases loss → increase the weight.

For a simple linear model y_pred = w * x + b with MSE loss:

pythonCopy
def compute_gradients(X, y_true, w, b):
    n = len(y_true)
    y_pred = w * X + b
    dL_dw = (2/n) * np.sum((y_pred - y_true) * X)  # dLoss/dw
    dL_db = (2/n) * np.sum(y_pred - y_true)         # dLoss/db
    return dL_dw, dL_db

The Update Rule: Taking a Step Downhill

pythonCopy
w = w - learning_rate * dL_dw
b = b - learning_rate * dL_db

Subtract because the gradient points uphill - we want to move downhill.

learning_rate (often called lr or alpha) controls step size:

• Too large: overshoot the minimum, loss bounces around or diverges
• Too small: training takes too long, may get stuck

Typical values: 0.001 to 0.1, depending on the problem. Start at 0.01 and adjust based on the loss curve.

Full Gradient Descent From Scratch

pythonCopy
import numpy as np
import matplotlib.pyplot as plt

# Synthetic data: y = 2x + 1 + noise
np.random.seed(42)
X = np.random.randn(100)
y = 2 * X + 1 + 0.3 * np.random.randn(100)

# Initialize weights
w, b = 0.0, 0.0
lr = 0.05
epochs = 100
losses = []

for epoch in range(epochs):
    # Forward pass
    y_pred = w * X + b

    # Loss
    loss = np.mean((y_pred - y) ** 2)
    losses.append(loss)

    # Gradients
    n = len(y)
    dL_dw = (2/n) * np.sum((y_pred - y) * X)
    dL_db = (2/n) * np.sum(y_pred - y)

    # Update
    w -= lr * dL_dw
    b -= lr * dL_db

    if epoch % 10 == 0:
        print(f"Epoch {epoch}: loss={loss:.4f}, w={w:.3f}, b={b:.3f}")

print(f"\nTrue: w=2.0, b=1.0")
print(f"Learned: w={w:.3f}, b={b:.3f}")

After 100 epochs, w will be close to 2.0 and b close to 1.0. The model has learned the underlying relationship purely from data.

Batch vs. Stochastic vs. Mini-Batch

Batch gradient descent: Compute the gradient on the entire dataset. Accurate but slow for large datasets.

Stochastic gradient descent (SGD): Compute the gradient on one example at a time. Fast but noisy - the loss bounces around because each example gives a different gradient estimate.

Mini-batch gradient descent: Compute the gradient on a small batch (32-256 examples). Balances speed and stability. This is what deep learning frameworks use in practice.

pythonCopy
batch_size = 32
for epoch in range(epochs):
    # Shuffle data
    idx = np.random.permutation(len(X))
    X_shuffled, y_shuffled = X[idx], y[idx]

    for i in range(0, len(X), batch_size):
        X_batch = X_shuffled[i:i+batch_size]
        y_batch = y_shuffled[i:i+batch_size]
        # Compute gradient on batch and update

Why Gradient Descent Sometimes Fails

Local minima: The landscape has multiple valleys. Gradient descent finds a nearby valley, not necessarily the global minimum. In practice, for large and well-regularized deep networks, local minima tend to be nearly as good as the global minimum - this is a reasonable empirical approximation, but it is not guaranteed for all architectures or problem sizes.

Saddle points: Points that are neither minima nor maxima - a plateau where the gradient is very small. Training can slow dramatically here.

Vanishing gradients: In deep networks, gradients can become extremely small as they propagate backward through many layers. The early layers learn very slowly. This is why activation functions, batch normalization, and residual connections matter.

Where to Go Next

This is the foundation for understanding how every neural network learns. The next natural post is Backpropagation, which applies the chain rule to compute gradients for networks with many layers.