Neural Networks From Scratch

Every modern AI system - image classifiers, language models, speech recognition, recommendation engines - is built on neural networks. Understanding how they work at the mathematical level is not optional: you need it to debug training failures, reason about capacity, and make architectural decisions. This module builds the foundation from first principles.

The Perceptron: One Neuron

A single neuron takes a vector of inputs, multiplies each by a weight, adds a bias, and applies an activation function:

output = activation(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)
       = activation(wᵀx + b)

In code:

pythonCopy
import numpy as np

def neuron(x, w, b, activation='relu'):
    z = np.dot(w, x) + b
    if activation == 'relu':
        return np.maximum(0, z)
    elif activation == 'sigmoid':
        return 1 / (1 + np.exp(-z))
    elif activation == 'linear':
        return z

A single neuron is a generalized linear classifier. With a sigmoid activation, it is exactly logistic regression. The power of neural networks comes from stacking many neurons - each layer applies a linear transformation followed by a nonlinearity, allowing the network to compose complex functions.

Forward Pass: How Inputs Become Predictions

A neural network is a sequence of layers. Each layer transforms its input:

Layer 1: h₁ = activation(W₁x + b₁)
Layer 2: h₂ = activation(W₂h₁ + b₂)
Output:  ŷ  = softmax(W₃h₂ + b₃)

Implementing from scratch in NumPy to build intuition:

pythonCopy
class DenseLayer:
    def __init__(self, in_dim, out_dim, activation='relu'):
        # He initialization: appropriate for ReLU activations
        self.W = np.random.randn(out_dim, in_dim) * np.sqrt(2.0 / in_dim)
        self.b = np.zeros((out_dim, 1))
        self.activation = activation

    def forward(self, x):
        self.input = x
        self.z = self.W @ x + self.b
        if self.activation == 'relu':
            self.out = np.maximum(0, self.z)
        elif self.activation == 'softmax':
            exp_z = np.exp(self.z - self.z.max(axis=0, keepdims=True))  # stable softmax
            self.out = exp_z / exp_z.sum(axis=0, keepdims=True)
        elif self.activation == 'linear':
            self.out = self.z
        return self.out

The forward pass computes predictions. The backward pass (backpropagation) computes gradients to update parameters.

Activation Functions

The activation function is what makes neural networks nonlinear - without it, a 10-layer network collapses to a single linear transformation.

pythonCopy
import matplotlib.pyplot as plt

x = np.linspace(-4, 4, 200)

activations = {
    'ReLU': lambda x: np.maximum(0, x),
    'Sigmoid': lambda x: 1 / (1 + np.exp(-x)),
    'Tanh': np.tanh,
    'GELU': lambda x: x * 0.5 * (1 + np.tanh(np.sqrt(2/np.pi) * (x + 0.044715 * x**3))),
    'Leaky ReLU': lambda x: np.where(x >= 0, x, 0.01 * x),
}

fig, axes = plt.subplots(1, len(activations), figsize=(15, 3))
for ax, (name, fn) in zip(axes, activations.items()):
    ax.plot(x, fn(x))
    ax.set_title(name); ax.grid(True)

Activation	Use Case	Key Property
ReLU	Default hidden layers	Fast, sparse activation, dying ReLU risk
Leaky ReLU	When dying ReLU is a problem	Small gradient for negatives
GELU	Transformers (BERT, GPT)	Smooth, outperforms ReLU in practice
Sigmoid	Binary output layer	Saturates → vanishing gradient in hidden layers
Tanh	RNN hidden states	Zero-centered, saturates
Softmax	Multi-class output layer	Normalizes to probability distribution

Backpropagation: How the Network Learns

Backprop computes gradients of the loss with respect to every parameter using the chain rule. You propagate the gradient backward through each layer.

pythonCopy
class DenseLayer:
    def backward(self, grad_out):
        # Gradient through activation
        if self.activation == 'relu':
            grad_z = grad_out * (self.z > 0)  # ReLU derivative: 1 if z > 0, else 0
        elif self.activation == 'linear':
            grad_z = grad_out

        # Gradient with respect to parameters
        self.grad_W = grad_z @ self.input.T / self.input.shape[1]
        self.grad_b = grad_z.mean(axis=1, keepdims=True)

        # Gradient with respect to input (propagated to previous layer)
        return self.W.T @ grad_z

You rarely implement backprop manually - PyTorch's autograd does it automatically. But understanding what .backward() does is essential for debugging shape errors, gradient vanishing, and custom architectures.

A Minimal Neural Network in PyTorch

pythonCopy
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset

# Define architecture
class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dims, output_dim, dropout=0.3):
        super().__init__()
        layers = []
        prev_dim = input_dim
        for h in hidden_dims:
            layers.extend([nn.Linear(prev_dim, h), nn.ReLU(), nn.Dropout(dropout)])
            prev_dim = h
        layers.append(nn.Linear(prev_dim, output_dim))
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        return self.net(x)

model = MLP(input_dim=50, hidden_dims=[256, 128, 64], output_dim=1)

# Training loop
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
criterion = nn.BCEWithLogitsLoss()

X_t = torch.FloatTensor(X_train.values)
y_t = torch.FloatTensor(y_train.values).unsqueeze(1)
dataset = TensorDataset(X_t, y_t)
loader  = DataLoader(dataset, batch_size=256, shuffle=True)

for epoch in range(100):
    model.train()
    total_loss = 0
    for batch_x, batch_y in loader:
        optimizer.zero_grad()
        logits = model(batch_x)
        loss = criterion(logits, batch_y)
        loss.backward()          # compute gradients
        optimizer.step()         # update parameters
        total_loss += loss.item()
    if epoch % 10 == 0:
        print(f"Epoch {epoch}: loss={total_loss/len(loader):.4f}")

BCEWithLogitsLoss is numerically more stable than BCELoss(sigmoid(logits)) - it fuses the sigmoid and binary cross-entropy into a single stable computation.

Weight Initialization

Bad initialization causes training to fail before it starts:

• Too small: activations collapse to zero, no gradient signal (vanishing gradients).
• Too large: activations explode, loss diverges (exploding gradients).

pythonCopy
# He initialization (Kaiming): designed for ReLU
# Variance = 2 / n_input
nn.init.kaiming_uniform_(layer.weight, nonlinearity='relu')

# Xavier/Glorot: designed for Tanh/Sigmoid
# Variance = 2 / (n_input + n_output)
nn.init.xavier_uniform_(layer.weight)

# PyTorch applies sensible defaults automatically - you only need this for custom layers

The Universal Approximation Theorem

Any continuous function can be approximated to arbitrary accuracy by a neural network with a single hidden layer and enough neurons. This is why neural networks are so powerful in principle.

In practice: the depth (number of layers) matters more than width. Deep networks compose simpler functions hierarchically - edge detectors → shape detectors → object parts → objects. This inductive bias makes them data-efficient for structured data like images and text.

Common Mistakes and Bad Instincts

Using sigmoid activations in hidden layers. Sigmoid saturates for large positive or negative inputs - the gradient becomes nearly zero, killing the signal during backprop. Use ReLU or GELU for hidden layers; sigmoid only for binary output layers.

Forgetting optimizer.zero_grad(). PyTorch accumulates gradients by default. Without zeroing them each step, gradients from multiple batches accumulate and training diverges.

Not using BCEWithLogitsLoss. Combining sigmoid + BCELoss loses numerical precision. Always use the fused version.

Training with batch size 1 or full-batch. Batch size 1 is extremely noisy. Full batch has no stochasticity to escape local minima. Use mini-batches of 32–512 (larger batches train faster but sometimes generalize worse).

Skipping gradient clipping for deep networks. Exploding gradients cause loss to suddenly jump to infinity. Add torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) before optimizer.step().

Where to Go Next

• Module 17 (Training Deep Networks) covers the practical training techniques - optimizers, learning rate schedules, regularization, and batch normalization - that make deep networks converge reliably.
• Module 18 (CNNs) and Module 19 (RNNs) apply these foundations to image and sequence data.
• The standalone post neural-network-fundamentals in the foundations track covers the math behind backprop in more detail.