Regularization: L1, L2, and Dropout Without the Formulas

What Problem Regularization Solves

A model with enough capacity can memorize the training data - achieving near-zero training loss while failing on any data it has not seen. This is overfitting. The model has not learned the underlying pattern; it has learned the specific training examples.

Regularization adds a constraint or modification that discourages this memorization and encourages the model to learn patterns that generalize.

L2 Regularization (Ridge): Penalize Large Weights

L2 regularization adds a penalty to the loss that is proportional to the sum of squared weights. The optimizer is now minimizing two things simultaneously: prediction error and weight magnitude.

The effect: weights are pushed toward zero but never exactly zero. This prevents any single feature from dominating the prediction, which tends to reduce overfitting.

pythonCopy
from sklearn.linear_model import Ridge, LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier

# For linear/logistic regression: alpha or C controls regularization strength
ridge = Ridge(alpha=1.0)         # Higher alpha = stronger regularization
logistic = LogisticRegression(C=0.1)  # Lower C = stronger regularization

# For neural networks in PyTorch:
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
# weight_decay is the L2 regularization coefficient

When to use L2: Almost always. It is a safe default that slightly improves generalization in most settings. The main hyperparameter is the regularization strength (alpha, lambda, or 1/C depending on the library).

L1 Regularization (Lasso): Sparsity

L1 regularization penalizes the sum of absolute values of weights. Unlike L2, it drives some weights exactly to zero - producing a sparse model where many features have no influence.

This is useful for feature selection: after training with L1 regularization, the features with non-zero weights are the ones the model considers relevant.

pythonCopy
from sklearn.linear_model import Lasso, LogisticRegression

lasso = Lasso(alpha=0.1)
sparse_logistic = LogisticRegression(penalty='l1', solver='liblinear', C=0.5)

lasso.fit(X_train, y_train)
# feature_names is the ordered list of column names you passed to lasso.fit,
# e.g.: feature_names = X_train.columns.tolist()
selected_features = [name for name, coef in zip(feature_names, lasso.coef_) if coef != 0]

When to use L1: When you have many features and suspect most are irrelevant. L1 will automatically zero out the irrelevant ones, producing a simpler, more interpretable model.

Elastic net: A combination of L1 and L2 that gets some sparsity with some stability. sklearn.linear_model.ElasticNet.

Dropout: Regularization for Neural Networks

Dropout is specific to neural networks. During training, each neuron is randomly set to zero with probability p (usually 0.1-0.5). This prevents neurons from co-adapting - relying on each other in ways that are specific to the training data.

The intuition: if any neuron might be dropped out at any time, the network is forced to learn redundant representations. Multiple pathways encode the same information. This redundancy helps generalization.

pythonCopy
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Dropout(p=0.3),   # 30% dropout after first hidden layer
    nn.Linear(128, 64),
    nn.ReLU(),
    nn.Dropout(p=0.3),
    nn.Linear(64, num_classes),
)

Important: dropout is active during training and disabled during evaluation. PyTorch handles this automatically when you call model.train() and model.eval().

When to use dropout: Any neural network where you are seeing a training/validation gap. Start with p=0.1-0.2 and increase if overfitting persists. Do not use it on every layer - typically only on fully connected layers, not on embedding layers.

Early Stopping: A Free Regularizer

Early stopping is not a penalty - it is a training procedure. Monitor validation loss during training and stop when it starts increasing, even if training loss continues to decrease.

pythonCopy
best_val_loss = float('inf')
patience = 10
patience_counter = 0

for epoch in range(max_epochs):
    train_loss = train_one_epoch(model, train_loader)
    val_loss = evaluate(model, val_loader)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        patience_counter = 0
        torch.save(model.state_dict(), 'best_model.pt')  # Save best checkpoint
    else:
        patience_counter += 1

    if patience_counter >= patience:
        print(f"Early stopping at epoch {epoch}")
        break

model.load_state_dict(torch.load('best_model.pt'))  # Restore best checkpoint

Early stopping is effectively free - no additional hyperparameter and no increase in training time (often a decrease). Use it always.

Choosing Regularization Strength

All regularization methods have a strength parameter. Too little regularization leaves overfitting uncorrected. Too much pushes the model toward underfitting.

Tune with cross-validation or a held-out validation set:

pythonCopy
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import Ridge

param_grid = {'alpha': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]}
ridge_cv = GridSearchCV(Ridge(), param_grid, cv=5, scoring='neg_mse')
ridge_cv.fit(X_train, y_train)
print(f"Best alpha: {ridge_cv.best_params_['alpha']}")

The pattern for all regularization decisions: if the training error is much lower than validation error, increase regularization. If both are high, decrease regularization (or use a more complex model).

Common Mistakes

Applying L1 and L2 simultaneously without understanding the interaction. Elastic net combines L1 and L2 penalties, and the interaction between them is non-trivial: L2 stabilizes the solution when features are correlated while L1 drives a subset to exactly zero. If you add both penalties without tuning both hyperparameters independently, you likely get neither the sparsity benefit of pure L1 nor the stability benefit of pure L2. Treat elastic net as a deliberate choice, not a safe default.

Tuning regularization strength on the test set. Choosing the regularization hyperparameter (lambda or alpha) by evaluating multiple values on the test set and picking the best one is a form of test set data leakage. The selected lambda will be overfit to the test set, and reported test performance will be optimistic. Always tune regularization using cross-validation on the training set and evaluate once on the held-out test set.

Expecting L1 to zero out features when using solvers that do not enforce exact sparsity. L1 regularization theoretically induces exact zeros in the coefficient vector, but only when the optimization solver reaches the exact subgradient solution. Some solvers (e.g., gradient descent without proper treatment of the L1 subgradient at zero) produce near-zero but not exactly-zero coefficients. If feature selection is the goal, verify that your solver produces exact zeros and threshold small coefficients explicitly if needed.

What to Practice Next

• Train a linear model with L1 regularization (Lasso) and L2 regularization (Ridge) on a dataset with 50+ features; plot the coefficient paths for both and identify which features are zeroed out by L1 first as regularization strength increases.
• Use cross-validation to tune the regularization strength for both Lasso and Ridge; report the optimal lambda and the resulting test set performance for each.
• Implement Elastic Net with two different alpha (L1 ratio) settings and compare sparsity of the resulting coefficient vectors to confirm the L1 ratio controls the degree of feature selection.