Supervised Learning Foundations

Every production ML system you will build - churn models, fraud detectors, recommendation engines, document classifiers - is a supervised learning system at its core. Before you reach for XGBoost or a neural net, you need a clean mental model of what supervised learning is actually doing, why it fails in predictable ways, and how to structure the training loop correctly. This module builds that foundation.

What Supervised Learning Is Actually Doing

You have a dataset of (input, label) pairs: (x₁, y₁), (x₂, y₂), ..., (xₙ, yₙ). Supervised learning is the process of finding a function f such that f(x) ≈ y for new, unseen inputs.

The catch: you want generalization, not memorization. A function that perfectly reproduces every training label but fails on new data is useless in production. Every design decision in supervised ML - regularization, cross-validation, early stopping - exists to close the gap between training performance and real-world performance.

The hypothesis class is the set of all functions your model could learn. A linear model can only find linear decision boundaries. A deep neural net can approximate nearly any function. Choosing the right hypothesis class for your problem is the first real design decision.

The Training, Validation, Test Split

This is the single most misunderstood concept in applied ML. Here is the correct mental model:

• Training set: the model sees labels and updates its parameters.
• Validation set: you see performance, use it to tune hyperparameters and make architectural decisions.
• Test set: held out completely until the final evaluation. You use it once. If you run it more than once and use the results to make any decision, it becomes a second validation set and your reported performance is optimistic.

pythonCopy
from sklearn.model_selection import train_test_split

X, y = load_dataset()

# First split: hold out test set
X_trainval, X_test, y_trainval, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42, stratify=y
)

# Second split: carve out validation from training
X_train, X_val, y_train, y_val = train_test_split(
    X_trainval, y_trainval, test_size=0.15, random_state=42, stratify=y_trainval
)

print(f"Train: {len(X_train)}, Val: {len(X_val)}, Test: {len(X_test)}")

Use stratify=y for classification so each split has the same class proportions - especially important with imbalanced data.

For time-series or any temporal data, never random-split. The test set must be chronologically later than the training set, or you have a leakage problem that makes your offline metrics meaningless.

The Loss Function as Objective

Training is numerical optimization. The model has parameters θ and minimizes a loss function L(θ) over the training data. Two canonical choices:

Cross-entropy loss (classification):

L = -1/n Σ [yᵢ log(p̂ᵢ) + (1 - yᵢ) log(1 - p̂ᵢ)]

Mean squared error (regression):

L = 1/n Σ (yᵢ - ŷᵢ)²

The optimizer (gradient descent and its variants) adjusts θ to reduce L on the training set. The metric you care about (accuracy, AUC, F1) is not directly minimized - it is evaluated separately. This distinction matters: you can have low training loss and bad AUC if the model is well-calibrated but predicts the wrong rank order.

sklearn's fit/predict API

sklearn provides a uniform interface across all supervised algorithms. Learning it once lets you swap algorithms with a single line change.

pythonCopy
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

# Any estimator follows the same pattern
model = RandomForestClassifier(n_estimators=100, random_state=42)

# Fit: learn parameters from training data
model.fit(X_train, y_train)

# Predict: apply learned function to new inputs
y_pred = model.predict(X_val)
y_prob = model.predict_proba(X_val)[:, 1]  # probability of positive class

print(classification_report(y_val, y_pred))

The contract: fit() takes (X, y) and modifies the object in place. predict() takes X and returns predictions. predict_proba() (available on classifiers) returns calibrated probabilities.

Overfitting, Underfitting, and Learning Curves

Overfitting: your model has memorized noise in the training data. Training performance is high, validation performance is substantially lower.

Underfitting: your model is too simple for the data. Both training and validation performance are poor.

Learning curves reveal which problem you have:

pythonCopy
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve

train_sizes, train_scores, val_scores = learning_curve(
    model, X_trainval, y_trainval,
    train_sizes=np.linspace(0.1, 1.0, 10),
    cv=5, scoring='roc_auc', n_jobs=-1
)

train_mean = train_scores.mean(axis=1)
val_mean = val_scores.mean(axis=1)

plt.plot(train_sizes, train_mean, label='Train AUC')
plt.plot(train_sizes, val_mean, label='Validation AUC')
plt.xlabel('Training set size')
plt.ylabel('AUC')
plt.legend()

Interpretation:

• Large gap between train and val curves → overfitting → need regularization, simpler model, or more data.
• Both curves low and converged → underfitting → need a more expressive model or better features.
• Both curves high and close together → good fit → now focus on the test set.

Cross-Validation for Reliable Estimates

A single train/val split has high variance - a lucky (or unlucky) split can mislead you. K-fold cross-validation uses k non-overlapping val folds and averages the results.

pythonCopy
from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(
    model, X_trainval, y_trainval,
    cv=cv, scoring='roc_auc', n_jobs=-1
)

print(f"AUC: {scores.mean():.4f} ± {scores.std():.4f}")

The standard deviation tells you how stable the model is. If it is large (> 0.02 on AUC), your data may have high variance or your splits are poorly designed.

Do not run cross-validation on your held-out test set. CV is for model development and hyperparameter selection. The test set is for the single final evaluation.

Pipelines: The Correct Way to Chain Preprocessing

The most common supervised learning bug is applying preprocessing transformations before the train/val split - this leaks validation statistics into training. sklearn Pipelines prevent this by fitting the entire chain inside CV folds.

pythonCopy
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.ensemble import GradientBoostingClassifier

pipe = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
    ('model', GradientBoostingClassifier(n_estimators=200, max_depth=4))
])

# fit() runs the full chain: imputer.fit_transform → scaler.fit_transform → model.fit
pipe.fit(X_train, y_train)

# predict() runs: imputer.transform → scaler.transform → model.predict
y_pred = pipe.predict(X_val)

When you pass a Pipeline to cross_val_score, sklearn re-fits the entire pipeline (including the scaler and imputer) inside each fold. This is the only correct way to do it.

The Model Selection Workflow

A disciplined workflow prevents overfitting to the validation set:

1. Define your metric before you look at any model output. Use the metric your business problem actually cares about.
2. Establish a baseline: DummyClassifier or a simple rule. Know what "random" looks like for your problem.
3. Train a fast cheap model (logistic regression, shallow tree). This reveals data quality issues before you invest compute.
4. Tune hyperparameters with CV on the training set. Use RandomizedSearchCV or Optuna rather than full grid search.
5. Pick one final model and evaluate on the test set once.
6. Report results with confidence intervals - a point estimate without variance is not a reliable evaluation.

pythonCopy
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

param_dist = {
    'model__n_estimators': randint(50, 500),
    'model__max_depth': randint(2, 8),
    'model__learning_rate': uniform(0.01, 0.3),
}

search = RandomizedSearchCV(
    pipe, param_dist, n_iter=50,
    cv=5, scoring='roc_auc',
    n_jobs=-1, random_state=42
)
search.fit(X_trainval, y_trainval)

print(f"Best CV AUC: {search.best_score_:.4f}")
print(f"Test AUC: {roc_auc_score(y_test, search.predict_proba(X_test)[:, 1]):.4f}")

Common Mistakes and Bad Instincts

Using test set for model selection. If you look at test performance and then decide which model to deploy, you have contaminated the test set. Your reported performance will be higher than production performance.

Shuffling time-series data. Temporal data has autocorrelation. Shuffling before splitting makes train and test non-representative of the actual deployment scenario. Always split chronologically.

Forgetting stratify=y on imbalanced data. With 1% positive rate, a random split can put 0% positives in the validation fold by chance, making AUC undefined.

Reporting accuracy on imbalanced problems. A classifier that always predicts the majority class achieves 99% accuracy on a 99/1 split. Always report precision, recall, AUC, or F1 alongside accuracy.

Tuning too many hyperparameters. Each hyperparameter you tune "overfits" slightly to the validation distribution. Tune the 2–3 most impactful parameters and leave the rest at defaults.

Where to Go Next

• Module 11 (Classical ML Algorithms) covers the specific algorithm families - decision trees, gradient boosting, logistic regression - and when each one wins.
• Module 12 (Feature Engineering) covers how to build the features that go into these models without leaking information.
• Module 13 (Evaluation Metrics) goes deeper on choosing and interpreting the right metric for your specific problem.