Cross-Validation Done Right

Why Train/Test Split Is Not Enough

A single train/test split evaluates model performance on one specific subset of the data. How do you know that subset is representative? How do you know your reported performance is not lucky - that the model happened to be tested on easy examples?

Cross-validation evaluates the model's performance across multiple different subsets of data, producing a more reliable estimate of how it will perform on truly unseen data.

K-Fold Cross-Validation

In k-fold CV, the data is split into k equal folds. The model is trained k times - each time using k-1 folds for training and the remaining fold for validation. Performance is reported as the mean (and standard deviation) across all k runs.

pythonCopy
from sklearn.model_selection import cross_val_score, KFold
from sklearn.ensemble import RandomForestClassifier
import numpy as np

model = RandomForestClassifier(n_estimators=100, random_state=42)
kf = KFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(model, X, y, cv=kf, scoring='roc_auc')
print(f"AUC: {scores.mean():.4f} ± {scores.std():.4f}")
# Output: AUC: 0.8423 ± 0.0187

The standard deviation tells you how stable the performance estimate is. A small std (0.01-0.03) is good. A large std (>0.05) suggests high variance in the model.

Stratified K-Fold for Classification

For classification problems, especially with imbalanced classes, use stratified k-fold. It ensures each fold contains approximately the same proportion of each class as the full dataset.

pythonCopy
from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf, scoring='roc_auc')

Without stratification, one fold might contain mostly the majority class and give an artificially inflated score.

The Leakage Trap: Fitting Preprocessors Before CV

This is the most common cross-validation mistake, and it invalidates every result it touches:

pythonCopy
# WRONG: Preprocessor sees all data before CV
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # Leakage! Test fold statistics affect scaling
scores = cross_val_score(model, X_scaled, y, cv=5)

# CORRECT: Preprocessor fitted only on training folds, via Pipeline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', RandomForestClassifier(n_estimators=100))
])
scores = cross_val_score(pipeline, X, y, cv=5, scoring='roc_auc')

When you use Pipeline, sklearn ensures the scaler is fitted only on the training folds for each split. The test fold is transformed using statistics from the training data - the correct behavior.

This matters more than it seems. If you fit a standard scaler on the full dataset, the mean and variance used for scaling incorporate information about the test fold - a form of data leakage that makes your performance estimates optimistically biased.

Hyperparameter Tuning + CV: The Nested CV Problem

If you use the same CV procedure to both tune hyperparameters and report performance, your reported performance is biased - you have optimized hyperparameters for that specific cross-validation split.

The correct approach for an unbiased final estimate:

pythonCopy
from sklearn.model_selection import GridSearchCV, cross_val_score

# Inner CV for hyperparameter tuning
inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
# Outer CV for performance estimation
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

param_grid = {'model__max_depth': [3, 5, 10, None]}
pipeline = Pipeline([('model', RandomForestClassifier())])

grid_search = GridSearchCV(pipeline, param_grid, cv=inner_cv, scoring='roc_auc')
outer_scores = cross_val_score(grid_search, X, y, cv=outer_cv, scoring='roc_auc')

print(f"Nested CV AUC: {outer_scores.mean():.4f} ± {outer_scores.std():.4f}")

This is nested cross-validation: the inner loop tunes hyperparameters, the outer loop estimates performance. It is more expensive but produces unbiased estimates.

Time-Series Cross-Validation

For time-series data, future data must not inform past predictions. Random k-fold CV violates this and leaks future information. Use TimeSeriesSplit:

pythonCopy
from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)
for train_idx, val_idx in tscv.split(X):
    X_train, X_val = X[train_idx], X[val_idx]
    # Training set is always before validation set in time

What a Good CV Report Looks Like

Model: Random Forest (n_estimators=100, max_depth=10)
CV strategy: Stratified 5-fold, preprocessor inside pipeline
Metric: ROC-AUC

Fold scores: [0.841, 0.855, 0.838, 0.862, 0.847]
Mean AUC: 0.849 ± 0.009

Interpretation: Stable performance (low std). Acceptable for this problem's quality bar.

Always report the std alongside the mean. A model with AUC 0.85 ± 0.02 is more reliable than one with 0.87 ± 0.08.

What to Practice Next

• Take a pipeline that includes a preprocessing step (e.g., StandardScaler or TF-IDF) and deliberately fit it on the full dataset before CV; measure the resulting score inflation, then fix it by wrapping everything in sklearn.pipeline.Pipeline.
• Implement GroupKFold on a dataset where samples from the same entity (user, patient, store) appear in multiple rows - confirm that no group appears in both train and validation folds by printing fold assignments.
• Compare StratifiedKFold vs. plain KFold on a dataset with a 90/10 class imbalance across 5 folds - measure variance in the per-fold positive class ratio to make the leakage risk tangible.