The Train / Val / Test Split: Common Mistakes

Why the Split Matters

The fundamental promise of ML evaluation: "this is how the model will perform on data it has never seen." Keeping that promise requires that the test set is genuinely unseen and representative. Every mistake in the split corrupts this promise - sometimes subtly, sometimes catastrophically.

Mistake 1: Using Test Set Performance to Make Decisions

The moment you look at test set performance and make a decision based on it - tune a hyperparameter, add a feature, choose between two models - you have "used up" the test set. Every look leaks information.

The rule: the test set is touched once, at the end, to report final performance. All decision-making uses the validation set.

This is why you need three sets:

• Train: Model learns from this
• Validation: Used for all decisions (hyperparameter tuning, feature selection, model selection)
• Test: Reported once at the end

pythonCopy
from sklearn.model_selection import train_test_split

X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.15, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=0.18, random_state=42)
# Result: roughly 70% train, 15% val, 15% test

Mistake 2: Random Split on Time-Series Data

For any data with a temporal dimension, random splitting leaks future information into the training set.

Wrong: Randomly shuffling all transactions, then splitting. The model sees December 2024 transactions during training and is tested on January 2024 - the past.

Right: Split by time. Everything before date X is train. Everything between X and Y is validation. Everything after Y is test.

pythonCopy
df_sorted = df.sort_values('timestamp')
n = len(df_sorted)
train_end = int(n * 0.70)
val_end = int(n * 0.85)

df_train = df_sorted.iloc[:train_end]
df_val = df_sorted.iloc[train_end:val_end]
df_test = df_sorted.iloc[val_end:]

Mistake 3: Leaking Group Information Across Splits

If your data has groups (users, patients, devices) with multiple rows per group, random row splitting puts different rows from the same group in both train and test.

The model can "recognize" a group from its training rows when it sees test rows from the same group - inflating performance. The model is not generalizing to new groups; it is memorizing group-specific patterns.

pythonCopy
from sklearn.model_selection import GroupShuffleSplit

gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(X, y, groups=df['user_id']))
X_train, X_test = X[train_idx], X[test_idx]

Mistake 4: Fitting the Preprocessor on All Data Before Splitting

This is the leakage mistake discussed in the cross-validation post. Even with a correct train/test split, fitting a preprocessor on the full dataset before splitting leaks test statistics into training.

The fix: always use sklearn Pipeline and fit the pipeline on training data only. Use transform (not fit_transform) to apply the preprocessing to validation and test sets.

Mistake 5: Using the Wrong Split Size for Small Datasets

The standard 80/20 or 70/15/15 split assumes a reasonably large dataset. For small datasets (< 1,000 examples), a 20% test set might contain only 200 examples - producing unreliable performance estimates.

For small datasets: use k-fold cross-validation instead of a single train/test split. The performance estimate is averaged over k different subsets, reducing variance.

What a Good Split Looks Like

Before fitting any model, verify:

• Class distribution is approximately equal across train, val, and test (for classification)
• Time distribution is non-overlapping (for time series)
• No user/group appears in both train and test (for group-structured data)
• Preprocessors are not fitted on val or test data

A 5-minute check here prevents weeks of debugging a model that "works" in evaluation but fails in production.

What to Practice Next

• Take a dataset with a temporal column and implement a time-based split (all rows before date X for train, after for test) - compare the model's test performance against a random split and explain why they differ.
• Introduce a deliberate target-leakage feature into a dataset, train a model, and observe the inflated validation metric - then remove the feature and compare; this makes leakage visceral rather than abstract.
• Audit a Kaggle notebook that reports a high leaderboard score: trace how the train/val/test split was done and identify at least one methodological choice that might inflate the reported metric.