Leakage-Proof Feature Pipelines

Data leakage is the most insidious bug in machine learning. A model with leakage will appear to work brilliantly during development: high offline metrics, clean validation curves, confident predictions. Then you deploy it, and it fails. Or worse, it does not obviously fail - it just makes confident wrong decisions that erode business trust slowly.

This article covers the three types of leakage, concrete examples of each, how to audit a pipeline, and the sklearn patterns that prevent them.

Type 1: Pipeline Leakage (Preprocessing on the Full Dataset)

Pipeline leakage occurs when a preprocessing step that "learns" from data is fit on the entire dataset - including the test set - before the train/test split.

pythonCopy
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X = np.random.randn(1000, 10)
y = (X[:, 0] + X[:, 1] > 0).astype(int)

# LEAKY: scaler sees test data before split
scaler_bad = StandardScaler()
X_scaled_bad = scaler_bad.fit_transform(X)  # uses test set statistics!
X_train_bad, X_test_bad, y_train, y_test = train_test_split(
    X_scaled_bad, y, test_size=0.2, random_state=42)
model_bad = LogisticRegression()
model_bad.fit(X_train_bad, y_train)
acc_bad = accuracy_score(y_test, model_bad.predict(X_test_bad))

# CORRECT: split first, then fit scaler on train only
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler_good = StandardScaler()
X_train_scaled = scaler_good.fit_transform(X_train)
X_test_scaled = scaler_good.transform(X_test)  # transform only - critical!
model_good = LogisticRegression()
model_good.fit(X_train_scaled, y_train)
acc_good = accuracy_score(y_test, model_good.predict(X_test_scaled))

print(f"Leaky accuracy: {acc_bad:.4f}")
print(f"Correct accuracy: {acc_good:.4f}")

For most standardization steps the effect is small. But for imputation, feature selection, or target encoding, it can inflate your reported accuracy by several percentage points.

Type 2: Temporal Leakage (Using Future Information)

Temporal leakage occurs when you use information from the future to predict the past - which is impossible in production where the future has not happened yet.

pythonCopy
import pandas as pd

# E-commerce dataset: predict whether a user will churn this month
df = pd.DataFrame({
    'user_id': range(1000),
    'signup_date': pd.date_range('2022-01-01', periods=1000, freq='D'),
    'total_orders_lifetime': np.random.randint(1, 100, 1000),
    'orders_next_30d': np.random.randint(0, 5, 1000),  # FUTURE FEATURE
    'churned': (np.random.rand(1000) > 0.8).astype(int)
})

# LEAKY: 'orders_next_30d' is unknowable at prediction time
X_leaky = df[['total_orders_lifetime', 'orders_next_30d']]

# CORRECT: only use features available at prediction time
X_correct = df[['total_orders_lifetime']]

# Also: random split on temporal data is leaky
# WRONG for time-series
X_train_bad, X_test_bad = train_test_split(X_correct, test_size=0.2, random_state=42)

# RIGHT: respect temporal order
cutoff_date = df['signup_date'].quantile(0.8)
train_df = df[df['signup_date'] <= cutoff_date]
test_df = df[df['signup_date'] > cutoff_date]

The key question for every feature: "At the time this prediction is made in production, is this value available?" If the answer is "sometimes no," you have a temporal leakage risk.

Type 3: Target Leakage (Features Derived From the Label)

Target leakage is the most dangerous type because it is often invisible in the feature list. It occurs when a feature is causally downstream of the target, or when the feature is the same signal as the target measured slightly differently.

pythonCopy
# Medical example: predict whether a patient will be diagnosed with a condition
patients = pd.DataFrame({
    'age': np.random.randint(20, 80, 500),
    'symptom_score': np.random.randint(0, 10, 500),
    'treatment_prescribed': np.random.binomial(1, 0.3, 500),  # LEAKY
    'diagnosis': np.random.binomial(1, 0.25, 500)
})

# 'treatment_prescribed' is caused by the diagnosis
# If you predict "diagnosis" using "treatment_prescribed",
# you are using the outcome to predict itself.
# This feature would have near-zero value in production:
# treatment is prescribed AFTER diagnosis, not before.

# Audit: check correlation of every feature with the target
correlation = patients.corr(numeric_only=True)['diagnosis'].sort_values(ascending=False)
print(correlation)
# High correlation between treatment_prescribed and diagnosis is a red flag

Target leakage is especially common with:

• Status fields that update when the event occurs (e.g., "claim_status" for predicting fraud)
• Aggregates computed over periods that include the prediction window
• Any field filled in by the same person who records the outcome

Sklearn Pipeline: The Correct-by-Construction Solution

The sklearn Pipeline class makes pipeline leakage impossible by design. When you use cross_val_score or fit on a Pipeline, the preprocessing steps are automatically refit on each training fold.

pythonCopy
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score

# Define preprocessing per column type
numeric_features = ['age', 'total_orders_lifetime']
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

preprocessor = ColumnTransformer(transformers=[
    ('num', numeric_transformer, numeric_features),
])

# Full pipeline: preprocessing + model
pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', GradientBoostingClassifier(n_estimators=100, random_state=42))
])

# This is leakage-proof: preprocessing fits on train fold only in each CV split
X = patients[numeric_features]
y = patients['diagnosis']

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipeline, X, y, cv=cv, scoring='roc_auc')
print(f"ROC-AUC: {scores.mean():.4f} ± {scores.std():.4f}")

Using a Pipeline does not protect you from temporal or target leakage - those require reasoning about what information exists at prediction time.

Auditing a Pipeline for Leakage

A systematic audit checks three things:

1. Does any preprocessing step see test data before split? Use Pipeline.
2. Does any feature contain information that would not exist at prediction time in production?
3. Is any feature causally downstream of the target?

Common red flags: features with implausibly high correlation to the target (>0.9), features with names containing "result," "outcome," "final," "post," or the name of the target event, and features computed over windows that overlap the prediction period.

Common Mistakes

Fitting transformers outside a Pipeline: Even if you call fit_transform on training data and transform on test data manually, it is easy to make mistakes. Let Pipeline manage it.

Target encoding without cross-fitting: Mean encoding the target into a feature requires careful out-of-fold computation or it becomes target leakage.

Ignoring temporal ordering in financial or medical data: Random splits are almost always wrong for time-series prediction tasks.

Where to Go Next

• data-contracts-quality-features - validate feature distributions before they enter the pipeline
• evaluation-metrics-error-analysis-systems - understand what inflated metrics look like and how to catch them
• training-pipelines-experiment-strategy - reproducible experiments that make leakage audits easier