Feature Engineering and Data Leakage Defense

Feature engineering is where most of the real performance gains in production ML come from. Switching from random forest to gradient boosting might give you 0.01 AUC. Adding the right feature can give you 0.05. The flip side: incorrectly engineered features cause leakage - the most dangerous silent bug in ML, because it produces impressive-looking metrics that collapse in production.

What Features Actually Are

A feature is anything you measure about an entity before the time of prediction. For a loan default model: the applicant's credit history, income, age of oldest account. Not: whether they defaulted (that's the label), not their payment behavior in the 30 days after the loan (that's future data leakage).

Raw data almost never comes in model-ready form. Feature engineering is the process of transforming raw data into a numeric matrix where each row is an example and each column is a signal for the model.

Numerical Features

Scaling: Tree-based models don't need scaling. Linear models and neural networks do.

pythonCopy
from sklearn.preprocessing import StandardScaler, RobustScaler

# StandardScaler: (x - mean) / std - sensitive to outliers
# RobustScaler: (x - median) / IQR - better for skewed distributions

Log transform for skewed distributions: Income, purchase amounts, and click counts follow heavy-tailed distributions. Log transform compresses the long tail.

pythonCopy
import numpy as np
import pandas as pd

df['log_amount'] = np.log1p(df['purchase_amount'])  # log1p handles zeros

Binning: Convert continuous into ordered categorical to allow the model to find non-linear thresholds.

pythonCopy
df['age_bucket'] = pd.cut(df['age'], bins=[18, 25, 35, 50, 65, 100], labels=False)

Clipping outliers: Cap extreme values to prevent single examples from dominating the loss.

pythonCopy
p1, p99 = df['revenue'].quantile([0.01, 0.99])
df['revenue_clipped'] = df['revenue'].clip(p1, p99)

Categorical Features

One-hot encoding: For nominal categories with low cardinality (< 20 unique values). Creates a binary column per category.

pythonCopy
from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
encoded = enc.fit_transform(df[['city', 'device_type']])

Ordinal encoding: For categories with natural order (low/medium/high, bronze/silver/gold).

pythonCopy
from sklearn.preprocessing import OrdinalEncoder

ord_enc = OrdinalEncoder(categories=[['low', 'medium', 'high']])
df['risk_level_ord'] = ord_enc.fit_transform(df[['risk_level']])

Target encoding: Replace category with the mean target value for that category. Powerful for high-cardinality categories (zip code, product ID) but must be done inside CV folds to avoid leakage.

pythonCopy
# WRONG: fit on all data before splitting - leaks target into features
means = df.groupby('zip_code')['churned'].mean()
df['zip_target_enc'] = df['zip_code'].map(means)

# RIGHT: use sklearn's TargetEncoder which handles cross-fitting automatically
from sklearn.preprocessing import TargetEncoder

enc = TargetEncoder(smooth='auto')
# fit_transform inside Pipeline will cross-fit correctly

Frequency encoding: Replace category with its count or proportion in the training set. Useful as a complement to target encoding.

pythonCopy
freq = df['product_id'].value_counts(normalize=True)
df['product_freq'] = df['product_id'].map(freq).fillna(0)

Datetime Features

Time columns contain a wealth of signal that needs to be extracted explicitly.

pythonCopy
df['event_ts'] = pd.to_datetime(df['event_ts'])

df['hour'] = df['event_ts'].dt.hour
df['dayofweek'] = df['event_ts'].dt.dayofweek     # 0=Monday
df['is_weekend'] = df['dayofweek'].isin([5, 6]).astype(int)
df['month'] = df['event_ts'].dt.month
df['days_since_signup'] = (df['event_ts'] - df['signup_ts']).dt.days
df['days_to_renewal'] = (df['renewal_ts'] - df['event_ts']).dt.days

Cyclical encoding for hour/day to preserve the periodic structure:

pythonCopy
df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24)
df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24)

Interaction and Aggregate Features

Models can learn interactions, but making them explicit often helps, especially for linear models.

pythonCopy
# Ratio features capture relationships between numerics
df['spend_per_session'] = df['total_spend'] / (df['session_count'] + 1)
df['return_rate'] = df['return_count'] / (df['purchase_count'] + 1)

# Aggregate features from related entities (user history)
user_stats = df.groupby('user_id').agg(
    user_purchase_count=('order_id', 'count'),
    user_avg_order_value=('order_value', 'mean'),
    user_last_purchase_days=('days_since_purchase', 'min')
).reset_index()
df = df.merge(user_stats, on='user_id', how='left')

The Three Forms of Data Leakage

Leakage means your model has access to information during training that it would not have at prediction time. It produces inflated validation metrics that disappear in production.

Pipeline Leakage

Fitting a transformer (scaler, imputer, encoder) on the full dataset before the train/val split leaks validation statistics into training.

pythonCopy
# WRONG: scaler sees validation data during fit
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # val mean/std leak into train
X_train_sc, X_val_sc = train_test_split(X_scaled)

# RIGHT: scaler only sees training data
X_train, X_val = train_test_split(X)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_val_sc = scaler.transform(X_val)  # transform only, not fit_transform

The bulletproof solution is always: use sklearn Pipelines and let cross_val_score handle the fitting correctly.

Temporal Leakage

Using future data to predict the past. Any feature derived from data that occurs after the prediction timestamp introduces temporal leakage.

pythonCopy
# WRONG: rolling average includes the current row's future purchases
df['rolling_7d_purchases'] = df.groupby('user_id')['purchases'].transform(
    lambda x: x.rolling(7).mean()
)

# RIGHT: rolling average is exclusive of the current row
df = df.sort_values(['user_id', 'event_date'])
df['rolling_7d_purchases'] = df.groupby('user_id')['purchases'].transform(
    lambda x: x.shift(1).rolling(7).mean()  # shift(1) excludes current row
)

For train/test splits on temporal data, always split by time:

pythonCopy
cutoff = pd.Timestamp('2024-01-01')
train = df[df['event_date'] < cutoff]
test  = df[df['event_date'] >= cutoff]

Target Leakage

Using a feature that is derived from or highly correlated with the label in a way that wouldn't be available before the label is observed.

Example: in a loan default model, late_payment_count (measured after the loan period) leaks the outcome. credit_score_at_application (measured before) is fine.

Rule: every feature's timestamp must be before the prediction timestamp. Document when each feature is computed and verify it.

Building a Leakage-Proof Feature Pipeline

pythonCopy
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer

numeric_features = ['age', 'income', 'log_amount', 'days_since_signup']
categorical_features = ['city', 'device_type', 'account_tier']

numeric_pipe = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
])

categorical_pipe = Pipeline([
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('encoder', OneHotEncoder(sparse_output=False, handle_unknown='ignore')),
])

preprocessor = ColumnTransformer([
    ('num', numeric_pipe, numeric_features),
    ('cat', categorical_pipe, categorical_features),
])

full_pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('model', lgb.LGBMClassifier(n_estimators=300, random_state=42)),
])

# cross_val_score re-fits the entire pipeline in each fold - no leakage
from sklearn.model_selection import cross_val_score
scores = cross_val_score(full_pipeline, X_trainval, y_trainval, cv=5, scoring='roc_auc')

Feature Selection

Not all features help. Irrelevant features add noise and slow training. Three practical approaches:

Permutation importance: Shuffle one feature at a time and measure the drop in validation metric. Unlike tree impurity importance, this is unbiased toward high-cardinality features.

pythonCopy
from sklearn.inspection import permutation_importance

result = permutation_importance(
    model, X_val, y_val, n_repeats=10, scoring='roc_auc', random_state=42
)
importance_df = pd.DataFrame({
    'feature': X_val.columns,
    'importance_mean': result.importances_mean,
    'importance_std': result.importances_std,
}).sort_values('importance_mean', ascending=False)

# Drop features with negative permutation importance - they hurt
bad_features = importance_df[importance_df['importance_mean'] < 0]['feature'].tolist()

Correlation-based removal: Drop one of any two features with correlation > 0.95.

L1 regularization: Logistic regression or Lasso with penalty='l1' zeroes out irrelevant coefficients automatically.

Common Mistakes and Bad Instincts

Computing aggregates over the entire dataset before splitting. If mean_purchase_value is computed from all rows including the test set, test rows contribute to the statistics used to train on. Always compute aggregates in the training set and join them to validation/test.

Forgetting shift(1) in time-series rolling features. The current row's value at time t should not be included in the window used to predict at time t. Use .shift(1) to exclude it.

Using target encoding without cross-fitting. Target encoding without using TargetEncoder inside a Pipeline will overfit dramatically - the model will learn to encode rare categories with high target rates (memorized from training) that don't generalize.

Adding every feature you can compute. Feature bloat increases training time, makes debugging harder, and can hurt performance via noise. Add features deliberately with a hypothesis about why they help, and verify with permutation importance.

One-hot encoding high-cardinality features. A product_id column with 100,000 unique values creates 100,000 binary columns. Use frequency encoding or target encoding instead.

Where to Go Next

• Module 13 (Evaluation Metrics) covers how to tell whether your engineered features actually helped - using the right metrics for your problem type.
• Module 10 (Supervised Learning Foundations) covers how to structure the training pipeline these features feed into.
• The post data-cleaning-and-feature-engineering covers data quality issues (missing values, outliers) in more depth.