ML FoundationsINTERMEDIATE13 min read

Data Cleaning and Feature Engineering

Better data beats better algorithms every time. This guide covers missing values, outliers, categorical encoding, feature engineering, and the silent killer - data leakage.

The Inconvenient Truth About Model Performance

In every major ML competition study and industry survey, the finding is the same: data quality and feature engineering account for most of the performance gain. Model architecture and hyperparameter tuning account for a fraction of it.

This is inconvenient because data cleaning is unglamorous. It involves looking at histograms, reading error logs, and talking to the team that owns the data pipeline. But it is where the work is, and skipping it is the fastest way to build a model that fails silently in production.

Understanding Your Data Before Touching It

Before any cleaning, answer these questions:

What does each row represent? A user? A transaction? An event? An ambiguous grain causes every downstream calculation to be wrong.
What time period does it cover? Data from three years ago may not represent today's population.
How was it collected? Surveys have self-selection bias. Clickstream data has recency bias. Labels from support tickets reflect what users bothered to report.
What would a correct label look like? If you do not understand the labeling process, you cannot trust the labels.

Document the answers. If you cannot answer them, talk to a domain expert before building anything.

Missing Values: Strategy Over Imputation

Missing values are not just a technical problem - they carry information. A missing income field on a credit application is not random. It correlates with the applicant's creditworthiness. Imputing it with the mean hides that signal.

Types of Missingness

MCAR (Missing Completely At Random): The missingness has no relationship to any variable. Rare in real data. Safe to impute.

MAR (Missing At Random): The missingness is related to observed variables but not the missing value itself. Acceptable to impute, but include a missingness indicator feature.

MNAR (Missing Not At Random): The missingness is related to the missing value itself. The most dangerous type. Do not impute without thought - the missing indicator may be your most predictive feature.

Imputation Strategies

Strategy	When to Use
Median	Numerical features with skew or outliers
Mean	Symmetrically distributed numerical features
Mode	Categorical features
Constant (e.g., "Unknown")	Categorical features where missingness is informative
KNN imputation	When relationships between features are strong
Model-based imputation	High-stakes features where accuracy of imputation matters

Always add a binary indicator column feature_was_missing alongside imputed values. This lets the model learn that missingness itself is a signal.

python
import pandas as pd
import numpy as np

def impute_with_indicator(df: pd.DataFrame, col: str) -> pd.DataFrame:
    df = df.copy()
    df[f"{col}_was_missing"] = df[col].isna().astype(int)
    df[col] = df[col].fillna(df[col].median())
    return df

Outliers: Understand Before Removing

Outliers are not always noise. In fraud detection, outliers are the signal. In sensor data, outliers may be real events. In survey data, outliers are often data entry errors. The decision to remove, cap, or keep an outlier depends on what it represents.

Detection Methods

IQR method: Values below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR are potential outliers. Robust to non-Gaussian distributions.

Z-score: Values more than 3 standard deviations from the mean. Works well for roughly Gaussian distributions. Sensitive to extreme values (the outlier inflates the standard deviation, masking itself).

Isolation Forest: A model-based approach that identifies points that are easy to isolate in the feature space. Good for multivariate outlier detection.

Handling Strategies

Capping (winsorization): Replace values beyond a percentile threshold with the threshold value. Preserves the observation, reduces extreme influence.

python
lower = df["revenue"].quantile(0.01)
upper = df["revenue"].quantile(0.99)
df["revenue_capped"] = df["revenue"].clip(lower, upper)

Log transform: Compresses large values and makes skewed distributions more Gaussian. Common for revenue, count, and price features.

python
df["log_revenue"] = np.log1p(df["revenue"])  # log1p handles zeros

Categorical Encoding: Choosing the Right Representation

How you encode categorical variables affects both model performance and training stability.

One-Hot Encoding

Creates a binary column for each category. Safe for nominal categories with low cardinality (< 20 unique values). Dangerous for high-cardinality features: encoding 10,000 cities as one-hot creates 10,000 sparse columns.

python
pd.get_dummies(df["city"], prefix="city", drop_first=True)

Ordinal Encoding

Maps categories to integers. Use only when categories have a natural order (e.g., Low=0, Medium=1, High=2). Using it for nominal categories (like city names) tells the model that one city is "between" two others, which is meaningless.

Target Encoding

Replaces a category with the mean target value for that category. Powerful for high-cardinality features. Dangerous if done naively - you must use out-of-fold statistics to avoid leakage.

python
# Wrong: leaks target information
df["city_encoded"] = df.groupby("city")["target"].transform("mean")

# Correct: use cross-validation folds
from category_encoders import TargetEncoder
enc = TargetEncoder(cols=["city"])
X_train_enc = enc.fit_transform(X_train, y_train)  # fits only on train
X_val_enc = enc.transform(X_val)                   # transforms using train stats

Feature Engineering: Creating Signal

Feature engineering is the process of creating new features that make the underlying patterns more accessible to a model. A model trained on raw features and a model trained on well-engineered features can show dramatically different performance - often larger than the difference between model families.

Interaction Features

Multiply or combine two features to capture their joint effect. Example: spend_per_session = total_spend / num_sessions captures user value in a way that neither feature alone does.

Date/Time Features

Extract components that have predictive power:

Day of week (captures weekly seasonality)
Hour of day (captures daily patterns)
Days since last event (recency)
Is holiday (binary flag)
Days until next holiday (anticipation effect)

python
df["day_of_week"] = df["timestamp"].dt.dayofweek
df["hour"] = df["timestamp"].dt.hour
df["is_weekend"] = (df["day_of_week"] >= 5).astype(int)

Aggregation Features

Compute statistics about a group at the row level. For user-level prediction tasks, these are often the most predictive features:

python
user_stats = df.groupby("user_id").agg(
    total_purchases=("purchase_amount", "sum"),
    avg_purchase=("purchase_amount", "mean"),
    purchase_count=("purchase_amount", "count"),
    days_since_first=("date", lambda x: (x.max() - x.min()).days),
).reset_index()

Data Leakage: The Silent Killer

Data leakage occurs when your model has access to information during training that it would not have at prediction time. It is the most common cause of models that perform well in evaluation but fail in production.

Types of Leakage

Target leakage: A feature that is directly derived from or caused by the target variable. Example: using claim_filed as a feature to predict is_fraudulent - claims are only filed because something suspicious happened.

Temporal leakage: Using future data to predict past events. Example: computing a user's total lifetime spend as a feature, when at prediction time you only know their spend up to the event date.

Pipeline leakage: Fitting a preprocessor (scaler, imputer, encoder) on the full dataset before splitting into train/val/test.

How to Prevent It

Always split before fitting any preprocessor - use sklearn Pipeline to make this automatic.
For time-series data, split by time, not randomly.
Ask: "would I have this feature at prediction time?" for every feature you add.
Sanity check: if your model is suspiciously good (AUC > 0.99 on a hard problem), you likely have leakage.

A Feature Engineering Workflow

python
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import GradientBoostingClassifier

numeric_pipeline = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])

preprocessor = ColumnTransformer([
    ("num", numeric_pipeline, numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
])

full_pipeline = Pipeline([
    ("preprocess", preprocessor),
    ("model", GradientBoostingClassifier()),
])

full_pipeline.fit(X_train, y_train)

This pipeline structure makes leakage prevention automatic - the preprocessor only sees training data when fitted.

Common Mistakes and Bad Instincts

Imputing before splitting. This leaks validation statistics (mean, median) into training. Always split first.

Dropping all rows with missing values. Unless missingness is truly MCAR, this introduces selection bias. Impute with a strategy appropriate to the missingness mechanism.

One-hot encoding high-cardinality features. A feature with 10,000 unique values becomes 10,000 columns. Use target encoding or embedding-based approaches instead.

Not checking for temporal leakage. Ask for every feature: "at the time I would use this model, would this feature be available?" If not, it is leakage.

Treating feature engineering as a one-time step. It is iterative. The first round of features tells you which signals matter, which guides the next round of engineering.

Where to Go Next

Feature engineering is covered deeply in Module 3 of the College Student path (Feature Engineering and Data Leakage Defense) and Module 6 of the SWE path (Classical ML Model Selection and Feature Engineering). The hands-on deliverable in both modules requires producing a reproducible feature pipeline on a real dataset with documented leakage checks.

A Feature Engineering Case Study

Suppose you are predicting subscription churn. The raw data includes sign-up date, last login, number of support tickets, plan type, payment failures, and product usage events.

Naive features might include:

Total logins
Total tickets
Current plan
Total payment failures

Better features encode behavior relative to a decision point:

Logins in the last 7, 30, and 90 days
Days since last meaningful action
Ticket count before the prediction timestamp
Failed payments before the prediction timestamp
Change in weekly usage over the last month
Plan tenure at prediction time

The difference is time awareness. A churn model should only use information that would have been available when the prediction was made. If you compute "total logins" using activity after churn, you have leakage.

Cleaning Is a Modeling Decision

Data cleaning is not janitorial work. Every cleaning choice encodes an assumption.

Problem	Bad default	Better question
Missing values	Fill everything with zero	Does missingness itself carry meaning?
Outliers	Remove all extreme values	Are they errors, fraud, VIPs, or rare but real cases?
Duplicates	Drop blindly	Are duplicates retries, events, or ingestion bugs?
Categories	One-hot everything	Are rare categories stable enough to keep?
Dates	Extract month only	What time window matches the decision?

Strong ML engineers document these choices because they affect both performance and ethics.

Leakage Patterns to Hunt

Leakage is the most dangerous feature engineering failure because it makes your model look better during evaluation.

Common leakage sources:

Features updated after the label event
Aggregates computed using the full dataset before splitting
Target encodings fit outside cross-validation folds
Duplicated users or entities split across train and test
Text fields containing label-like phrases
Operational status fields that are consequences of the outcome

Ask this for every feature: would this exact value exist at prediction time?

Feature Store Thinking Without a Feature Store

You do not need a feature store to think clearly about features. Create a feature contract:

Name
Description
Source table or event
Computation window
Refresh frequency
Owner
Allowed range or categories
Known caveats

This turns features from notebook inventions into maintained product assets.

Evaluation After Feature Changes

Do not only ask whether a feature improves the average metric. Ask:

Does it improve the target business slice?
Does it increase leakage risk?
Is it available at inference time?
Is it stable over time?
Is it explainable to stakeholders?
Does it introduce fairness concerns?

A feature that adds 0.5% AUC but requires fragile infrastructure may be worse than a simple feature that is stable, cheap, and explainable.

Exercise

Pick a real dataset and create a feature table with ten candidate features. For each one, write:

Prediction-time availability
Cleaning rule
Leakage risk
Expected direction of effect
Monitoring signal

This exercise will improve your modeling more than trying five algorithms blindly.

Closing Thought

The practical standard is not memorization. It is whether you can use the idea to make a better engineering decision, explain that decision to someone else, and notice when reality disagrees with your assumptions.

What to Do Next

Turn this article into a small artifact. Write a checklist, run a tiny experiment, sketch the architecture, or review an old project using the concepts above. Learning becomes durable when it changes what you inspect before you trust a result.

For a portfolio or team setting, save that artifact next to the code or decision memo. Future reviewers should be able to see not only what you built, but how you reasoned about correctness, risk, and tradeoffs.

Team Review Prompts

Before treating this work as complete, ask a teammate to review it using three prompts:

What assumption is most likely to break in production?
What evidence would make you trust the result?
What simpler approach should we compare against?

These questions are deliberately plain. They work because they force the discussion away from tool enthusiasm and back toward judgment, evidence, and maintainability.

Final Rule

Features should be useful, available, stable, and explainable. If a feature fails one of those tests, document the risk before using it. Most production model failures are not caused by exotic algorithms. They are caused by ordinary data assumptions that nobody wrote down.

Label Quality

Feature work cannot compensate for broken labels. Before modeling, inspect how labels are created:

Are they human judgments, business events, heuristics, or delayed outcomes?
Do different reviewers agree?
Are labels missing for certain groups?
Does the label happen before or after the product action?
Has the label definition changed over time?

For example, "customer churned" may mean cancellation date, no login for 60 days, non-renewal, or account downgrade. Those are different labels. Choosing one changes the model and the product action.

Data Contracts

For production pipelines, create data contracts between upstream producers and ML consumers. A contract should define required fields, types, freshness, allowed null rates, and owner escalation. Without contracts, upstream changes silently become model regressions.

The best feature engineering is boring in production: stable, documented, monitored, and understood by both data producers and model owners.

Text, Time, and Categorical Features

Different feature types need different handling.

Text features can be converted into counts, TF-IDF vectors, embeddings, or LLM-derived labels. Start simple. If a support ticket subject line contains enough signal for a bag-of-words model, you may not need a transformer.

Time features require special care. Extracting month or day_of_week may help, but the more important question is windowing. A feature such as "orders in the last 30 days" is meaningful because it is tied to a prediction timestamp. Without that timestamp, time features often leak.

Categorical features need stability. Rare categories can overfit. New categories can appear in production. A robust pipeline handles unknown categories without crashing and groups very rare categories when appropriate.

Monitoring Feature Health

Feature engineering does not end at training. Monitor:

Missing-value rate
Distribution drift
Cardinality changes
New unseen categories
Feature freshness
Upstream pipeline failures
Correlation with the target over time

If a high-value feature stops updating, the model may keep serving predictions with no obvious infrastructure error. Feature monitoring catches failures that normal service monitoring misses.

Documentation Template

For every important feature, document:

text
Feature: usage_30d
Definition: Count of meaningful product events in the 30 days before prediction_time
Source: product_events
Refresh: hourly
Owner: growth-data
Known risks: bot activity can inflate counts
Prediction-time available: yes

This makes features reviewable. It also helps future engineers avoid rebuilding the same logic with slightly different semantics.