Classical ML Model Selection and Feature Engineering

This module is for software engineers who know how to build systems but are developing ML judgment. It covers the practical art of choosing the right model and engineering the right features - the two decisions that dominate real-world ML performance far more than hyperparameter tuning.

The Model Selection Decision Framework

Most tabular production ML problems are solved with gradient-boosted trees. But knowing when to use what separates practitioners from people who just apply the same algorithm to every problem.

pythonCopy
# Decision heuristic for tabular data
def choose_model(n_rows, n_features, interpretability_required, latency_ms_budget):
    if interpretability_required:
        if n_rows < 10_000:
            return "LogisticRegression with L1 (feature selection built in)"
        else:
            return "DecisionTree(max_depth=5) for rules, or LR for probabilities"
    if n_rows < 5_000:
        return "LogisticRegression or SVM - too little data for GBM to shine"
    if latency_ms_budget < 5:
        return "LogisticRegression - tree ensembles are ~10-50ms at scale"
    # Default: gradient boosting
    if n_rows > 1_000_000:
        return "LightGBM with hist algorithm - fastest at scale"
    return "LightGBM with default settings - best accuracy/time tradeoff"

The most common production mistake is training a neural network on a 10K-row dataset. Gradient boosting will outperform it, train 100x faster, and be easier to debug.

Feature Engineering With an Engineer's Discipline

Experienced engineers make better feature engineers because they think systematically about what information is available at prediction time and what is being modeled.

pythonCopy
import pandas as pd
import numpy as np

def engineer_features(df: pd.DataFrame, reference_date: pd.Timestamp) -> pd.DataFrame:
    """
    All features are computed as of reference_date to prevent leakage.
    """
    features = df[['user_id']].copy()

    # Temporal features - computed relative to reference_date
    features['account_age_days'] = (reference_date - df['signup_date']).dt.days
    features['days_since_last_login'] = (reference_date - df['last_login_date']).dt.days
    features['days_since_last_purchase'] = (reference_date - df['last_purchase_date']).dt.days

    # Ratio features - prevent scale sensitivity
    features['purchase_frequency'] = (
        df['purchase_count'] / (features['account_age_days'] + 1)
    )
    features['avg_order_value'] = df['total_spend'] / (df['purchase_count'] + 1)

    # Log transforms for heavy-tailed distributions
    for col in ['total_spend', 'page_views']:
        features[f'log_{col}'] = np.log1p(df[col])

    # Interaction features
    features['spend_per_login'] = (
        df['total_spend'] / (df['login_count'] + 1)
    )

    return features

The reference_date parameter is the critical design element here. Every feature is computed relative to this anchor timestamp, making the pipeline deployable for both training (use historical dates) and serving (use today).

Calibration: When Probabilities Must Be Trusted

Gradient boosting outputs are not well-calibrated probabilities by default. If your product uses the score for decisions (risk thresholds, pricing tiers, prioritization), calibration matters:

pythonCopy
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
import matplotlib.pyplot as plt

# Train + calibrate
base_model = lgb.LGBMClassifier(n_estimators=300)
calibrated = CalibratedClassifierCV(base_model, method='isotonic', cv=5)
calibrated.fit(X_train, y_train)

# Verify calibration
prob_true, prob_pred = calibration_curve(y_cal, calibrated.predict_proba(X_cal)[:, 1], n_bins=10)
plt.plot(prob_pred, prob_true, 'o-', label='Calibrated')
plt.plot([0, 1], [0, 1], '--', label='Perfect')

After calibration, predict_proba(x) = 0.7 should mean 70% of those users actually exhibit the predicted behavior.

Temporal Leakage: The SWE Blind Spot

Engineers who are used to deterministic systems often underestimate temporal leakage because their instinct is to use the most data possible. The rule is simple: every feature's value must be knowable before the prediction timestamp.

pythonCopy
# WRONG: rolling features that look into the future
df = df.sort_values(['user_id', 'date'])
df['rolling_spend_7d'] = df.groupby('user_id')['spend'].transform(
    lambda x: x.rolling(7).mean()  # includes the current row!
)

# RIGHT: use shift(1) to exclude current row
df['rolling_spend_7d'] = df.groupby('user_id')['spend'].transform(
    lambda x: x.shift(1).rolling(7).mean()
)

# WRONG: train/test split before temporal aggregation
X_train, X_test = train_test_split(df, test_size=0.2, random_state=42)
# Now compute features on X_train and X_test - test statistics leaked into train

# RIGHT: temporal split first, then feature computation
cutoff = pd.Timestamp('2024-01-01')
train = df[df['event_date'] < cutoff]
test  = df[df['event_date'] >= cutoff]
# Compute user aggregates from train, join to test (no leakage)

Common Mistakes and Bad Instincts

One-hot encoding high-cardinality features. A product_sku with 50K unique values becomes 50K binary columns. Use frequency encoding, target encoding (inside a Pipeline), or embeddings.

Not checking if your "improvement" is within statistical noise. An AUC improvement of 0.002 with ± 0.008 standard deviation across 5 folds is not a real improvement. Use a paired t-test.

Adding features without a hypothesis. Every feature you add should come with a hypothesis about why it should help. Validate with permutation importance. Remove features that don't contribute - they add noise and slow serving.

Where to Go Next

• Module 7 (Evaluation and Experimentation) covers how to rigorously measure whether these feature changes actually improve your model.
• Module 8 (ML Debugging) covers how to investigate when a model is underperforming despite good features.