Model Debugging and Error Analysis

Building a model that reaches some initial metric is the easy part. Systematically improving it - and understanding why it fails - is the engineering work that separates prototype from production. Model debugging is structured investigation: you form a hypothesis about why the model is wrong, gather evidence, and make a targeted fix. Random experimentation without diagnosis is expensive and rarely works.

The Debugging Mindset

When your model underperforms, the cause is almost always one of four things:

1. Wrong label or data quality issue - the model is learning from corrupted signal.
2. Missing or leaking feature - the model doesn't have the information it needs, or has information it shouldn't.
3. Distribution mismatch - the model was trained on a distribution different from the one it's evaluated on.
4. Model capacity / regularization - the model is too simple to capture the pattern, or overfitting noise.

Work through these in order. Data issues are more common than model capacity issues.

Step 1: Read Your Errors, Don't Just Measure Them

The confusion matrix tells you how many errors you're making. Looking at the actual examples tells you why.

pythonCopy
import pandas as pd
import numpy as np

y_prob = model.predict_proba(X_val)[:, 1]
y_pred = (y_prob >= 0.5).astype(int)

# Annotate validation set with predictions
val_df = X_val.copy()
val_df['true_label']   = y_val.values
val_df['pred_label']   = y_pred
val_df['pred_prob']    = y_prob
val_df['error_type']   = np.where(
    (val_df['true_label'] == 1) & (val_df['pred_label'] == 0), 'FN',
    np.where(
        (val_df['true_label'] == 0) & (val_df['pred_label'] == 1), 'FP', 'correct'
    )
)

# Look at your worst false negatives - high-confidence wrong predictions
fn_examples = val_df[val_df['error_type'] == 'FN'].sort_values('pred_prob')
print("Worst false negatives (model was most confident these were negative):")
print(fn_examples.head(10))

Manually read 20–50 false positives and 20–50 false negatives. You will find patterns - specific feature ranges, edge cases, or data quality issues - that no aggregate metric will reveal.

Step 2: Sliced Evaluation to Find Failure Modes

Global metrics hide failure modes in subpopulations. A model with AUC=0.88 overall might have AUC=0.61 on a particular customer segment that represents 15% of your revenue.

pythonCopy
def evaluate_slice(df, mask, label_col='true_label', prob_col='pred_prob', min_size=50):
    from sklearn.metrics import roc_auc_score, average_precision_score
    sub = df[mask]
    if len(sub) < min_size or sub[label_col].nunique() < 2:
        return None
    return {
        'n': len(sub),
        'positive_rate': sub[label_col].mean(),
        'roc_auc': roc_auc_score(sub[label_col], sub[prob_col]),
        'pr_auc': average_precision_score(sub[label_col], sub[prob_col]),
    }

# Slice by categorical features
for col in ['device_type', 'account_tier', 'region']:
    for value in val_df[col].unique():
        mask = val_df[col] == value
        result = evaluate_slice(val_df, mask)
        if result:
            print(f"{col}={value}: AUC={result['roc_auc']:.4f} (n={result['n']})")

When you find a weak slice: check if it has enough training examples, whether it has different feature distributions, and whether the label quality is lower for that segment.

Step 3: Feature Importance Analysis

Tree feature importance (mean decrease in impurity) is a fast starting point but biased. Use permutation importance or SHAP for reliable attribution.

pythonCopy
from sklearn.inspection import permutation_importance

result = permutation_importance(
    model, X_val, y_val, n_repeats=20, scoring='roc_auc',
    n_jobs=-1, random_state=42
)

imp_df = pd.DataFrame({
    'feature': X_val.columns,
    'mean': result.importances_mean,
    'std': result.importances_std,
}).sort_values('mean', ascending=False)

print(imp_df.head(15).to_string())

# Any feature with negative permutation importance is actively hurting the model
harmful = imp_df[imp_df['mean'] < -0.005]
print(f"\nPotentially harmful features: {harmful['feature'].tolist()}")

Interpreting results:

• Top features with high importance: confirm these make intuitive sense. If an unexpected feature ranks first, investigate - it may be a proxy for the label (leakage) or a data quality artifact.
• Features with zero importance: they contribute nothing. Remove them to simplify the model.
• Features with negative importance: shuffling them improves performance, meaning they're adding noise.

Step 4: SHAP for Individual Prediction Explanations

SHAP (SHapley Additive exPlanations) decomposes each prediction into per-feature contributions, additive to the model output.

pythonCopy
import shap

# TreeExplainer is fast for tree-based models
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_val)

# Global summary plot: feature importance + direction of effect
shap.summary_plot(shap_values, X_val, plot_type='bar', max_display=15)

# Detailed summary: direction and magnitude
shap.summary_plot(shap_values, X_val, max_display=15)

# Single prediction explanation
idx = fn_examples.index[0]  # explain a false negative
shap.force_plot(
    explainer.expected_value, shap_values[idx], X_val.loc[idx]
)

SHAP is especially useful for debugging a specific misprediction. It tells you exactly which features pushed the model toward or away from the positive class for that example.

Step 5: Learning Curve Diagnosis

Learning curves reveal whether the problem is overfitting or underfitting - which determines whether you need more data, more regularization, or a more powerful model.

pythonCopy
from sklearn.model_selection import learning_curve
import matplotlib.pyplot as plt

train_sizes, train_scores, val_scores = learning_curve(
    model, X_trainval, y_trainval,
    train_sizes=np.linspace(0.1, 1.0, 10),
    cv=5, scoring='roc_auc', n_jobs=-1
)

plt.figure(figsize=(8, 5))
plt.plot(train_sizes, train_scores.mean(axis=1), label='Train AUC')
plt.plot(train_sizes, val_scores.mean(axis=1), label='Val AUC')
plt.fill_between(train_sizes,
    val_scores.mean(axis=1) - val_scores.std(axis=1),
    val_scores.mean(axis=1) + val_scores.std(axis=1), alpha=0.2)
plt.xlabel('Training set size')
plt.ylabel('AUC')
plt.legend()

Large gap, val curve still rising: overfitting. Add regularization (reduce max_depth, increase min_child_samples, add L2/L1 penalty) or get more data.

Both curves low and converged: underfitting. Add features, use a more expressive model, reduce regularization.

Both curves high and converged: you've found the ceiling. More data won't help. Focus on feature engineering or architectural changes.

Step 6: Residual Analysis for Regression Models

pythonCopy
residuals = y_val - y_pred

# Plot residuals vs. predicted values - should be random scatter
plt.scatter(y_pred, residuals, alpha=0.3)
plt.axhline(0, color='red', linestyle='--')
plt.xlabel('Predicted value')
plt.ylabel('Residual')

# Check for systematic errors in specific feature ranges
for col in numeric_features:
    corr = np.corrcoef(X_val[col], residuals)[0, 1]
    if abs(corr) > 0.1:
        print(f"Residual correlated with {col}: r={corr:.3f}  -  model is missing a signal")

Non-random residuals mean your model is systematically wrong in a predictable direction - there is a signal in the data that your current model cannot capture. This guides your next feature engineering iteration.

Step 7: Calibration Check

If your model's probabilities are used for decisions (pricing, risk scoring, recommendation), miscalibration is a silent bug.

pythonCopy
from sklearn.calibration import calibration_curve, CalibratedClassifierCV

# Check calibration
prob_true, prob_pred = calibration_curve(y_val, y_prob, n_bins=10)

for pt, pp in zip(prob_true, prob_pred):
    print(f"Predicted {pp:.2f} → Actual positive rate {pt:.2f}  (diff: {pt-pp:+.3f})")

# Fix calibration with isotonic regression
calibrated = CalibratedClassifierCV(model, method='isotonic', cv='prefit')
calibrated.fit(X_cal, y_cal)  # use a dedicated calibration set, not the training set

Debugging Checklist

Before concluding your model has a capacity problem, go through this list:

□ Read 50 false positives + 50 false negatives manually
□ Run sliced evaluation on all major categorical features
□ Verify no temporal leakage (features derived from future data)
□ Check label quality - are there mislabeled examples?
□ Compute permutation importance - are any features harmful?
□ Plot learning curves - is the problem overfitting or underfitting?
□ Check calibration if probabilities are used for decisions
□ Compare feature distributions between train and val - distribution shift?
□ Inspect residuals (regression) for systematic patterns

Common Mistakes and Bad Instincts

Jumping to model complexity before fixing data. A common pattern: model performs poorly → try neural net → still performs poorly → spend a week tuning → find the problem was mislabeled training examples. Data debugging should come first.

Trusting aggregate metrics on imbalanced data. AUC=0.85 with a 1% positive rate might mean the model has no ability to distinguish positives. Always check the PR curve and look at actual example predictions.

Not looking at your validation examples. Aggregate metrics hide everything. Manually reading 50 examples you got wrong is the highest-ROI debugging action you can take.

Attributing poor performance to model without checking features. A feature that is correlated with the label but computed from future data will give you an incredibly high offline metric that disappears in production. SHAP + data timestamps can catch this.

Fixing the metric without understanding the root cause. Adding regularization because AUC improved, without understanding why, makes the next failure harder to debug. Document your hypotheses and what you found.

Where to Go Next

• Module 13 (Evaluation Metrics) covers choosing the right metric to measure what you're debugging toward.
• Module 15 (Unsupervised Learning) introduces techniques (PCA, clustering) useful for understanding structure in your errors.
• The post model-debugging-and-iteration in the applied track covers production debugging - when your model regresses after a training data update.