Evaluation, Metrics, and Experimental Design

The most important question in applied ML is not "which algorithm should I use?" It is "how do I know if my model is actually better?" Choosing the wrong metric can make a broken model look good, cause you to optimize in the wrong direction, and result in a system that fails the business objective it was built for.

Why Metric Choice Is a Design Decision

Every metric encodes an assumption about what matters. Accuracy assumes all errors cost the same. F1 treats precision and recall as equally important. AUC measures rank ordering but not calibration. None of these assumptions are true by default - you must select the metric that matches your actual business problem.

A fraud detection model that flags 100% of transactions as fraud achieves 100% recall. A model that flags nothing achieves 100% specificity. Neither is useful. The right metric forces the model to balance both.

Classification Metrics

Confusion matrix components:

• TP (True Positive): predicted positive, actually positive
• FP (False Positive): predicted positive, actually negative (Type I error)
• TN (True Negative): predicted negative, actually negative
• FN (False Negative): predicted negative, actually positive (Type II error)

pythonCopy
from sklearn.metrics import (
    confusion_matrix, classification_report,
    roc_auc_score, average_precision_score, f1_score
)
import matplotlib.pyplot as plt
import seaborn as sns

y_pred = model.predict(X_val)
y_prob = model.predict_proba(X_val)[:, 1]

# Confusion matrix
cm = confusion_matrix(y_val, y_pred)
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
            xticklabels=['Pred Neg', 'Pred Pos'],
            yticklabels=['True Neg', 'True Pos'])

# Full classification report
print(classification_report(y_val, y_pred, digits=4))

# Threshold-independent metrics
print(f"ROC-AUC:  {roc_auc_score(y_val, y_prob):.4f}")
print(f"PR-AUC:   {average_precision_score(y_val, y_prob):.4f}")

Precision: Of all predicted positives, what fraction are actually positive? TP / (TP + FP). Optimize this when false positives are expensive (spam filters - don't delete real email).

Recall (Sensitivity): Of all actual positives, what fraction did you catch? TP / (TP + FN). Optimize this when false negatives are expensive (cancer screening - don't miss a case).

F1 Score: Harmonic mean of precision and recall. Use when you need a single number and neither precision nor recall dominates.

ROC-AUC: The probability that a randomly chosen positive is ranked higher than a randomly chosen negative. Threshold-independent, measures rank order quality. Good default metric for binary classification. Ranges 0.5 (random) to 1.0 (perfect).

PR-AUC (Average Precision): Area under the precision-recall curve. More informative than ROC-AUC when positive class is rare (< 5%). A model that trivially classifies everything as negative can look good in ROC space but poor in PR space.

pythonCopy
# When to use which:
# Balanced classes → ROC-AUC
# Highly imbalanced (fraud, rare disease) → PR-AUC
# Need exact threshold trade-off → plot PR curve and pick operating point
from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_val, y_prob)

# Find threshold that achieves recall >= 0.90 with maximum precision
target_recall = 0.90
idx = np.where(recall >= target_recall)[0][-1]
operating_threshold = thresholds[idx]
print(f"At recall={recall[idx]:.3f}: precision={precision[idx]:.3f}, threshold={operating_threshold:.3f}")

Regression Metrics

pythonCopy
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

y_pred = model.predict(X_val)

mae  = mean_absolute_error(y_val, y_pred)
rmse = mean_squared_error(y_val, y_pred, squared=False)
r2   = r2_score(y_val, y_pred)
mape = np.mean(np.abs((y_val - y_pred) / (y_val + 1e-8))) * 100

print(f"MAE:  {mae:.2f}")    # mean absolute error - in original units, robust to outliers
print(f"RMSE: {rmse:.2f}")   # root mean squared error - penalizes large errors more
print(f"R²:   {r2:.4f}")     # fraction of variance explained - context-dependent
print(f"MAPE: {mape:.2f}%")  # mean absolute percentage error - relative, but breaks near zero

MAE: Easy to interpret in original units. Treats all errors equally. Use when large errors are not catastrophically worse.

RMSE: Penalizes large errors quadratically. Sensitive to outliers. Use when large errors are disproportionately bad (inventory overstock).

R²: Proportion of variance explained relative to a mean-predicting baseline. R² = 0.9 does not mean "90% accurate" - it means your model explains 90% of the variance. Context-dependent: R² = 0.6 might be excellent for stock returns and terrible for predicting manufacturing yield.

MAPE: Useful when relative error matters (e.g., forecasting). Undefined when actuals are zero; breaks near zero. Use MASE (mean absolute scaled error) for time-series forecasting instead.

Multi-Class Metrics

pythonCopy
from sklearn.metrics import f1_score, classification_report

# micro: aggregate TP/FP/FN across all classes - biased toward frequent classes
# macro: unweighted average per class - treats rare and common classes equally
# weighted: weighted by class support - standard for imbalanced multi-class

f1_macro    = f1_score(y_val, y_pred, average='macro')
f1_weighted = f1_score(y_val, y_pred, average='weighted')

Use macro when each class matters equally regardless of support (rare category errors should be caught). Use weighted when frequent classes matter more.

Experimental Design: Measuring Real Improvement

The core question: "is model B actually better than model A, or did we just get lucky on this particular validation fold?"

Statistical Significance in Offline Experiments

pythonCopy
from scipy import stats
import numpy as np
from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)

scores_a = cross_val_score(model_a, X_trainval, y_trainval, cv=cv, scoring='roc_auc')
scores_b = cross_val_score(model_b, X_trainval, y_trainval, cv=cv, scoring='roc_auc')

# Paired t-test: fold-by-fold comparison removes fold-to-fold variation
t_stat, p_value = stats.ttest_rel(scores_b, scores_a)

print(f"Model A AUC: {scores_a.mean():.4f} ± {scores_a.std():.4f}")
print(f"Model B AUC: {scores_b.mean():.4f} ± {scores_b.std():.4f}")
print(f"Delta: {scores_b.mean() - scores_a.mean():.4f}")
print(f"p-value: {p_value:.4f} ({'significant' if p_value < 0.05 else 'not significant'})")

A p-value < 0.05 with 10-fold CV means the difference is unlikely to be due to random fold assignment. A delta of 0.002 AUC may be statistically significant but practically irrelevant - always report both.

Calibration: Do Probabilities Mean What They Say?

A model outputting 0.7 for a prediction should be right 70% of the time. Poor calibration means your probabilities cannot be used for decision-making.

pythonCopy
from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

prob_true, prob_pred = calibration_curve(y_val, y_prob, n_bins=10)

plt.plot(prob_pred, prob_true, marker='o', label='Model')
plt.plot([0, 1], [0, 1], linestyle='--', label='Perfect calibration')
plt.xlabel('Mean predicted probability')
plt.ylabel('Fraction of positives')
plt.legend()

Recalibrate with CalibratedClassifierCV if your model is systematically overconfident (Platt scaling) or underconfident (isotonic regression).

Sliced Evaluation

Aggregate metrics hide failures in subgroups. Always evaluate separately on meaningful slices:

pythonCopy
# Evaluate on subgroups
for segment in ['new_user', 'returning_user', 'mobile', 'desktop']:
    mask = X_val['segment'] == segment
    if mask.sum() < 50:
        continue
    auc = roc_auc_score(y_val[mask], y_prob[mask])
    print(f"{segment}: AUC={auc:.4f} (n={mask.sum()})")

If model B improves aggregate AUC by 0.01 but drops AUC on mobile users by 0.05, that is not an improvement - it is a regression for a subpopulation.

Online Evaluation: The Offline-Online Gap

Offline metrics measure performance on historical data. Online metrics measure performance in production. They often disagree because:

• Distribution shift: production data distribution has drifted from training data.
• Feedback loops: model predictions influence future data.
• Engagement vs. relevance: a recommendation system with high CTR offline may show clickbait in production.

Always A/B test before fully deploying. Measure the metric you actually care about (revenue, user retention, error rate) - not just the model's offline score.

Choosing Your Evaluation Strategy

Scenario	Recommended Metric	Evaluation Strategy
Balanced binary classification	ROC-AUC	5-fold CV + test set
Imbalanced (< 5% positive)	PR-AUC	Stratified 5-fold CV
Multi-class, equal class importance	Macro F1	Stratified CV
Regression, outliers present	MAE	5-fold CV
Regression, large errors critical	RMSE	5-fold CV
Time-series forecasting	MASE	Walk-forward validation
Ranking/retrieval	NDCG@k	Offline eval + A/B test

Common Mistakes and Bad Instincts

Optimizing for accuracy on imbalanced data. A 99/1 split makes accuracy useless. A model predicting all negatives achieves 99% accuracy while catching nothing. Use PR-AUC or F1.

Comparing models on different random splits. If model A was evaluated on one val split and model B on another, the comparison is not valid. Always use the same folds.

Reporting only mean AUC without variance. "AUC = 0.873" without standard deviation is not a reliable evaluation. Use 5- or 10-fold CV and report mean ± std.

Picking the threshold at 0.5 by default. The optimal threshold depends on the cost asymmetry between FP and FN. Plot the precision-recall curve and choose the operating point that matches your business requirement.

Not checking calibration before using probabilities in decisions. Gradient boosting outputs are not well-calibrated by default (they tend to be overconfident at the tails). If you use model.predict_proba() to make dollar decisions, calibrate first.

Where to Go Next

• Module 14 (Model Debugging) covers what to do when your metrics reveal a problem - how to diagnose error types, analyze failure modes, and systematically improve.
• Module 10 (Supervised Learning Foundations) covers the train/val/test split structure these metrics are computed on.
• The post evaluation-metrics-for-machine-learning in the foundations track goes deeper on specific metric computations.