Evaluation Metrics and Error Analysis Systems

Accuracy is 94%. Ship it.

This sentence has caused more production failures than almost any other mistake in ML engineering. The choice of evaluation metric is not a technical detail to resolve at the end of training. It is a design decision that encodes what you care about, what you are willing to sacrifice, and what failure looks like for your system.

This article explains when to use which metric and how to build an error analysis system that tells you where your model fails and why.

The Problem With Accuracy

Accuracy is the fraction of correct predictions. It is the right metric when your classes are balanced and the cost of false positives equals the cost of false negatives. That combination is rare in practice.

pythonCopy
import numpy as np
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
                              f1_score, roc_auc_score, average_precision_score,
                              confusion_matrix, classification_report)
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Imbalanced dataset: 5% positive class (fraud, disease, churn)
X, y = make_classification(n_samples=10000, n_features=20,
                            weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

# A naive model that predicts all-negative
y_pred_naive = np.zeros(len(y_test), dtype=int)
print(f"Naive accuracy: {accuracy_score(y_test, y_pred_naive):.4f}")  # 0.95!

# An actual model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)[:, 1]

print(f"Model accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"Model recall:   {recall_score(y_test, y_pred):.4f}")
print(f"Model precision: {precision_score(y_test, y_pred):.4f}")
print(f"Model F1: {f1_score(y_test, y_pred):.4f}")

The naive model gets 95% accuracy. Your real model might get 96%. That 1-point improvement hides the fact that the naive model catches zero fraud cases.

Precision, Recall, and F1: The Triad

Precision: Of the examples the model labeled positive, what fraction actually are? (Measures false alarm rate) Recall: Of all actual positives, what fraction did the model catch? (Measures miss rate) F1: Harmonic mean of precision and recall. Balanced measure when both matter.

pythonCopy
print(classification_report(y_test, y_pred, target_names=['negative', 'positive']))

# When to prioritize each:
# High recall needed: cancer screening, fraud detection (missing a case is costly)
# High precision needed: email spam filter (false positives annoy users)
# F1 balanced: when both errors have similar cost

The precision-recall trade-off is controlled by the classification threshold, not by retraining:

pythonCopy
thresholds = np.arange(0.1, 0.9, 0.1)
for t in thresholds:
    y_t = (y_proba >= t).astype(int)
    if y_t.sum() > 0:  # avoid zero-division
        p = precision_score(y_test, y_t, zero_division=0)
        r = recall_score(y_test, y_t)
        print(f"Threshold {t:.1f}: precision={p:.3f}, recall={r:.3f}")

Always plot the precision-recall curve and choose the threshold that satisfies your business constraint before reporting a single metric.

ROC-AUC vs. PR-AUC

pythonCopy
roc_auc = roc_auc_score(y_test, y_proba)
pr_auc = average_precision_score(y_test, y_proba)

print(f"ROC-AUC: {roc_auc:.4f}")
print(f"PR-AUC:  {pr_auc:.4f}")

ROC-AUC measures the model's ability to rank positives above negatives. It is threshold-independent and handles class imbalance reasonably. Use it when you care about ranking quality.

PR-AUC (Average Precision) focuses on the performance at the positive class. It is more sensitive to class imbalance and better reflects real-world performance when positives are rare. Use it for fraud detection, medical diagnosis, and any task where the positive class is small.

A model with 0.98 ROC-AUC on a 1% positive class dataset can still have 0.15 PR-AUC - meaning it is nearly useless at actually finding positives.

RMSE vs. MAE: Regression Metrics

pythonCopy
from sklearn.metrics import mean_squared_error, mean_absolute_error
import numpy as np

y_true = np.array([100, 200, 150, 300, 50])
y_pred = np.array([110, 190, 160, 500, 55])  # one large outlier (300 → 500)

rmse = np.sqrt(mean_squared_error(y_true, y_pred))
mae = mean_absolute_error(y_true, y_pred)

print(f"RMSE: {rmse:.2f}")  # heavily penalizes the 500 prediction
print(f"MAE:  {mae:.2f}")   # treats all errors equally

# RMSE: use when large errors are disproportionately costly (financial prediction)
# MAE: use when all errors are equally bad (delivery time estimation)
# MAPE: use when you care about relative error (demand forecasting)

Building an Error Analysis System

A single summary metric tells you how good the model is on average. Error analysis tells you where it fails and why. This is where most of the actionable improvement comes from.

pythonCopy
import pandas as pd
from sklearn.inspection import permutation_importance

# Build an error analysis DataFrame
error_df = pd.DataFrame({
    'y_true': y_test,
    'y_pred': y_pred,
    'y_proba': y_proba,
    'error': y_test - y_pred,
})
# Attach original features for sliced analysis
feature_df = pd.DataFrame(X_test, columns=[f'f{i}' for i in range(X_test.shape[1])])
error_df = pd.concat([error_df.reset_index(drop=True), feature_df], axis=1)

# False negatives: model missed actual positives
false_negatives = error_df[(error_df['y_true'] == 1) & (error_df['y_pred'] == 0)]
false_positives = error_df[(error_df['y_true'] == 0) & (error_df['y_pred'] == 1)]

print(f"False negatives: {len(false_negatives)} ({len(false_negatives)/y_test.sum():.1%} of positives missed)")
print(f"False positives: {len(false_positives)}")

# Analyze FN vs TP: what makes false negatives different?
true_positives = error_df[(error_df['y_true'] == 1) & (error_df['y_pred'] == 1)]
print("\nFalse Negative vs True Positive feature means:")
for col in ['f0', 'f1', 'f2']:
    fn_mean = false_negatives[col].mean()
    tp_mean = true_positives[col].mean() if len(true_positives) > 0 else float('nan')
    print(f"  {col}: FN={fn_mean:.3f}, TP={tp_mean:.3f}")

The output tells you: are false negatives clustered in a specific region of feature space? Do they share a common characteristic? That insight drives targeted data collection or feature engineering, not blind hyperparameter tuning.

Common Mistakes

Optimizing the wrong metric: If your business cares about recall (catching all fraudsters), and you tune for accuracy, you are optimizing in the wrong direction.

Reporting AUC without threshold analysis: AUC is great for model selection. But in production you need a fixed threshold. Choose it based on business constraints, not the point that maximizes F1 on your validation set.

Not doing error analysis: Most engineers go from metrics directly to "let me add more features." Error analysis takes 30 minutes and usually reveals a structural problem more important than any single feature.

Using the same metric for model selection and business reporting: You might select models by PR-AUC but report performance to stakeholders as "we catch 87% of fraud." These are different things. Know which is which.

Where to Go Next

• leakage-proof-feature-pipelines - make sure your evaluation metrics are measuring real performance, not leakage artifacts
• applied-stats-engineering-decisions - add statistical significance to your metric comparisons
• experiment-tracking-reproducibility-systems - track metrics across runs so you can compare them rigorously