Evaluation, Experimentation, and Decision Metrics

Engineers from software backgrounds are used to tests that are binary: pass or fail. ML evaluation is a spectrum - models are better or worse, not correct or incorrect. This module covers how to define success before you build, choose metrics that reflect what actually matters, and design experiments that produce trustworthy conclusions.

Defining Success Before You Build

The most common ML project failure mode is not technical - it is building the wrong thing. Define the success criteria before you start, and get explicit agreement from stakeholders:

pythonCopy
# Document your evaluation plan before training
evaluation_plan = {
    "primary_metric": "PR-AUC",
    "rationale": "Dataset is 2% positive rate - ROC-AUC doesn't penalize the trivial 'predict all negatives' classifier enough",
    "minimum_threshold": 0.35,  # PR-AUC baseline for random classifier = positive_rate = 0.02
    "comparison_baseline": "current_rule_based_system",
    "test_set_size": 5000,
    "test_set_period": "2024-Q4",
    "subgroup_evals": ["mobile_users", "new_users_30d", "enterprise_accounts"],
    "online_metric": "14_day_retention_rate",
    "offline_online_correlation": "to_be_measured_in_A/B_test"
}

Getting explicit alignment on these decisions prevents the post-hoc "but AUC doesn't matter for this problem" argument.

Choosing the Right Metric for Your Problem

Problem	Bad Default	Better Choice	Why
Imbalanced classification (1% positive)	Accuracy	PR-AUC	Accuracy of 99% is trivially achieved
Fraud detection	[email protected]	Recall at fixed precision	FN (missed fraud) is far worse than FP
Recommender system	MSE on ratings	NDCG@10	Rank order of top items matters more than rating accuracy
Regression with outliers	RMSE	MAE	RMSE heavily penalizes the few outliers you can't improve
LLM feature	Human rating	Automated + human calibration	Human ratings don't scale; auto ratings need validation

Experimental Design: A/B Testing Fundamentals

pythonCopy
from scipy import stats
import numpy as np

def analyze_ab_test(
    control_outcomes: np.ndarray,
    treatment_outcomes: np.ndarray,
    alpha: float = 0.05
) -> dict:
    """
    Analyze a binary outcome A/B test.
    Returns whether the treatment shows a statistically significant improvement.
    """
    n_control = len(control_outcomes)
    n_treatment = len(treatment_outcomes)
    p_control = control_outcomes.mean()
    p_treatment = treatment_outcomes.mean()

    # Two-proportion z-test
    p_pooled = (control_outcomes.sum() + treatment_outcomes.sum()) / (n_control + n_treatment)
    se = np.sqrt(p_pooled * (1 - p_pooled) * (1/n_control + 1/n_treatment))
    z_score = (p_treatment - p_control) / se
    p_value = 2 * (1 - stats.norm.cdf(abs(z_score)))  # two-tailed

    # Minimum detectable effect (for power calculation)
    lift_absolute = p_treatment - p_control
    lift_relative = lift_absolute / p_control if p_control > 0 else float('inf')

    return {
        "control_rate": p_control,
        "treatment_rate": p_treatment,
        "absolute_lift": lift_absolute,
        "relative_lift_pct": lift_relative * 100,
        "p_value": p_value,
        "significant": p_value < alpha,
        "z_score": z_score,
        "n_control": n_control,
        "n_treatment": n_treatment,
    }

# Usage after running a 2-week A/B test
result = analyze_ab_test(
    control_outcomes=np.array([0, 1, 0, 0, 1, ...]),   # did user convert?
    treatment_outcomes=np.array([1, 0, 1, 0, 1, ...]),
)
print(result)

Sample Size Estimation: How Long to Run the Test?

pythonCopy
from statsmodels.stats.power import NormalIndPower

def required_sample_size(
    baseline_rate: float,
    minimum_detectable_effect: float,
    alpha: float = 0.05,
    power: float = 0.80
) -> int:
    """
    Calculate required sample size per variant.
    """
    analysis = NormalIndPower()
    effect_size = minimum_detectable_effect / np.sqrt(
        baseline_rate * (1 - baseline_rate)
    )
    n = analysis.solve_power(
        effect_size=effect_size, alpha=alpha, power=power, alternative='two-sided'
    )
    return int(np.ceil(n))

# Example: 5% conversion baseline, want to detect a 10% relative lift (0.5pp absolute)
n = required_sample_size(baseline_rate=0.05, minimum_detectable_effect=0.005)
print(f"Need {n} users per variant")
# → ~15,000 per variant

Run the test for the full duration implied by this calculation. Peeking at results early and stopping when they look good inflates Type I error rate.

The Offline-Online Gap

Offline metrics and online metrics frequently disagree. Common causes:

• Feedback loops: your model recommendation influences what data you collect, which biases the next training set.
• Logging bias: only logged outcomes (clicks, conversions) can become labels - but users who didn't see the item had no chance to engage with it.
• Distribution shift: the traffic distribution in the A/B test differs from the test set distribution.

Always calibrate your offline metric against historical A/B tests: "when our offline AUC improved by X, our online conversion rate improved by Y." This lets you predict whether an offline improvement is worth shipping.

Common Mistakes and Bad Instincts

Running the A/B test until it looks significant. This is p-hacking. Decide sample size before the test, run to that sample size, then evaluate once. Sequential testing methods (SPRT) are valid alternatives but must be planned in advance.

Not separating evaluation from training data. Evaluation data must be temporally after training data for time-series problems. Re-using training data for evaluation is data snooping.

Comparing models with different test sets. If model A was tested on Q1 data and model B on Q3 data, the comparison is not valid. Always compare on identical held-out data.

Where to Go Next

• Module 8 (ML Debugging) covers what to do when your metric reveals a problem - how to diagnose which component is failing.
• Module 9 (sklearn Pipelines and Reproducibility) covers how to structure experiments so they are reproducible and comparable.