Model Evaluation and Experiment Tracking

Why Evaluation Is the Hardest Part of ML

Training a model is straightforward once you understand the mechanics. Knowing whether the model is actually good - whether it will perform in production, whether it is better than what you have, whether the improvement is real or noise - is significantly harder.

Poor evaluation causes models to pass review and fail in production. It causes teams to deploy marginal improvements at high cost, or to discard genuinely good models because they measured the wrong thing. This post covers the discipline of doing it right.

Metric Selection: Start With the Business Problem

Never pick a metric because it is the default. Pick it because it aligns with how errors will actually cost you.

The Cost of Error Matrix

Before choosing a metric, fill out this matrix:

	Predicted Positive	Predicted Negative
Actually Positive	True Positive (benefit)	False Negative (cost A)
Actually Negative	False Positive (cost B)	True Negative (benefit)

What is cost A? What is cost B? The ratio of these costs determines whether you should optimize for precision, recall, or a specific F-beta score.

Example: In fraud detection, a false negative (missed fraud) costs hundreds of dollars. A false positive (blocked legitimate transaction) costs a fraction of that. You should optimize for recall (catching fraud) even at some cost to precision.

Example: In a job recommendation system, a false positive (showing a bad job) is mildly annoying. A false negative (missing a great job) loses a user. Again, recall matters more - but the balance is different from fraud.

Metric Reference

Metric	Formula	Best when
Accuracy	(TP + TN) / N	Classes are balanced, all errors equally costly
Precision	TP / (TP + FP)	False positives are costly
Recall	TP / (TP + FN)	False negatives are costly
F1	2 × (P × R) / (P + R)	Both matter equally, imbalanced classes
F-beta	(1+β²) × (P × R) / (β²×P + R)	Recall matters β times more than precision
AUC-ROC	Area under ROC curve	Ranking quality, imbalanced binary
PR-AUC	Area under PR curve	Very imbalanced binary, rare positive class
RMSE	√(mean((y-ŷ)²))	Large errors disproportionately costly
MAE	mean(	y-ŷ

Confidence Intervals on Model Performance

A single number on a test set is not a reliable estimate of production performance. It is a sample statistic with uncertainty. You need a confidence interval.

Bootstrap Confidence Intervals

pythonCopy
import numpy as np
from sklearn.metrics import roc_auc_score

def bootstrap_auc(y_true, y_score, n_bootstrap=1000, alpha=0.05):
    aucs = []
    n = len(y_true)
    for _ in range(n_bootstrap):
        idx = np.random.choice(n, size=n, replace=True)
        try:
            auc = roc_auc_score(y_true[idx], y_score[idx])
            aucs.append(auc)
        except ValueError:
            pass  # Skip bootstrap samples with only one class
    lower = np.percentile(aucs, 100 * alpha / 2)
    upper = np.percentile(aucs, 100 * (1 - alpha / 2))
    return np.mean(aucs), lower, upper

mean_auc, lower, upper = bootstrap_auc(y_test, y_pred_prob)
print(f"AUC: {mean_auc:.4f} ({lower:.4f}, {upper:.4f})")

If the confidence interval of model B overlaps with the confidence interval of model A, the improvement is not statistically meaningful - you need more test data or a different model.

Comparing Models: Avoiding Common Statistical Errors

Paired Tests

When comparing two models on the same test set, use a paired test. Each test example produces a prediction from both models. The difference in their errors is the quantity of interest.

The McNemar test is appropriate for binary classification. The Wilcoxon signed-rank test is a non-parametric option for continuous metrics.

Multiple Comparisons

If you train 10 models and report the best one, you are almost certainly reporting an optimistically biased result. This is the multiple comparisons problem (also called p-hacking or data dredging).

Mitigations:

• Report all experiments, not just the winners
• Apply Bonferroni correction to p-values when running many comparisons
• Reserve the test set until the end; make decisions on validation performance

A/B Testing: The Only Way to Know for Sure

Offline evaluation tells you about past data. Online A/B testing tells you about real users in production.

Basic Structure

• Control group (A): Existing model or behavior
• Treatment group (B): New model or feature
• Assignment: Random, often by user ID hash for consistency
• Duration: Long enough to achieve statistical power, typically 1–2 full weekly cycles to capture day-of-week effects

Sample Size Calculation

Before starting an A/B test, calculate the required sample size:

pythonCopy
from scipy.stats import norm

def min_sample_size(baseline_rate, expected_lift, alpha=0.05, power=0.8):
    """Returns minimum sample size per variant."""
    p1 = baseline_rate
    p2 = baseline_rate * (1 + expected_lift)
    pooled_p = (p1 + p2) / 2
    z_alpha = norm.ppf(1 - alpha / 2)
    z_beta = norm.ppf(power)
    n = (z_alpha + z_beta) ** 2 * 2 * pooled_p * (1 - pooled_p) / (p2 - p1) ** 2
    return int(np.ceil(n))

n = min_sample_size(baseline_rate=0.05, expected_lift=0.10)
print(f"Minimum {n} users per variant required")

Starting an A/B test without calculating sample size means you will stop too early (underpowered) or waste time running longer than needed.

Common A/B Testing Mistakes

Peeking: Checking results before the planned end date and stopping early if they look significant. This inflates false positive rates massively. Commit to the end date in advance.

Not randomizing at the right level: If users see both variants across sessions, you contaminate both groups.

Ignoring novelty effects: Users engage with new things simply because they are new. Run the test long enough that novelty effects dissipate.

Experiment Tracking: Making ML Work Reproducible

An ML experiment without tracking is unreproducible. You cannot go back to it, compare it to others, or explain what produced the result.

What to Track for Every Experiment

Category	Examples
Code version	Git commit SHA
Data	Dataset version or hash
Hyperparameters	All tunable parameters
Environment	Python version, package versions
Metrics	Train, val, and test metrics
Artifacts	Model file, feature importance plots
Notes	What changed and why

MLflow: Practical Experiment Tracking

pythonCopy
import mlflow
import mlflow.sklearn

with mlflow.start_run():
    mlflow.log_param("model_type", "xgboost")
    mlflow.log_param("n_estimators", 200)
    mlflow.log_param("max_depth", 6)
    mlflow.log_param("learning_rate", 0.05)

    model = XGBClassifier(n_estimators=200, max_depth=6, learning_rate=0.05)
    model.fit(X_train, y_train)

    val_auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])
    mlflow.log_metric("val_auc", val_auc)
    mlflow.sklearn.log_model(model, "model")

    print(f"Val AUC: {val_auc:.4f}")

Weights & Biases (wandb) is a popular alternative with richer visualization, particularly for deep learning experiments.

Common Mistakes and Bad Instincts

Reporting accuracy on imbalanced data. A model that predicts "not fraud" for every transaction achieves 99.9% accuracy if fraud is 0.1% of data. This is not a good model.

Not separating validation from test sets. Tuning hyperparameters on the test set means your test performance is optimistically biased. Reserve the test set.

Comparing models trained with different random seeds. Training variance can be larger than the difference between models. Average over multiple seeds for fair comparison.

Skipping statistical testing. "Model B had AUC 0.8501 vs 0.8499 for model A, so B wins" is not a defensible conclusion without a significance test.

Not tracking experiments. "I ran a bunch of experiments and this was the best one" is not reproducible engineering. Log everything.

Where to Go Next

Evaluation and experiment tracking are core skills in Module 4 (Evaluation, Metrics, and Experimental Design) of the College Student path and Module 7 of the SWE path. Both modules require producing an evaluation report with confidence intervals, error analysis, and an experiment log for a real ML problem.

Evaluation Starts Before Training

Good evaluation is designed before the model exists. Decide:

• What decision will the model influence?
• What metric matches that decision?
• What slices must not regress?
• What baseline must be beaten?
• What failure cost is acceptable?
• What evidence is needed before deployment?

If you choose metrics after seeing results, you are negotiating with the outcome.

Offline, Online, and Human Evaluation

Different evaluation types answer different questions.

Evaluation type	Question it answers
Offline validation	Did the model generalize on held-out historical data?
Slice analysis	Did it work for important subgroups and edge cases?
Human review	Are the outputs acceptable by domain standards?
Shadow mode	Does it behave safely on production traffic without affecting users?
A/B test	Does it improve real product outcomes?
Monitoring	Does it keep working after launch?

No single metric covers all of this. Production ML requires a chain of evidence.

Experiment Tracking Schema

Track every run with enough detail to reproduce or reject it:

jsonCopy
{
  "run_id": "2026-04-30-baseline-03",
  "git_commit": "abc123",
  "data_snapshot": "s3://bucket/churn/2026-04-01",
  "config": "configs/xgboost_depth4.yaml",
  "metrics": {
    "roc_auc": 0.84,
    "pr_auc": 0.31,
    "precision_at_500": 0.42
  },
  "notes": "Improved enterprise slice, regressed new users"
}

The notes matter. Numbers without interpretation do not create learning.

Avoiding False Progress

ML teams often fool themselves with:

• Reusing the test set repeatedly
• Comparing models trained on different data
• Optimizing a metric that does not match product value
• Ignoring confidence intervals
• Reporting only average performance
• Celebrating tiny gains that are not operationally meaningful

Add a promotion checklist. A model should not advance because it has the highest score in a spreadsheet. It should advance because it passes agreed criteria.

Error Budgets for Model Quality

Software teams use error budgets for reliability. ML teams can use quality budgets. For example:

• False negative rate must stay below 8% for high-value customers
• Median latency must stay below 300 ms
• Weekly drift alert must be reviewed within one business day
• Human-rated answer quality must stay above 4 out of 5

This turns evaluation from a one-time notebook event into an operating system.

What to Put in an Evaluation Report

A strong report includes:

• Problem and decision context
• Dataset and split description
• Baseline comparison
• Primary and guardrail metrics
• Slice results
• Error analysis with examples
• Risks and unknowns
• Deployment recommendation
• Monitoring plan

That report is often more valuable than the model artifact because it captures the reasoning behind the decision.

Quick Self-Assessment

You understand this topic when you can explain the main tradeoff, name the most likely failure mode, and describe how you would test the work before trusting it. Do that in writing. Short written explanations expose vague thinking quickly.

Final Rule

Evaluation is not a score at the end of training. It is the discipline that connects model behavior to product risk. If you cannot explain why a metric matters, you are not ready to optimize it.

Evaluating Data Changes Separately

When a metric changes, separate model changes from data changes. A new model trained on new data cannot be cleanly compared with an old model trained on old data unless you control the experiment.

Useful comparisons:

• Old model on old data
• Old model on new data
• New model on old data
• New model on new data

This matrix shows whether improvement came from architecture, fresher data, label changes, or distribution shift. It is slower than a single leaderboard number, but far more trustworthy.

Statistical Confidence

When two models are close, the difference may be noise. Report uncertainty. For cross-validation, include the mean and standard deviation. For online experiments, use confidence intervals or Bayesian credible intervals. For small datasets, be especially cautious about declaring winners.

Also distinguish statistical significance from practical significance. A huge product can make tiny differences statistically significant. That does not mean the change is worth the engineering cost, added latency, or operational risk.

Guardrail Metrics

Every primary metric needs guardrails. A recommender may optimize clicks while reducing diversity. A fraud model may improve recall while overwhelming investigators. An LLM assistant may improve answer completeness while increasing latency and cost.

Guardrails make tradeoffs visible before launch instead of after damage has happened.