Experiment Tracking and Reproducibility Systems

Three months from now, a stakeholder will ask why you switched from Model A to Model B. If you cannot answer that question with a precise comparison - same data, same evaluation protocol, tracked metrics, logged config - you do not have an experiment tracking system. You have a pile of notebooks.

Reproducibility is not bureaucracy. It is the difference between ML engineering and ML tinkering.

What Must Be Tracked in Every Experiment

A reproducible experiment requires tracking five categories of information:

Category	What to Record	How
Code	Git commit hash	git rev-parse HEAD
Data	Hash or DVC ref	pd.util.hash_pandas_object() or dvc run
Config	All hyperparameters	YAML file logged as artifact
Metrics	Train, val, test scores	MLflow log_metric()
Artifacts	Model file, plots, feature importance	MLflow log_artifact()

If you are missing any row in this table, you cannot fully reproduce the run.

MLflow Experiment Tracking Patterns

pythonCopy
import mlflow
import mlflow.sklearn
import subprocess
import hashlib
import yaml
import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.metrics import classification_report

def get_git_hash() -> str:
    try:
        return subprocess.check_output(
            ['git', 'rev-parse', 'HEAD'], stderr=subprocess.DEVNULL
        ).decode().strip()
    except subprocess.CalledProcessError:
        return "unknown"

def get_dataframe_hash(df: pd.DataFrame) -> str:
    return hashlib.md5(
        pd.util.hash_pandas_object(df, index=True).values
    ).hexdigest()[:12]

def run_experiment(config_path: str, data_path: str):
    with open(config_path) as f:
        cfg = yaml.safe_load(f)

    df = pd.read_parquet(data_path)
    X = df.drop(columns=[cfg['target_col']])
    y = df[cfg['target_col']]

    mlflow.set_experiment(cfg.get('experiment_name', 'default'))

    with mlflow.start_run(run_name=cfg.get('run_name', None)):
        # Track provenance
        mlflow.set_tag('git_hash', get_git_hash())
        mlflow.set_tag('data_hash', get_dataframe_hash(df))
        mlflow.set_tag('data_path', data_path)

        # Log all hyperparameters from config
        mlflow.log_params(cfg['model_params'])
        mlflow.log_param('cv_folds', cfg.get('cv_folds', 5))
        mlflow.log_artifact(config_path)

        # Train and evaluate
        model = GradientBoostingClassifier(**cfg['model_params'])
        cv = StratifiedKFold(n_splits=cfg.get('cv_folds', 5),
                             shuffle=True, random_state=42)
        scores = cross_val_score(model, X, y, cv=cv, scoring='roc_auc')

        mlflow.log_metric('roc_auc_mean', scores.mean())
        mlflow.log_metric('roc_auc_std', scores.std())

        # Fit on full training set and log model
        model.fit(X, y)
        mlflow.sklearn.log_model(model, "model",
                                  registered_model_name=cfg.get('model_name'))

        print(f"ROC-AUC: {scores.mean():.4f} ± {scores.std():.4f}")
        return scores.mean()

Now every run is identified by its git hash and data hash. Two runs with the same hash on both should produce the same metrics.

Why Notebooks Fail Reproducibility

Jupyter notebooks are excellent for exploration. They are terrible for reproducible experiments. The problems:

Cell execution order is not guaranteed: You can run cells out of order, accumulate state across multiple executions of the same cell, and produce results that cannot be reproduced by re-running the notebook top-to-bottom.

No clean input/output contract: A notebook blends data loading, transformation, training, evaluation, and visualization in a linear sequence that makes it hard to extract just the training logic.

State persists across kernel restarts: Variables survive even after you think you have "reset" the notebook.

The solution is to write exploration notebooks freely, but extract the final pipeline logic into a Python script before tracking the experiment.

notebooks/
  EDA_user_features.ipynb       # exploration only, not tracked
  model_experiments.ipynb       # early experiments, not reproducible
src/
  train.py                       # canonical training script, version controlled
  features.py                    # feature engineering, version controlled
  evaluate.py                    # evaluation logic
configs/
  experiment_001.yaml
  experiment_002.yaml

Structuring Runs for Comparison

The value of MLflow is not logging a single run. It is comparing 20 runs and knowing exactly what changed between them. To enable this:

pythonCopy
# experiment_001.yaml - baseline
experiment_name: "churn-prediction"
run_name: "gbm-baseline"
target_col: "churned"
cv_folds: 5
model_params:
  n_estimators: 100
  max_depth: 3
  learning_rate: 0.1
  random_state: 42

# experiment_002.yaml - deeper trees
experiment_name: "churn-prediction"
run_name: "gbm-deeper-trees"
target_col: "churned"
cv_folds: 5
model_params:
  n_estimators: 100
  max_depth: 6          # changed
  learning_rate: 0.05   # changed
  random_state: 42

By diffing the config files, you immediately know what changed between runs. The MLflow UI lets you compare metric tables across all runs in an experiment.

Comparing Runs Three Months Later

The scenario: it is March. You ran 40 experiments in December. A production incident shows the model is degrading. You need to know which run is deployed and whether any December run with a different data version might be better.

This is only tractable if:

• Every run is tagged with its git hash and data hash
• Config is logged as an artifact, not just parameters
• Model artifacts are stored (not just metrics)
• Runs are organized in named experiments by problem, not by date

pythonCopy
# Retrieve all runs for an experiment
import mlflow

client = mlflow.tracking.MlflowClient()
experiment = client.get_experiment_by_name("churn-prediction")
runs = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    order_by=["metrics.roc_auc_mean DESC"],
    max_results=10
)

for run in runs:
    print(f"Run: {run.info.run_name}")
    print(f"  ROC-AUC: {run.data.metrics.get('roc_auc_mean', 'N/A'):.4f}")
    print(f"  Git: {run.data.tags.get('git_hash', 'N/A')[:8]}")
    print(f"  Data: {run.data.tags.get('data_hash', 'N/A')}")

Common Mistakes

Logging metrics without logging config: A metric number without its config is useless for reproduction.

Using the same run name for different configurations: If two runs have the same name, you cannot tell them apart in the UI three months later.

Not storing model artifacts: If you only store metrics, you cannot deploy the best run. Always log the model.

One big "experiments" MLflow experiment: Organize experiments by prediction task, not by calendar period.

Where to Go Next

• training-pipelines-experiment-strategy - the pipeline structure that makes tracking work
• evaluation-metrics-error-analysis-systems - what to track beyond a single summary metric
• milestone-gate-1-ml-core-transition - assess whether you are ready to move to production ML

What to Practice Next

• Add MLflow tracking to a training script you already have: log hyperparameters with mlflow.log_params, metrics per epoch with mlflow.log_metric, and the final model artifact with mlflow.sklearn.log_model - then open the UI and compare two runs.
• Create a reproducibility checklist for your project: pin library versions in requirements.txt, seed NumPy and PyTorch random states, and log the git commit hash as an MLflow tag on every run.
• Set up a W&B Sweep for a small hyperparameter search (learning rate × batch size) and compare the parallel-coordinates plot against a manual grid search - note what the visualization reveals that a table does not.