Training Pipelines and Experiment Strategy

Here is a scenario that every ML engineer eventually lives through: you ran an experiment two weeks ago that beat your baseline by 4 points. Now you cannot reproduce it. You do not remember the exact preprocessing you used, the config you passed in, or whether you were on the main branch or a throwaway notebook. The result is gone.

Reproducibility is not a nice-to-have. It is the foundation of trustworthy ML work.

The Anatomy of a Reproducible Experiment

A reproducible ML experiment has five tracked components:

1. Code version - the exact git commit hash used for training
2. Data version - a hash or tagged snapshot of the dataset
3. Configuration - all hyperparameters and pipeline settings, serialized
4. Environment - Python version, dependency versions
5. Artifacts - the trained model, evaluation outputs, feature importance plots

If any of these five are missing, you cannot reproduce the result.

Config-Driven Training

The first step is removing magic numbers from your code. Every hyperparameter, path, and flag should come from a config file.

pythonCopy
# config.yaml
model:
  type: random_forest
  n_estimators: 200
  max_depth: 10
  random_state: 42

data:
  train_path: data/train.parquet
  val_path: data/val.parquet
  target_col: label

training:
  cv_folds: 5
  scoring: f1_weighted

pythonCopy
# train.py
import yaml
import hashlib
import json
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import pandas as pd

def load_config(path: str) -> dict:
    with open(path) as f:
        return yaml.safe_load(f)

def get_data_hash(df: pd.DataFrame) -> str:
    return hashlib.md5(pd.util.hash_pandas_object(df).values).hexdigest()

def train(config_path: str):
    cfg = load_config(config_path)
    df = pd.read_parquet(cfg['data']['train_path'])
    data_hash = get_data_hash(df)

    X = df.drop(columns=[cfg['data']['target_col']])
    y = df[cfg['data']['target_col']]

    model = RandomForestClassifier(**cfg['model'])
    scores = cross_val_score(model, X, y, cv=cfg['training']['cv_folds'],
                             scoring=cfg['training']['scoring'])

    return {
        'config': cfg,
        'data_hash': data_hash,
        'cv_mean': scores.mean(),
        'cv_std': scores.std()
    }

Now every experiment is fully described by config.yaml plus the git hash.

Experiment Tracking with MLflow

MLflow gives you a UI and API to log parameters, metrics, and artifacts across runs so you can compare them systematically.

pythonCopy
import mlflow
import mlflow.sklearn
import subprocess

def get_git_hash() -> str:
    return subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode().strip()

def train_with_tracking(config_path: str):
    cfg = load_config(config_path)
    df = pd.read_parquet(cfg['data']['train_path'])

    X = df.drop(columns=[cfg['data']['target_col']])
    y = df[cfg['data']['target_col']]

    mlflow.set_experiment("random-forest-baseline")

    with mlflow.start_run():
        # Log everything
        mlflow.log_params(cfg['model'])
        mlflow.log_param('git_hash', get_git_hash())
        mlflow.log_param('data_hash', get_data_hash(df))
        mlflow.log_artifact(config_path)

        model = RandomForestClassifier(**cfg['model'])
        scores = cross_val_score(model, X, y,
                                 cv=cfg['training']['cv_folds'],
                                 scoring=cfg['training']['scoring'])

        mlflow.log_metric('cv_f1_mean', scores.mean())
        mlflow.log_metric('cv_f1_std', scores.std())

        # Fit on full training data and log model
        model.fit(X, y)
        mlflow.sklearn.log_model(model, "model")

    print(f"CV F1: {scores.mean():.4f} ± {scores.std():.4f}")

Run mlflow ui and navigate to localhost:5000 to compare all your runs in a table.

Dataset Versioning with DVC

DVC (Data Version Control) treats data files like git treats code: you can tag versions, push to remote storage, and check out any historical dataset version.

bashCopy
# Initialize DVC in your repo
dvc init

# Track a dataset file
dvc add data/train.parquet

# Push to remote (S3, GCS, or local)
dvc remote add -d myremote s3://my-ml-bucket/data
dvc push

# In your experiment, record the DVC commit
git add data/train.parquet.dvc .gitignore
git commit -m "experiment: rf baseline v1"

Now each git commit that references a .dvc file pins an exact dataset version.

Exploration vs. Reproducible Experiments

There is a useful distinction between exploration and reproducible experiments:

• Exploration: Jupyter notebooks, throwaway code, eyeballing distributions. Fast, messy, disposable. Use notebooks here.
• Reproducible experiment: Config-driven script, MLflow tracking, git-committed code, DVC-tracked data. Slow to set up once, then very fast to compare and rerun.

The mistake engineers make is treating every experiment as exploration. Once you have a baseline and are comparing alternatives, switch to the reproducible experiment pattern.

Common Mistakes

Not logging the config: Saving only the metric without the config that produced it means you cannot reproduce the run even if you remember the number.

Using notebooks as the canonical training script: Notebook cell execution order is not guaranteed. Extract your training logic into a .py script once you move past exploration.

Treating MLflow as optional: Until you have 10+ runs to compare, tracking feels like overhead. At run 11, you will wish you had started at run 1.

Where to Go Next

• experiment-tracking-reproducibility-systems - deeper patterns for tracking and comparing runs at scale
• leakage-proof-feature-pipelines - make sure what you are training on is correct before tracking it
• milestone-gate-1-ml-core-transition - check your readiness to go beyond ML basics