Scikit-learn Pipelines, Reproducibility, and ML Project Structure

Experienced engineers know that reproducibility is not an optional quality - it is fundamental to trusting your results and shipping maintainable systems. This module covers the three practices that make ML projects reproducible: sklearn Pipelines, config-driven experiments, and clean project structure.

Why Pipelines Are Not Optional

The most common ML reproducibility bug: fitting a scaler or encoder on the full dataset before splitting, then using it on validation. sklearn Pipelines prevent this by ensuring transformers are fit only on training data:

pythonCopy
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OrdinalEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold

numeric_features = ['age', 'income', 'account_age_days', 'log_spend']
categorical_features = ['device_type', 'account_tier', 'region']

numeric_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
])

categorical_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='constant', fill_value='unknown')),
    ('encoder', OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)),
])

preprocessor = ColumnTransformer([
    ('num', numeric_transformer, numeric_features),
    ('cat', categorical_transformer, categorical_features),
])

model = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', GradientBoostingClassifier(n_estimators=200, max_depth=4)),
])

# cross_val_score re-fits the entire pipeline (including preprocessing) inside each fold
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv, scoring='roc_auc')
print(f"CV AUC: {scores.mean():.4f} ± {scores.std():.4f}")

The guarantee: all preprocessing statistics (means, standard deviations, encoder mappings) are derived exclusively from training fold data. Validation folds are transformed using training fold statistics.

Serialization: Saving and Loading Pipelines Correctly

pythonCopy
import joblib

# Save the fitted pipeline
joblib.dump(model, 'models/churn_pipeline_v3.pkl')

# Load and predict in serving
loaded_model = joblib.load('models/churn_pipeline_v3.pkl')
predictions = loaded_model.predict_proba(X_new)[:, 1]

Critical: save the entire pipeline, not just the model weights. The pipeline includes the scaler and encoder. If you save only the GBM weights and separately serialize the scaler, you risk version mismatch when loading.

Config-Driven Experiments

Hardcoded hyperparameters prevent reproducible comparisons. Use config files:

yamlCopy
# configs/experiment_v3.yaml
experiment:
  name: churn_lgbm_v3
  seed: 42
  cv_folds: 5

data:
  train_path: data/processed/train_2024Q4.parquet
  val_path: data/processed/val_2024Q4.parquet
  features:
    numeric: [age, income, account_age_days, log_spend, purchase_frequency]
    categorical: [device_type, account_tier, region]
  label: churned_30d

model:
  type: LGBMClassifier
  params:
    n_estimators: 300
    learning_rate: 0.05
    max_depth: 6
    min_child_samples: 20
    subsample: 0.8
    colsample_bytree: 0.8
    random_state: 42

training:
  early_stopping_rounds: 50
  eval_metric: auc

pythonCopy
import yaml
from dataclasses import dataclass, field
from typing import Any

def load_experiment_config(path: str) -> dict:
    with open(path) as f:
        return yaml.safe_load(f)

def run_experiment(config_path: str):
    config = load_experiment_config(config_path)
    seed = config['experiment']['seed']

    # Load data
    train_df = pd.read_parquet(config['data']['train_path'])
    features = config['data']['features']['numeric'] + config['data']['features']['categorical']
    label = config['data']['label']

    # Log everything
    import mlflow
    mlflow.log_params(config['model']['params'])
    mlflow.log_artifact(config_path)  # save the config that produced this run

    # Train with config params
    model = build_pipeline(config)
    model.fit(train_df[features], train_df[label])

    return model

Every run logs the config file that produced it. You can reproduce any run by replaying its config.

Clean Project Structure

my-ml-project/
├── configs/
│   ├── baseline.yaml
│   └── experiment_v3.yaml
├── data/
│   ├── raw/          # never modified
│   ├── processed/    # pipeline outputs
│   └── .gitignore    # don't commit large files
├── src/
│   ├── __init__.py
│   ├── features.py   # feature engineering functions
│   ├── pipeline.py   # sklearn pipeline construction
│   ├── train.py      # training entrypoint
│   └── evaluate.py   # evaluation and reporting
├── tests/
│   ├── test_features.py
│   ├── test_pipeline.py
│   └── test_model.py
├── notebooks/        # exploration only, not production
├── Makefile
├── requirements.txt  # pinned with pip-compile
└── README.md

makefileCopy
# Makefile
.PHONY: features train evaluate test

features:
	python -m src.features --config configs/baseline.yaml

train: features
	python -m src.train --config configs/baseline.yaml

evaluate:
	python -m src.evaluate --config configs/baseline.yaml

test:
	pytest tests/ -v

make train runs the full pipeline from features through training. A new team member can clone the repo and reproduce results with one command.

Cross-Validation and Serialization Workflow

pythonCopy
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform
import joblib

param_dist = {
    'classifier__n_estimators': randint(100, 500),
    'classifier__max_depth': randint(3, 8),
    'classifier__learning_rate': uniform(0.01, 0.2),
}

search = RandomizedSearchCV(
    model,           # the full Pipeline
    param_dist,
    n_iter=30,
    cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
    scoring='roc_auc',
    n_jobs=-1,
    random_state=42,
    verbose=1,
)
search.fit(X_trainval, y_trainval)

print(f"Best CV AUC: {search.best_score_:.4f}")
print(f"Best params: {search.best_params_}")

# Save the best pipeline
joblib.dump(search.best_estimator_, 'models/churn_pipeline_best.pkl')

Common Mistakes and Bad Instincts

Not pinning dependencies. numpy, lightgbm, and scikit-learn all have breaking changes between minor versions. Use pip-compile or pip freeze > requirements.txt to pin exact versions. A model trained on sklearn 1.3 may produce different predictions when loaded in sklearn 1.4.

Putting business logic in notebooks. Notebooks are for exploration. Feature engineering functions, pipeline constructors, and training loops belong in src/ as importable, testable Python. Extract them as soon as they are more than exploratory.

Random seeds inconsistently applied. Set random_state=42 in every stochastic component: the train/test split, the cross-validation shuffle, the model, and any random search. A missing seed somewhere in the chain means you cannot reproduce your best result.

Not testing the serialization/deserialization cycle. Save the pipeline, load it back, and confirm predictions match before shipping. Serialization bugs are rare but catastrophic when they occur silently.

Where to Go Next

• Module 10 (Neural Networks for Practitioners) begins the deep learning portion of the SWE path.
• The earlier Module 8 (ML Debugging) is where these reproducibility practices pay off: when a model regresses, you need a reliable baseline to diff against.