Python for ML Workflow (Deep)

Why This Matters

Most ML portfolios fail in hiring loops because they are not reproducible, not maintainable, and not production-oriented. A strong ML engineer is not just a model trainer; they are a system builder.

This post teaches you how to construct a workflow where another engineer can clone your repo, run one command, and reproduce your results with confidence.

Prerequisites

• Basic Python, virtual environments, and CLI usage
• Familiarity with Git and repository structure
• Intro knowledge of ML training scripts

Learning Outcomes

After this module, check that you can:

• Design a clean ML project architecture with clear boundaries
• Enforce reproducibility with dependency and config discipline
• Track experiments and artifacts for auditable decisions
• Separate exploration code from production pipelines
• Build training/inference interfaces safe for deployment handoff

Core Concepts

1) Project Architecture

• src/ for reusable code
• notebooks/ for exploration, not production logic
• configs/ for parameterized runs
• tests/ for data and feature correctness checks

2) Reproducibility

• Pinned dependencies
• Deterministic seeds
• Immutable data/version references
• Config snapshots for each run

3) Experiment Lineage

• Track code hash, data snapshot, config, metrics, artifacts
• Keep promotion decisions linked to experiment evidence

4) Training/Inference Contract

• Explicit input schema
• Stable preprocessing pipeline
• Versioned model artifact
• Clear runtime assumptions

Mental Models and Tradeoffs

Mental Model: "Notebooks discover; modules deliver"

Notebook insights should move into tested modules quickly. If a pipeline depends on notebook state, it is a deployment risk.

Mental Model: "Every run is an audit event"

If you cannot explain which code+data+config produced a model, you do not have a reliable engineering process.

Tradeoffs

• Speed of exploration vs maintainability
• One-off scripts vs shared pipeline components
• Strict interfaces vs quick prototyping flexibility
• Comprehensive tests vs team velocity

Details

A) Reference structure

textCopy
ml-project/
  data/
    raw/
    processed/
  notebooks/
  src/
    data/
    features/
    models/
    training/
    inference/
  configs/
    train.yaml
    eval.yaml
  tests/
  pyproject.toml
  Makefile

B) Config-driven runs

A config-first setup allows consistent reruns and controlled changes. Avoid hidden hyperparameters in scripts.

C) Data + feature versioning

You do not need heavy tooling initially. Even robust file hashing and manifest snapshots can dramatically improve reliability.

D) Inference safety

Training-time preprocessing must be serialized and reused in inference path. Any divergence creates silent quality failures.

Implementation Walkthrough

1. Scaffold project structure above
2. Create typed config loader and run entrypoint (train.py)
3. Build preprocessing module with fit/transform split
4. Train baseline model from CLI config
5. Save model artifact + metadata manifest
6. Create predict.py using same preprocessing artifact
7. Add integration test to verify training/inference compatibility

Common Failure Modes

• Hardcoded paths and hidden notebook state
• Different preprocessing in train vs infer
• Mutable default configs and accidental overrides
• Missing seed control causing irreproducible benchmarks
• No artifact metadata (who/what/when)

Interview Depth

Be ready to explain:

• How your architecture enables collaboration and handoff
• How you guarantee experiment reproducibility
• How you prevent train/serve skew
• How you would scale this workflow for a 5-engineer ML squad

Hands-On Lab

Lab Task

Build ml-workflow-lab with:

• Config-driven training
• Experiment log JSON per run
• Artifact registry folder with versioned metadata
• Inference API contract test

Required Deliverables

• Repo tree screenshot or tree output
• Two reproducible experiment runs
• One train/serve skew prevention test
• One-page architecture tradeoff note

Milestone Checklist

• Fresh clone reproduces core experiment
• Train and inference share same preprocessing path
• Artifacts are versioned with metadata
• Config changes are explicit and tracked
• Tests catch at least one realistic workflow regression

Next Step in the Path

Use this workflow foundation to build robust feature pipelines in Data Cleaning and Feature Engineering (Deep).

---

Diagram

Workbook

Download:

• /workbooks/ml_workflow_checklist.md

Code Snippets

Minimal config-driven training entrypoint

pythonCopy
import json
from dataclasses import dataclass

@dataclass
class TrainConfig:
    lr: float
    seed: int

def train(cfg: TrainConfig) -> dict:
    dataset = load_dataset(cfg.data_path)
    X, y = build_features(dataset)
    model = build_model(cfg)
    model.fit(X, y)
    score = evaluate(model, X, y)
    return {"lr": cfg.lr, "seed": cfg.seed, "metric": score}

def main():
    cfg = TrainConfig(lr=1e-2, seed=42)
    result = train(cfg)
    with open("runs/latest.json", "w") as f:
        json.dump({"config": cfg.__dict__, "result": result}, f, indent=2)

if __name__ == "__main__":
    main()

Train/serve skew guard test (conceptual)

pythonCopy
# Assert that preprocess() used in training and inference is the same version.
assert training_preprocessor.version == inference_preprocessor.version

Focus questions:

• What files must be snapshotted to reproduce a run?
• What is the minimum evidence packet for promoting a model?
• Where do tests add the most leverage in ML code?