Python for ML: Practical Workflow

The Gap Nobody Tells You About

Python is the dominant language in ML/AI, but most ML tutorials teach you to use Python the wrong way for production work. They start with a Jupyter notebook, stuff everything into cells, and produce code that runs once, on one machine, and cannot be reproduced by anyone else.

This post covers the Python workflow habits that distinguish ML engineers who can be trusted with production systems from those who can only run tutorials.

Virtual Environments: Non-Negotiable From Day One

Every ML project must have its own isolated Python environment. Without this, package versions conflict across projects, your code stops working when you install something new for a different project, and teammates cannot reproduce your results.

Using venv

bashCopy
python -m venv .venv
source .venv/bin/activate      # Mac/Linux
.venv\Scripts\activate         # Windows
pip install -r requirements.txt

Pin your dependencies:

bashCopy
pip freeze > requirements.txt

Or, better, use pyproject.toml with a tool like uv or poetry that produces a lockfile. A lockfile records exact versions, not just ranges, so anyone who installs from it gets identical packages.

The Golden Rule

Never install packages globally for project work. Never. The first time you skip this rule you will spend hours debugging a version conflict that venv would have prevented.

Project Structure: Scripts Over Notebooks

Jupyter notebooks are excellent for exploration. They are terrible for anything that needs to be reproduced, tested, versioned, or run by someone else.

The workflow that works:

1. Use a notebook to explore data, test ideas, and visualize
2. Once an approach works, extract it into a Python module
3. Write a script that calls the module
4. Put the notebook in notebooks/ and never import from it

A Practical ML Project Layout

my_ml_project/
├── pyproject.toml          # Project metadata and dependencies
├── requirements.txt        # Pinned dependencies for CI
├── README.md
├── notebooks/              # Exploration only, never imported
│   └── 01_eda.ipynb
├── src/
│   └── my_ml_project/
│       ├── __init__.py
│       ├── data/
│       │   ├── __init__.py
│       │   ├── loader.py   # Data loading functions
│       │   └── cleaner.py  # Preprocessing functions
│       ├── features/
│       │   └── engineer.py # Feature engineering pipeline
│       ├── models/
│       │   ├── train.py    # Training script entry point
│       │   └── evaluate.py # Evaluation functions
│       └── utils/
│           └── config.py   # Configuration management
├── tests/
│   └── test_cleaner.py
└── data/
    ├── raw/                # Never modify
    └── processed/          # Output of preprocessing

The raw/ data directory is read-only. Preprocessing scripts read from it and write to processed/. This makes data provenance clear and prevents accidental overwrites.

Python Idioms That Matter in ML Work

List Comprehensions and Generator Expressions

pythonCopy
# Slow: building a list with append
squares = []
for x in range(1000):
    squares.append(x ** 2)

# Fast: list comprehension (also more readable)
squares = [x ** 2 for x in range(1000)]

# Memory-efficient: generator (use when you only iterate once)
squares_gen = (x ** 2 for x in range(1000))

In ML, you often process millions of records. Generators avoid loading everything into memory at once.

Type Hints

Type hints make your ML code self-documenting and let tools like mypy catch bugs before runtime:

pythonCopy
import pandas as pd
import numpy as np
from sklearn.base import BaseEstimator

def clean_features(df: pd.DataFrame, target_col: str) -> pd.DataFrame:
    """Drop rows missing the target and apply standard preprocessing."""
    return df.dropna(subset=[target_col]).copy()

def train_model(
    X_train: np.ndarray,
    y_train: np.ndarray,
    model: BaseEstimator,
) -> BaseEstimator:
    return model.fit(X_train, y_train)

When a teammate looks at train_model, they immediately know the expected types without reading the implementation.

Dataclasses for Configuration

Avoid magic dictionaries for experiment configuration. Use dataclasses:

pythonCopy
from dataclasses import dataclass

@dataclass
class TrainConfig:
    model_type: str = "random_forest"
    n_estimators: int = 100
    max_depth: int | None = None
    test_size: float = 0.2
    random_seed: int = 42

This is version-controllable, type-checked, and trivially serializable. Libraries like pydantic and hydra build on this pattern for larger experiment management.

Debugging Without Print Statements

Print-statement debugging does not scale beyond toy scripts. Learn two better tools:

The Python Debugger (pdb)

pythonCopy
def compute_features(df):
    breakpoint()  # Drops into interactive debugger here
    result = df.groupby("user_id")["event"].count()
    return result

Inside the debugger: n steps to next line, c continues, p variable_name prints a value, l lists surrounding code.

Logging Instead of Printing

pythonCopy
import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)

def train(config: TrainConfig) -> None:
    logger.info("Starting training with config: %s", config)
    # ...
    logger.info("Training complete. Val accuracy: %.4f", val_acc)

Logs are filterable by level, timestamped, and go to files - print statements do none of these things.

Reproducibility: The Most Underrated Skill

A model that trains on your machine but produces different results on your teammate's machine is not a working model. Reproducibility requires:

1. Fixed random seeds everywhere

pythonCopy
import random
import numpy as np
import torch

def set_seed(seed: int = 42) -> None:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)

Call this at the start of every training script.

2. Pinned dependencies - use a lockfile, not just requirements.txt with version ranges.

3. Data versioning - raw data should be immutable and referenced by hash or version tag. Tools: DVC, LakeFS.

4. Explicit configuration - never hardcode paths or hyperparameters inline. Put them in a config object that gets logged with each run.

Pandas Patterns That Prevent Leakage

Data leakage (using future information to predict the past) is one of the most common and dangerous bugs in ML. Python Pandas workflows that prevent it:

Always use a pipeline for preprocessing:

pythonCopy
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer

preprocessor = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])

# Fit ONLY on training data
preprocessor.fit(X_train)

# Transform both splits using the training statistics
X_train_processed = preprocessor.transform(X_train)
X_val_processed = preprocessor.transform(X_val)

If you fit_transform on the full dataset before splitting, you have leaked validation statistics into training. Pipelines make the correct approach the default.

Common Mistakes and Bad Instincts

Doing everything in one notebook. The longer a notebook gets, the harder it is to reproduce, test, and debug. Extract logic into modules early.

Not pinning dependencies. pandas>=1.0 installs the latest version today. Six months later, when pandas releases a breaking change, your code silently breaks.

Using global variables for configuration. LEARNING_RATE = 0.01 at the top of a script is invisible to your experiment tracker. Use config objects.

Committing data files to Git. Data files belong in object storage or version-controlled data systems (DVC, S3), not in your code repository. Large files make Git slow and history useless.

Skipping the if __name__ == "__main__": guard. Without it, running python module.py executes everything at import time when another module imports from it. The guard separates script behavior from library behavior.

Where to Go Next

These workflow habits are the prerequisite for everything else. The College Student → ML/AI path covers this in Module 2 (Linux, Git, CLI, and Reproducible Dev Environments) and Module 1 (Python for ML Engineers). The SWE path covers it in Module 2 with a focus on production-aligned patterns. Pick the path that matches your starting point and build these habits before writing a single line of model code.

A Practical Workflow You Can Reuse

A reliable ML workflow has three loops: exploration, training, and serving. Exploration is where you ask questions and discover structure. Training is where you turn those discoveries into repeatable code. Serving is where the model receives real inputs and returns outputs under constraints.

The mistake beginners make is mixing all three in one notebook. That feels fast at first, then becomes impossible to reproduce. A better structure is:

textCopy
project/
  notebooks/
    01_explore_data.ipynb
  src/
    data.py
    features.py
    train.py
    evaluate.py
    predict.py
  tests/
    test_features.py
  configs/
    baseline.yaml

Use notebooks for discovery, but move reusable logic into src/ as soon as it matters. Your future self should be able to run the project without replaying notebook cells in the correct order.

The Boundary Between Notebook and Code

Keep these in notebooks:

• Rough plots
• Hypothesis checks
• Temporary analysis
• Explanatory markdown
• One-off diagnostics

Move these into modules:

• Data loading rules
• Cleaning logic
• Feature transformations
• Train/validation splitting
• Metric computation
• Prediction code

The rule is simple: if the model depends on it, it belongs in versioned code.

Reproducibility Checklist

Before you compare models, capture the conditions that produced each result:

• Git commit hash
• Dataset version or extract date
• Feature code version
• Train/validation/test split logic
• Random seed
• Hyperparameters
• Metric definitions
• Environment dependencies

Without this metadata, "model A beat model B" is not a fact. It is a memory.

Example: Config-Driven Training

Instead of editing constants inside a script, make parameters explicit:

pythonCopy
from dataclasses import dataclass

@dataclass
class TrainConfig:
    data_path: str
    target: str
    test_size: float
    random_state: int
    model_type: str

Now each experiment is a named configuration, not a mystery hidden in code. This also makes reviews easier. A teammate can see what changed between runs without reading the entire pipeline.

Tests That Matter in ML Projects

ML tests are different from normal unit tests. You are rarely testing that a model achieves exactly one score. You are testing that the pipeline has not silently broken.

High-value tests include:

• Schema tests: required columns exist with expected types
• Range tests: ages are non-negative, probabilities are between 0 and 1
• Split tests: no user appears in both train and test when leakage matters
• Feature tests: transformations handle missing values
• Prediction tests: predict() returns the documented shape and type
• Smoke tests: a tiny training run completes in CI

These tests do not guarantee a good model, but they prevent many embarrassing failures.

Production Hand-Off

Every serious ML project needs an inference contract. Write down:

• Input fields and allowed values
• Output format
• Model version
• Preprocessing version
• Expected latency
• Fallback behavior
• Known limitations

If training and inference use different preprocessing, the model you evaluated is not the model users experience. Treat that as a production bug, not a cleanup task.

Closing Thought

The practical standard is not memorization. It is whether you can use the idea to make a better engineering decision, explain that decision to someone else, and notice when reality disagrees with your assumptions.

Quick Self-Assessment

You understand this topic when you can explain the main tradeoff, name the most likely failure mode, and describe how you would test the work before trusting it. Do that in writing. Short written explanations expose vague thinking quickly.

Review Questions

Use these questions to test whether the workflow is real:

1. Can a teammate reproduce your best run without asking you anything?
2. Can you explain which data, code, and config created a model?
3. Can tests catch a broken feature transformation?
4. Can inference run without importing a notebook?
5. Can you compare two experiments honestly?

If the answer is yes, your Python workflow is no longer just code that runs. It is engineering work that another person can trust.

The One-Command Standard

A good ML project should have one obvious way to run each core workflow:

textCopy
make train
make evaluate
make test
make predict-example

This is not cosmetic. It reduces onboarding time, makes CI easier, and forces hidden assumptions into explicit commands. If a project requires a private explanation to run, it is not reproducible yet.

Daily Development Habits

Small habits make ML work much easier to trust:

• Start every run from a clean environment.
• Save raw data separately from processed data.
• Never edit raw data by hand.
• Keep one command for training and one command for evaluation.
• Log metrics to a file, not only the terminal.
• Add a short NOTES.md entry after meaningful experiments.

These habits sound boring until a model regresses and you need to know why. Professional ML work is mostly controlled comparison. Python is the tool, but discipline is the advantage.