ML for Software Engineers: Mental Model Reset

The Instincts That Will Mislead You

Software engineers entering ML bring powerful habits: clean abstractions, deterministic tests, reproducible deployments. Many of these habits transfer directly. But some core instincts actively mislead you in ML systems, and the engineers who struggle most in this transition are usually the ones who do not notice the mismatches early.

This module maps the specific differences between software engineering and ML engineering - so you can decide when to apply your existing habits and when to override them.

Deterministic vs. Probabilistic Systems

Software systems produce the same output for the same input. If a function is wrong, it is consistently wrong - testable, debuggable, fixable.

ML systems are different in a fundamental way: the behavior of the system is a function of the training data, not just the code. Two identical training scripts with different random seeds produce different models with different behaviors. A model that performs well today may perform worse in six months because user behavior changed - without any change to the code.

pythonCopy
# In software: same input → same output, always
def calculate_discount(price: float, tier: str) -> float:
    rates = {"gold": 0.20, "silver": 0.10, "basic": 0.05}
    return price * rates.get(tier, 0)

# In ML: same input at different times may give different outputs
# because the model may have been retrained, or drift may have occurred
def predict_churn_risk(user_features: dict) -> float:
    return model.predict_proba([user_features])[0, 1]
# This output depends on: when the model was trained, what data it saw,
# what random seed was used, and the current feature distribution

The implication: your testing strategy must change. You cannot unit-test a model's outputs as fixed values. You test properties: is the output in [0,1]? Does it decrease when tenure increases? Does it degrade gracefully on missing input?

The Training - Serving Gap

In software, you write code and deploy it. Code runs the same in development and production.

In ML, you have two separate processes: training (where the model learns) and serving (where it makes predictions). The gap between them is where most ML production failures originate.

	Software	ML
Bug source	Code logic	Data, labels, or distribution shift
Failure mode	Exception / wrong output	Silent degradation
Test validity	Unit tests are reliable	Eval on held-out data is necessary but not sufficient
Deployment	Code is the artifact	Model + preprocessing pipeline + code are the artifact
"It worked before"	Means it still works	Model trained 6 months ago may not work today

The preprocessing pipeline must be identical between training and serving. Fitting a StandardScaler on the training set and not saving it means the scaler in production will have different statistics than the scaler used in training - a silent, hard-to-diagnose bug.

pythonCopy
import joblib
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Train: fit the pipeline, save it
pipeline = Pipeline([("scaler", StandardScaler()), ("model", XGBClassifier())])
pipeline.fit(X_train, y_train)
joblib.dump(pipeline, "models/pipeline_v3.pkl")   # Save the entire fitted pipeline

# Serve: load the same fitted pipeline
pipeline = joblib.load("models/pipeline_v3.pkl")   # Includes trained scaler
prediction = pipeline.predict_proba([features])[0, 1]

Evaluation Is Not Testing

In software, tests pass or fail. In ML, evaluation produces metrics that exist on a spectrum. The question is not "does it work?" but "how well does it work, on what data, for which subgroups, and compared to what baseline?"

Strong ML engineers bring software-style rigor to ML evaluation:

• Define the metric before training, based on the business cost of each error type
• Use a held-out test set that was not touched during training or tuning
• Report performance on subgroups, not just overall
• Compare to a baseline - random, majority-class, or rule-based

pythonCopy
# Software-style: binary pass/fail
assert discount == expected_discount

# ML-style: metric-based evaluation with explicit baseline and subgroup analysis
results = {
    "overall_auc": roc_auc_score(y_val, y_pred_prob),
    "baseline_auc": roc_auc_score(y_val, [y_val.mean()] * len(y_val)),  # majority class
    "mobile_auc": roc_auc_score(y_val[mobile_idx], y_pred_prob[mobile_idx]),
    "desktop_auc": roc_auc_score(y_val[desktop_idx], y_pred_prob[desktop_idx]),
}

Data Dependence: The Instinct That Needs Updating

Software engineers are trained to distrust external state. Pure functions are preferable; global state is a bug. In ML, data is the external state - and you cannot avoid it. You must learn to work with messy, biased, shifting data as a core engineering discipline.

The instinct to clean up "impure" functions that reference external data leads to over-abstraction in ML. The instinct to write lots of tests before implementing leads to testing before you understand the data well enough to write meaningful tests.

The better sequence for ML:

1. Understand the data (EDA)
2. Build a minimal working pipeline (data → model → evaluation)
3. Add structure and tests as the pipeline stabilizes

Distribution Shift: The Invisible Production Failure

A model trained on January - March data, deployed in June, serves requests from a June population that may be meaningfully different from the training population. This is distribution shift - one of the most common causes of ML production failures, and one that has no direct analogue in software.

pythonCopy
# Detecting distribution shift in production
from scipy.stats import ks_2samp

def check_feature_drift(
    reference_data: pd.Series,
    live_data: pd.Series,
    threshold: float = 0.05,
) -> dict:
    statistic, p_value = ks_2samp(reference_data.dropna(), live_data.dropna())
    return {
        "feature": reference_data.name,
        "drift_detected": p_value < threshold,
        "ks_statistic": statistic,
        "p_value": p_value,
    }

Software engineers' instinct: monitoring means checking error rates and latency. ML engineers must additionally monitor data distributions and model output distributions - because the system can be "healthy" (no exceptions, normal latency) while producing degraded predictions.

What Transfers Directly

Many software engineering habits improve ML work significantly:

Version control - Git everything: code, configs, requirements, notebooks. Avoid committing large data files, but version everything else.

Code organization - Single-responsibility functions, clear module boundaries, and type hints improve ML code just as much as software code.

CI/CD discipline - ML pipelines benefit from automated testing, linting, and deployment workflows. The pipeline is just a different kind of service.

Observability - Logging, metrics, and alerts are equally important in ML - you just need additional ML-specific signals (drift, prediction distribution, model performance).

Incremental delivery - A simple baseline model deployed quickly provides more value than a complex model that takes six months to ship. Software engineers' bias toward shipping early is an asset in ML.

Where to Go Next

Module 2 (Python for Experienced Engineers) moves quickly through Python tooling and packaging - compressing what the College path spends two modules on, since you already know how to write and organize code. The focus is on the ML-specific Python patterns that differ from backend or systems development.