Classical ML Algorithms and When to Use Them

Most production ML systems - churn prediction, fraud detection, pricing, lead scoring - do not use neural networks. They use gradient-boosted trees or logistic regression, trained in minutes, interpretable by stakeholders, and debuggable when they fail. Knowing which algorithm to reach for first and when each one wins is a core production ML skill.

Decision Trees: The Building Block

A decision tree splits the feature space recursively using binary rules: "is age > 35?" → yes/no → next split. Each leaf node predicts a class (or regression value).

pythonCopy
from sklearn.tree import DecisionTreeClassifier, export_text
import pandas as pd

# Depth-limited tree is interpretable; unlimited tree overfits badly
tree = DecisionTreeClassifier(max_depth=4, min_samples_leaf=20, random_state=42)
tree.fit(X_train, y_train)

# Read the actual rules
print(export_text(tree, feature_names=list(X_train.columns)))

Strengths: Interpretable rules, handles mixed types, no scaling needed, fast to train.

Weaknesses: Single trees overfit easily. Axis-aligned splits are bad at diagonal decision boundaries. Unstable - a small data change can change the entire tree structure.

Use a single tree when: you need to audit the exact decision logic (compliance, healthcare, credit decisions).

The key hyperparameters that prevent overfitting: max_depth (keep ≤ 5 for interpretability), min_samples_leaf (≥ 20 prevents splits on noise), min_samples_split.

Random Forest: Variance Reduction Through Bagging

Random Forest trains n_estimators trees, each on a bootstrap sample of the data and a random subset of features. It predicts by majority vote (classification) or average (regression). The randomness reduces variance while keeping bias low.

pythonCopy
from sklearn.ensemble import RandomForestClassifier
import numpy as np

rf = RandomForestClassifier(
    n_estimators=300,
    max_depth=None,       # let trees grow deep - bagging controls variance
    max_features='sqrt',  # each split considers sqrt(n_features) features
    min_samples_leaf=5,
    n_jobs=-1,
    random_state=42
)
rf.fit(X_train, y_train)

# Feature importance (mean decrease in impurity - fast but biased toward high-cardinality)
importances = pd.Series(rf.feature_importances_, index=X_train.columns)
print(importances.sort_values(ascending=False).head(10))

Strengths: Robust to outliers and noisy features, works well out-of-the-box with little tuning, handles missing values via imputation in pipeline, provides feature importance.

Weaknesses: Slower to predict than a single tree (300 trees × depth), less accurate than gradient boosting on most tabular problems, feature importances are biased toward high-cardinality features.

Use Random Forest when: you want a strong baseline quickly, you have noisy features, or you need robustness without careful hyperparameter tuning.

Gradient Boosting: The Production Workhorse

Gradient boosting builds trees sequentially, each one fitting the residual errors of the previous ensemble. XGBoost and LightGBM are the standard implementations - both are fast, regularized, and handle missing values natively.

pythonCopy
import lightgbm as lgb
from sklearn.metrics import roc_auc_score

model = lgb.LGBMClassifier(
    n_estimators=1000,
    learning_rate=0.05,
    max_depth=6,
    num_leaves=31,          # key LightGBM parameter: controls complexity
    min_child_samples=20,
    subsample=0.8,          # row subsampling (stochastic gradient boosting)
    colsample_bytree=0.8,   # feature subsampling
    reg_alpha=0.1,          # L1 regularization
    reg_lambda=1.0,         # L2 regularization
    random_state=42,
    n_jobs=-1
)

# Use early stopping to find optimal n_estimators automatically
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(50), lgb.log_evaluation(100)]
)

val_auc = roc_auc_score(y_val, model.predict_proba(X_val)[:, 1])
print(f"Val AUC: {val_auc:.4f}, Best iteration: {model.best_iteration_}")

Strengths: Best performance on tabular data in most Kaggle competitions and production benchmarks, handles heterogeneous feature types, built-in regularization, fast with LightGBM.

Weaknesses: More hyperparameters to tune, prone to overfitting without regularization, training is sequential (can't parallelize across trees, only within each tree), less interpretable than a single tree.

Use gradient boosting when: you want maximum predictive performance on tabular data and can afford ~30 minutes of hyperparameter tuning.

Logistic Regression: The Underrated Baseline

Logistic regression models log-odds as a linear function of features: log(p/(1-p)) = wᵀx + b. It is fast, interpretable via coefficients, and well-calibrated out of the box.

pythonCopy
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Logistic regression requires scaled features
lr_pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('lr', LogisticRegression(C=1.0, penalty='l2', solver='lbfgs', max_iter=500))
])
lr_pipe.fit(X_train, y_train)

# Coefficients are directly interpretable (after scaling: relative importance)
coefs = pd.Series(
    lr_pipe.named_steps['lr'].coef_[0],
    index=X_train.columns
).sort_values()
print(coefs.head(5), "\n", coefs.tail(5))

C is the inverse of regularization strength: small C → heavy regularization → fewer non-zero coefficients. Use penalty='l1' for automatic feature selection.

Use logistic regression when: you need a calibrated probability, you need interpretable coefficients for stakeholders, you have a linear problem, or you want a reliable baseline to beat.

SVM: When Margins Matter

Support Vector Machines find the maximum-margin hyperplane separating classes. With the RBF kernel, they implicitly operate in a high-dimensional feature space without computing it explicitly.

pythonCopy
from sklearn.svm import SVC
from sklearn.pipeline import Pipeline

svm_pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('svm', SVC(C=1.0, kernel='rbf', gamma='scale', probability=True))
])
svm_pipe.fit(X_train, y_train)

Use SVM when: you have a small-to-medium dataset (< 100K rows), high-dimensional sparse features (text classification with TF-IDF), or you know the problem has a clear margin.

Avoid SVM when: you have > 100K rows (quadratic training time), you need fast predictions, or you need feature importance.

KNN: The Lazy Learner

K-Nearest Neighbors stores all training examples and classifies new points by the majority class among the k nearest neighbors. It has no training phase but slow prediction and poor performance in high dimensions (curse of dimensionality).

pythonCopy
from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=5, metric='euclidean', n_jobs=-1)
knn.fit(X_train, y_train)

Use KNN when: you have a small dataset, interpretability of "these are the most similar past examples" matters (recommendation, anomaly explanation), or as a baseline for embedding-space retrieval.

Avoid KNN when: you have > 50K rows, high-dimensional features, or need fast inference.

Algorithm Selection Framework

Is your data tabular (rows × features)?
  ├─ Yes, < 1K rows → Logistic Regression or SVM
  ├─ Yes, 1K–1M rows → LightGBM (primary), Random Forest (interpretability fallback)
  └─ No (images, text, sequences) → Neural networks (covered in Phase 3)

Do you need a probability output?
  ├─ Yes, calibration critical → Logistic Regression or calibrated GBM
  └─ No → any classifier

Do you need coefficients/rules for compliance?
  ├─ Yes → Logistic Regression or single Decision Tree
  └─ No → LightGBM

Do you have time for tuning?
  ├─ No → Random Forest (good defaults)
  └─ Yes → LightGBM with RandomizedSearchCV

Comparing Algorithms Fairly

Always compare on the same CV folds. Use the same preprocessing pipeline. Report uncertainty:

pythonCopy
from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
models = {
    'LogReg': lr_pipe,
    'RandomForest': rf,
    'LightGBM': lgb.LGBMClassifier(**lgb_params),
}

for name, m in models.items():
    scores = cross_val_score(m, X_trainval, y_trainval, cv=cv, scoring='roc_auc', n_jobs=-1)
    print(f"{name:15s}  AUC {scores.mean():.4f} ± {scores.std():.4f}")

A difference of 0.003 AUC with ± 0.005 standard deviation means the models are statistically equivalent. Do not over-engineer to gain 0.1% AUC in CV if the simpler model is more maintainable.

Common Mistakes and Bad Instincts

Jumping straight to neural networks. For structured tabular data, a well-tuned LightGBM model almost always beats a neural net and is far easier to debug, serve, and explain. Neural nets are the right call for images, text, and sequences - not for 50-column CSV files.

Skipping the logistic regression baseline. If logistic regression achieves 0.88 AUC and your tuned GBM achieves 0.89, the 0.01 gain may not be worth the operational complexity. Always know what the simple model achieves.

Using feature_importances_ from Random Forest to select features. Mean decrease in impurity is biased toward high-cardinality features. Use permutation_importance or SHAP for reliable importance estimates.

Not using early stopping with gradient boosting. Setting n_estimators=1000 and not using early stopping will overfit. Always use eval_set + early_stopping_rounds.

Forgetting to scale before logistic regression or SVM. Unscaled features make gradient descent slow and regularization unequal across features.

Where to Go Next

• Module 12 (Feature Engineering) covers how to build and transform features that dramatically improve any of these algorithms.
• Module 14 (Model Debugging) covers how to diagnose when these algorithms fail and what to do about it.
• The standalone post regularization-explained goes deeper on L1/L2 regularization across all these model families.