Model Selection Guide: When to Use Which ML Algorithm

The most common mistake beginners make is reaching for the most complex model first. The most common mistake experienced practitioners make is choosing based on familiarity rather than fit. This guide gives you a structured way to think about algorithm selection.

The Decision Framework

Before picking an algorithm, answer four questions:

1. What is your output? (regression, binary classification, multi-class, ranking, generation)
2. How much data do you have? (< 1K, 1K–100K, 100K–10M, > 10M)
3. What are your constraints? (inference latency, memory, interpretability, training time)
4. Do you need to understand why the model makes predictions? (interpretability requirement)

Linear Models

When to use: tabular data, need interpretability, limited data, features have linear relationship with target.

Logistic Regression (classification):

• Strong baseline - always start here
• Fast to train and serve
• Works well when features are already engineered
• Calibrated probabilities out of the box
• Regularization: L1 (sparse features), L2 (correlated features)

Linear/Ridge Regression (regression):

• Same advantages as logistic regression
• Ridge (L2 regularization) handles multicollinearity well
• Lasso (L1) does automatic feature selection

When to move on: when residuals show clear non-linear patterns, or model plateaus well below task ceiling.

Tree-Based Models

Decision Trees

Pure decision trees are rarely the final model - they overfit badly. But they are the foundation for ensemble methods and are useful for quick feature importance checks.

Random Forest

Best for: tabular data with moderate size, mixed feature types, need fast training, some interpretability.

Key properties:

• Ensemble of decorrelated trees (bagging + feature subsampling)
• Naturally handles missing values (with missing_action='zero' in sklearn)
• Provides reliable feature importance estimates
• Robust to hyperparameters - decent performance with defaults

Hyperparameters that matter most: n_estimators (more is better until diminishing returns), max_features (try sqrt(d) for classification, d/3 for regression).

Gradient Boosting (XGBoost, LightGBM, CatBoost)

Best for: tabular data, you care about winning Kaggle competitions, production ML with structured data.

This is the default choice for most tabular ML problems in production:

• Generally outperforms random forest on the same dataset
• Handles categorical features natively (CatBoost, LightGBM with cat_features)
• Fast inference - low-latency serving
• Mature tooling for feature importance, SHAP explanations

Which to choose:

• XGBoost: battle-tested, widest ecosystem, good default
• LightGBM: faster training, better with large datasets, leaf-wise splitting
• CatBoost: best for datasets with many categorical features, minimal preprocessing required

Key hyperparameters: n_estimators + learning_rate (they interact: more trees, lower rate), max_depth (shallow trees - 3–6 - often win), subsample, colsample_bytree.

Support Vector Machines

Best for: high-dimensional sparse data (text), small-to-medium datasets (< 100K), when kernel trick is useful.

• Works well for text classification with TF-IDF features
• Kernel SVM can model non-linear boundaries in lower-dimensional spaces
• Does not scale well to large datasets (training is O(n²) to O(n³))
• Does not output calibrated probabilities by default

Use SVM when: you have < 50K samples, high-dimensional features, and need a strong non-linear baseline before investing in deep learning.

k-Nearest Neighbors

Best for: baselines, recommendation systems with small item sets, anomaly detection.

• Simple: no training, prediction is just a distance lookup
• Requires storing all training data at inference time
• Slow at scale: O(n) per prediction without approximate methods (use FAISS or Annoy for large scale)
• Sensitive to feature scaling - always normalize first

Use case in production: nearest-neighbor retrieval for recommendations and semantic search via approximate nearest neighbor (ANN) libraries.

Neural Networks

When to Use Deep Learning

Move from classical ML to deep learning when:

1. Your data is unstructured (images, audio, text, video)
2. You have > 100K examples (more is better)
3. You have GPU budget for training
4. Classical models have plateaued and you have evidence deep learning can improve

Feed-Forward Networks (MLPs)

For tabular data, MLPs rarely beat gradient boosting. But they work well when:

• You need to share representations across tasks (multi-task learning)
• Your feature space is very large or sparse (ad click prediction)
• You want to jointly learn embeddings and a prediction head

CNNs (Convolutional Neural Networks)

Use for: images, audio spectrograms, any 2D spatial structure.

• Translation invariance via local filters
• Standard architectures: ResNet, EfficientNet, ViT (for large datasets)
• Transfer learning from ImageNet pre-trained models cuts training time dramatically

RNNs / LSTMs

Use for: sequential data where order matters - time series, sensor data.

• LSTMs handle long-range dependencies better than vanilla RNNs
• Largely replaced by Transformers for text tasks
• Still competitive for short-horizon time series with small datasets

Transformers

Use for: text (always), code (always), images with enough data (ViT), audio.

• Dominate all NLP benchmarks
• Self-attention captures long-range dependencies
• Pre-training then fine-tuning is the standard workflow
• Require significant compute and data to train from scratch - use pre-trained models

Quick Selection Table

Data Type	Small Data (< 10K)	Medium (10K–1M)	Large (> 1M)
Tabular	Logistic Reg / RF	XGBoost / LGBM	LGBM / NN
Text	Logistic Reg + TF-IDF	Fine-tune BERT family	Fine-tune large LM
Images	Pre-trained CNN (frozen)	Fine-tune CNN	Train or fine-tune
Time series	ARIMA, LSTM	XGBoost, Prophet	Transformer, N-BEATS
Ranking	LambdaMART	LambdaMART / NN	Two-tower NN

The Process

1. Establish a baseline: simplest possible model (majority class, mean predictor, or logistic regression). Know what "no skill" looks like.
2. Try gradient boosting: if tabular data, XGBoost or LightGBM almost always wins over everything else.
3. Add deep learning: if you have enough data and the problem warrants it.
4. Ensemble: stack or blend models when you need to squeeze out the last few percent.
5. Simplify for production: choose the simplest model that meets your accuracy requirement and satisfies latency/memory constraints.

The best model is the one that ships reliably and meets product requirements - not the one with the highest offline validation score.

What to Practice Next

• Take a problem you are working on and fill out a model selection matrix: list candidate models (at least three), then score each on accuracy ceiling, inference latency, training cost, and interpretability - make the trade-offs explicit before touching code.
• Train a logistic regression and a gradient-boosted tree (XGBoost or LightGBM) on the same dataset and compare validation AUC, inference time per 1000 rows, and feature importance outputs - document when you would prefer the simpler model.
• Find a real production case study (ML engineering blog post, paper, or talk) where a simpler model was chosen over a more complex one for non-accuracy reasons - summarize the deciding factor in one sentence.