Math and Statistics for Practical ML Judgment

The Math You Need, Not the Math You Were Taught

This module does not attempt to teach a semester of linear algebra and statistics. It covers the specific concepts that come up in ML design discussions, interviews, and debugging sessions - presented from an engineer's perspective: what does this tell me, when does it matter, and when does the intuition break down.

Linear Algebra: What Models Compute

Everything in a neural network is linear algebra. A forward pass is matrix multiplication. A training step is gradient computation. An embedding lookup is vector retrieval.

The three concepts that explain almost everything:

Dot product - similarity and projection:

pythonCopy
import numpy as np

a = np.array([0.6, 0.8, 0.0])   # Normalized user embedding
b = np.array([0.7, 0.7, 0.1])   # Normalized item embedding

# Cosine similarity: how aligned are they?
cos_sim = np.dot(a, b)           # 0.42 + 0.56 + 0.0 = 0.98 (already normalized)

Attention scores, similarity search, and linear model predictions are all dot products. Understanding this makes systems like vector databases and two-tower recommendation architectures intuitive.

Matrix multiplication - transformation:

pythonCopy
X = np.random.randn(1000, 512)   # 1000 token embeddings, 512 dimensions
W = np.random.randn(512, 256)    # Weight matrix for projection

# Project all tokens at once
projected = X @ W                # Shape: (1000, 256)
# Each row of projected = X[i] @ W - the token projected through W

A nn.Linear(512, 256) layer does exactly this plus a bias term. Understanding this shape arithmetic prevents the dimension-mismatch bugs that dominate early deep learning debugging.

Norms - scale and regularization:

pythonCopy
w = np.array([0.5, -2.3, 0.1, 1.8])

l2_norm = np.linalg.norm(w)             # Sqrt of sum of squares: ≈ 2.97
l1_norm = np.linalg.norm(w, ord=1)     # Sum of absolute values: 4.7

# L2 regularization penalizes large weights but keeps all non-zero
# L1 regularization pushes small weights to exactly zero (feature selection)

These two norms produce different regularization effects because their geometry differs - L1 has corners at the axes where sparse solutions live, L2 has a smooth circular constraint.

Probability: Interpreting Model Outputs

A classifier outputs probabilities - not because ML is magic, but because the model is learning the conditional distribution P(label | features).

Calibration - when probabilities mean what they say:

pythonCopy
from sklearn.calibration import calibration_curve

fraction_positive, mean_predicted = calibration_curve(y_true, y_prob, n_bins=10)
# Perfect calibration: mean_predicted ≈ fraction_positive (diagonal)

A model with AUC 0.90 but poor calibration is fine for ranking (finding the top-k at-risk users) but unreliable for threshold decisions ("flag anyone above 0.7"). The choice of evaluation metric must reflect how the model will actually be used.

Base rates - why precision is low:

pythonCopy
# 1% fraud rate, 95% recall, 5% FPR
p_fraud = 0.01
p_flag_given_fraud = 0.95
p_flag_given_legit = 0.05

p_flag = p_flag_given_fraud * p_fraud + p_flag_given_legit * (1 - p_fraud)
p_fraud_given_flag = (p_flag_given_fraud * p_fraud) / p_flag
print(f"Precision: {p_fraud_given_flag:.3f}")  # ≈ 0.161

A 16% precision on fraud might sound bad, but it is entirely expected given the base rate. The key metric is whether 16% is acceptable cost for 95% fraud capture - which is a business decision, not a model quality judgment.

Confidence intervals - how much to trust an evaluation result:

pythonCopy
def bootstrap_ci(y_true, y_score, n=1000, alpha=0.05):
    from sklearn.metrics import roc_auc_score
    rng = np.random.default_rng(42)
    aucs = [
        roc_auc_score(y_true[idx := rng.integers(0, len(y_true), size=len(y_true))],
                      y_score[idx])
        for _ in range(n)
    ]
    return np.percentile(aucs, [100 * alpha / 2, 100 * (1 - alpha / 2)])

lower, upper = bootstrap_ci(y_test, y_pred_prob)
print(f"AUC 95% CI: [{lower:.4f}, {upper:.4f}]")

A model that beats the baseline by 0.002 AUC is not meaningfully better if the confidence intervals overlap. This framing - "is the improvement outside the noise floor?" - is how good teams make deployment decisions.

Calculus: Gradients and Why Training Fails

You do not need to compute derivatives by hand. You need to understand what the gradient is telling you and why it sometimes goes wrong.

The gradient points uphill - training moves downhill:

w ← w - α × ∇L(w)

The gradient ∇L(w) is the vector of partial derivatives - "if each weight increases slightly, how much does the loss change?" Gradient descent subtracts a fraction of this vector, moving weights toward lower loss.

Three things that go wrong:

pythonCopy
# 1. Learning rate too large: loss oscillates or diverges
# Symptom: loss increases or becomes NaN after a few steps
# Fix: reduce lr by 10×, add gradient clipping

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

# 2. Vanishing gradients: early layers do not learn
# Symptom: loss decreases but model output quality stays poor
# Fix: use ReLU activations, BatchNorm, residual connections

# 3. Exploding gradients: weights become very large
# Symptom: loss becomes NaN
# Fix: gradient clipping (above), lower learning rate, check initialization

Understanding these failure modes means you can diagnose training problems from the loss curve rather than running random hyperparameter searches.

The chain rule as backpropagation:

Each layer's gradient is the product of the layer's local gradient and the gradient flowing in from the next layer. In a 50-layer network, the gradient for layer 1 is the product of 50 local gradients. If any of them is small (sigmoid derivatives top out at 0.25), the product shrinks exponentially. This is why deep networks use ReLU (gradient = 1 when active, no shrinkage) rather than sigmoid in hidden layers.

Statistics: Honest Evaluation

Hypothesis testing in practice:

Use it when comparing models on the same held-out data:

pythonCopy
from scipy import stats

model_a_scores = [0.841, 0.855, 0.838, 0.862, 0.847]  # CV folds
model_b_scores = [0.856, 0.871, 0.853, 0.878, 0.862]

t_stat, p_value = stats.ttest_rel(model_b_scores, model_a_scores)
improvement = np.mean(model_b_scores) - np.mean(model_a_scores)
print(f"Mean improvement: {improvement:.4f}, p={p_value:.4f}")
# Report both - statistical significance and effect size

The multiple comparisons problem:

If you train 20 models and report the best one, some of that gain is noise. The Bonferroni correction: divide your significance threshold by the number of comparisons. More practically: reserve a test set and report performance only once, on the model chosen using validation performance.

Where to Go Next

Module 4 (Data Workflows: NumPy, Pandas, SQL, and Data Quality) applies these mathematical intuitions to real data work: the tools and patterns for turning raw data into reliable model inputs. With the math and tooling from Modules 1–4, you are ready for Module 5's end-to-end supervised learning workflow.