Math for ML II: Probability, Statistics, and Uncertainty

Why Probability Is the Language of ML

A supervised model does not predict labels. It estimates probabilities - how likely is a sample to belong to each class, given its features? If you do not understand probability, you cannot interpret what your model is outputting, design appropriate loss functions, or evaluate results honestly.

This module covers the probability concepts that appear in ML every week, at the depth you need to use them in code.

Probability Fundamentals

A probability is a number between 0 and 1 representing the likelihood of an event:

• P(event) ≥ 0 - probabilities are non-negative
• P(entire sample space) = 1 - something always happens
• For mutually exclusive events A and B: P(A or B) = P(A) + P(B)

Conditional probability: P(A | B) - probability of A given that B has occurred.

pythonCopy
# Concrete example: fraud detection
p_fraud = 0.001              # 0.1% of transactions are fraud
p_flagged_given_fraud = 0.95  # model catches 95% of fraud
p_flagged_given_legit = 0.05  # 5% false positive rate

p_legit = 1 - p_fraud
p_flagged = (p_flagged_given_fraud * p_fraud
             + p_flagged_given_legit * p_legit)
print(f"P(flagged) = {p_flagged:.4f}")  # ≈ 0.0509

Bayes' Theorem: The Foundation of ML Reasoning

P(A | B) = P(B | A) × P(A) / P(B)

Given that a transaction was flagged, what is the probability it is actually fraud?

pythonCopy
p_fraud_given_flagged = (p_flagged_given_fraud * p_fraud) / p_flagged
print(f"P(fraud | flagged) = {p_fraud_given_flagged:.4f}")  # ≈ 0.0187

Only ~1.9% of flagged transactions are actual fraud, even though the model has 95% recall. This is the base rate problem: when the positive class is rare, the prior dominates.

This explains why precision is often low on imbalanced datasets despite strong recall - the model is right almost never in absolute terms, even when it is detecting a high fraction of true positives. Every ML classifier is implicitly doing Bayesian reasoning. Understanding the base rates in your data is not optional.

Distributions You Will Encounter Repeatedly

Gaussian (Normal): The bell curve, parameterized by mean μ and standard deviation σ.

pythonCopy
import numpy as np

rng = np.random.default_rng(42)
samples = rng.normal(loc=0.0, scale=1.0, size=10000)

# The 68-95-99.7 rule
print(f"Within 1σ: {(np.abs(samples) < 1).mean():.3f}")   # ≈ 0.683
print(f"Within 2σ: {(np.abs(samples) < 2).mean():.3f}")   # ≈ 0.954
print(f"Within 3σ: {(np.abs(samples) < 3).mean():.3f}")   # ≈ 0.997

Appears in: weight initialization, residuals of linear regression, measurement noise, and - due to the Central Limit Theorem - the distribution of many sample statistics.

Bernoulli: Single binary event with probability p.

• Binary cross-entropy loss is the negative log-likelihood under this distribution
• E[X] = p, Var(X) = p(1-p)

Categorical: Generalizes Bernoulli to k mutually exclusive classes.

• Softmax outputs represent a categorical distribution over k classes
• Categorical cross-entropy (the standard multi-class loss) is its log-likelihood

Poisson: Number of events in a fixed time interval given average rate λ.

• Use for modeling counts: support tickets per hour, page views per session
• E[X] = λ, Var(X) = λ

pythonCopy
from scipy import stats

# Example: a webpage averages 150 visits per hour
lam = 150
dist = stats.poisson(lam)
print(f"P(fewer than 100 visits) = {dist.cdf(100):.4f}")  # probability of unusual quiet hour

Maximum Likelihood Estimation: Why Loss Functions Exist

When we train a model, we are finding parameter values that maximize the probability of observing the training data. This is Maximum Likelihood Estimation (MLE).

For binary classification with Bernoulli labels, the log-likelihood is:

log L(θ) = Σᵢ [ yᵢ log p̂ᵢ + (1-yᵢ) log(1-p̂ᵢ) ]

Maximizing this is equivalent to minimizing binary cross-entropy (multiply by −1). The loss function is not arbitrary - it is the negative log-likelihood under the assumed data distribution.

Distribution assumption	Loss function
Bernoulli labels	Binary cross-entropy
Categorical labels	Categorical cross-entropy
Gaussian noise around predictions	Mean squared error
Laplace noise around predictions	Mean absolute error

When you choose a loss function, you make an implicit assumption about the data distribution. If that assumption is wrong, you are optimizing for the wrong thing.

Expectation and Variance

Expectation E[X]: Probability-weighted average of all possible values. For a dataset, this is the mean.

Variance Var(X) = E[(X − E[X])²]: Average squared deviation from the mean - measures spread.

Standard deviation σ = sqrt(Var(X)): Same units as the original variable; more interpretable than variance.

These connect directly to the bias-variance decomposition of expected test error:

E[test error] = Bias² + Variance + Irreducible noise

• Bias²: Squared difference between the model's expected prediction and the true value (systematic error)
• Variance: How much model predictions vary across different training sets (instability)
• Irreducible noise: Inherent label randomness that no model can remove

pythonCopy
# Demonstrating via simulation
rng = np.random.default_rng(42)
true_fn = lambda x: np.sin(x)
noise = 0.3

# Simulate training many different models on different samples
predictions_at_x0 = []
x0 = 1.0
for _ in range(200):
    x_train = rng.uniform(0, 2 * np.pi, size=30)
    y_train = true_fn(x_train) + rng.normal(0, noise, size=30)
    # (fit a model and predict at x0  -  conceptual)
    predictions_at_x0.append(rng.normal(true_fn(x0), 0.2))  # simulate one model prediction

bias_sq = (np.mean(predictions_at_x0) - true_fn(x0)) ** 2
variance = np.var(predictions_at_x0)
print(f"Bias²: {bias_sq:.4f}, Variance: {variance:.4f}")

The Law of Large Numbers and the Central Limit Theorem

Law of Large Numbers (LLN): As sample size increases, the sample mean converges to the population mean.

This is why evaluation metrics become more reliable with larger test sets, and why a model's performance on 100 test examples is a much noisier estimate than on 10,000 examples.

Central Limit Theorem (CLT): The mean of n independent random variables approaches a Gaussian distribution as n grows, regardless of the original distribution.

pythonCopy
# Demonstrate CLT: uniform distribution, but sample means are Gaussian
n_experiments = 5000
sample_size = 30

sample_means = [
    rng.uniform(0, 1, size=sample_size).mean()
    for _ in range(n_experiments)
]

# Expected: mean ≈ 0.5, std ≈ sqrt(1/12 / sample_size) ≈ 0.048
print(f"Mean: {np.mean(sample_means):.3f}")
print(f"Std: {np.std(sample_means):.3f}")

The CLT explains why confidence intervals, t-tests, and bootstrap methods work: they rely on sample statistics being approximately Gaussian at reasonable sample sizes.

Hypothesis Testing for Model Comparison

When comparing two models, you want to know: is the observed performance difference real, or could it arise from sampling variation?

pythonCopy
from scipy import stats

# 5-fold cross-validation scores for two models
model_a = [0.841, 0.855, 0.838, 0.862, 0.847]
model_b = [0.861, 0.875, 0.855, 0.878, 0.860]

# Paired t-test: each fold is paired between A and B
t_stat, p_value = stats.ttest_rel(model_b, model_a)
print(f"Mean improvement: {np.mean(model_b) - np.mean(model_a):.4f}")
print(f"p-value: {p_value:.4f}")

The paired test is appropriate because both models are evaluated on the same folds - the pairing reduces variance from fold-to-fold differences.

A p-value below 0.05 means: if there were no real difference, we would observe a gap this large less than 5% of the time. It is evidence against the null hypothesis, not proof of a practically meaningful improvement. Always report the effect size (mean improvement) alongside the p-value.

Calibration: Are Probabilities Trustworthy?

A well-calibrated model is one where predicted probabilities match empirical frequencies. If the model assigns 70% probability to outcomes, roughly 70% of those outcomes should actually occur.

pythonCopy
from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

fraction_positive, mean_predicted = calibration_curve(
    y_true, y_prob_scores, n_bins=10
)

# Perfect calibration: fraction_positive ≈ mean_predicted (a diagonal line)
plt.plot(mean_predicted, fraction_positive, marker='o', label='Model')
plt.plot([0, 1], [0, 1], linestyle='--', label='Perfect calibration')
plt.xlabel("Mean predicted probability")
plt.ylabel("Fraction of positives")
plt.legend()

Poor calibration does not affect ranking metrics like AUC-ROC, but it breaks any decision that depends on the raw probability - threshold selection, risk-weighted decisions, and expected-value calculations. Always check calibration before using model probabilities in downstream decisions.

Bootstrap Confidence Intervals

A single test-set score is a point estimate with uncertainty. Bootstrap confidence intervals quantify that uncertainty:

pythonCopy
from sklearn.metrics import roc_auc_score

def bootstrap_auc(y_true, y_score, n_bootstrap=1000, alpha=0.05):
    rng = np.random.default_rng(42)
    n = len(y_true)
    aucs = []
    for _ in range(n_bootstrap):
        idx = rng.integers(0, n, size=n)
        try:
            auc = roc_auc_score(y_true[idx], y_score[idx])
            aucs.append(auc)
        except ValueError:
            pass  # Skip samples with only one class
    lower = np.percentile(aucs, 100 * alpha / 2)
    upper = np.percentile(aucs, 100 * (1 - alpha / 2))
    return float(np.mean(aucs)), lower, upper

mean_auc, lower, upper = bootstrap_auc(y_test, y_pred_prob)
print(f"AUC: {mean_auc:.4f} ({lower:.4f} – {upper:.4f})")

If the confidence interval of model B overlaps with model A's interval, the improvement is not statistically meaningful on this test set.

Common Mistakes and Bad Instincts

Ignoring base rates. The rarity of the positive class is not a technicality - it directly controls what your model is learning and what your metrics mean. A 99% accuracy on a 1% positive-class dataset tells you almost nothing. Check class imbalance before writing any model code.

Confusing p-value with practical significance. A small p-value means the null hypothesis is implausible given the data. It does not mean the effect is large, important, or worth acting on. A 0.001% improvement in accuracy can be statistically significant with enough data and economically meaningless.

Not checking model calibration. A model used for threshold-based decisions (approve/deny, alert/ignore) must be calibrated. An uncalibrated model that reports "95% confidence" when the true rate is 40% will cause systematic decision errors that are hard to trace back to the model.

Treating confidence intervals as probability intervals. A 95% CI does not mean "there is a 95% chance the true value is in this range." It means that if you repeated the experiment 100 times and computed a CI each time, 95 of them would contain the true value.

Where to Go Next

Module 5 completes the mathematical foundation with calculus and optimization - the third pillar needed to understand why models train the way they do, why some optimizers converge faster, and how to diagnose training failures. After Module 5, you have the math needed to follow the internals of any ML algorithm you will encounter in the curriculum.