Math for ML: The Cheat Sheet Every Practitioner Needs

You do not need a mathematics PhD to do ML. But you do need to be fluent in a specific subset of math: the part that describes data, models, and learning. This reference covers that subset precisely - not exhaustively, but practically. Every formula here shows up in real ML work.

Linear Algebra

Vectors

A vector is an ordered list of numbers. In ML, a data point, an embedding, or a gradient is a vector.

x = [x₁, x₂, ..., xₙ]   -  a column vector in Rⁿ

Dot product: measures how aligned two vectors are.

x · y = x₁y₁ + x₂y₂ + ... + xₙyₙ = Σ xᵢyᵢ

L2 norm (Euclidean length):

‖x‖₂ = √(x₁² + x₂² + ... + xₙ²)

Cosine similarity: dot product normalized by lengths - used in embeddings and NLP.

cos(x, y) = (x · y) / (‖x‖₂ · ‖y‖₂)    range: [-1, 1]

Matrices

A matrix is a 2D array of numbers. Weights in a neural network are matrices.

Matrix multiplication - the shape rule: (m × k) · (k × n) = (m × n).

pythonCopy
import numpy as np
A = np.array([[1, 2], [3, 4]])   # shape (2, 2)
B = np.array([[5, 6], [7, 8]])   # shape (2, 2)
C = A @ B                         # shape (2, 2)

Transpose: flip rows and columns. Aᵀ[i,j] = A[j,i].

Identity matrix I: square matrix with 1s on the diagonal, 0s elsewhere. A · I = A.

Matrix inverse A⁻¹: A · A⁻¹ = I. Only square, non-singular matrices have inverses.

Eigenvalues and eigenvectors: Av = λv. Used in PCA and spectral methods.

Calculus

Derivatives

The derivative measures how a function changes with its input.

Function	Derivative
f(x) = xⁿ	f'(x) = nxⁿ⁻¹
f(x) = eˣ	f'(x) = eˣ
f(x) = ln(x)	f'(x) = 1/x
f(x) = sigmoid(x)	f'(x) = σ(x)(1 - σ(x))

Chain rule - the most important rule in backpropagation:

d/dx f(g(x)) = f'(g(x)) · g'(x)

Example: if L = (Wx + b - y)², then dL/dW = 2(Wx + b - y) · x.

Gradients

The gradient of a function f: Rⁿ → R is a vector of partial derivatives:

∇f(x) = [∂f/∂x₁, ∂f/∂x₂, ..., ∂f/∂xₙ]

The gradient points in the direction of steepest ascent. Gradient descent moves opposite to it:

θ ← θ - η · ∇L(θ)

where η (eta) is the learning rate and L is the loss.

Jacobian and Hessian

Jacobian: matrix of partial derivatives for vector-valued functions. Used in batch gradient computation.

Hessian: matrix of second-order partial derivatives. Used in second-order optimizers like L-BFGS.

Probability and Statistics

Core Probability

Probability axioms: 0 ≤ P(A) ≤ 1, P(Ω) = 1, P(A ∪ B) = P(A) + P(B) if A, B disjoint.

Conditional probability: P(A|B) = P(A ∩ B) / P(B).

Bayes' theorem:

P(A|B) = P(B|A) · P(A) / P(B)

Independence: A ⊥ B ↔ P(A ∩ B) = P(A) · P(B).

Expectation and Variance

E[X] = Σ x · P(X = x)              (discrete)
E[X] = ∫ x · f(x) dx               (continuous)

Var(X) = E[(X - E[X])²] = E[X²] - (E[X])²

Std(X) = √Var(X)

Covariance: how two variables move together.

Cov(X, Y) = E[(X - μX)(Y - μY)]

Key Distributions

Distribution	Use in ML
Normal N(μ, σ²)	Weight initialization, noise modeling, Gaussian processes
Bernoulli Bern(p)	Binary classification labels
Categorical	Multi-class labels, softmax outputs
Uniform U(a, b)	Random initialization
Exponential	Time-to-event modeling

Gaussian PDF: f(x) = (1/√(2πσ²)) · exp(-(x-μ)²/(2σ²))

Information Theory

Entropy - average surprise in a distribution:

H(P) = -Σ P(x) log P(x)

Cross-entropy - loss for classification:

H(P, Q) = -Σ P(x) log Q(x)

In practice: L = -Σ yᵢ log(ŷᵢ) where y is true label, ŷ is model probability.

KL divergence - how different Q is from P:

D_KL(P‖Q) = Σ P(x) log(P(x)/Q(x))    ≥ 0, not symmetric

Used in VAEs, policy gradients, and distillation.

Optimization

Gradient descent update rule:

θ_{t+1} = θ_t - η · ∇L(θ_t)

Adam optimizer (what most practitioners use):

m_t = β₁ m_{t-1} + (1-β₁) g_t          # first moment
v_t = β₂ v_{t-1} + (1-β₂) g_t²         # second moment
m̂_t = m_t / (1-β₁ᵗ)                    # bias correction
v̂_t = v_t / (1-β₂ᵗ)
θ_{t+1} = θ_t - η · m̂_t / (√v̂_t + ε)

Defaults: β₁ = 0.9, β₂ = 0.999, η = 1e-3, ε = 1e-8.

Convexity: a function f is convex if f(λx + (1-λ)y) ≤ λf(x) + (1-λ)f(y). Convex functions have global minima - gradient descent is guaranteed to converge. Neural network losses are not convex, but gradient descent still works well in practice.

Saddle points vs. local minima: in high dimensions, most critical points are saddle points, not local minima. This is why neural networks are trainable despite non-convexity.

Common ML Notation

Symbol	Meaning
x	input feature vector
y	true label
ŷ	predicted label
θ, W, b	model parameters (weights, biases)
L, J	loss function
η, α	learning rate
n	number of training examples
d	feature dimension
k	number of classes
T	number of training steps
Σ	sum
∇	gradient operator
∂	partial derivative
∈	element of
Rⁿ	n-dimensional real space

Common Mistakes

Memorizing formulas without understanding when to apply them. Knowing the formula for KL divergence by heart is useless if you do not know that it measures information lost when approximating one distribution with another, and that it is asymmetric. Focus on the intuition behind each formula - what problem does it solve, when is it appropriate, what does a large value mean - before worrying about derivation.

Ignoring numerical stability. In ML implementations, mathematical formulas that are equivalent on paper can differ dramatically in floating-point stability. The naive implementation of log(sum(exp(x_i))) overflows for large x values; the log-sum-exp trick gives the same result stably. Sigmoid computed as 1/(1+exp(-x)) underflows for large negative x; the numerically stable form avoids this. When you implement a formula from scratch, always ask: "does this overflow or underflow at extreme values?"

Confusing sample variance with population variance in ML contexts. Population variance divides by n; sample variance (unbiased estimator) divides by n-1. The distinction matters for small datasets and for understanding what sklearn's StandardScaler does by default (it uses population variance, not sample variance). Misidentifying which formula you are using leads to subtle normalization errors that compound across preprocessing pipelines.

What to Practice Next

• Derive the gradient of MSE loss with respect to the weight vector w from first principles; verify your result matches the gradient computed by PyTorch autograd on the same inputs.
• Implement the log-sum-exp trick in Python and confirm it produces the same result as the naive implementation for small inputs but avoids overflow for inputs in the range [500, 510].
• Compute the gradient of cross-entropy loss with respect to softmax logits by hand; verify numerically that it equals (predicted probabilities - one-hot labels).