Math for ML I: Linear Algebra That Actually Matters

Linear Algebra Without the Textbook

Most linear algebra courses teach abstract vector spaces and proof techniques. This module reverses that order. Every concept here has a direct, recurrent ML application. The goal is not fluency with proofs - it is the geometric and computational intuition that lets you read logits = X @ W.T + b and know exactly what it is doing.

Vectors: Points in Feature Space

A vector is a list of numbers representing a point in space. In ML, almost everything is a vector:

• A training example: [age=25, income=55000, tenure_days=365] is a point in 3D feature space
• A word embedding: a 768-dimensional vector encoding word semantics
• A model weight row: a 512-dimensional vector defining one neuron's learned transformation

pythonCopy
import numpy as np

a = np.array([1.0, 2.0, 3.0])
b = np.array([4.0, 5.0, 6.0])

# Addition: combine two vectors (e.g., residual connections skip a layer)
c = a + b              # [5., 7., 9.]

# Scalar multiplication: scale a direction
scaled = 2.0 * a       # [2., 4., 6.]

# Norms: measure magnitude
l2 = np.linalg.norm(a)            # sqrt(1² + 2² + 3²) ≈ 3.74
l1 = np.linalg.norm(a, ord=1)     # |1| + |2| + |3| = 6.0
linf = np.linalg.norm(a, ord=np.inf)  # max(|1|, |2|, |3|) = 3.0

L1 vs. L2 norm in practice:

• L2 (Euclidean) penalizes large individual components - pushes weights toward small but nonzero values
• L1 (Manhattan) penalizes total magnitude equally - can push weights exactly to zero, producing sparsity

This is exactly why L1 and L2 regularization have different effects on models.

The Dot Product: The Engine of ML

The dot product of two vectors is their element-wise product summed:

pythonCopy
dot = np.dot(a, b)     # 1×4 + 2×5 + 3×6 = 32
# Equivalent: (a * b).sum()

Geometrically, the dot product measures how much two vectors point in the same direction:

• Maximum when they are parallel (same direction)
• Zero when they are perpendicular (orthogonal - no shared direction)
• Negative when they point in opposite directions

This geometric property drives three critical ML computations:

1. Attention scores: scores = Q @ K.T - "how relevant is key kⱼ to query qᵢ?"
2. Cosine similarity: cos_sim = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)) - measures angle between vectors; used in semantic search and recommendation
3. Linear layer forward pass: output = input @ W.T + b - each output neuron is the dot product of the input with one row of W

pythonCopy
# A linear layer forward pass - what nn.Linear(d_in, d_out) does:
def linear_layer(x: np.ndarray, W: np.ndarray, b: np.ndarray) -> np.ndarray:
    """
    x: input of shape (d_in,)
    W: weight matrix of shape (d_out, d_in)
    b: bias vector of shape (d_out,)
    Returns: output of shape (d_out,)
    """
    return W @ x + b

Matrix Multiplication: Batched Transformations

A matrix is a stack of row vectors. Matrix multiplication transforms a collection of vectors simultaneously - this is the core operation in every neural network forward pass.

pythonCopy
X = np.random.randn(100, 10)    # 100 training examples, 10 features each
W = np.random.randn(5, 10)      # Weight matrix for 5 output neurons

# Transform all 100 examples at once
output = X @ W.T                # Shape: (100, 5)
# output[i, j] = dot product of example i with weight row j

Shape arithmetic - the rule that prevents errors:

• (m, n) @ (n, p) → (m, p)
• Inner dimensions must match; outer dimensions become the result shape

Why the transpose? If W has shape (d_out, d_in), then W.T has shape (d_in, d_out). The multiplication (n, d_in) @ (d_in, d_out) produces (n, d_out) - one output vector per input example.

A common shape error: (100, 10) @ (5, 10) fails because inner dimensions (10 ≠ 5). Fix: X @ W.T makes it (100, 10) @ (10, 5).

Transpose: Swapping Rows and Columns

pythonCopy
A = np.array([[1, 2, 3],
              [4, 5, 6]])  # Shape: (2, 3)

A.T                         # Shape: (3, 2)
# [[1, 4],
#  [2, 5],
#  [3, 6]]

You will use transpose constantly to align dimensions for matrix multiplication. Knowing the rule: (A @ B).T = B.T @ A.T - order reverses on transpose.

Matrix Inverse: When It Exists and When It Does Not

The inverse A⁻¹ of a matrix A satisfies A @ A⁻¹ = I (the identity matrix). This appears in the closed-form solution to linear regression:

β = (X.T @ X)⁻¹ @ X.T @ y

pythonCopy
# Computing the inverse
A_inv = np.linalg.inv(A)      # Fails if A is singular (non-invertible)

# Better for linear systems: solve directly (more numerically stable)
beta = np.linalg.solve(X.T @ X, X.T @ y)

When does the inverse not exist? When the matrix is singular - its determinant is zero. This happens when:

• Two or more features are perfectly correlated (one is a linear combination of others)
• There are more features than training examples

Ridge regression (L2 regularization) adds a small value to the diagonal: (X.T @ X + λI)⁻¹. This makes the matrix invertible regardless of correlation - a key practical reason L2 regularization stabilizes linear models.

Eigenvalues and Eigenvectors: Principal Directions

An eigenvector of matrix A is a vector v that only gets scaled (not rotated) when multiplied by A:

A @ v = λ × v

Where λ is the eigenvalue - the scaling factor.

Why this matters in ML: The covariance matrix of your features has eigenvectors pointing in the directions of maximum variance. The eigenvalues tell you how much variance lies in each direction. This is PCA.

pythonCopy
# Covariance matrix of centered features
X_centered = X - X.mean(axis=0)
cov = (X_centered.T @ X_centered) / (len(X) - 1)

# Eigendecomposition
eigenvalues, eigenvectors = np.linalg.eigh(cov)  # eigh for symmetric matrices

# Sort by eigenvalue descending (most variance first)
idx = np.argsort(eigenvalues)[::-1]
eigenvalues = eigenvalues[idx]
eigenvectors = eigenvectors[:, idx]

# How much variance does each component explain?
explained = eigenvalues / eigenvalues.sum()
cumulative = np.cumsum(explained)
n_for_90pct = int(np.searchsorted(cumulative, 0.90)) + 1
print(f"{n_for_90pct} components explain 90% of variance")

# Project onto the top-k components
k = 2
X_pca = X_centered @ eigenvectors[:, :k]   # Shape: (n_samples, k)

In practice, use sklearn.decomposition.PCA. The manual computation is educational for understanding what sklearn is doing internally.

Singular Value Decomposition: The Universal Factorization

SVD decomposes any matrix M into: M = U Σ V.T

• U: orthogonal matrix of left singular vectors (input directions)
• Σ: diagonal matrix of singular values in descending order (scaling factors)
• V.T: orthogonal matrix of right singular vectors (output directions)

pythonCopy
U, sigma, VT = np.linalg.svd(M, full_matrices=False)
# sigma is a 1D array of singular values, largest first

# Low-rank approximation: keep only top-k singular values
k = 50
M_approx = U[:, :k] @ np.diag(sigma[:k]) @ VT[:k, :]

Where SVD appears in ML:

• Matrix factorization for recommendation: user-item matrix decomposed into user and item embeddings
• Truncated SVD as an alternative to PCA (numerically more stable, works on sparse matrices)
• Condition number (sigma[0] / sigma[-1]) measures how sensitive a linear system is to numerical errors

Norms and Distances: Measuring Similarity

pythonCopy
a = np.array([1.0, 0.0, 0.0])
b = np.array([0.8, 0.6, 0.0])

# L2 (Euclidean) distance
l2_dist = np.linalg.norm(a - b)                     # sqrt((0.2)² + (0.6)²) ≈ 0.632

# Cosine similarity (angle-based, range [-1, 1])
cos_sim = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))  # 0.8

# Cosine distance (for nearest-neighbor search)
cos_dist = 1 - cos_sim                              # 0.2

Cosine similarity measures angle, not magnitude - it is robust to differences in vector scale. This is why semantic search uses cosine similarity: a long document and a short document about the same topic should have high similarity regardless of length.

Common Mistakes and Bad Instincts

Confusing matrix multiplication order. A @ B ≠ B @ A in general. Always track shapes explicitly when debugging shape errors - write them as comments next to operations.

Using np.linalg.inv to solve linear systems. Explicitly computing the inverse is numerically unstable (especially when the matrix is nearly singular) and slower. Use np.linalg.solve(A, b) instead.

Not centering data before PCA. PCA assumes zero-mean data. If you skip centering, the first principal component will capture the mean offset, not the direction of maximum variance.

Treating SVD as black-box magic. SVD is the foundation of matrix factorization, low-rank approximations, and numerical stability analysis. Understanding it at the level of "input directions, output directions, scaling factors" makes it useful rather than mysterious.

Where to Go Next

Module 4 covers probability and statistics with the same applied emphasis: not theory for its own sake, but the reasoning tools that make evaluation, experimentation, and model trust defensible. The linear algebra in this module is the substrate - probability and statistics are what you layer on top to interpret what models are actually doing.