Unsupervised Learning, Representation, and Dimensionality Reduction

Most real-world data has no labels. You have clickstream logs, transaction records, images, text - but no one has annotated which transactions are anomalies, which users will churn, or which documents are topically similar. Unsupervised learning extracts structure from this unlabeled data. It is also foundational to modern deep learning: embedding models, autoencoders, and contrastive learning are all unsupervised methods that power production recommendation systems, search, and anomaly detection.

Why Unsupervised Learning Matters in Production

Three common production use cases:

1. Clustering for segmentation: Group customers by behavioral patterns without pre-defined segments. The output drives marketing, product, and support prioritization.
2. Dimensionality reduction for visualization and features: Reduce 500-dimensional embeddings to 2D for exploration. Feed compressed representations as features to downstream models.
3. Anomaly detection: Identify transactions, log entries, or sensor readings that don't fit the learned data distribution.

Principal Component Analysis (PCA)

PCA finds the orthogonal directions of maximum variance in your data. It rotates the feature space so that the first principal component (PC1) explains the most variance, PC2 the second most, and so on.

pythonCopy
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import numpy as np

# PCA requires scaled features - variance is the signal
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

pca = PCA()
pca.fit(X_scaled)

# Explained variance ratio: how much variance each PC captures
explained = pca.explained_variance_ratio_
cumulative = np.cumsum(explained)

plt.figure(figsize=(8, 4))
plt.plot(range(1, len(explained) + 1), cumulative, marker='o')
plt.axhline(0.90, linestyle='--', color='red', label='90% threshold')
plt.xlabel('Number of components')
plt.ylabel('Cumulative explained variance')
plt.legend()

n_components_90 = np.argmax(cumulative >= 0.90) + 1
print(f"Components needed for 90% variance: {n_components_90}")

When to use PCA:

• Reduce feature dimensionality before a linear model or distance-based algorithm (KNN, SVM).
• Visualize high-dimensional data in 2D for exploration.
• Remove correlated features (multicollinearity) before regression.
• Speed up training when you have many features (> 500) and some are redundant.

Limitation: PCA is linear. It finds linear combinations of features. Nonlinear structure (e.g., a Swiss roll or a manifold in embedding space) requires nonlinear methods.

pythonCopy
# Project to 2D for visualization
pca_2d = PCA(n_components=2)
X_2d = pca_2d.fit_transform(X_scaled)

plt.scatter(X_2d[:, 0], X_2d[:, 1], c=y, cmap='tab10', alpha=0.5)
plt.xlabel(f'PC1 ({pca_2d.explained_variance_ratio_[0]:.1%} variance)')
plt.ylabel(f'PC2 ({pca_2d.explained_variance_ratio_[1]:.1%} variance)')

t-SNE and UMAP: Nonlinear Visualization

t-SNE (t-distributed Stochastic Neighbor Embedding) and UMAP (Uniform Manifold Approximation and Projection) reduce high-dimensional data to 2D or 3D while preserving local neighborhood structure. They reveal clusters and structure that PCA misses.

pythonCopy
from sklearn.manifold import TSNE
import umap  # pip install umap-learn

# t-SNE: good at revealing clusters, but slow for > 10K points
tsne = TSNE(n_components=2, perplexity=30, random_state=42, n_jobs=-1)
X_tsne = tsne.fit_transform(X_scaled[:5000])  # limit for speed

# UMAP: faster, better at preserving global structure
reducer = umap.UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
X_umap = reducer.fit_transform(X_scaled)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
ax1.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y[:5000], cmap='tab10', alpha=0.5, s=5)
ax1.set_title('t-SNE')
ax2.scatter(X_umap[:, 0], X_umap[:, 1], c=y, cmap='tab10', alpha=0.5, s=5)
ax2.set_title('UMAP')

Critical caveat: t-SNE and UMAP distort distances. Cluster shape and cluster distance between clusters are not meaningful. Only cluster membership (which points are neighbors) is preserved. Do not make quantitative claims about cluster spread from a t-SNE plot.

Use t-SNE/UMAP for: exploratory visualization of embeddings (word vectors, image representations), verifying that class clusters exist before training a classifier, identifying outlier groups.

Do not use for: dimensionality reduction as preprocessing for a downstream model (use PCA instead - UMAP/t-SNE are not invertible and the axes have no interpretable meaning).

K-Means Clustering

K-means partitions data into k clusters by alternating between: (1) assign each point to its nearest centroid, (2) recompute centroids as cluster means. It minimizes within-cluster variance.

pythonCopy
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

# Find optimal k using the elbow method and silhouette score
inertias = []
silhouettes = []
k_range = range(2, 12)

for k in k_range:
    km = KMeans(n_clusters=k, random_state=42, n_init=10)
    labels = km.fit_predict(X_scaled)
    inertias.append(km.inertia_)
    silhouettes.append(silhouette_score(X_scaled, labels, sample_size=5000))

# Plot elbow: look for the "kink"
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.plot(k_range, inertias, marker='o')
ax1.set_xlabel('k'); ax1.set_ylabel('Inertia'); ax1.set_title('Elbow Method')
ax2.plot(k_range, silhouettes, marker='o')
ax2.set_xlabel('k'); ax2.set_ylabel('Silhouette Score'); ax2.set_title('Silhouette (higher=better)')

# Train final model
best_k = k_range[np.argmax(silhouettes)]
km_final = KMeans(n_clusters=best_k, random_state=42, n_init=20)
cluster_labels = km_final.fit_predict(X_scaled)

Profile your clusters to make them actionable:

pythonCopy
df['cluster'] = cluster_labels

# Describe each cluster by its average feature values
cluster_profile = df.groupby('cluster').agg({
    'age': 'mean', 'purchase_frequency': 'mean',
    'avg_order_value': 'mean', 'churn_rate': 'mean',
}).round(2)
print(cluster_profile)

Limitations of K-means:

• Assumes spherical, equal-sized clusters. Fails on elongated or irregularly shaped clusters.
• Sensitive to initialization - use n_init=20 to run multiple random starts.
• Requires specifying k in advance.
• Sensitive to scale - always scale features first.

DBSCAN: Density-Based Clustering

DBSCAN finds clusters as regions of high density separated by low-density regions. It discovers clusters of arbitrary shape and identifies outliers as points that don't belong to any cluster.

pythonCopy
from sklearn.cluster import DBSCAN

# eps: maximum distance between two points to be considered neighbors
# min_samples: minimum points to form a dense region (core point)
dbscan = DBSCAN(eps=0.5, min_samples=10, n_jobs=-1)
labels = dbscan.fit_predict(X_scaled)

n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
n_outliers  = (labels == -1).sum()
print(f"Clusters: {n_clusters}, Outliers: {n_outliers} ({n_outliers/len(labels):.1%})")

# Outlier detection: points labeled -1 are anomalies
outlier_mask = labels == -1
anomalies = df[outlier_mask]

Use DBSCAN for: anomaly detection (the -1 points), geographic clustering (natural boundaries), data with noise where you don't know k in advance.

Limitation: The eps parameter is sensitive and hard to choose. Use a k-nearest-neighbor distance plot to guide selection.

Autoencoders: Learning Compressed Representations

An autoencoder is a neural network trained to reconstruct its input after passing through a bottleneck. The bottleneck layer learns a compressed representation (embedding). The reconstruction error can identify anomalies.

pythonCopy
import torch
import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, input_dim, latent_dim):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 128),
            nn.ReLU(),
            nn.Linear(128, latent_dim)
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 128),
            nn.ReLU(),
            nn.Linear(128, input_dim)
        )

    def forward(self, x):
        z = self.encoder(x)
        return self.decoder(z), z

# Anomaly detection: high reconstruction error = anomaly
model = Autoencoder(input_dim=X_scaled.shape[1], latent_dim=16)
# ... train with MSE loss on normal data only ...

X_tensor = torch.FloatTensor(X_scaled)
recon, embeddings = model(X_tensor)
recon_error = ((X_tensor - recon) ** 2).mean(dim=1).detach().numpy()

threshold = np.percentile(recon_error[y_train == 0], 95)  # 95th percentile of normal
anomaly_pred = (recon_error > threshold).astype(int)

Embeddings as Learned Representations

Modern production systems use dense embeddings - learned via supervised or self-supervised training - as the foundation for recommendation, search, and classification.

pythonCopy
# Word/sentence embeddings from a pretrained model
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer('all-MiniLM-L6-v2')
texts = ["The model failed during inference", "Training loss diverged at epoch 3"]
embeddings = encoder.encode(texts)  # shape: (n_texts, 384)

# Semantic similarity via cosine similarity
from sklearn.metrics.pairwise import cosine_similarity
sim_matrix = cosine_similarity(embeddings)
print(sim_matrix)  # [1.0, 0.72]  -  similar topics

# Store and retrieve with a vector database (e.g., pgvector, Pinecone)
# Index embeddings → query by vector similarity for semantic search

Embeddings differ from classical features in one key way: they are dense (all dimensions non-zero) and carry semantic meaning in the geometry. The dot product between two embeddings measures similarity - this is what attention mechanisms and vector search exploit.

Choosing the Right Unsupervised Method

Goal	Method	Notes
Visualize clusters	UMAP or t-SNE	Not for downstream model input
Reduce features for linear model	PCA	Scales to large datasets
Segment customers	K-means or DBSCAN	Profile clusters after fitting
Detect anomalies	DBSCAN, Isolation Forest, or Autoencoder	Choose based on data type
Generate features for supervised model	PCA, Autoencoder, or pretrained embeddings	Prefer supervised embeddings if available

Common Mistakes and Bad Instincts

Using t-SNE output as features for a downstream model. t-SNE is stochastic, non-deterministic across runs, and the axes have no meaning. Use PCA or an autoencoder for feature compression.

Trusting the elbow method alone for choosing k. The "elbow" is often ambiguous. Always complement with silhouette score and, most importantly, manual inspection of cluster profiles.

Scaling after PCA instead of before. PCA finds directions of maximum variance. Without scaling, features with large units (income in thousands) dominate over features with small units (age). Scale first, every time.

Interpreting DBSCAN outliers as errors without investigation. Outlier points may be genuine anomalies, or may be data entry errors, or may be edge cases that are perfectly valid but sparse. Always sample and read the outliers before acting on them.

Treating unsupervised clusters as ground truth labels. Clusters are hypotheses about structure. They need validation - do these clusters correspond to known segments? Do they predict an outcome you care about? Use them as hypotheses, not facts.

Where to Go Next

• Phase 3 (Modules 16–23) covers neural networks and deep learning, where learned representations and embeddings become the primary modeling paradigm.
• The post embeddings-and-vector-representations in the foundations track goes deeper on embedding geometry and vector search.
• Module 14 (Model Debugging) covers using unsupervised analysis (PCA visualization, cluster analysis of errors) as a debugging technique for supervised models.