Embeddings: What They Are and Why They Work

The Core Idea

An embedding is a dense, fixed-size vector of numbers that represents a discrete object - a word, a sentence, a user, a product, a code function. The key property: objects that are semantically similar should have similar vectors (as measured by cosine similarity or dot product).

This property emerges from training: embedding models are trained on tasks where the relationship between objects matters. Word2Vec learns word embeddings by predicting surrounding words in text. Sentence transformers learn by contrasting similar and dissimilar sentence pairs.

Why Not Just One-Hot Encode?

Before embeddings, NLP used one-hot encoding: a vector of zeros with a single 1 at the index of the word in the vocabulary. For a 50,000-word vocabulary, each word is a 50,000-dimensional sparse vector.

Problems with one-hot encoding:

• No semantic relationship: "car" and "automobile" are as different as "car" and "banana"
• Dimensionality: 50,000 dimensions is expensive to compute with
• Sparsity: Most values are zero - wasteful

Embeddings solve all three: dense (128-1536 dimensions), low-dimensional, and capturing semantic relationships.

Word Embeddings: word2vec and GloVe

Word2Vec trains a model to predict surrounding words given a center word (or vice versa). As a side effect, word vectors with similar contexts cluster together.

The famous demonstration: king - man + woman ≈ queen

This works because the directions in embedding space correspond to semantic relationships. The "royalty" direction is similar for both king and queen. The "gender" direction distinguishes them.

pythonCopy
from gensim.models import Word2Vec

sentences = [sentence.split() for sentence in corpus]
model = Word2Vec(sentences, vector_size=100, window=5, min_count=1)

# Get the embedding for a word
word_vector = model.wv["machine"]

# Find similar words
similar_words = model.wv.most_similar("machine", topn=5)

Sentence Embeddings: Semantic Search

Modern sentence embedding models (e.g., sentence-transformers/all-MiniLM-L6-v2) embed entire sentences into a dense vector such that semantically similar sentences have high cosine similarity.

pythonCopy
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')

sentences = [
    "The cat sat on the mat",
    "A feline rested on the rug",
    "The stock market crashed today",
]

embeddings = model.encode(sentences)

# Similarity between sentences 0 and 1 (should be high)
sim_01 = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
# Similarity between sentences 0 and 2 (should be low)
sim_02 = cosine_similarity([embeddings[0]], [embeddings[2]])[0][0]

print(f"Cat/Feline similarity: {sim_01:.3f}")  # ~0.85
print(f"Cat/Stock market similarity: {sim_02:.3f}")  # ~0.05

Embeddings in Practice: Semantic Search

pythonCopy
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')

# Knowledge base
documents = [
    "Our return policy allows returns within 30 days of purchase.",
    "Free shipping is available on orders over $50.",
    "Customer support is available Monday through Friday, 9am-5pm EST.",
    "We accept Visa, Mastercard, American Express, and PayPal.",
]

# Encode knowledge base once
doc_embeddings = model.encode(documents)

def semantic_search(query: str, top_k: int = 2) -> list[tuple[str, float]]:
    query_embedding = model.encode([query])
    sims = cosine_similarity(query_embedding, doc_embeddings)[0]
    top_indices = np.argsort(sims)[::-1][:top_k]
    return [(documents[i], sims[i]) for i in top_indices]

results = semantic_search("Can I return a product?")
for doc, score in results:
    print(f"Score: {score:.3f}  -  {doc}")

User and Item Embeddings for Recommendation

In recommendation systems, users and items are embedded into the same space. A user's embedding represents their taste; an item's embedding represents its characteristics. High dot product between user and item embeddings means the user is likely to engage with the item.

Matrix factorization (the underlying technique) decomposes the user-item interaction matrix into user embeddings and item embeddings. Modern neural recommendation systems (YouTube DNN, Two-Tower models) learn richer embeddings through deep learning.

What Makes a Good Embedding?

Discrimination: Similar things should be close, dissimilar things should be far. Measure with precision@K on a retrieval benchmark.

Consistency: The same object should produce the same embedding each time (for deterministic models, this is guaranteed).

Transferability: Embeddings trained for one task should be useful for related tasks. Pre-trained embeddings are often fine-tuned for specific applications.

The embedding model you use matters significantly. Always evaluate on your specific data, not just general benchmarks.

Common Mistakes

Not L2-normalizing before cosine similarity. Cosine similarity measures the angle between two vectors, which requires them to live on the unit sphere. If your embeddings have variable magnitudes, the dot product will be dominated by magnitude rather than direction, and your similarity rankings will be wrong in proportion to how much magnitudes vary across your corpus. Always normalize unless the model documentation explicitly states that raw dot product is preferred.

Using the same embedding model for all tasks. Embedding models are trained with specific contrastive objectives on specific data distributions. A model fine-tuned for semantic textual similarity excels at paraphrase detection but may perform poorly on asymmetric retrieval (short query vs. long document) or code search. Evaluate multiple models on a small labeled sample of your task before committing to one.

Treating embedding dimensions as interpretable features. Individual dimensions of an embedding vector do not correspond to human-interpretable concepts - the representation is distributed across all dimensions simultaneously. You cannot inspect dimension 42 to understand what it encodes. This matters when debugging: poor retrieval quality requires whole-vector analysis (PCA, UMAP projections) rather than inspecting individual dimensions.

What to Practice Next

• Embed five sentences that form a semantic cluster and five that are unrelated; compute all pairwise cosine similarities and verify that semantically similar pairs have higher scores than unrelated pairs.
• Apply UMAP or PCA to a set of 100+ document embeddings and visualize the result; identify whether natural topic clusters are visible in the lower-dimensional projection.
• Swap the embedding model in a small retrieval system and remeasure recall@5 on your eval set; confirm that model choice has a measurable effect on retrieval quality.