How Recommendation Systems Work

The Invisible Curator

Every time Netflix surfaces a show you end up watching, Spotify builds a playlist that fits your mood, or an e-commerce site shows you exactly what you were about to search for - that is a recommendation system at work.

These systems feel almost psychic. But the underlying logic is more mechanical than magical, and understanding it helps you reason clearly about both their value and their risks.

The Two Core Approaches

Collaborative Filtering: People Like You Also Liked

The fundamental idea: find users whose behavior closely resembles yours, and recommend things they engaged with that you have not tried yet.

The system does not need to understand the content at all. It only needs enough history of who liked what. If you and another user gave similar ratings to fifty movies, and they gave five stars to a film you have not seen, the system bets you will like it too.

At scale - hundreds of millions of users with years of interaction history - these bets become remarkably precise. Netflix, Spotify, and Amazon all use collaborative filtering as a core layer. The patterns are real: people who share tastes on five things often share tastes on the sixth.

The limitation: collaborative filtering needs prior behavior. A brand-new user has no history. A brand-new product has no ratings. Both cases produce unhelpful recommendations.

Content-Based Filtering: Similar Items

This approach looks at the properties of items rather than the behavior of users. If you liked a slow-burn thriller set in 1970s New York, the system finds other films with similar attributes: genre, pacing, era, director, tone.

Content-based filtering works well for new items (they can be described before anyone has rated them) and for users with unusual tastes that few others share. Its limitation is that it stays close to what you already like - it tends to recommend more of the same rather than genuinely surprising you.

Most production systems blend both approaches: collaborative filtering for users with rich history, content-based as a fallback for new users and new items, with additional layers on top.

The Cold-Start Problem

New users and new items create gaps that neither core approach can fill cleanly. This is the cold-start problem, and every recommendation team deals with it.

Consider a music streaming service. A new subscriber from Brazil signs up. The system has no playback history, no ratings, no follows. It has no idea what to show them.

Common solutions: ask new users a few preference questions during onboarding. Use demographic or geographic signals as a rough proxy. Fall back to popularity - show what is trending among users who share a few early signals. As the user generates more data, transition from population-level guesses to personal recommendations.

The cold-start experience sets the first impression, which shapes early retention. It is worth deliberate design, not just a fallback.

Why Netflix Recommendations Feel So Good

Netflix combines a remarkable number of signals: ratings, watch history, completion rates, rewatch behavior, time of day, device type, title browsing without watching, what you added to a list versus what you actually started.

They retrain models frequently on fresh behavioral data. They run large-scale A/B tests to measure whether a new recommendation model actually improves completion rates and subscriber satisfaction - not just clicks.

And they have years of history on hundreds of millions of subscribers. The volume of signal is staggering. The "psychic" feeling is the output of an enormous amount of data and years of iteration, not a clever trick.

The Feedback Loop and the Filter Bubble

Here is the quiet risk in recommendation systems: they create a loop.

The system shows you items it predicts you will engage with. You engage with them. That engagement becomes training data for the next version of the system. Over time, the system gets increasingly good at predicting what you will click - and increasingly narrow in what it shows you.

This is the filter bubble. It is not a conspiracy. It is a predictable outcome of optimizing for engagement. If you engage with content from one political perspective, the system learns to show you more of it. Your exposure to diverse content shrinks even as the total available content explodes.

The filter bubble is an optimization problem with a value misalignment: the system maximizes what you will click, which is not the same as what you would consciously choose if asked, or what serves your long-term interests.

Relevance vs. Diversity: The Real Tension

Purely optimizing for relevance produces a system that shows you things you are nearly certain to like - but that narrows your world. Introducing diversity deliberately shows you things that are a small step outside your current taste profile, which can improve long-term satisfaction even if it reduces short-term engagement metrics.

The most thoughtful recommendation teams define their objective carefully. They might combine an engagement score with a novelty score, or measure long-term retention alongside clicks, to avoid optimizing their way into a filter bubble.

Common Misunderstandings

"The algorithm knows what I want better than I do." It knows what you have engaged with before, and it predicts more of that. It has no model of what is actually good for you, or what you would want if your past behavior were different.

"Recommendation systems are neutral." They embed choices: what signals to use, what to optimize for, whose behavior counts more. Those choices have consequences for what different users see and which content gets amplified.

"The cold-start problem is solved." It is managed, not solved. Every product with new users faces it. Explicit onboarding, demographic signals, and popularity fallbacks are mitigations, not eliminations.

What to Explore Next

• /posts/what-are-embeddings - how users and items are represented as vectors in modern recommendation systems
• /posts/designing-recommendation-systems - a systems-level look at production recommendation architecture
• /posts/ai-in-everyday-products - how recommendation is one of several AI layers in the products you use daily

Common Mistakes

Assuming tighter personalization always improves the product. Extreme personalization creates filter bubbles where users only see items similar to what they already clicked on. This reduces serendipity and can degrade long-term engagement as users get bored seeing a narrow slice of the catalog.

Ignoring popularity bias in collaborative filtering. Collaborative filtering learns that popular items are liked by many users, so it keeps recommending them - compounding their popularity while systematically burying niche or new items. Correcting for this requires explicit popularity debiasing or exploration strategies.

Underestimating the cold-start problem at launch. A recommendation system deployed with no historical interaction data falls back to popularity-based ranking, which is just "show everyone the same thing." This failure mode is predictable and needs an explicit strategy - onboarding surveys, content-based fallbacks, or curated starter lists - before the first user arrives.