System Design for AI Products

What Makes AI System Design Different

In a traditional system design interview, you design for availability, consistency, partition tolerance, and latency. In an AI system design interview, you design for all of those - plus quality, data freshness, evaluation infrastructure, model versioning, distribution shift, and the probabilistic nature of model outputs.

The additional complexity is real, not just interview theater. Every design decision in an AI system has quality implications. Choosing to serve a model synchronously vs. asynchronously affects which use cases are viable. Choosing how to chunk documents affects retrieval quality. Choosing a reranker model affects precision. This post gives you a framework for reasoning through these decisions.

The Core AI System Stack

Most AI products compose the same layers:

User Interface
      ↓
Application Logic / Orchestration
      ↓
AI Services (LLM API, Embedding API, Reranker)
      ↓
Data Services (Vector Store, Feature Store, Cache)
      ↓
Data Pipeline (Ingestion, Processing, Indexing)
      ↓
Storage (Document Store, Data Warehouse, Object Store)

Each layer has its own design considerations. The art of AI system design is choosing implementations for each layer that satisfy the product requirements jointly.

Latency vs. Quality: The Central Tradeoff

Almost every decision in AI system design involves trading latency for quality. Understanding this tradeoff explicitly prevents you from designing systems that are technically correct but unusable.

Latency Budgets

Start with user-facing latency requirements:

• Interactive (conversational): < 500ms to first token, < 5s to complete response
• Batch (document processing): Minutes to hours acceptable
• Real-time (ad serving, fraud): < 50ms end-to-end

Work backward from the user-facing requirement to allocate a latency budget across each system component.

Where Latency Comes From in AI Systems

Component	Typical Latency
Embedding a query	20–100ms
Vector search (ANN)	5–50ms
LLM generation (first token)	100–500ms
LLM generation (complete, 200 tokens)	2–10s
Reranker model	50–200ms
Cache hit	< 5ms

For a RAG system with a 2-second budget: embedding (50ms) + vector search (20ms) + LLM generation (1.5s) + overhead (200ms) = ~1.8s. This works if you start streaming immediately and the user sees the first token within 500ms.

Key Design Decisions and How to Make Them

Synchronous vs. Asynchronous Serving

Synchronous: User waits for the result. Required for interactive use cases. Requires model serving with low p99 latency.

Asynchronous: Request is queued; result is returned later. Appropriate for batch processing, document ingestion, email/report generation. Allows much heavier computation and cheaper serving infrastructure.

Design decision: If the user expects an immediate answer, serve synchronously. If they can accept "your report will be ready in 5 minutes," serve asynchronously.

Caching Strategy

LLM responses are expensive and often idempotent. Cache aggressively:

Exact match cache: Cache response by hash of (model, prompt). Works for FAQ systems, structured queries.

Semantic cache: Use embeddings to find cached responses to semantically similar queries. Works for conversational systems where exact matches are rare. Requires a similarity threshold that trades freshness for cost savings.

Prompt caching: Cache the static prefix of your prompt (system prompt, examples) with the model provider. Anthropic and OpenAI both offer this. Saves 80%+ of input token costs for systems with long, repeated system prompts.

Model Selection

Do not default to the largest, most expensive model. Model selection should be driven by the task requirements:

Task Type	Appropriate Model
Binary classification of short text	Fine-tuned small model (BERT-class)
Semantic routing / intent detection	Small LLM or embedding + classifier
Structured extraction	Mid-size LLM with tool use
Complex reasoning, long generation	Large frontier model
Embedding generation	Dedicated embedding model

A common pattern: use a small, cheap model for routing and simple classification; reserve the large model for complex generation tasks. This reduces cost by 60–80% on most workloads.

Retrieval-Augmented Generation: The Architecture

RAG is the dominant pattern for building LLM applications that need to reference specific, up-to-date, or proprietary knowledge. The architecture:

Query → Embed Query → Vector Search → Retrieve Chunks
      ↓
Context Assembly → LLM Generation → Response

The Indexing Pipeline (Offline)

1. Ingest: Load documents from sources (PDF, web, database, API)
2. Parse: Extract text, preserve structure
3. Chunk: Split into retrieval units. Common strategies: fixed size, sentence-based, semantic (split on topic shifts)
4. Embed: Convert chunks to vectors using an embedding model
5. Index: Store in a vector database (Pinecone, Weaviate, Qdrant, pgvector)

The Query Pipeline (Online)

1. Embed query: Same embedding model as indexing
2. Retrieve: ANN search returns top-K candidates
3. Rerank (optional): A cross-encoder reranker rescores candidates based on query-document relevance. Improves precision but adds latency.
4. Assemble context: Format retrieved chunks into the prompt
5. Generate: LLM produces grounded response
6. Cite: Link claims to source documents

Key Architectural Decisions

Chunk size: Smaller chunks = higher precision, lower recall. Larger chunks = more context per result, potentially lower precision. A common starting point: 256–512 tokens with 50-token overlap.

Number of retrieved chunks (top-K): More chunks = more recall, more context tokens, higher LLM cost. Start with K=5 and tune based on eval results.

Reranker: A cross-encoder reranker (e.g., bge-reranker) significantly improves precision over pure vector search. Add it when precision problems are identified in evaluation, not by default.

Sparse + dense fusion: Hybrid search combines BM25 (keyword-based, sparse) with vector search (dense). Outperforms either alone on most benchmarks. Use RRF (Reciprocal Rank Fusion) to merge results.

Evaluation Infrastructure

You cannot operate an AI system you cannot measure. Evaluation infrastructure must be built before or alongside the application, not after.

What to Measure

Retrieval quality: Precision@K, Recall@K, MRR (Mean Reciprocal Rank). Requires a labeled set of query → relevant document pairs.

Generation quality: Factual consistency (does the response match the retrieved context?), helpfulness (does it address the query?), groundedness (are claims backed by retrieved context?).

System health: Latency percentiles (p50, p95, p99), error rates, cache hit rate, cost per query.

User feedback: Thumbs up/down signals, correction rates, conversation abandonment.

Offline vs. Online Evaluation

Build a representative eval set (50–500 examples) for offline evaluation before every deployment. Use LLM-as-judge with calibrated prompts for automatic scoring.

For online evaluation: implement a feedback mechanism, log all interactions, and run A/B tests when making significant changes.

Data Pipeline Design

AI systems require a data pipeline to keep the knowledge base fresh and the models informed.

Ingestion frequency: How often does source data change? Daily batch is common for document knowledge bases. Real-time streaming is needed for news, stock prices, support tickets.

Incremental vs. full re-indexing: Full re-indexing is simple but expensive. Incremental updates require detecting changed documents (checksums, timestamps) and updating only the affected chunks and embeddings.

Schema evolution: Embedding models get updated. When you switch to a new embedding model, you must re-index everything - the old and new embeddings live in different geometric spaces and are not comparable.

Common Mistakes and Bad Instincts

Defaulting to the largest model. Cost and latency matter in production. Start with the smallest model that meets quality requirements.

Not building evaluation infrastructure. Without eval, you cannot measure quality, detect regressions, or justify model upgrades.

Ignoring cold-start problems. Systems that depend on user history break for new users. Plan the cold-start experience explicitly.

Chunking without measurement. Different chunking strategies produce different retrieval quality. Measure chunking decisions with a retrieval eval set, do not pick intuitively.

Assuming vector search is always fast enough. ANN search at scale (hundreds of millions of vectors) requires careful index configuration and hardware. Profile before committing to a vector database choice.

The Interview Framework

In an ML system design interview, structure your answer around:

1. Clarify requirements: Who are the users? What is the latency SLA? What scale? What quality threshold?
2. Propose a high-level architecture: Draw the key components and their interactions
3. Identify the core design decisions: What are the main tradeoffs (latency/quality/cost)?
4. Go deep on one or two areas: Retrieval architecture, evaluation infrastructure, or serving
5. Address failure modes: What breaks first at scale? How do you detect and recover?

Where to Go Next

AI system design is covered in Module 28 of the College Student path (AI System Design and Product Tradeoffs) and Module 23 of the SWE path (AI System Design: Quality, Cost, Latency, and Safety Tradeoffs). Both modules require producing a system design document with explicit design decisions, tradeoff reasoning, and an evaluation plan.

A Framework for AI Product Design

Start every AI system design with five questions:

1. What user decision or workflow changes because of the model?
2. What data is available at prediction time?
3. What quality level is required for the product to be useful?
4. What happens when the model is wrong?
5. How will the system improve after launch?

These questions force you to design the model as part of a product system.

Reference Architecture

Most AI product systems have these layers:

textCopy
Client
  -> API gateway
  -> Application service
  -> Feature/context service
  -> Model or LLM service
  -> Policy and validation layer
  -> Storage, logs, metrics, and feedback

The model service is only one box. The surrounding system handles identity, rate limits, privacy, retries, caching, monitoring, and user experience.

Quality, Cost, Latency, and Risk

AI design is tradeoff management.

Constraint	Typical design response
Low latency	Smaller model, caching, precomputation, streaming
High quality	Stronger model, retrieval, reranking, human review
Low cost	Routing, batching, shorter context, cheaper model tiers
High risk	Human approval, policy checks, conservative fallback
Fresh knowledge	Retrieval, indexing pipeline, source citations
Personalization	User features, consent, privacy controls

Strong designs state the tradeoff explicitly. Weak designs pretend one architecture optimizes everything.

Designing for Failure

AI systems fail differently from normal software. A service can return 200 OK and still produce a bad answer. Your design needs semantic failure handling:

• Confidence thresholds
• Retrieval score checks
• Output validation
• Human escalation
• Safe default responses
• User feedback capture
• Post-launch review queues

For high-stakes domains, the system should be designed to avoid overconfident automation. The safest product might be a decision-support tool rather than full automation.

Feedback Loops

A product AI system should learn from use, but carefully. Capture:

• User corrections
• Thumbs up/down
• Human reviewer labels
• Task completion outcomes
• Repeated queries
• Abandoned flows

Do not blindly train on user feedback. Feedback can be biased, sparse, adversarial, or influenced by the current model. Treat it as signal to inspect, not truth to ingest automatically.

Design Review Checklist

A complete AI system design should include:

• Problem framing and non-goals
• Data sources and freshness requirements
• Model choice and alternatives
• Evaluation plan
• Serving architecture
• Latency and cost model
• Privacy and compliance handling
• Failure modes and fallbacks
• Monitoring and incident response
• Feedback loop and improvement plan

This is what separates a demo from a product architecture. Keep improving it deliberately with evidence.

Operating Note

The work is finished only when ownership is clear. Name who reviews quality, who responds to incidents, who approves changes, and who decides when the system should be paused.

That ownership turns architecture from a diagram into a living product system.

Closing Thought

The practical standard is not memorization. It is whether you can use the idea to make a better engineering decision, explain that decision to someone else, and notice when reality disagrees with your assumptions.

What to Do Next

Turn this article into a small artifact. Write a checklist, run a tiny experiment, sketch the architecture, or review an old project using the concepts above. Learning becomes durable when it changes what you inspect before you trust a result.

For a portfolio or team setting, save that artifact next to the code or decision memo. Future reviewers should be able to see not only what you built, but how you reasoned about correctness, risk, and tradeoffs.

Evidence Habit

When in doubt, prefer evidence over confidence. Keep the smallest repeatable test that proves the idea works, and revisit it whenever data, users, models, or requirements change.

In real reviews, also name the tradeoff you are most uncomfortable with. That honesty often leads to the most useful design discussion.

Interview Defense Checklist

For system design, prepare to defend:

• Why this architecture fits the user workflow
• Which component is most likely to fail first
• How the design changes at 10x traffic
• How cost scales with usage
• How privacy and permissions are enforced
• How quality regressions are detected
• What gets rolled back during an incident

The strongest answers are not the most complex. They are the ones where every component has a reason to exist and every risk has an owner.

Final Rule

System design for AI is not "where do we put the model?" It is the design of a complete decision system: data, model, product action, human review, monitoring, feedback, privacy, cost, and recovery. The best designs make uncertainty visible and manageable.

Architecture Defense Practice

Take any AI product idea and write three versions of the architecture:

1. A prototype that can be built in one week
2. A production version for ten thousand users
3. A regulated or high-risk version with audit requirements

Compare what changes. You will usually add stronger identity controls, better logging, stricter evaluation, human review, fallbacks, cost controls, and incident response. This exercise builds the habit that system design interviews reward: adapting architecture to constraints instead of memorizing one diagram.

Example Design: Fraud Detection

Fraud detection has different constraints from support automation. It is adversarial, latency-sensitive, and cost-sensitive. A useful design often combines:

• Real-time rules for obvious blocks
• Streaming features for recent activity
• Batch features for account history
• Supervised model for risk scoring
• Graph features for connected behavior
• Manual review queue for uncertain cases
• Feedback loop from investigator decisions

The model should not simply output "fraud" or "not fraud." It should support actions:

• Allow
• Step-up verification
• Hold for review
• Block

Each action has a different cost. Blocking a legitimate customer is expensive. Allowing fraud is expensive. Sending too many cases to review overwhelms operations. System design must optimize the whole decision workflow, not only model score.

Example Design: Search and Ranking

A search system usually has stages:

1. Query understanding
2. Candidate retrieval
3. Filtering
4. Ranking
5. Personalization
6. Blending business rules
7. Logging and feedback

The first model does not need to be perfect. It needs to be measurable. Track relevance, latency, zero-result rate, click behavior, long-clicks, reformulations, and user satisfaction. Ranking systems improve through iteration, so logging quality is part of the architecture.

Build the System Around Decisions

For every AI product, map model output to action:

Output	Product action	Risk control
High confidence answer	Respond automatically	Citation and feedback
Low confidence answer	Ask clarifying question	No unsupported claim
High fraud risk	Step-up verification	Manual appeal path
Medium churn risk	Sales outreach	Contact frequency cap
Unclear recommendation	Diversify results	Exploration budget

This table exposes missing product thinking. If there is no action, the prediction has no job.

Example Design: AI Support Assistant

Consider an assistant that answers customer support questions from company documentation.

Requirements:

• Answer common questions quickly
• Cite official docs
• Escalate account-specific or high-risk issues
• Avoid leaking private customer data
• Stay under a fixed cost per conversation

Architecture:

textCopy
User message
  -> intent classifier
  -> policy check
  -> retrieval from approved docs
  -> answer generation with citations
  -> output validation
  -> escalation or response
  -> feedback and logging

Key design choices:

• Use retrieval because docs change often.
• Use metadata filters to restrict sources by product, locale, and permission.
• Use a smaller model for routing and a stronger model for final answers.
• Escalate billing disputes, legal questions, abusive content, and low-confidence answers.
• Cache common answered questions to reduce latency and cost.

Capacity and Cost Planning

System design interviews and real launches both require rough math. Estimate:

• Requests per day
• Peak requests per second
• Average input and output tokens
• Retrieval calls per request
• Model cost per thousand or million tokens
• Cache hit rate
• Retry rate
• Storage growth for logs and indexes

Even approximate numbers improve design quality. They reveal whether a proposed architecture is plausible.

Privacy and Governance

AI systems often touch sensitive data. Design must answer:

• What data is sent to external providers?
• Is user consent required?
• Are prompts and outputs logged?
• How long are logs retained?
• Can users request deletion?
• Are employees allowed to paste customer data?
• Which outputs require audit trails?

Governance is not a final checklist. It shapes architecture from the beginning.

Multi-Model Architecture

Many products should not use one model for everything. A practical system may use:

• Rules for obvious blocks
• Small classifier for routing
• Embedding model for retrieval
• Reranker for relevance
• Large model for final response
• Deterministic validator for output checks

This is cheaper, safer, and easier to debug than asking one large model to do every job.

Post-Launch Operations

Launch is the start of the real test. Define:

• Quality review cadence
• Incident severity levels
• Escalation owners
• Evaluation dataset refresh process
• Prompt and model change review
• Cost review threshold
• User feedback triage

An AI product without operations is a demo with users attached.