Capstone: Build and Operate a Production-Style AI System

The capstone is not a tutorial. It is a self-directed project where you define the problem, make all the engineering decisions, encounter real failures, and ship something that runs. The goal is not a perfect system - it is demonstrable end-to-end competency: you can frame a problem, choose tools, write production-quality code, evaluate honestly, deploy, and monitor.

This module gives you the capstone specification, a reference architecture you can adapt, and the criteria that separate strong capstone projects from weak ones.

The Capstone Specification

Duration: 4–8 weeks of part-time work (10–20 hours/week).

Required components (all must be present):

Component	What it shows
Problem definition + success metric	Ability to frame ML problems from business requirements
Data pipeline with versioning	Data engineering instincts, reproducibility
Trained model with offline evaluation	ML fundamentals, proper evaluation methodology
Serving endpoint	Production engineering, not just notebooks
Monitoring / eval harness	Operational maturity
README with decisions documented	Communication, architectural reasoning

Not required: perfect accuracy, novel architecture, GPU training, large datasets. A well-executed project on a simple problem beats a poorly-executed project on a complex one.

Reference Architecture: AI-Powered Document Q&A System

This reference capstone demonstrates the full stack in a self-contained, deployable system.

System Overview

User submits question
      ↓
FastAPI endpoint
      ↓
Redis semantic cache check (exact + semantic)
      ↓ (cache miss)
Query embedding (bge-small-en-v1.5)
      ↓
Hybrid retrieval (pgvector cosine + PostgreSQL FTS)
      ↓
Cross-encoder reranking (top-5 of top-20)
      ↓
Prompt assembly with citations
      ↓
LLM call (gpt-4o-mini, temp=0.0, structured output)
      ↓
Response validation (Pydantic, confidence check)
      ↓
Logging (request, response, sources, latency, tokens)
      ↓
Return answer to user

Repository Structure

capstone-rag/
├── src/
│   ├── ingest.py          ← Document chunking + embedding + storing
│   ├── retrieve.py        ← Hybrid search + reranking
│   ├── generate.py        ← Prompt assembly + LLM call
│   ├── serve.py           ← FastAPI app
│   └── monitor.py         ← Eval harness + drift checks
├── data/
│   └── documents.dvc      ← DVC pointer to document corpus
├── evals/
│   ├── eval_set.json      ← 50 hand-labeled Q&A pairs
│   └── run_evals.py       ← Evaluation script
├── tests/
│   ├── test_retrieve.py   ← Unit tests with fixture data
│   └── test_serve.py      ← Integration tests (test client)
├── migrations/
│   └── 001_create_schema.sql
├── Dockerfile
├── docker-compose.yml     ← postgres + pgvector + redis + app
├── requirements.txt
└── README.md

Key Implementation Pieces

Document ingestion pipeline:

pythonCopy
from src.ingest import ingest_documents
import mlflow

with mlflow.start_run(run_name="document-ingestion"):
    n_chunks = ingest_documents(
        source_dir="data/raw/",
        chunk_size=512,
        chunk_overlap=64,
    )
    mlflow.log_metric("n_chunks_ingested", n_chunks)
    mlflow.log_param("chunk_size", 512)

Serving with observability:

pythonCopy
# serve.py (excerpt)
import time
import structlog

logger = structlog.get_logger()

@app.post("/ask", response_model=AnswerResponse)
async def ask(req: QuestionRequest):
    start = time.perf_counter()

    cached = semantic_cache.lookup(req.question)
    if cached:
        return AnswerResponse(**cached, cache_hit=True)

    candidates = retrieve(req.question, k=20)
    top_chunks = rerank(req.question, candidates, top_n=5)
    result = generate_answer(req.question, top_chunks)

    latency_ms = (time.perf_counter() - start) * 1000
    logger.info("request_completed",
                question=req.question[:50],
                latency_ms=latency_ms,
                n_sources=len(result.sources),
                confidence=result.confidence,
                tokens=result.usage.total_tokens)

    return AnswerResponse(**result.dict(), cache_hit=False, latency_ms=latency_ms)

Evaluation harness:

pythonCopy
# evals/run_evals.py
import json
from src.retrieve import retrieve, rerank
from src.generate import generate_answer

with open("evals/eval_set.json") as f:
    eval_set = json.load(f)

results = []
for example in eval_set:
    candidates = retrieve(example["question"], k=20)
    top_chunks = rerank(example["question"], candidates, top_n=5)
    result = generate_answer(example["question"], top_chunks)

    # Check retrieval: did the correct source appear in top-5?
    retrieval_hit = any(s in example["expected_sources"] for s in result.sources)

    results.append({
        "question": example["question"],
        "answer": result.answer,
        "expected": example["expected_answer"],
        "retrieval_hit": retrieval_hit,
        "confidence": result.confidence,
    })

retrieval_recall = sum(r["retrieval_hit"] for r in results) / len(results)
avg_confidence = sum(r["confidence"] for r in results) / len(results)
print(f"Retrieval recall@5: {retrieval_recall:.3f}")
print(f"Average confidence: {avg_confidence:.3f}")

What Makes a Capstone Strong

Strong capstone projects share these properties:

Honest evaluation. Measure what actually matters. If your RAG system gets 60% retrieval recall@5, say so and explain what limits it. Interviewers trust engineers who understand their system's failure modes.

Documented decisions. Your README should explain why you chose pgvector over a dedicated vector database, why you used hybrid search rather than pure vector search, and what you would do differently with more time.

Production-grade structure. Structured logging, health endpoints, Docker Compose for local dev, DVC for data versioning, and a Dockerfile separate you from the "works in my notebook" cohort.

A real eval set. Fifty hand-labeled Q&A pairs where you manually verified the expected answer is more valuable than automated metrics on noisy data. It demonstrates rigor.

The README That Gets You Interviews

markdownCopy
# Capstone: Document Q&A System

## Problem
[2 sentences: what problem, who has it, why it matters]

## Approach
[Technical summary: retrieval strategy, model choices, why not alternatives]
- Chose hybrid search because: [specific reason based on benchmarking]
- Chose gpt-4o-mini because: [cost/quality tradeoff reasoning]
- Chose pgvector over Pinecone because: [operational simplicity]

## Results
- Retrieval recall@5: 73%
- Average response confidence: 0.82
- p99 latency: 380ms
- Cache hit rate: 41%

## What I Would Do Differently
- Add query expansion to improve low-recall queries
- Fine-tune the embedding model on domain-specific data
- Implement query classification to route complex questions to gpt-4o

## How to Run
[Working setup instructions - this must work on first try]

Common Mistakes and Bad Instincts

Picking a complex problem to look impressive. A capstone on "autonomous trading system" that is half-finished is weaker than a completed, honest Q&A system. Pick a problem you can ship in 4-6 weeks.

Skipping the eval set. "The model gives good answers" is not an evaluation. Write 50 test questions, record expected answers, measure retrieval recall and answer quality. This is what separates engineers from notebook hobbyists.

Not deploying. A model that runs locally in a notebook is a demo. A FastAPI endpoint with a Dockerfile that can be deployed in five minutes is evidence of production engineering capability.

README as a tutorial. Your README should document your decisions and results, not explain how RAG works. Assume the reader knows what a vector database is - they want to understand your specific choices.

Where to Go Next

You have completed the Software Engineer → ML/AI Engineer curriculum. The next phase is execution:

1. Pick one of the capstone options and commit to shipping it in 6 weeks
2. Run the eval harness and document your results honestly
3. Apply the portfolio framing from Module 25 to your resume and GitHub profile
4. Practice the interview framework from Module 26 until it is fluent

The brewYourAgent curriculum has given you the knowledge. The capstone gives you the evidence. The interview preparation converts both into the job.