Milestone Gate 2: Applied AI Engineering

Milestone gates exist because finishing modules is not the same as being able to apply the skills. This gate covers applied AI engineering: RAG, LLM product features, agents, and embeddings. Use it as a self-assessment before moving to more advanced topics.

This is not a quiz. It is a practical rubric. For each area, be honest about whether you can do it unassisted - not whether you have read about it.

What This Gate Covers

The modules leading to this gate cover:

• Transformer architecture and attention mechanics
• Embedding models and vector search
• Retrieval-Augmented Generation (RAG): chunking, indexing, retrieval quality
• Building LLM-powered product features in production
• Tool-using agents and guardrails

Self-Assessment Rubric

RAG Systems

Pass criteria:

• Can build a complete RAG pipeline from scratch: document ingestion → chunking → embedding → vector index → retrieval → generation
• Can measure recall@k on a labeled test set and knows how to improve it
• Can explain why bad retrieval causes hallucination, not just that it does
• Has chosen between at least two chunking strategies and can justify the choice for a given document type

Not ready if: You have only followed a LangChain tutorial. Re-implement one piece manually - the chunker, the embedder, or the retrieval query - to verify you understand what the library is doing.

LLM Product Engineering

Pass criteria:

• Can structure LLM outputs with Pydantic + instructor or native response_format
• Has implemented retry logic with exponential backoff
• Has tracked per-feature LLM costs and knows the cost per 1K tokens for two models
• Has a working fallback for when the LLM API is unavailable

Not ready if: You cannot explain what happens to your feature when the LLM API returns a 429. Define the fallback behavior explicitly.

Embeddings

Pass criteria:

• Can generate embeddings from two different models and compare them on a retrieval task
• Knows the difference between symmetric and asymmetric embedding tasks
• Can set up pgvector with an HNSW index and run a cosine similarity query

Not ready if: You cannot write the SQL to create an HNSW index on a vector column. That is table stakes.

Agents

Pass criteria:

• Has built an agent that calls at least two tools in sequence
• Has implemented input validation for every tool
• Has a max_turns budget that terminates runaway agents
• Can describe three failure modes of tool-using agents from experience or deliberate testing

Not ready if: You have built an agent that works in the happy path but have never deliberately tested what happens when a tool returns an error.

Practical Check: The 30-Minute Build

If you are uncertain whether you pass, do this: build a RAG system in 30 minutes using only the standard library + openai + numpy. No LangChain, no vector DB library. Use numpy for cosine similarity, build the chunker yourself, store embeddings in a list.

This will tell you what you actually understand versus what you have borrowed from a framework.

pythonCopy
import openai
import numpy as np

client = openai.OpenAI()

def embed(texts: list[str]) -> np.ndarray:
    resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return np.array([e.embedding for e in resp.data])

def chunk_text(text: str, size: int = 400, overlap: int = 50) -> list[str]:
    words = text.split()
    chunks = []
    for i in range(0, len(words), size - overlap):
        chunks.append(" ".join(words[i:i + size]))
    return chunks

def retrieve(query: str, chunks: list[str], embeddings: np.ndarray, k: int = 3):
    q_emb = embed([query])[0]
    q_emb /= np.linalg.norm(q_emb)
    norm_embs = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)
    scores = norm_embs @ q_emb
    top_k = scores.argsort()[-k:][::-1]
    return [chunks[i] for i in top_k]

# Build index
document = open("your_doc.txt").read()
chunks = chunk_text(document)
embeddings = embed(chunks)

# Query
context = "\n\n".join(retrieve("What is the main argument?", chunks, embeddings))
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": f"Answer based on context:\n{context}"},
        {"role": "user", "content": "What is the main argument?"},
    ]
)
print(response.choices[0].message.content)

If you can write something like this from scratch, you are ready.

If You Are Not Ready

Do not rush past the gate. The modules in the next phase assume you can build these systems independently. Return to the modules that feel uncertain, build one more small project, and run the 30-minute check again.

Specific remediation:

• Weak on RAG: rebuild a mini RAG system without LangChain
• Weak on agents: build a two-tool agent with explicit error handling and test the failure paths
• Weak on cost/production: deploy a small LLM feature to a staging environment and watch the logs for one day

Where to Go Next

See also: [rag-evaluation-failure-analysis], [llm-app-engineering-production], [tool-using-agents-guardrails]