How to Evaluate an LLM Application

Why LLM Evaluation Is Harder Than Supervised ML Evaluation

In supervised ML, you have ground truth labels. Evaluation is a comparison: did the model predict the correct label? In LLM applications, there is often no single correct answer. A good summary can take many forms. A helpful response to a question might be expressed dozens of ways.

This makes LLM evaluation part science, part judgment - and entirely necessary before deployment.

The Evaluation Stack

Think of LLM evaluation in four layers, from fastest/cheapest to slowest/most reliable:

1. Automated metrics: Fast, cheap, imperfect
2. LLM-as-judge: Moderate cost, good coverage, needs calibration
3. Human evaluation: Expensive, authoritative, required for calibration
4. Online A/B testing: The ultimate ground truth, requires deployment

Build all four. Use automated metrics and LLM-as-judge for continuous evaluation during development. Use human evaluation to calibrate the automated methods. Use online A/B tests before major releases.

Layer 1: Automated Metrics

For text similarity (summarization, translation):

ROUGE measures recall: what fraction of reference n-grams appear in the prediction?

pythonCopy
from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
scores = scorer.score(reference, prediction)
print(f"ROUGE-L: {scores['rougeL'].fmeasure:.3f}")

BERTScore measures semantic similarity using BERT embeddings:

pythonCopy
from bert_score import score

P, R, F1 = score([prediction], [reference], lang='en')
print(f"BERTScore F1: {F1.mean():.3f}")

Automated metrics are fast to run but correlate imperfectly with human judgment. Use them for regression testing (did a change hurt performance?), not for absolute quality assessment.

For structured output (extraction, classification):

pythonCopy
def evaluate_extraction(predictions: list[dict], ground_truth: list[dict]) -> dict:
    all_keys = set()
    for item in ground_truth:
        all_keys.update(item.keys())

    results = {}
    for key in all_keys:
        correct = sum(
            1 for pred, gt in zip(predictions, ground_truth)
            if pred.get(key) == gt.get(key)
        )
        results[f"{key}_accuracy"] = correct / len(ground_truth)
    return results

Layer 2: LLM-as-Judge

Use a capable LLM to evaluate responses on specified dimensions. This scales better than human evaluation while capturing nuance that rule-based metrics miss.

pythonCopy
import anthropic
import json

judge_client = anthropic.Anthropic()

def llm_judge(
    question: str,
    response: str,
    context: str = None,
    dimensions: list[str] = None,
) -> dict:
    if dimensions is None:
        dimensions = ["accuracy", "completeness", "clarity"]

    context_section = f"\nContext provided to the model:\n{context}\n" if context else ""

    prompt = f"""Evaluate the following AI response on each dimension below.
Score each dimension from 1 (very poor) to 5 (excellent).
Return JSON with keys for each dimension score and a "rationale" key.

Question: {question}{context_section}
Response to evaluate: {response}

Dimensions to score: {', '.join(dimensions)}

Return format: {{"dimension_name": score, ..., "rationale": "explanation"}}"""

    result = judge_client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=512,
        messages=[{"role": "user", "content": prompt}]
    )

    try:
        return json.loads(result.content[0].text)
    except json.JSONDecodeError:
        return {"error": "Failed to parse judge output", "raw": result.content[0].text}

# Example
eval_result = llm_judge(
    question="What is the refund policy?",
    response="Refunds are available within 30 days of purchase.",
    context="Our policy: Returns accepted within 30 days. Full refund to original payment method.",
    dimensions=["accuracy", "completeness", "groundedness"],
)
print(eval_result)

Known biases of LLM-as-judge: prefers longer responses, agrees with assertive framing, favors same-family models. Calibrate against human judgments before trusting fully.

Layer 3: Building an Eval Dataset

Your eval dataset is the asset that makes all other evaluation reliable. Invest in it early.

pythonCopy
import json
from datetime import datetime

class EvalDataset:
    def __init__(self, path: str):
        self.path = path
        self.cases = self._load()

    def _load(self) -> list[dict]:
        try:
            with open(self.path) as f:
                return json.load(f)
        except FileNotFoundError:
            return []

    def add_case(self, query: str, ideal_response: str, metadata: dict = None):
        case = {
            "id": len(self.cases) + 1,
            "query": query,
            "ideal_response": ideal_response,
            "added_at": datetime.utcnow().isoformat(),
            "metadata": metadata or {},
        }
        self.cases.append(case)
        self._save()

    def _save(self):
        with open(self.path, "w") as f:
            json.dump(self.cases, f, indent=2)

    def run_eval(self, pipeline_fn, judge_fn=None) -> dict:
        results = []
        for case in self.cases:
            response = pipeline_fn(case["query"])
            result = {"id": case["id"], "query": case["query"], "response": response}
            if judge_fn:
                result["scores"] = judge_fn(case["query"], response, case.get("ideal_response"))
            results.append(result)

        if judge_fn:
            avg_scores = {}
            score_keys = [k for k in results[0]["scores"] if k != "rationale" and k != "error"]
            for key in score_keys:
                avg_scores[key] = sum(r["scores"].get(key, 0) for r in results) / len(results)
            return {"results": results, "averages": avg_scores}
        return {"results": results}

Layer 4: Task-Specific Eval Patterns

For code generation: Execute the generated code and run tests.

pythonCopy
import subprocess
import tempfile

def eval_code_generation(prompt: str, test_cases: list[str]) -> dict:
    generated_code = generate_code(prompt)  # Your LLM function
    passed = 0
    for test in test_cases:
        with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
            f.write(generated_code + "\n" + test)
            fname = f.name
        result = subprocess.run(["python", fname], capture_output=True, timeout=10)
        if result.returncode == 0:
            passed += 1
    return {"pass_rate": passed / len(test_cases)}

For RAG: Measure retrieval hit rate separately from generation quality.

For classification: Standard precision/recall/F1 over a labeled test set.

Putting It Together: A Continuous Eval Pipeline

pythonCopy
def run_eval_suite(pipeline_fn, eval_dataset_path: str) -> dict:
    dataset = EvalDataset(eval_dataset_path)

    def judge(query, response, ideal):
        return llm_judge(query, response, dimensions=["accuracy", "helpfulness"])

    report = dataset.run_eval(pipeline_fn, judge_fn=judge)

    print(f"Eval results ({len(report['results'])} cases):")
    for metric, value in report.get("averages", {}).items():
        print(f"  {metric}: {value:.2f}/5.0")

    return report

Run this before every deployment. Track trends over time. A drop in eval scores is an early warning of a regression.

Common Mistakes

Using BLEU or ROUGE for open-ended generation. These metrics measure n-gram overlap with a reference string. For summarization or Q&A with a single reference they are weak proxies; for open-ended generation where many valid answers exist they are essentially noise. Use task-appropriate metrics: exact match for factual extraction, BERTScore for semantic similarity, or a calibrated LLM judge for qualitative tasks.

Evaluating only on "happy path" inputs. If your test set consists of well-formed, in-distribution queries, you will systematically miss the failure modes that matter most in production - adversarial prompts, ambiguous phrasing, empty inputs, or multilingual requests. Deliberately include edge cases and adversarial examples in every eval set.

Treating LLM-as-judge scores as ground truth without calibration. LLM judges have their own biases: position bias (preferring the first response), verbosity bias, and self-preference when the judge shares the same base model as the system under test. Always spot-check a random sample of judge scores against human labels and report inter-rater agreement before relying on automated scores.

What to Practice Next

• Build an eval harness with at least 20 test cases covering both happy-path and edge-case inputs; run it before and after making a prompt change and report the delta.
• Add your eval harness to CI so any regression is caught automatically on each prompt or model change.
• Score 20 outputs with an LLM judge and then score the same outputs yourself; compute Cohen's kappa and investigate cases where you disagreed.