← Back to Guides
5

Series

RAG Engineering in Production· Part 5

GuideFor: AI Engineers, ML Engineers, Platform Engineers, AI Systems Architects

Why Your RAG System Cannot Tell When It Is Wrong

Most RAG pipelines measure answer quality and ignore retrieval quality - which means the Retrieval Tax, Chunking Debt, and Precision Gap from Parts 1 through 4 are accumulating invisibly, query by query, until a production incident forces the question.

ai-engineeringretrieval-augmented-generation
#rag#evaluation#ragas#ares#deepeval#context-recall#faithfulness#production-monitoring#llm-infrastructure

Six months after shipping a contract review assistant, a legal team asked why the system had been recommending standard termination notice periods for a particular vendor category when the actual contract contained a 90-day exception clause. The answer was in the corpus. It had been ingested. The chunk was well-formed. The embedding was domain-aligned. The reranker had seen it.

No one had measured context recall since launch. No one knew whether the 90-day clause was being retrieved consistently. The system's answer quality metrics - user satisfaction ratings, thumbs up/down, spot-check audits - all looked fine because the LLM is excellent at generating plausible-sounding answers from approximate context. The answers were coherent, cited real documents, and were usually correct. "Usually" is not a word that belongs in a compliance workflow.

The failure had been accumulating for months before someone checked the right underlying contract. By that point, the team had no idea which other clauses the system had been silently wrong about.

This is the Evals Blind Spot: building the full four-layer RAG pipeline from Parts 1 through 4 - retrieval strategy, chunking, embedding, reranking - and then measuring only the final output, not the layers that determine whether that output is trustworthy. The LLM is too good at sounding right from wrong context for generation-level metrics to catch retrieval failures reliably. You need retrieval-layer metrics. Most teams do not have them.

The thesis of this article is specific: measuring answer quality tells you whether the LLM can write. Measuring retrieval quality tells you whether the system knows the truth. These are different instruments and most production RAG pipelines have only one of them.

Why Generation Metrics Cannot Catch Retrieval Failures

The standard production metrics for RAG are answer-level: user satisfaction ratings, thumbs up/down, response relevancy scores, spot-check accuracy audits. These are the right metrics for evaluating whether the generated text is useful. They are the wrong instruments for detecting retrieval failures.

The mismatch is structural. Large language models generate fluent, coherent text from whatever context they receive. When retrieval returns approximately-correct documents - the right topic, wrong time period; the right clause category, wrong vendor exception; the general policy, missing the specific carve-out - the LLM produces an answer that sounds authoritative and is calibrated to the retrieved content. The answer is faithful to what was retrieved. It is not faithful to what was true.

This is the RAGAS distinction between faithfulness and factual correctness that most teams conflate. Faithfulness measures whether the generated response is supported by the retrieved context. A score of 1.0 means every claim in the answer traces to the retrieved documents. Factual correctness measures whether those claims are true in the world. A high-faithfulness answer built on wrong retrieved documents is both faithful and wrong.

Your user satisfaction ratings cannot distinguish between these. Your thumbs up/down data cannot distinguish between these. Only retrieval-layer metrics can - and retrieval-layer metrics require measuring what the retriever returned, not what the LLM said about it.

The second structural problem: retrieval failures are often long-tail. They concentrate in edge-case queries, specialized document types, or recently updated content where the index has not caught up. Your spot-check audits will not find them unless you specifically design your evaluation set to cover these cases. A random sample of queries from your production logs will underrepresent edge cases. The system looks fine on the 80% and fails silently on the 20% where it matters most.

Named Concept: The Evals Blind Spot

The Evals Blind Spot is the gap between what you measure and what can actually fail. It has two dimensions:

Layer blindspot: Measuring final answer quality while retrieval quality is unmeasured. The Retrieval Tax from Part 1 accumulates invisibly. The Precision Gap from Part 4 is never quantified. You do not know your context recall, your context precision, or your nDCG@10. You only know whether users clicked thumbs up.

Temporal blindspot: Measuring at launch and not continuously. Knowledge bases drift - documents are added, updated, restructured, deleted. Retrieval quality degrades as the index diverges from the query distribution the embedding model was optimized for. Chunking Debt from Part 2 compounds as new document formats are added that the original splitting strategy handles poorly. Semantic Compression Loss from Part 3 worsens as new domain terminology enters the corpus that the embedding model was not trained on. None of this triggers an error. It accumulates silently until it reaches a threshold that produces a visible wrong answer.

Closing the Evals Blind Spot requires two things: a retrieval evaluation layer that runs before generation, and a continuous monitoring layer that runs in production. Neither is optional if the system's output has downstream consequences.

The Four Metrics You Actually Need

RAGAS (Es et al., EACL 2024, arXiv:2309.15217) established the canonical metric suite for RAG evaluation. The four core metrics cover retrieval quality and generation quality separately, with a critical property: three of the four require no ground truth labels.

Context Recall (retrieval metric, requires ground truth)

Measures whether the retrieved context contains all the information needed to answer the question. Computed against a reference answer: for each sentence in the reference, the metric checks whether the retrieved context contains the information needed to derive it.

Why it matters: This is the direct measurement of whether your retrieval pipeline is surfacing the right content. A context recall of 0.72 means 28% of your queries are missing information that exists in the corpus. That missing information is Chunking Debt made visible - it was probably there, just not retrievable because the chunk boundary destroyed the semantic unit.

Threshold: 0.8 and above is the production-ready signal. Below 0.7 warrants retrieval investigation before measuring anything else - generation metrics are meaningless on top of poor retrieval.

Context Precision (retrieval metric, reference-free)

Measures what fraction of the retrieved context is actually relevant to the query. If you retrieve 5 documents and 3 are relevant, context precision is 0.6. A pipeline with high recall but low precision is returning a lot of noise alongside the signal.

Why it matters: Low context precision is the Precision Gap from Part 4 made quantitative. It is also a cost metric: irrelevant context consumes tokens, increases latency, and degrades generation quality by introducing misleading information that the LLM has to reason around.

Faithfulness (generation metric, reference-free)

Measures whether all claims in the generated response are supported by the retrieved context. Each claim in the answer is checked against the context; faithfulness = (supported claims) / (total claims).

Why it matters: Faithfulness catches the specific failure mode where the LLM confabulates beyond the retrieved context. High faithfulness with low context recall means the system is faithfully reproducing wrong information that retrieval returned. The two metrics together tell you where the failure is: retrieval (low recall) or generation (low faithfulness with adequate recall).

Answer Relevancy (generation metric, reference-free)

Measures whether the generated response directly addresses the question asked. A response that is faithful to the context but does not answer the question scores low here.

Why it matters: This is the only metric that approximates what user satisfaction measures - but it is computed mechanically from the query and response, not from user feedback. It cannot catch factual errors, but it catches cases where the LLM has wandered off-topic despite having relevant context.

The Wrong Way: Measuring Only Generation Quality

code
# Wrong way: evaluating RAG quality by user feedback and spot-checks only# This is what most production teams do.# 1. User satisfaction (thumbs up/down) - measures whether the answer felt useful# Problem: users cannot tell when the answer is confidently wrong#          from a correct-feeling document. They only know when it is#          obviously wrong. Silent precision failures pass this filter.# 2. Spot-check audits on answer text# Problem: auditors sample from production queries.#          Edge-case retrieval failures are underrepresented.#          The 80% of easy queries look fine.# 3. Response time monitoring# Problem: measures latency, not quality.#          A fast wrong answer is still wrong.# What these three instruments together cannot detect:# - Context recall below 1.0 (some queries missing critical information)# - Context precision below 1.0 (noise in retrieved context)# - The Precision Gap: right documents not ranked first# - Retrieval drift as the knowledge base evolves# - Chunking Debt compounding as new document types are added# None of the above produce errors. None produce alert spikes.# They produce slightly worse answers on a fraction of queries,# invisible to all three instruments, until someone checks the# right underlying document and finds the system was wrong for months.

The Right Way: Retrieval-Layer Evaluation with RAGAS

code
from ragas import evaluatefrom ragas.metrics import (    context_recall,    context_precision,    faithfulness,    answer_relevancy,)from ragas.testset import TestsetGeneratorfrom ragas.llms import LangchainLLMWrapperfrom ragas.embeddings import LangchainEmbeddingsWrapperfrom langchain_openai import ChatOpenAI, OpenAIEmbeddingsfrom datasets import Dataset# Step 1: Generate a synthetic evaluation dataset from your corpus# RAGAS TestsetGenerator creates (question, ground_truth, context) triples# from your source documents without manual annotation.# This is how you bootstrap evals when you have no labeled data.def build_eval_dataset(documents: list, n_samples: int = 100) -> Dataset:    """    Generate synthetic evaluation test set from your corpus.        n_samples=100 is the minimum meaningful evaluation set.    Run once at launch, then refresh when the corpus changes significantly.        The generator creates three question types:      - Simple: single-hop factual questions      - Reasoning: multi-step inference questions        - Multi-context: questions requiring multiple documents    Balance across types to cover the full failure-mode surface.    """    generator_llm = LangchainLLMWrapper(        ChatOpenAI(model="gpt-4o-mini")  # Haiku or mini for cost control    )    generator_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())        generator = TestsetGenerator(        llm=generator_llm,        embedding_model=generator_embeddings,    )        testset = generator.generate_with_langchain_docs(        documents,        testset_size=n_samples,    )    return testset.to_dataset()# Step 2: Run your RAG pipeline against the eval datasetdef run_pipeline_on_testset(    testset: Dataset,    retriever,    llm,) -> Dataset:    """    Execute the RAG pipeline against the eval dataset and collect    the retrieved contexts and generated answers for each question.        These are the inputs RAGAS needs to compute metrics.    """    questions = testset["question"]    ground_truths = testset["ground_truth"]        contexts_list = []    answers_list = []        for question in questions:        # Retrieve - use your production retriever, not a test stub        # LangChain v0.2+: use .invoke() not .get_relevant_documents()        docs = retriever.invoke(question)        contexts = [doc.page_content for doc in docs]                # Generate        context_str = "\n\n".join(contexts)        prompt = f"Context:\n{context_str}\n\nQuestion: {question}\nAnswer:"        answer = llm.invoke(prompt).content                contexts_list.append(contexts)        answers_list.append(answer)        return Dataset.from_dict({        "question": questions,        "answer": answers_list,        "contexts": contexts_list,        "ground_truth": ground_truths,    })# Step 3: Score with RAGASdef evaluate_rag_pipeline(results: Dataset) -> dict:    """    Run the full RAGAS metric suite and return scores.        Metric interpretation:      context_recall:    0.8+ = production-ready; below 0.7 = retrieval problem      context_precision: 0.7+ = acceptable; below 0.5 = noise problem      faithfulness:      0.85+ = customer-facing; 0.7+ = internal tools      answer_relevancy:  0.8+ = acceptable; varies by use case        If context_recall is low, fix retrieval before debugging generation.    Measuring generation quality on top of poor retrieval is diagnostic noise.    """    score = evaluate(        results,        metrics=[            context_recall,       # Did we retrieve the right content?            context_precision,    # Did we retrieve only the right content?            faithfulness,         # Does the answer stay within the context?            answer_relevancy,     # Does the answer address the question?        ],    )    return score

Continuous Evaluation: The Production Monitoring Layer

A one-time evaluation at launch is not sufficient. Knowledge bases drift. Query distributions shift. Document formats change. Every change to the corpus is a potential regression in retrieval quality. You need a continuous evaluation layer that runs in production.

The minimum viable production monitoring stack has three components.

Component 1: Golden Dataset with Regression Gates

A golden dataset is a fixed set of (question, expected context, expected answer) triples that you evaluate on every significant pipeline change. It is your regression test suite for retrieval quality.

code
import jsonfrom datetime import datetimefrom pathlib import PathGOLDEN_DATASET_PATH = Path("evals/golden_dataset.json")METRICS_LOG_PATH = Path("evals/metrics_history.jsonl")# Thresholds - tune for your domain and risk toleranceTHRESHOLDS = {    "context_recall": 0.80,     # Hard gate: below this blocks deployment    "context_precision": 0.70,  # Hard gate    "faithfulness": 0.85,       # Hard gate for customer-facing    "answer_relevancy": 0.75,   # Soft gate: warning, not blocking}def run_regression_gate(    retriever,    llm,    block_on_failure: bool = True,) -> dict:    """    Run the golden dataset evaluation and check against thresholds.        Integrate into CI/CD: run this before any deployment that touches    chunking strategy, embedding model, retrieval config, or reranker.        If block_on_failure=True, raises ValueError on threshold breach.    Set False for monitoring-only mode (no deployment blocking).    """    with open(GOLDEN_DATASET_PATH) as f:        golden = json.load(f)        golden_dataset = Dataset.from_dict(golden)    results = run_pipeline_on_testset(golden_dataset, retriever, llm)    scores = evaluate_rag_pipeline(results)        # Log for trend tracking    log_entry = {        "timestamp": datetime.utcnow().isoformat(),        "scores": {k: float(v) for k, v in scores.items()},    }    with open(METRICS_LOG_PATH, "a") as f:        f.write(json.dumps(log_entry) + "\n")        # Check thresholds    failures = {        metric: (float(scores[metric]), threshold)        for metric, threshold in THRESHOLDS.items()        if float(scores[metric]) < threshold    }        if failures and block_on_failure:        failure_details = "\n".join(            f"  {metric}: {score:.3f} < {threshold} (threshold)"            for metric, (score, threshold) in failures.items()        )        raise ValueError(            f"RAG evaluation gate failed. Deployment blocked.\n"            f"Failing metrics:\n{failure_details}\n"            f"Fix retrieval regressions before deploying."        )        return {        "passed": len(failures) == 0,        "scores": log_entry["scores"],        "failures": failures,    }

Component 2: Production Sampling Evaluation

Run the RAGAS faithfulness check on a sample of live queries to detect degradation between golden-dataset evaluations. Context recall cannot be computed without ground truth, but faithfulness and answer relevancy can run reference-free on any live query.

code
import randomfrom typing import Callabledef production_sample_eval(    live_query_log: list[dict],  # [{question, retrieved_contexts, answer}]    sample_rate: float = 0.05,   # Evaluate 5% of live traffic    alert_callback: Callable | None = None,) -> dict:    """    Sample production queries and run reference-free RAGAS metrics.        Run this on a schedule (hourly or daily) to detect drift between    golden-dataset evaluations.        Faithfulness and answer_relevancy are reference-free - they require    only the question, retrieved contexts, and answer. No ground truth needed.        Context_recall is NOT included here because it requires ground truth.    Track context_recall only on your golden dataset.    """    sample = random.sample(        live_query_log,        k=max(1, int(len(live_query_log) * sample_rate)),    )        sample_dataset = Dataset.from_dict({        "question": [q["question"] for q in sample],        "answer": [q["answer"] for q in sample],        "contexts": [q["retrieved_contexts"] for q in sample],        # No ground_truth key - intentionally omitted for reference-free metrics.        # RAGAS v0.2+: context_recall and context_precision require ground_truth        # and are excluded here. Only faithfulness and answer_relevancy run        # reference-free. If your RAGAS version requires ground_truth in schema,        # add a placeholder: "ground_truth": [""] * len(sample)    })        scores = evaluate(        sample_dataset,        metrics=[faithfulness, answer_relevancy],    )        # Alert if faithfulness drops below threshold    if float(scores["faithfulness"]) < THRESHOLDS["faithfulness"]:        if alert_callback:            alert_callback(                f"Faithfulness degradation detected in production sample: "                f"{float(scores['faithfulness']):.3f} < "                f"{THRESHOLDS['faithfulness']}"            )        return {        "sample_size": len(sample),        "faithfulness": float(scores["faithfulness"]),        "answer_relevancy": float(scores["answer_relevancy"]),    }

Component 3: Retrieval Drift Detection

As the knowledge base evolves, track whether retrieval quality is drifting without running a full evaluation. The signal is the distribution of top-1 reranker scores over a moving window: if the distribution shifts downward, retrieval quality is degrading.

code
import numpy as npfrom collections import dequeclass RetrievalDriftMonitor:    """    Tracks reranker top-1 score distribution over time.    A downward shift in the distribution signals retrieval drift -    the knowledge base has changed in ways that make retrieval harder.        This is a lightweight signal, not a replacement for RAGAS evaluation.    Use it to trigger a full golden-dataset eval when drift is detected.    """        def __init__(self, window_size: int = 1000, alert_threshold: float = 0.05):        self.window_size = window_size        self.alert_threshold = alert_threshold        self.score_window: deque[float] = deque(maxlen=window_size)        self.baseline_mean: float | None = None        def record(self, top1_reranker_score: float) -> None:        self.score_window.append(top1_reranker_score)        def set_baseline(self) -> None:        """Call after collecting initial window of scores at launch."""        if len(self.score_window) >= self.window_size:            self.baseline_mean = float(np.mean(self.score_window))        def check_drift(self) -> dict:        """        Returns drift signal based on mean shift from baseline.        A mean drop of > alert_threshold triggers a drift warning.        """        if self.baseline_mean is None or len(self.score_window) < self.window_size:            return {"status": "insufficient_data"}                current_mean = float(np.mean(self.score_window))        delta = self.baseline_mean - current_mean                return {            "baseline_mean": self.baseline_mean,            "current_mean": current_mean,            "delta": delta,            "drift_detected": delta > self.alert_threshold,            "recommendation": (                "Run full golden-dataset evaluation to confirm and diagnose."                if delta > self.alert_threshold                else "No significant drift detected."            ),        }

Diagnosing Retrieval Failures from Metric Signals

The metrics create a diagnostic matrix. When a metric is low, the pattern of which metrics are low tells you where in the pipeline to look:

mermaid
flowchart TD
    START[RAGAS evaluation complete] --> CR{Context\nRecall}
    CR -- "Below 0.7" --> RET[Retrieval failure\nThe right content\nexists but is not retrieved]
    RET --> D1{Chunking\nDebt?}
    D1 -- Yes --> FIX1[Re-chunk corpus\nPart 2: recursive 400 tokens\nor structure-aware splitting]
    D1 -- No --> D2{Semantic\nCompression Loss?}
    D2 -- Yes --> FIX2[Evaluate domain-specific\nembedding model\nPart 3]
    D2 -- No --> FIX3[Expand candidate set\nIncrease retrieve-k\nCheck BM25 coverage]
    CR -- "0.7 or above" --> CP{Context\nPrecision}
    CP -- "Below 0.6" --> NOISE[Too much noise\nin retrieved context]
    NOISE --> FIX4[Reduce retrieve-k\nIncrease reranker threshold\nAdd metadata filtering\nPart 4: tune relevance floor]
    CP -- "0.6 or above" --> FA{Faithfulness}
    FA -- "Below 0.8" --> HALLU[Generation\nhallucination]
    HALLU --> FIX5[Tighten system prompt\nAdd citation instructions\nReduce max generation tokens]
    FA -- "0.8 or above" --> AR{Answer\nRelevancy}
    AR -- "Below 0.75" --> REL[Answer misses\nthe question]
    REL --> FIX6[Improve query routing\nCheck query rewriting\nReview system prompt]
    AR -- "0.75 or above" --> GOOD[Pipeline healthy\nNo action needed]

    style START fill:#4A90E2,color:#fff
    style CR fill:#7B68EE,color:#fff
    style CP fill:#7B68EE,color:#fff
    style FA fill:#7B68EE,color:#fff
    style AR fill:#7B68EE,color:#fff
    style D1 fill:#9B59B6,color:#fff
    style D2 fill:#9B59B6,color:#fff
    style RET fill:#E74C3C,color:#fff
    style NOISE fill:#FFA07A,color:#333
    style HALLU fill:#E74C3C,color:#fff
    style REL fill:#FFD93D,color:#333
    style FIX1 fill:#6BCF7F,color:#fff
    style FIX2 fill:#6BCF7F,color:#fff
    style FIX3 fill:#6BCF7F,color:#fff
    style FIX4 fill:#6BCF7F,color:#fff
    style FIX5 fill:#6BCF7F,color:#fff
    style FIX6 fill:#6BCF7F,color:#fff
    style GOOD fill:#6BCF7F,color:#fff

The diagnostic flow embodies a core principle: always start with context recall. Generation metrics are diagnostic noise when retrieval is broken. If context recall is below 0.7, do not debug faithfulness or answer relevancy - they will both look problematic downstream of a broken retriever, and fixing generation will not close the Evals Blind Spot.

Integrating Evaluation into CI/CD

Evaluation without automation is evaluation you will stop running. The golden-dataset regression gate must run automatically on every change to the pipeline - not as a post-launch audit, but as a deployment blocker.

code
# .github/workflows/rag-eval-gate.ymlname: RAG Evaluation Gateon:  push:    paths:      - 'src/retrieval/**'      - 'src/chunking/**'      - 'src/embedding/**'      - 'src/reranking/**'      - 'config/rag_config.yaml'jobs:  evaluation-gate:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v3      - name: Set up Python        uses: actions/setup-python@v4        with:          python-version: '3.11'      - name: Install dependencies        run: pip install ragas langchain openai datasets      - name: Run RAG evaluation gate        env:          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}        run: |          python scripts/run_eval_gate.py \            --golden-dataset evals/golden_dataset.json \            --block-on-failure \            --thresholds evals/thresholds.json
code
# scripts/run_eval_gate.py# This script imports run_regression_gate from your evaluation module,# which in turn uses run_pipeline_on_testset and evaluate_rag_pipeline# defined in your eval library. The imports below are illustrative -# replace with your actual module paths.import argparseimport jsonimport sys# Wire up your production retriever, LLM, and eval functions:# from src.retrieval import build_retriever# from src.llm import build_llm# from src.eval import run_regression_gatedef main():    parser = argparse.ArgumentParser()    parser.add_argument("--golden-dataset", required=True)    parser.add_argument("--block-on-failure", action="store_true")    parser.add_argument("--thresholds", required=True)    args = parser.parse_args()        with open(args.thresholds) as f:        thresholds = json.load(f)        # Build production retriever and LLM    # retriever = build_retriever()    # llm = build_llm()        try:        result = run_regression_gate(            retriever=retriever,            llm=llm,            block_on_failure=args.block_on_failure,        )                print(f"Evaluation passed: {result['passed']}")        print("Scores:")        for metric, score in result["scores"].items():            threshold = thresholds.get(metric, 0)            status = "PASS" if score >= threshold else "FAIL"            print(f"  {metric}: {score:.3f} [{status}]")                if not result["passed"] and args.block_on_failure:            sys.exit(1)        except ValueError as e:        print(f"Gate failed: {e}")        sys.exit(1)if __name__ == "__main__":    main()

The Evaluation Checklist

At launch (non-negotiable):

  • Synthetic eval dataset generated using RAGAS TestsetGenerator: minimum 100 samples, balanced across simple, reasoning, and multi-context question types
  • Baseline scores established: context recall, context precision, faithfulness, answer relevancy
  • Context recall baseline documented: if below 0.8, diagnose and fix retrieval before shipping
  • Thresholds set per domain: customer-facing systems warrant faithfulness >= 0.85; internal tools 0.70 acceptable
  • Golden dataset saved: this is the regression test suite for all future changes

On every pipeline change:

  • CI/CD regression gate configured: any change to chunking, embedding model, retrieval config, or reranker triggers eval
  • Hard gates set on context recall and context precision: failures block deployment
  • Soft gate on answer relevancy: warning, not blocking
  • Score delta logged: track direction of change across deployments, not just absolute pass/fail

Continuously in production:

  • Production sample evaluation running: faithfulness and answer relevancy on 5% of live queries, hourly or daily
  • Retrieval drift monitor configured: reranker top-1 score distribution tracked; baseline set at launch; alerts on mean drop > 0.05
  • Drift alert triggers full golden-dataset evaluation: automated, not manual
  • Context recall on golden dataset re-run monthly: or whenever knowledge base has major updates

When metrics degrade:

  • Start with context recall: if below threshold, retrieval is the problem - do not debug generation
  • Use the diagnostic flowchart: context recall → context precision → faithfulness → answer relevancy in order
  • Trace failures back to the pipeline layer: Chunking Debt (Part 2), Semantic Compression Loss (Part 3), Precision Gap (Part 4)
  • Fix upstream, re-evaluate before deploying

Where the Series Stands

Five parts in, the diagnostic vocabulary covers every layer of the RAG pipeline:

  • The Retrieval Tax (Part 1): wrong retrieval strategy per query - measured by nDCG@10 before and after routing
  • Chunking Debt (Part 2): bad chunking decisions compounding - signals as low context recall at evaluation time
  • Semantic Compression Loss (Part 3): domain-misaligned embeddings - signals as retrieval drift on specialized terminology
  • The Precision Gap (Part 4): bi-encoder ranking error - measured as the delta between pre- and post-reranking context precision
  • The Evals Blind Spot (Part 5 - this article): measuring the wrong things - closed by adding retrieval-layer metrics and continuous monitoring

Each named concept is a failure mode at a specific layer. Each has a metric that detects it. Each has a fix in the relevant part of this series.

Part 6 covers what happens when you wrap all five layers in an agentic loop: the cost and latency implications of multi-step retrieval, and why agentic RAG systems require their own evaluation and cost governance layer that is distinct from single-pass evaluation.

References

Retrieval Augmented Generation

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:

Comments