← Back to Guides
7

Series

RAG Engineering in Production· Part 7

GuideFor: AI Engineers, ML Engineers, Platform Engineers, AI Systems Architects

Why Your RAG Knowledge Base Is Lying About What It Knows

The vector index is a snapshot of your documents at ingestion time. Semantic similarity has no relationship to temporal validity. A document from 18 months ago can score 0.94 cosine similarity and still be completely wrong today - and nothing in the standard RAG pipeline raises a flag.

ai-engineeringretrieval-augmented-generation
#rag#index-freshness#staleness#incremental-indexing#cdc#streaming-rag#production-ai#llm-infrastructure

A user asked the internal developer portal about the SSO configuration process. The system returned a confident, detailed, step-by-step answer. The steps described an authentication flow that had been deprecated fourteen months earlier when the company migrated to a new identity provider. The old documentation was still in the index. The new documentation was also in the index. The retriever returned the old version because that page happened to use slightly more similar vocabulary to the query.

No error was thrown. No staleness signal was logged. The RAGAS evaluation had been passing for months because it was built against the same query set at the same time as the old documentation was current. The evaluation kept passing as the documentation decayed. The answer was faithful to what was retrieved. It was not faithful to what was true.

This is the Staleness Gap: the time between when a document changes in the source system and when that change is reflected in the vector index. During the Staleness Gap, every query that reaches the affected document is answered from outdated context - confidently, without qualification, with no downstream signal that anything is wrong.

The Staleness Gap is the failure mode that the first six parts of this series did not address. Parts 1 through 6 covered what happens at ingestion time (chunking, embedding) and at query time (retrieval strategy, reranking, evaluation, agentic governance). This part covers what happens between those two moments, across the full lifecycle of every document in your corpus.

The thesis is specific: the vector index is not a live mirror of your knowledge base. It is a point-in-time snapshot that ages from the moment ingestion completes. Most teams treat it as the former and architect it as the latter - and the gap between those two assumptions is where production failures accumulate.

Why the Standard Evaluation Stack Cannot Catch Staleness

Before diagnosing the failure, it is worth understanding why the Evals Blind Spot from Part 5 makes staleness invisible. The RAGAS metrics (context recall, context precision, faithfulness, answer relevancy) all share a common assumption: the retrieved documents are currently true. Faithfulness measures whether the answer is supported by the retrieved context. A faithfulness score of 1.0 means every claim in the answer traces to a retrieved document. It says nothing about whether that document is still accurate.

If your golden evaluation dataset was built when your documents were fresh, it will keep passing even as the underlying documents decay. The retriever still returns semantically similar content - the old SSO guide is just as similar to "how do I configure SSO" as the new one. The LLM still generates faithful, coherent, well-structured answers from that stale context. Faithfulness: 0.94. Context recall: 0.88. Answer relevancy: 0.91. Production outcome: user follows instructions for a system that no longer exists.

This is why staleness requires its own detection layer on top of the Part 5 evaluation framework. RAGAS measures correctness given retrieval. Staleness detection measures whether retrieval is working from current ground truth.

Named Concept: The Staleness Gap

The Staleness Gap has three dimensions that compound:

The freshness window. The time between when a document changes and when the change is reflected in the index. In a nightly batch architecture, this is up to 24 hours. In an hourly batch architecture, up to 60 minutes. In a streaming architecture with CDC, seconds to minutes. Every query during this window is answered from the pre-change version.

The accumulation rate. As the corpus grows, the proportion of stale documents at any given time increases if the re-indexing strategy does not scale proportionally. A system handling 1,000 documents might maintain sub-hour freshness. At 100,000 documents, the same architecture produces 12-hour staleness windows. At 1 million documents, multi-day delays.

The detection void. Standard RAG evaluation cannot distinguish between a faithful answer from current context and a faithful answer from stale context. The evaluation stack from Part 5 passes on stale content because it was calibrated on fresh content. Without a separate staleness detection layer, degradation accumulates invisibly - the same failure mode as the Evals Blind Spot, one layer deeper.

The Staleness Gap also interacts with prior failure modes from this series. Chunking Debt from Part 2 accumulates as new document formats enter the corpus that the original chunking strategy handles poorly - each new document type added to a live corpus inherits the original chunking configuration. The Evals Blind Spot from Part 5 compounds because the golden dataset itself goes stale: the ground-truth answers were valid when the dataset was built, and the evaluation keeps passing long after the source documents change.

The Four Document Lifecycle Events That Break RAG Silently

Staleness is not a single failure mode. Four distinct events in the document lifecycle each produce different failure signatures.

Event 1: Update - The Silent Version Conflict

A document is modified in the source system. The old version remains in the vector index until the next re-indexing run. During the Staleness Gap, the retriever returns whichever version has higher cosine similarity to the query - which is not deterministically the newer one. Old and new versions of the same document may co-exist in the index for the duration of the freshness window.

At scale, with hourly or nightly re-indexing, this means that for any rapidly changing document - pricing, policy, API documentation - there is a predictable window during every update cycle where the wrong version surfaces.

Event 2: Deletion - The Orphaned Vector

A document is removed from the source system. The embedding vectors remain in the index as orphans. The retriever continues to return them. The LLM receives chunks that reference a document that no longer exists. In the best case, the document URL is dead and the user gets a 404. In the worst case, the document has been superseded by a policy change and the old content is actively misleading.

Document deletion is operationally harder than update for most vector databases. Hierarchical Navigable Small World (HNSW) indexes mark deleted vectors as tombstones rather than removing them - true removal requires index rebuilding. Incremental upsert handles additions cleanly; deletions require explicit tombstone management or periodic full re-index.

Event 3: Supersession - The Version Collision

A new version of a document is published while the old version remains in the index. Unlike an update (which replaces), supersession adds a new document while the old one persists. Regulatory filings are a classic example: the 2024 filing and the 2025 filing are both in the index. A query about current requirements may retrieve either, depending on which happens to be more similar to the query text.

The retriever has no concept of document authority or version precedence. It returns the document that is semantically closest to the query. When two versions of the same document are semantically close to each other, the retriever's choice between them is effectively noise.

Event 4: Temporal Invalidity - The Confidence Cliff

Some documents do not require an update event to become wrong. They become wrong because the world changed while they stayed the same. Pricing documents, rate limit documentation, regulatory compliance guides, market data summaries. These have implicit expiration windows that the index has no mechanism to model.

Every document in your knowledge base has an implicit shelf life. Core values documentation: stable for years. API rate limits: stable for weeks to months. Regulatory filings: superseded quarterly. Market data: stale within minutes. The vector index treats all of these equally. A document about 2023 pricing that has not been edited has the same indexing status as a document about architectural principles that has not changed in two years. Both have a freshness timestamp of their last ingestion date. The temporal invalidity of the pricing document is not represented anywhere in the index.

The Wrong Way: Treat the Index as a Live Mirror

code
# Wrong way: build once at launch, re-index on a fixed schedule# This is the architecture most RAG systems ship with.import scheduleimport timefrom langchain_community.vectorstores import Chromafrom langchain_openai import OpenAIEmbeddings# Built at launch - complete, correct, current.vectorstore = Chroma(    collection_name="knowledge_base",    embedding_function=OpenAIEmbeddings(),)def nightly_full_reindex(document_loader, chunker):    """    Full corpus re-index at midnight.    Problems:        1. Freshness window: up to 24 hours.       A policy updated at 2 PM is still returning the old version at 11 PM.       For compliance or pricing content, 24 hours is a long time to be wrong.        2. Cost does not scale with change rate.       Re-indexing 50,000 documents at midnight costs the same whether       1 document changed or 10,000 changed.       At $300-650 per 10M tokens (Voyage/OpenAI), nightly full re-index       of a large corpus is a meaningful recurring cost with no proportionality.        3. Deletions are not handled.       Documents removed from the source since the last re-index       will be removed from the new index - but only if the re-index       starts from a clean slate. If the pipeline appends rather than replaces,       deleted documents persist as orphans indefinitely.        4. The evaluation stack does not detect this.       RAGAS faithfulness measures grounding in retrieved context.       A faithful answer from a stale document passes evaluation.       The golden dataset itself goes stale alongside the corpus.    """    docs = document_loader.load_all()    chunks = chunker.split_documents(docs)    vectorstore.add_documents(chunks)# Run at midnight every day regardless of how much changedschedule.every().day.at("00:00").do(nightly_full_reindex)while True:    schedule.run_pending()    time.sleep(60)

The Three Ingestion Architectures and When Each Is Correct

Architecture 1: Full Batch Re-index with Atomic Swap

Build the new index completely offline, validate it against a benchmark query set, then atomically swap an alias so all queries transition to the new index instantaneously. The old index is retained for rollback. No query ever sees a partial index.

This is the correct architecture for corpora where:

  • The total corpus size makes streaming operationally impractical
  • Freshness windows of 12-24 hours are acceptable for the use case
  • Index correctness is more important than freshness latency
code
import hashlibfrom datetime import datetimefrom langchain_community.vectorstores import Qdrantfrom langchain_openai import OpenAIEmbeddingsfrom qdrant_client import QdrantClientfrom qdrant_client.models import Distance, VectorParamsclient = QdrantClient(url="http://localhost:6333")embeddings = OpenAIEmbeddings()def build_index_offline(    documents: list,    chunker,    collection_name: str,) -> str:    """    Build a complete new index in a staging collection.    Do not touch the live collection until validation passes.        Returns the staging collection name.    """    timestamp = datetime.utcnow().strftime("%Y%m%d_%H%M%S")    staging_name = f"{collection_name}_staging_{timestamp}"        # Create staging collection    client.create_collection(        collection_name=staging_name,        vectors_config=VectorParams(            size=1536,  # text-embedding-3-small dimensions            distance=Distance.COSINE,        ),    )        # Chunk and embed into staging    chunks = chunker.split_documents(documents)    staging_store = Qdrant(        client=client,        collection_name=staging_name,        embeddings=embeddings,    )    staging_store.add_documents(chunks)        return staging_namedef validate_staging_index(    staging_name: str,    golden_queries: list[dict],    recall_threshold: float = 0.80,) -> bool:    """    Run the golden dataset against the staging index.    Only promote to production if recall threshold is met.        This is the same gate as the CI/CD regression gate from Part 5,    applied to index promotion rather than code deployment.    """    staging_store = Qdrant(        client=client,        collection_name=staging_name,        embeddings=embeddings,    )        hits = 0    for item in golden_queries:        results = staging_store.similarity_search(item["query"], k=5)        retrieved_texts = [r.page_content for r in results]        if any(item["expected_content"] in text for text in retrieved_texts):            hits += 1        recall = hits / len(golden_queries)    print(f"Staging index recall@5: {recall:.3f} (threshold: {recall_threshold})")    return recall >= recall_thresholddef promote_staging_to_production(    staging_name: str,    production_alias: str,    previous_collection: str | None,) -> None:    """    Atomically swap the production alias to the new index.    Previous collection retained for rollback window (e.g., 48 hours).        Queries pointing to the alias transition instantly.    No partial index is ever served.    """    # Point alias at new collection (atomic in Qdrant)    client.update_collection_aliases(        change_aliases_operations=[            {                "create_alias": {                    "collection_name": staging_name,                    "alias_name": production_alias,                }            }        ]    )        print(f"Promoted {staging_name} to {production_alias}")        # Schedule deletion of previous collection after rollback window    if previous_collection:        print(            f"Previous collection {previous_collection} retained for rollback. "            f"Delete after 48h if no issues."        )

Architecture 2: Incremental Upsert with Hash-Based Change Detection

Only re-embed documents that have changed since the last indexing run. Use content hashing (MD5 or SHA-256) to detect changes without fetching and comparing full document content.

This is the correct architecture when:

  • Freshness windows of 1-4 hours are acceptable
  • The change rate is low relative to corpus size (<5% daily)
  • Operational simplicity is valued over sub-minute freshness
code
import hashlibimport jsonfrom pathlib import Pathfrom datetime import datetimeHASH_STORE_PATH = Path("index_state/document_hashes.json")def compute_content_hash(content: str) -> str:    """SHA-256 hash of document content. Changes iff content changes."""    return hashlib.sha256(content.encode()).hexdigest()def load_hash_store() -> dict[str, str]:    """Load the persisted doc_id -> content_hash mapping."""    if HASH_STORE_PATH.exists():        with open(HASH_STORE_PATH) as f:            return json.load(f)    return {}def save_hash_store(store: dict[str, str]) -> None:    HASH_STORE_PATH.parent.mkdir(parents=True, exist_ok=True)    with open(HASH_STORE_PATH, "w") as f:        json.dump(store, f)def incremental_update(    current_documents: list[dict],  # [{id, content, metadata}]    chunker,    vectorstore,) -> dict:    """    Update only changed documents. Skip unchanged ones.    Handle deletions by comparing current doc IDs against the hash store.        Returns a summary of what changed.    """    hash_store = load_hash_store()    current_ids = {doc["id"] for doc in current_documents}        updated = []    unchanged = []    deleted_ids = [        doc_id for doc_id in hash_store        if doc_id not in current_ids    ]        for doc in current_documents:        doc_id = doc["id"]        content_hash = compute_content_hash(doc["content"])                if hash_store.get(doc_id) == content_hash:            unchanged.append(doc_id)            continue  # Skip: content has not changed                # Content changed or new document - re-embed        chunks = chunker.split_text(doc["content"])                # Remove old chunks for this document first        # (prevents duplicate vectors from stale version)        try:            vectorstore.delete(filter={"doc_id": doc_id})        except Exception:            pass  # Document may not exist yet if new                # Add new chunks        vectorstore.add_texts(            texts=chunks,            metadatas=[{                "doc_id": doc_id,                "indexed_at": datetime.utcnow().isoformat(),                "content_hash": content_hash,                **doc.get("metadata", {}),            }] * len(chunks),        )                hash_store[doc_id] = content_hash        updated.append(doc_id)        # Handle deletions - remove orphaned vectors    for doc_id in deleted_ids:        try:            vectorstore.delete(filter={"doc_id": doc_id})            del hash_store[doc_id]        except Exception as e:            print(f"Warning: could not delete vectors for {doc_id}: {e}")        save_hash_store(hash_store)        return {        "updated": len(updated),        "unchanged": len(unchanged),        "deleted": len(deleted_ids),        "updated_ids": updated,        "deleted_ids": deleted_ids,    }

Architecture 3: Streaming CDC Pipeline

Connect directly to the document source system via Change Data Capture. Every insert, update, or delete event in the source database triggers an immediate re-embedding and index update. Sub-minute freshness. Three times the operational complexity of batch re-indexing.

This is the correct architecture when:

  • The use case cannot tolerate >5 minute staleness (compliance, pricing, live documentation)
  • The team has the operational capacity to run and monitor a streaming pipeline
  • The change rate is high enough that incremental batch is also expensive
code
# Conceptual CDC pipeline - implementation depends on your source system# This example uses PostgreSQL logical replication + asyncpg# Depends on: compute_content_hash, load_hash_store, save_hash_store# defined in the incremental upsert block above.import asyncioimport asyncpgimport jsonfrom dataclasses import dataclassfrom typing import Literal@dataclassclass DocumentChangeEvent:    event_type: Literal["insert", "update", "delete"]    doc_id: str    content: str | None  # None for deletes    metadata: dictasync def process_change_event(    event: DocumentChangeEvent,    chunker,    vectorstore,    hash_store: dict,) -> None:    """    Process a single document change event from the CDC stream.    Called immediately when a change occurs in the source system.    Freshness: seconds to minutes vs. hours for batch.    """    if event.event_type == "delete":        # Remove all vectors for this document        vectorstore.delete(filter={"doc_id": event.doc_id})        hash_store.pop(event.doc_id, None)        print(f"Deleted vectors for {event.doc_id}")        return        # Insert or update: re-embed the document    content_hash = compute_content_hash(event.content)        # Skip if content unchanged (can happen with metadata-only updates)    if hash_store.get(event.doc_id) == content_hash:        return        # Remove stale vectors before adding new ones    try:        vectorstore.delete(filter={"doc_id": event.doc_id})    except Exception:        pass        chunks = chunker.split_text(event.content)        from datetime import datetime    vectorstore.add_texts(        texts=chunks,        metadatas=[{            "doc_id": event.doc_id,            "event_type": event.event_type,            "indexed_at": datetime.utcnow().isoformat(),            "content_hash": content_hash,            **event.metadata,        }] * len(chunks),    )        hash_store[event.doc_id] = content_hash    print(f"Re-indexed {event.doc_id} ({event.event_type})")async def run_cdc_listener(    postgres_dsn: str,    chunker,    vectorstore,) -> None:    """    Listen to PostgreSQL logical replication stream.    Every change in the source triggers immediate re-embedding.        Operational requirements:    - PostgreSQL WAL level must be 'logical'    - Replication slot must be created    - Pipeline must be monitored for lag (CDC lag = Staleness Gap)        Alternative sources: Kafka (document topic), webhook from CMS,    S3 event notifications, Confluence/Notion webhooks.    """    conn = await asyncpg.connect(postgres_dsn)    hash_store = load_hash_store()        async def handle_replication_message(connection, pid, channel, payload):        event = DocumentChangeEvent(**json.loads(payload))        await process_change_event(event, chunker, vectorstore, hash_store)        save_hash_store(hash_store)        await conn.add_listener("document_changes", handle_replication_message)        print("CDC listener active. Waiting for document changes...")    while True:        await asyncio.sleep(1)

Adding Document Shelf Life to Metadata

The three architectures above address the operational Staleness Gap - the delay between a change event and the index update. They do not address temporal invalidity: documents that become wrong without a change event because the world moved on.

The solution is to classify documents by shelf life at ingestion time and surface the classification in retrieval.

code
from enum import Enumfrom datetime import datetime, timedeltaclass ShelfLife(Enum):    PERMANENT = "permanent"      # Core values, architecture docs: years    ANNUAL = "annual"            # Annual reports, legal filings: 12 months    QUARTERLY = "quarterly"      # Product roadmaps, pricing: 3 months    MONTHLY = "monthly"          # API docs, integration guides: 30 days    WEEKLY = "weekly"            # Release notes, changelogs: 7 days    VOLATILE = "volatile"        # Market data, live status: hoursSHELF_LIFE_DAYS = {    ShelfLife.PERMANENT: 730,    ShelfLife.ANNUAL: 365,    ShelfLife.QUARTERLY: 90,    ShelfLife.MONTHLY: 30,    ShelfLife.WEEKLY: 7,    ShelfLife.VOLATILE: 0,  # Never cache; always verify freshness}def classify_shelf_life(document_metadata: dict) -> ShelfLife:    """    Classify document shelf life based on source, type, and content signals.    Override with explicit metadata field if present in source system.    """    if doc_type := document_metadata.get("doc_type"):        type_map = {            "pricing": ShelfLife.QUARTERLY,            "api_reference": ShelfLife.MONTHLY,            "release_notes": ShelfLife.WEEKLY,            "architecture": ShelfLife.PERMANENT,            "compliance_policy": ShelfLife.ANNUAL,            "market_data": ShelfLife.VOLATILE,        }        if doc_type in type_map:            return type_map[doc_type]        # Fallback: classify by source system    source = document_metadata.get("source", "")    if "confluence" in source:        return ShelfLife.MONTHLY    if "github" in source:        return ShelfLife.WEEKLY    if "legal" in source or "compliance" in source:        return ShelfLife.ANNUAL        return ShelfLife.MONTHLY  # Safe defaultdef ingest_with_shelf_life(    document: dict,    chunker,    vectorstore,) -> None:    """    Ingest a document with shelf life metadata attached to every chunk.        The shelf life enables two things at query time:    1. Freshness filtering: exclude documents past their expiry    2. Staleness warnings: flag documents approaching expiry in responses    """    shelf_life = classify_shelf_life(document.get("metadata", {}))    max_days = SHELF_LIFE_DAYS[shelf_life]        indexed_at = datetime.utcnow()    expires_at = (        indexed_at + timedelta(days=max_days)        if max_days > 0        else indexed_at  # Volatile: expires immediately    )        chunks = chunker.split_text(document["content"])        vectorstore.add_texts(        texts=chunks,        metadatas=[{            "doc_id": document["id"],            "shelf_life": shelf_life.value,            "indexed_at": indexed_at.isoformat(),            "expires_at": expires_at.isoformat(),            "days_until_expiry": max_days,        }] * len(chunks),    )def retrieve_with_freshness_filter(    query: str,    vectorstore,    k: int = 10,    exclude_expired: bool = True,) -> list:    """    Retrieve with metadata filter excluding expired documents.        The freshness filter is applied at the vector database level,    not post-hoc - expired documents are not retrieved at all,    which prevents them from displacing fresh content in the    re-ranked results.    """    now = datetime.utcnow().isoformat()        filter_dict = {"expires_at": {"$gt": now}} if exclude_expired else {}        return vectorstore.similarity_search(        query,        k=k,        filter=filter_dict,    )

Measuring and Monitoring the Staleness Gap

The retrieval drift monitor introduced in Part 5 (reranker top-1 score distribution) is the early warning system for staleness. A downward shift in that distribution signals that the index is diverging from the query distribution - which can be caused by corpus growth, chunking strategy changes, embedding model drift, or staleness accumulation.

Beyond drift monitoring, add two production metrics specific to staleness:

code
from datetime import datetime, timezoneimport statisticsdef compute_staleness_metrics(    vectorstore,    sample_size: int = 1000,) -> dict:    """    Sample the index metadata and compute staleness distribution.        Implementation note: metadata sampling is vector-database-specific.    - Chroma: vectorstore.get(limit=sample_size, include=["metadatas"])    - Qdrant: client.scroll(collection_name, limit=sample_size, with_payload=True)    - Pinecone: requires a fetch by known IDs; use a separate metadata store        The example below uses a database-agnostic wrapper.    Adapt the `fetch_metadata_sample()` call to your vector DB.        Run on a schedule (daily) and alert on:    - stale_fraction above 5% for compliance content    - mean_age_days exceeding the corpus's expected shelf life    - orphan_count above zero (docs deleted from source but still in index)        This is a metadata scan, not a retrieval quality measurement.    It catches index state problems that RAGAS cannot detect.    """    # Database-agnostic metadata fetch:    # Replace with your vector DB's scroll/list/get API    def fetch_metadata_sample(limit: int) -> list[dict]:        """        Chroma: return vectorstore.get(limit=limit, include=["metadatas"])["metadatas"]        Qdrant:            records, _ = client.scroll(                collection_name="knowledge_base",                limit=limit,                with_payload=True,            )            return [r.payload for r in records]        """        raise NotImplementedError("Implement for your vector database")        metadatas = fetch_metadata_sample(sample_size)        now = datetime.now(timezone.utc)    ages_days = []    stale_count = 0    expiry_counts = {shelf.value: 0 for shelf in ShelfLife}        for meta in metadatas:        if not meta:            continue                if indexed_at := meta.get("indexed_at"):            try:                indexed_dt = datetime.fromisoformat(indexed_at).replace(                    tzinfo=timezone.utc                )                age = (now - indexed_dt).days                ages_days.append(age)            except ValueError:                pass                if expires_at := meta.get("expires_at"):            try:                expires_dt = datetime.fromisoformat(expires_at).replace(                    tzinfo=timezone.utc                )                if expires_dt < now:                    stale_count += 1            except ValueError:                pass                if shelf := meta.get("shelf_life"):            expiry_counts[shelf] = expiry_counts.get(shelf, 0) + 1        return {        "sample_size": len(metadatas),        "mean_age_days": statistics.mean(ages_days) if ages_days else 0,        "max_age_days": max(ages_days) if ages_days else 0,        "stale_fraction": stale_count / max(len(ages_days), 1),        "stale_count": stale_count,        "shelf_life_distribution": expiry_counts,        "measured_at": now.isoformat(),    }

The Staleness Gap Decision Guide

mermaid
flowchart TD
    A[Choose ingestion architecture] --> B{What is the acceptable\nfreshness window?}
    B -- "24 hours acceptable" --> C{Corpus size?}
    C -- "Under 100K docs" --> D[Full batch re-index\nwith atomic alias swap\nNightly, validated against\ngolden dataset]
    C -- "Over 100K docs" --> E[Hash-based incremental\nupsert + nightly\norphan cleanup pass]
    B -- "1-4 hours acceptable" --> F[Hash-based incremental\nupsert on hourly schedule\nDeletion tracking required]
    B -- "Under 5 minutes" --> G{Operational capacity\nfor streaming pipeline?}
    G -- Yes --> H[CDC streaming pipeline\nPostgres WAL or Kafka\nSub-minute freshness]
    G -- No --> I[Incremental upsert\nevery 5-15 minutes\nAccept residual gap]
    D --> J[Add shelf life metadata\nto all chunks at ingestion]
    E --> J
    F --> J
    H --> J
    I --> J
    J --> K[Add freshness filter\nto retrieval queries]
    K --> L[Add staleness metrics\nto production monitoring\nAlert on stale fraction above 5pct]

    style A fill:#4A90E2,color:#fff
    style B fill:#7B68EE,color:#fff
    style C fill:#7B68EE,color:#fff
    style G fill:#7B68EE,color:#fff
    style D fill:#6BCF7F,color:#fff
    style E fill:#6BCF7F,color:#fff
    style F fill:#6BCF7F,color:#fff
    style H fill:#6BCF7F,color:#fff
    style I fill:#FFD93D,color:#333
    style J fill:#98D8C8,color:#333
    style K fill:#98D8C8,color:#333
    style L fill:#98D8C8,color:#333

The Pre-Ingestion Staleness Checklist

Architecture selection (before first ingestion):

  • Freshness SLA defined per document tier: what is the maximum acceptable Staleness Gap for each content category?
  • Corpus size estimated at 12-month horizon: will batch re-indexing still be affordable at that scale?
  • Change rate estimated: what fraction of documents change daily/weekly?
  • Ingestion architecture selected based on the above: batch, incremental, or streaming
  • Deletion handling strategy defined: how are documents removed from the source reflected in the index?

Metadata (at ingestion time):

  • Shelf life classification added to every chunk: PERMANENT, ANNUAL, QUARTERLY, MONTHLY, WEEKLY, or VOLATILE
  • indexed_at timestamp stored per chunk in metadata
  • expires_at computed and stored based on shelf life classification
  • content_hash stored per document for change detection
  • doc_id stored per chunk for targeted deletion and upsert

At query time:

  • Freshness filter applied: expired chunks excluded from retrieval results
  • VOLATILE tier documents bypassed from cache entirely
  • Staleness warning surfaced in response when best-match document is approaching expiry

Production monitoring:

  • Staleness metrics running on schedule: stale_fraction, mean_age_days, orphan_count
  • Alert threshold configured: stale fraction above 5% for compliance content, 10% for general content
  • Retrieval drift monitor from Part 5 already running: staleness is a leading cause of drift signal
  • Golden dataset updated on a cadence that matches the corpus change rate - stale golden dataset is a separate failure mode

How the Staleness Gap Compounds the Prior Six Failure Modes

Every failure mode in this series is worsened by an aging index.

The Retrieval Tax from Part 1 compounds because the routing logic was calibrated at ingestion time. If your SQL RAG agent for structured data was built against the schema at launch, and the schema has since changed, the routing logic routes to the wrong backend or generates invalid queries against a schema that no longer matches.

Chunking Debt from Part 2 accumulates as new document formats enter the corpus after initial ingestion. Documents added after launch inherit the original chunking configuration. If those documents have different structural properties than the launch corpus, they accrue Chunking Debt from day one of their ingestion.

Semantic Compression Loss from Part 3 worsens as new terminology enters the domain after the embedding model was selected. An embedding model calibrated against your domain's 2024 vocabulary will exhibit increasing compression loss as 2025 and 2026 terminology is added to the corpus without a corresponding model update.

The Evals Blind Spot from Part 5 deepens as the golden dataset ages. The evaluation framework keeps passing on stale content because the ground-truth answers were valid when the dataset was built. Both the corpus and the evaluation dataset need freshness governance.

The Orchestration Overhead from Part 6 increases because agentic loops retrieve more aggressively when first-pass results are insufficient. A stale index produces more insufficient first-pass results. More iterations. Higher cost. The Staleness Gap is a multiplier on the Orchestration Overhead.

Treating document freshness as a first-class architectural concern from day one is cheaper than adding it later. The ingestion architecture, the metadata schema, the shelf life classification, the monitoring stack - all of these are harder to retrofit than to build correctly at the start.

References

Retrieval Augmented Generation

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:

Comments