The Problem: Traditional LLM Debugging Focuses on the Wrong Layer
Your agent generates a wrong answer in production. Your first instinct is to examine the prompt. You look at conversation history, check if instructions were clear, inspect the model's reasoning chain. You spend hours tweaking prompts, adding examples, adjusting temperature. The problem persists.
The actual issue: the agent was working with stale customer data from a misconfigured MCP server. The prompt was fine. The model reasoning was fine. The context was wrong. But you spent your time debugging the model layer when the failure happened in the infrastructure layer.
This is the observability gap in MCP-based systems. Traditional LLM debugging assumes context is correct and focuses on prompts, model outputs, and reasoning chains. MCP changes the architecture fundamentally—context becomes dynamic infrastructure assembled from distributed sources. When something goes wrong, the failure is usually in context assembly: wrong version fetched, cache served stale data, provider timeout returned partial results, or resource permissions changed.
Most teams apply traditional logging approaches to MCP servers. They log requests and responses. They capture prompt templates. They record model outputs. None of this helps debug context assembly failures because context assembly happens before prompts are constructed, in infrastructure that prompt logs don't capture.
The symptoms are subtle. An agent occasionally returns outdated information—turns out one MCP server's cache has a 10-minute TTL while others use 1 minute, creating version skew. An agent inconsistently accesses certain resources—permissions check in the MCP layer sometimes fails due to race conditions that aren't logged. An agent's responses degraded gradually over weeks—context providers silently started returning less relevant data as schemas evolved.
Traditional metrics don't catch these issues. Request counts, latency percentiles, error rates—all normal. The problem is data quality at the context layer, not API health at the request layer. You need observability specifically for context assembly: what context was assembled, from which providers, at what versions, with what filters, and how it differed from previous requests.
The Mental Model: Context Assembly as Critical Path, Not Background Infrastructure
Stop thinking of MCP servers as data APIs you log like HTTP endpoints. Start thinking of context assembly as the critical path you instrument like database query plans.
When a database query is slow, you don't just log "query executed in 5 seconds." You examine the query plan: which indexes used, how many rows scanned, where the cost concentrated. Context assembly needs the same detailed instrumentation.
The key abstraction: context assembly is a multi-stage pipeline with observable state at each stage.
Stage 1: Request routing. Agent requests context. MCP layer routes to appropriate providers. Observable: which providers selected, why others excluded, what routing logic applied.
Stage 2: Provider execution. Each provider fetches data. Observable: latency per provider, data volume fetched, cache hits/misses, errors or timeouts.
Stage 3: Context aggregation. Results from multiple providers merge into unified context. Observable: how much data from each source, any conflicts or duplicates, aggregation logic applied.
Stage 4: Context versioning. Assembled context gets version identifier. Observable: version hash, what changed from previous version, dependency versions.
Stage 5: Context delivery. Final context provided to agent. Observable: total size, token count, what was included vs. truncated.
Traditional logging captures stage 1 and 5. Production debugging requires visibility into all five stages, especially 2-4 where failures happen silently.
The debugging insight: context diffs reveal changes that cause behavior changes.
When agent behavior changes between requests, the first question isn't "what's different about the prompt?" It's "what's different about the assembled context?" Context diffs show exactly what changed: which resources updated, which providers returned different data, which version hashes changed.
This is like Git for context. Instead of debugging by re-running requests and guessing what changed, you diff context versions and see precisely what's different. "Customer data was v123 last request, v124 this request. Diff shows email changed. That's why agent generated different email template."
The drift insight: context quality degrades gradually through provider evolution.
Prompt drift—where prompts slowly diverge from optimal as models update—is well-known. Context drift is worse and harder to detect. Providers evolve: schemas change, data sources add fields, relevance algorithms update. Context quality degrades gradually. The agent doesn't suddenly fail; it slowly gets worse.
Traditional monitoring misses this. You're tracking error rates and latency, both stable. What's changing is semantic quality of assembled context. You need metrics that detect drift: context size trends, provider version changes, schema evolution, data distribution shifts.
Architecture: Observable Context Assembly Pipeline
MCP observability architecture instruments every stage of context assembly.
Component Responsibilities
Request Router logs which providers were selected and why. This captures routing logic decisions that affect what context gets assembled.
Provider Execution Logger instruments each provider: latency, data volume, cache behavior, errors. This is where most context assembly failures surface.
Context Aggregator Logger records how data from multiple providers combines: conflicts resolved, duplicates removed, priority applied.
Version Calculator computes content hash of assembled context. This enables detecting when context changed and enables diffs.
Context Diff Engine compares current context version with previous versions. This is the key to debugging behavior changes.
Observability Store persists all logs with structured data, queryable for debugging and analysis.
What Gets Logged
Traditional approach logs: request ID, timestamp, success/failure, latency.
MCP observability logs:
- Request context: agent ID, query, routing decision
- Provider execution: which providers called, order, parallelization
- Provider results: data size, cache status, version hash, latency
- Aggregation logic: merge strategy, conflicts, filtering
- Context version: hash, size, token estimate, dependencies
- Context diff: what changed from previous version
- Delivery: truncation applied, final size, compression
This is verbose. That's the point. You can't debug context assembly without detailed instrumentation.
Implementation: Building Observable MCP Infrastructure
Layer 1: Structured Context Logging
Every context assembly operation emits structured events.
from typing import Dict, List, Any, Optionalfrom dataclasses import dataclass, asdictfrom datetime import datetimeimport hashlibimport json@dataclassclass ProviderExecutionLog: """ Log entry for individual provider execution. This is the granular data you need for debugging. """ provider_name: str request_id: str started_at: datetime completed_at: datetime latency_ms: float # Data metrics data_size_bytes: int record_count: int # Cache behavior cache_hit: bool cache_key: Optional[str] # Version info data_version: str provider_version: str # Errors error: Optional[str] partial_result: bool def to_dict(self) -> Dict[str, Any]: """Convert to dictionary for structured logging""" return { **asdict(self), "started_at": self.started_at.isoformat(), "completed_at": self.completed_at.isoformat() }@dataclassclass ContextAssemblyLog: """ Complete log of context assembly operation. Links all provider executions together. """ request_id: str agent_id: str query: str # Routing selected_providers: List[str] routing_reason: Dict[str, str] # Provider results provider_logs: List[ProviderExecutionLog] # Aggregation total_data_bytes: int total_records: int conflicts_resolved: int duplicates_removed: int # Versioning context_version: str context_hash: str # Delivery final_size_bytes: int truncation_applied: bool token_estimate: int # Timing total_latency_ms: float def to_dict(self) -> Dict[str, Any]: """Convert to structured log format""" return { "request_id": self.request_id, "agent_id": self.agent_id, "query": self.query, "routing": { "selected_providers": self.selected_providers, "reasons": self.routing_reason }, "providers": [p.to_dict() for p in self.provider_logs], "aggregation": { "total_data_bytes": self.total_data_bytes, "total_records": self.total_records, "conflicts_resolved": self.conflicts_resolved, "duplicates_removed": self.duplicates_removed }, "versioning": { "version": self.context_version, "hash": self.context_hash }, "delivery": { "final_size_bytes": self.final_size_bytes, "truncation_applied": self.truncation_applied, "token_estimate": self.token_estimate }, "latency_ms": self.total_latency_ms }class ContextLogger: """ Logger specifically for context assembly. Not general-purpose—specialized for MCP debugging. """ def __init__(self, storage_backend): self.storage = storage_backend def log_assembly(self, log_entry: ContextAssemblyLog): """ Log complete context assembly operation. """ self.storage.write({ "event_type": "context_assembly", "timestamp": datetime.utcnow().isoformat(), **log_entry.to_dict() }) def query_by_request(self, request_id: str) -> Optional[Dict]: """Retrieve full assembly log for debugging""" return self.storage.query_one({ "request_id": request_id, "event_type": "context_assembly" }) def query_by_context_version( self, context_hash: str ) -> List[Dict]: """Find all requests that used specific context version""" return self.storage.query_many({ "versioning.hash": context_hash, "event_type": "context_assembly" })Production considerations:
Structured logging is non-negotiable. JSON format enables programmatic querying. Free-form text logs are useless for context debugging.
Every log entry includes request_id. This enables correlation across distributed providers and aggregation stages.
Provider-level logs are essential. When debugging slow context assembly, you need to know which provider was slow, not just aggregate latency.
Version hashes enable reproducibility. Given a context hash, you can reconstruct exactly what context was assembled.
Layer 2: Context Versioning and Diff Engine
Track what changed between context versions.
import hashlibimport jsonfrom typing import Dict, Any, List, Optionalfrom difflib import unified_diffclass ContextVersioner: """ Version context and enable diffs between versions. This is how you debug "what changed?" """ def __init__(self): self.versions: Dict[str, Dict[str, Any]] = {} def compute_version( self, context: Dict[str, Any], metadata: Dict[str, Any] ) -> str: """ Compute content-based hash for context. Same content = same hash, enabling change detection. """ # Serialize context deterministically serialized = json.dumps(context, sort_keys=True) # Hash the content hash_obj = hashlib.sha256(serialized.encode()) version_hash = hash_obj.hexdigest()[:16] # Store version for later diffing self.versions[version_hash] = { "content": context, "metadata": metadata, "created_at": datetime.utcnow().isoformat() } return version_hash def diff_versions( self, old_version: str, new_version: str ) -> Dict[str, Any]: """ Compute diff between two context versions. Returns structured diff showing what changed. """ if old_version not in self.versions or new_version not in self.versions: return {"error": "Version not found"} old_content = self.versions[old_version]["content"] new_content = self.versions[new_version]["content"] # Compute structured diff diff_result = { "old_version": old_version, "new_version": new_version, "changes": {} } # Find added, removed, modified keys old_keys = set(old_content.keys()) new_keys = set(new_content.keys()) # Added keys added = new_keys - old_keys if added: diff_result["changes"]["added"] = { k: new_content[k] for k in added } # Removed keys removed = old_keys - new_keys if removed: diff_result["changes"]["removed"] = { k: old_content[k] for k in removed } # Modified keys common = old_keys & new_keys modified = {} for key in common: if old_content[key] != new_content[key]: modified[key] = { "old": old_content[key], "new": new_content[key] } if modified: diff_result["changes"]["modified"] = modified return diff_result def text_diff( self, old_version: str, new_version: str ) -> str: """ Generate human-readable text diff. Useful for debugging in logs. """ if old_version not in self.versions or new_version not in self.versions: return "Version not found" old_content = self.versions[old_version]["content"] new_content = self.versions[new_version]["content"] # Convert to formatted strings old_str = json.dumps(old_content, indent=2, sort_keys=True) new_str = json.dumps(new_content, indent=2, sort_keys=True) # Generate unified diff diff = unified_diff( old_str.splitlines(keepends=True), new_str.splitlines(keepends=True), fromfile=f"version_{old_version}", tofile=f"version_{new_version}" ) return "".join(diff)class ContextDriftDetector: """ Detect gradual drift in context quality. """ def __init__(self): self.metrics_history: List[Dict[str, Any]] = [] def record_metrics( self, context_hash: str, metrics: Dict[str, Any] ): """ Record metrics for this context version. Metrics to track: size, token count, provider versions. """ self.metrics_history.append({ "timestamp": datetime.utcnow().isoformat(), "context_hash": context_hash, **metrics }) def detect_drift( self, metric_name: str, window_size: int = 100, threshold_percent: float = 20.0 ) -> Optional[Dict[str, Any]]: """ Detect if metric has drifted beyond threshold. Returns drift alert if detected, None otherwise. """ if len(self.metrics_history) < window_size: return None recent = self.metrics_history[-window_size:] # Extract metric values values = [ m.get(metric_name, 0) for m in recent if metric_name in m ] if not values: return None # Compute baseline (first half of window) baseline_values = values[:window_size // 2] recent_values = values[window_size // 2:] baseline_avg = sum(baseline_values) / len(baseline_values) recent_avg = sum(recent_values) / len(recent_values) # Calculate drift if baseline_avg == 0: return None drift_percent = ((recent_avg - baseline_avg) / baseline_avg) * 100 if abs(drift_percent) > threshold_percent: return { "metric": metric_name, "baseline_avg": baseline_avg, "recent_avg": recent_avg, "drift_percent": drift_percent, "threshold": threshold_percent, "window_size": window_size } return NoneProduction considerations:
Content hashing enables reproducibility. Given a hash, you retrieve exact context that was used. This is critical for debugging non-deterministic failures.
Diffs are structured, not just text. You can programmatically analyze what changed: which fields added/removed/modified.
Text diffs are human-readable. When debugging manually, you want to see side-by-side what changed.
Drift detection uses statistical baselines. Sudden changes trigger alerts. Gradual drift gets caught by comparing recent metrics to baseline.
Layer 3: Provider Version Tracking
Track provider versions and schema evolution.
from typing import Dict, List, Anyfrom datetime import datetimeclass ProviderVersionRegistry: """ Track provider versions and schema changes. Critical for understanding context evolution. """ def __init__(self): self.versions: Dict[str, List[Dict[str, Any]]] = {} def register_version( self, provider_name: str, version: str, schema: Dict[str, Any], changelog: str ): """ Register new provider version with schema. """ if provider_name not in self.versions: self.versions[provider_name] = [] self.versions[provider_name].append({ "version": version, "schema": schema, "changelog": changelog, "registered_at": datetime.utcnow().isoformat() }) def get_schema_diff( self, provider_name: str, old_version: str, new_version: str ) -> Dict[str, Any]: """ Compare schemas between provider versions. Shows what fields were added/removed/changed. """ provider_versions = self.versions.get(provider_name, []) old_schema = None new_schema = None for v in provider_versions: if v["version"] == old_version: old_schema = v["schema"] if v["version"] == new_version: new_schema = v["schema"] if not old_schema or not new_schema: return {"error": "Version not found"} # Compute schema diff old_fields = set(old_schema.get("fields", {}).keys()) new_fields = set(new_schema.get("fields", {}).keys()) return { "provider": provider_name, "old_version": old_version, "new_version": new_version, "added_fields": list(new_fields - old_fields), "removed_fields": list(old_fields - new_fields), "common_fields": list(old_fields & new_fields) } def detect_breaking_changes( self, provider_name: str, old_version: str, new_version: str ) -> List[str]: """ Identify breaking changes between versions. Breaking: removed fields, changed types. """ schema_diff = self.get_schema_diff( provider_name, old_version, new_version ) breaking_changes = [] # Removed fields are breaking if schema_diff.get("removed_fields"): breaking_changes.append( f"Removed fields: {', '.join(schema_diff['removed_fields'])}" ) # Check type changes in common fields provider_versions = self.versions.get(provider_name, []) old_schema = next( (v["schema"] for v in provider_versions if v["version"] == old_version), None ) new_schema = next( (v["schema"] for v in provider_versions if v["version"] == new_version), None ) if old_schema and new_schema: for field in schema_diff.get("common_fields", []): old_type = old_schema.get("fields", {}).get(field, {}).get("type") new_type = new_schema.get("fields", {}).get(field, {}).get("type") if old_type != new_type: breaking_changes.append( f"Field '{field}' type changed: {old_type} -> {new_type}" ) return breaking_changesProduction considerations:
Schema versioning is separate from data versioning. Provider can return same data with different schema structure.
Breaking change detection prevents silent failures. When provider removes a field that agents depend on, you get alerts.
Changelogs are human-readable documentation. When debugging context issues, you need to understand what changed and why.
Layer 4: Observability Dashboards
Query logs to understand context assembly patterns.
from typing import List, Dict, Any, Optionalfrom datetime import datetime, timedeltaclass ContextObservabilityQueries: """ Queries for common debugging scenarios. """ def __init__(self, storage_backend): self.storage = storage_backend def find_slow_providers( self, time_range_hours: int = 24, latency_threshold_ms: float = 1000.0 ) -> List[Dict[str, Any]]: """ Identify providers that are slow. Critical for performance debugging. """ since = datetime.utcnow() - timedelta(hours=time_range_hours) results = self.storage.query_many({ "timestamp": {"$gte": since.isoformat()}, "event_type": "context_assembly" }) # Aggregate latency by provider provider_latencies = {} for result in results: for provider_log in result.get("providers", []): name = provider_log["provider_name"] latency = provider_log["latency_ms"] if name not in provider_latencies: provider_latencies[name] = [] provider_latencies[name].append(latency) # Calculate stats slow_providers = [] for provider, latencies in provider_latencies.items(): avg_latency = sum(latencies) / len(latencies) p95_latency = sorted(latencies)[int(len(latencies) * 0.95)] if avg_latency > latency_threshold_ms: slow_providers.append({ "provider": provider, "avg_latency_ms": avg_latency, "p95_latency_ms": p95_latency, "sample_count": len(latencies) }) return sorted(slow_providers, key=lambda x: x["avg_latency_ms"], reverse=True) def find_context_changes( self, agent_id: str, time_range_hours: int = 24 ) -> List[Dict[str, Any]]: """ Find when context changed for specific agent. Useful for debugging behavior changes. """ since = datetime.utcnow() - timedelta(hours=time_range_hours) results = self.storage.query_many({ "agent_id": agent_id, "timestamp": {"$gte": since.isoformat()}, "event_type": "context_assembly" }) # Sort by timestamp results.sort(key=lambda x: x["timestamp"]) # Find version changes changes = [] prev_hash = None for result in results: current_hash = result["versioning"]["hash"] if prev_hash and current_hash != prev_hash: changes.append({ "timestamp": result["timestamp"], "request_id": result["request_id"], "old_version": prev_hash, "new_version": current_hash, "query": result["query"] }) prev_hash = current_hash return changes def analyze_cache_effectiveness( self, time_range_hours: int = 24 ) -> Dict[str, Any]: """ Calculate cache hit rates by provider. """ since = datetime.utcnow() - timedelta(hours=time_range_hours) results = self.storage.query_many({ "timestamp": {"$gte": since.isoformat()}, "event_type": "context_assembly" }) cache_stats = {} for result in results: for provider_log in result.get("providers", []): name = provider_log["provider_name"] if name not in cache_stats: cache_stats[name] = {"hits": 0, "misses": 0} if provider_log.get("cache_hit"): cache_stats[name]["hits"] += 1 else: cache_stats[name]["misses"] += 1 # Calculate hit rates for provider, stats in cache_stats.items(): total = stats["hits"] + stats["misses"] stats["hit_rate"] = (stats["hits"] / total * 100) if total > 0 else 0 stats["total_requests"] = total return cache_statsProduction considerations:
Pre-built queries for common scenarios. Don't make engineers write custom queries for every debugging session.
Time-based analysis shows trends. Performance degradation over time indicates drift or scaling issues.
Cache effectiveness analysis identifies optimization opportunities. Low hit rates mean expensive re-fetching.
Pitfalls & Failure Modes
Logging Prompts Instead of Context
Teams log final prompts sent to models but not the context that was assembled to construct those prompts.
Symptom: Can't debug why agent behavior changed. Prompts look identical but outputs differ. Missing visibility into what context changed between requests.
Why it happens: Traditional LLM debugging focuses on prompts. Teams apply same patterns to MCP systems.
Detection: Try debugging a context assembly issue. If you can't answer "what context was used" without re-running the request, logging is insufficient.
Prevention: Log context assembly pipeline: routing decisions, provider executions, aggregation, versions. Prompts are downstream artifacts, not root causes.
No Version Tracking
Teams assemble context but don't version it, making it impossible to detect changes or compare versions.
Symptom: Agent behavior changes but you can't identify what context changed. "It worked yesterday" debugging with no way to compare yesterday's context to today's.
Why it happens: Versioning seems like overhead. Teams skip it initially and never add it.
Detection: Ask "what context was used for request X?" If answer requires re-executing providers, you have no versioning.
Prevention: Content-hash every assembled context. Store versions for diffing. Treat context like code—version controlled and diffable.
Provider-Level Logging Missing
Teams log aggregate context assembly latency but not per-provider metrics.
Symptom: Know context assembly is slow but can't identify which provider. Debug by trial and error, disabling providers until slowness goes away.
Why it happens: Aggregate metrics are simpler to implement. Per-provider instrumentation requires more code.
Detection: Slow context assembly in production. Can you identify which provider in <1 minute without code changes? If not, logging is too coarse.
Prevention: Every provider execution gets logged individually: latency, data size, cache status, errors. Aggregate metrics are computed from these.
Ignoring Drift Until Crisis
Teams monitor for sudden failures but not gradual degradation in context quality.
Symptom: Agent quality slowly degrades over weeks. Users report increasing inaccuracy. No single incident, just slow decline.
Why it happens: Traditional monitoring focuses on availability and latency. Quality metrics require domain-specific instrumentation.
Detection: User complaints about quality without corresponding incident. No alerts fired but users unhappy.
Prevention: Track context quality metrics: size trends, provider version changes, schema evolution. Alert on drift, not just outages.
Not Correlating Context with Outcomes
Teams log context assembly and agent outputs separately, making it impossible to correlate context quality with output quality.
Symptom: Can see context was assembled and can see agent output, but can't determine if poor output was due to poor context.
Why it happens: Context logging and agent logging are separate systems with no correlation.
Detection: Trying to debug poor agent output. Can you retrieve the exact context that was used? If requires manual correlation via timestamps, linkage is missing.
Prevention: Every log entry includes request_id. Context logs and agent output logs share request_id for correlation.
Summary & Next Steps
MCP systems require observability specifically for context assembly, not just traditional request/response logging. Debug context, not prompts. Version assembled context for diffing. Track provider-level metrics for performance. Detect drift in context quality before it impacts users.
The key insights: context assembly is a multi-stage pipeline that must be instrumented at every stage. Content hashing enables versioning and diffing. Provider versions and schemas evolve, causing drift. Correlation between context and outcomes enables root cause analysis.
Start building MCP observability:
This week: Add structured logging to context assembly. Log routing decisions, provider executions, aggregation, versions. Make logs queryable with request_id.
Next sprint: Implement context versioning with content hashing. Build diff engine to compare versions. When debugging behavior changes, diff context versions first.
Within month: Deploy drift detection for context quality metrics. Track size trends, provider versions, schema changes. Alert on gradual degradation before users notice.
Test your observability: simulate provider failures, cache staleness, version skew. Verify you can identify root cause from logs alone without re-executing requests. If debugging requires reproduction, observability is insufficient.
The goal is making context assembly transparent and debuggable. When something goes wrong, you should be able to answer: which providers were called, what data they returned, how context was aggregated, what version was produced, and how it differed from previous requests. Without these answers, you're debugging blind.
Build observability that assumes context failures are the norm, not exceptions. Because in production MCP systems, they are.