Updated 30 October 2026: rewritten around a failure the article's own code reproduces, replacing an unsourced incident story; all code runs offline against LangGraph 1.2.18 and the OpenTelemetry Python SDK 1.45.2. 28 September 2026: corrected gen_ai.system to gen_ai.provider.name, moved custom metrics out of the spec's gen_ai.* namespace, and dropped the claim that LangSmith Fleet is an identity layer for coded fleets.
An invoice agent's model returns the amount as "4,200.00". The validator rejects anything float() cannot parse, so the agent extracts again, gets "4,200.00" again, and loops. I built that agent in LangGraph and instrumented it the way the OpenTelemetry GenAI semantic conventions describe, to see what standard agent observability shows while an agent fails.
Two hundred node executions in, I read the telemetry. One hundred chat spans, all successful. One hundred execute_tool spans, all successful. Zero spans with an error status. 41,200 input tokens counted. And on the two metrics the spec defines for exactly this situation, gen_ai.invoke_agent.duration and gen_ai.invoke_agent.tool_calls: zero data points.
Only LangGraph's default recursion limit ended the run, by raising GraphRecursionError after 10,007 node executions. By then the agent had made 5,004 model calls and spent 2,061,648 input tokens. That exception was the first agent-level signal the instrumentation produced. With a real model at one second per call, the dashboard would have been green for more than 80 minutes.
Why Agent Fleets Look Healthy While They Fail
I want to take apart a belief most platform teams hold: "We have tracing set up for every agent, so we have observability."
They do not. They have logging with a better user interface. Traces answer "what happened in this request?" after the request is over. Governing a fleet needs an answer to a different question - "is any agent doing the wrong thing right now?" - and the loop above shows that the agent-level span and the spec's agent metrics cannot answer it while the agent is still running.
So the thesis of this part, and the foundation of the series, is this: a fleet of agents needs a control plane, and its first component is a three-tier metric layer whose behavioral tier is measured in flight, per step. The agent-level span and the spec's agent metrics are recorded when an agent finishes, and fleet alerting runs on metrics, so an agent that fails by never finishing is invisible to both.
I call the coverage difference between what your monitoring captures and that full three-tier surface the Telemetry Surface Gap, and the rest of this series refers back to it. Policy enforcement in Part 2 has its own version of it, and the audit trail in Part 6 closes it from the compliance side.
Here is what each tier reported after 200 node executions of the looping run:
grid-rows: 1
grid-gap: 24
classes: {
ok: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
none: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
t1: "Tier 1: is it running?" {
label.near: top-center
grid-columns: 1
style: {fill: "#F4F6F6"; stroke: "#95A5A6"}
a: "Process up" {class: ok}
b: "No exception raised" {class: ok}
c: "Nothing for Tier 1\nto flag" {class: ok}
}
t2: "Tier 2: is it executing correctly?" {
label.near: top-center
grid-columns: 1
style: {fill: "#F4F6F6"; stroke: "#95A5A6"}
a: "100 chat spans, all OK" {class: ok}
b: "100 tool spans, all OK" {class: ok}
c: "0 retries, 0 error spans" {class: ok}
}
t3: "Tier 3: is it doing the right thing?" {
label.near: top-center
grid-columns: 1
style: {fill: "#F4F6F6"; stroke: "#95A5A6"}
a: "invoke_agent.duration:\nno data" {class: none}
b: "invoke_agent.tool_calls:\nno data" {class: none}
c: "no per-step signal" {class: none}
}
Nothing in the two green columns is lying. Every individual operation did succeed. The validator ran correctly and returned a validation failure as data, which is what validators are supposed to do. Retry counters stayed at zero because the loop is graph control flow, not a retry. What failed is the whole agent run, and that is the one level with no data.
Research supports the idea that this is not an exotic case. The MAST taxonomy (Cemri et al., arXiv 2503.13657, v3) was built from annotated benchmark traces of seven open-source multi-agent frameworks; it contains no production fleet data. In those traces, step repetition (FM-1.3) accounts for 15.7% of failures and "unaware of termination conditions" (FM-1.5) for another 12.4%.
The Three-Tier Metric Model for Agent Fleets
Splitting agent telemetry into layers is not new, and I am not claiming it. Galileo organises agent logging by granularity: spans, traces and sessions. AWS's Well-Architected Agentic AI Lens (June 2026) separates performance metrics from behavioral ones and warns against "monitoring only performance metrics without behavioral metrics". My tiers split along a different line: each one answers a different question, and each one has a characteristic way of looking healthy while the tier above it fails.
grid-rows: 4
grid-columns: 3
grid-gap: 10
h1: "Tier" {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#FFFFFF"; bold: true}}
h2: "Question it answers" {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#FFFFFF"; bold: true}}
h3: "Looks healthy while..." {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#FFFFFF"; bold: true}}
r1: "Tier 1\nInfrastructure" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
q1: "Is the system running?\nCPU, memory, GPU, HTTP errors" {style: {fill: "#F4F6F6"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
m1: "every agent makes\nwrong decisions at full speed" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
r2: "Tier 2\nHarness" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
q2: "Is each step executing correctly?\ntokens, step latency, tool results,\nretries, gate trips, validation" {style: {fill: "#F4F6F6"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
m2: "every step succeeds and\nthe run as a whole goes nowhere" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
r3: "Tier 3\nBehavioral" {style: {fill: "#C2185B"; stroke: "#8E1243"; font-color: "#FFFFFF"}}
q3: "Is the agent doing the right thing?\ngoal completion, quality score,\nloops, schema compliance" {style: {fill: "#F4F6F6"; stroke: "#95A5A6"; font-color: "#2C2C2A"}}
m3: "it is only measured\nwhen a run ends" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
Read the right-hand column first, because it is the useful part of the grid: each tier's blind spot is the next tier's job.
Tier 1, infrastructure, is what your existing Application Performance Monitoring (APM) stack already covers. It is necessary, and it was designed for services whose control flow you can read in the code. An agent at 100% uptime can be wrong at every step.
Tier 2, harness, covers the execution primitives from the Harness Engineering series: token cost per node, step latency, tool call results, retry counts, trip rates from the Gated Execution layer, failure rates from the Validation Layer, and circuit breaker trips from Retry and Fallback infrastructure. For one agent these show up in LangSmith without effort. For a fleet of 10 to 50 agents they need consistent names and attributes so one query covers every agent type.
Tier 3, behavioral, measures the distance between technical success and the right outcome: goal completion rate, tool-parameter accuracy, an LLM-as-judge quality score, output schema compliance, and repeated reasoning states. Record the quality score as fleet.agent.quality_score: a histogram with the same bounded attributes as every other fleet metric, one value per judged output. Part 4 of this series gates canary promotion on it. That only works if the tier exists.
That last cell in the grid, "it is only measured when a run ends", is the subject of the next section, and it is the part of the Telemetry Surface Gap that most teams with a Tier 3 still have.
Exit-Recorded Telemetry: Why Agent Metrics Miss a Running Loop
Look at where the agent-level signals in the GenAI semantic conventions come from. gen_ai.invoke_agent.duration is a histogram of "end-to-end duration of a single in-process agent invocation". gen_ai.invoke_agent.tool_calls counts the tool calls an agent makes "during a single invocation". Even the invoke_agent span itself is exported by the SDK's span processor only when the span ends. All three have a value only after the invocation is over.
I call this Exit-Recorded Telemetry: a signal recorded when the unit of work it describes exits, which makes it blind for that unit's entire run. The unit matters. Here it is the agent invocation, and the failure lives at that level. A duration is exit-recorded by definition. So is a per-invocation count, a final outcome attribute, and an LLM-as-judge score computed over the finished output. These signals are correct and useful. They are also structurally unable to report on a run that has not exited, and the run that has not exited is the one burning tokens.
The general effect is known from tracing long batch jobs. A Jaeger maintainer's answer to a user who could not see traces "until the parent Span has ended" was that the UI "only works with spans that have well-defined duration". What is specific to agents is where the effect lands. The GenAI spec defines its agent-level metrics per invocation, so they are exit-recorded by construction. And the agent invocation is exactly the level at which a loop fails, while every step inside it succeeds. The audit table below turns that into a method: for each agent-level signal, find the moment its value comes into existence, and give every exit-recorded one an in-flight partner.
Its opposite is an in-flight signal: one recorded on every step while the run is still going. Token counters are recorded after each model call, so relative to the agent invocation they are in flight, which is why the looping run above did show 41,200 input tokens after 200 node executions. They just carry no information about whether those tokens were well spent. The same problem can appear one level down: a single model call that hangs never ends its chat span. "In flight" for a metric still means within one export interval plus one scrape interval, which is 60 seconds by default for the Python SDK's periodic reader. That is slow for a request, and fast for a loop that would otherwise run for an hour.
Here is the agent from the opening. It uses spec names and attributes throughout, including the gen_ai.client.inference.usage.* token counters that replaced gen_ai.client.token.usage on the spec's main branch on 22 September 2026. A fake chat model stands in for the provider, so the code runs offline:
# Block 1: an invoice agent instrumented the way the GenAI spec describes.# Runs offline: a fake chat model stands in for the provider.import jsonimport timefrom typing import TypedDictfrom langchain_core.language_models.fake_chat_models import GenericFakeChatModelfrom langchain_core.messages import AIMessagefrom langgraph.graph import END, StateGraphfrom opentelemetry.sdk.metrics import MeterProviderfrom opentelemetry.sdk.metrics.export import InMemoryMetricReaderfrom opentelemetry.sdk.trace import TracerProviderfrom opentelemetry.sdk.trace.export import SimpleSpanProcessorfrom opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporterfrom opentelemetry.trace import SpanKindspans = InMemorySpanExporter()tracer_provider = TracerProvider()tracer_provider.add_span_processor(SimpleSpanProcessor(spans))tracer = tracer_provider.get_tracer("invoice-fleet")reader = InMemoryMetricReader()meter = MeterProvider(metric_readers=[reader]).get_meter("invoice-fleet")# Spec names (open-telemetry/semantic-conventions-genai, main, Oct 2026)agent_duration = meter.create_histogram("gen_ai.invoke_agent.duration", unit="s")agent_tool_calls = meter.create_histogram( "gen_ai.invoke_agent.tool_calls", unit="{tool_call}")input_tokens = meter.create_counter( "gen_ai.client.inference.usage.input_tokens", unit="{token}")output_tokens = meter.create_counter( "gen_ai.client.inference.usage.output_tokens", unit="{token}")PROVIDER, MODEL = "openai", "invoice-model"def fake_replies(): # The model keeps the thousands separator in the amount on every # attempt, so the validator rejects every attempt. while True: yield AIMessage( content='{"vendor": "Acme", "amount": "4,200.00"}', usage_metadata={ "input_tokens": 412, "output_tokens": 18, "total_tokens": 430, }, )llm = GenericFakeChatModel(messages=fake_replies())class InvoiceState(TypedDict): invoice_text: str fields: dict errors: list[str] tool_calls: intdef extract(state: InvoiceState) -> dict: prompt = f"Extract vendor and amount as JSON.\n{state['invoice_text']}" if state["errors"]: prompt += "\nFix these errors: " + "; ".join(state["errors"]) with tracer.start_as_current_span( f"chat {MODEL}", kind=SpanKind.CLIENT, attributes={ "gen_ai.operation.name": "chat", "gen_ai.provider.name": PROVIDER, "gen_ai.request.model": MODEL, }, ) as span: reply = llm.invoke(prompt) usage = reply.usage_metadata span.set_attribute("gen_ai.usage.input_tokens", usage["input_tokens"]) span.set_attribute("gen_ai.usage.output_tokens", usage["output_tokens"]) token_attrs = { "gen_ai.operation.name": "chat", "gen_ai.provider.name": PROVIDER, "gen_ai.request.model": MODEL, "gen_ai.token.modality": "text", } input_tokens.add(usage["input_tokens"], token_attrs) output_tokens.add(usage["output_tokens"], token_attrs) return {"fields": json.loads(reply.content)}def validate(state: InvoiceState) -> dict: with tracer.start_as_current_span( "execute_tool validate_invoice", attributes={ "gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": "validate_invoice", }, ): errors = [] try: float(state["fields"]["amount"]) except ValueError: errors.append("amount must be a number") return {"errors": errors, "tool_calls": state["tool_calls"] + 1}def route(state: InvoiceState) -> str: return "extract" if state["errors"] else ENDgraph = StateGraph(InvoiceState)graph.add_node("extract", extract)graph.add_node("validate", validate)graph.set_entry_point("extract")graph.add_edge("extract", "validate")graph.add_conditional_edges("validate", route, ["extract", END])invoice_agent = graph.compile()Token cost comes from usage_metadata on the returned AIMessage, which is where LangChain chat models report what the provider billed. Do not estimate it from the length of the reply text.
Block 2 wraps the graph in an invoke_agent span and records the two spec agent metrics in a finally clause, the same place any instrumentation library has to put them. It counts one step per node execution, reads the telemetry once after 200 of them while the agent is still looping, and once after it exits:
# Block 2: run the agent and look at the telemetry while it is still running.# One "step" here is one node execution (stream_mode="updates").from opentelemetry.trace import StatusCodedef points(name: str) -> list: data = reader.get_metrics_data() if data is None: return [] for rm in data.resource_metrics: for sm in rm.scope_metrics: for metric in sm.metrics: if metric.name == name: return list(metric.data.data_points) return []def snapshot(label: str) -> None: finished = spans.get_finished_spans() by_op = {} for s in finished: op = s.attributes.get("gen_ai.operation.name") by_op[op] = by_op.get(op, 0) + 1 failed = sum(1 for s in finished if s.status.status_code == StatusCode.ERROR) tokens = sum(p.value for p in points("gen_ai.client.inference.usage.input_tokens")) duration = len(points("gen_ai.invoke_agent.duration")) tool_calls = len(points("gen_ai.invoke_agent.tool_calls")) print(f"{label}:") print(f" finished spans by operation: {by_op}") print(f" spans with error status: {failed}") print(f" input tokens counted: {tokens:,}") print(f" invoke_agent.duration points: {duration}") print(f" invoke_agent.tool_calls points: {tool_calls}")def run_agent(state: InvoiceState, watch_at: int) -> None: agent_attrs = {"gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": "invoice_extractor"} with tracer.start_as_current_span( "invoke_agent invoice_extractor", attributes=agent_attrs ) as span: start = time.monotonic() tool_calls = 0 try: for step, update in enumerate( invoice_agent.stream(state, stream_mode="updates"), start=1 ): for node_update in update.values(): tool_calls = node_update.get("tool_calls", tool_calls) if step == watch_at: snapshot(f"while running, after {step} node executions") except Exception as exc: # Recorded here for the demo; production code should re-raise. span.set_status(StatusCode.ERROR) span.set_attribute("error.type", type(exc).__name__) print(f"stopped by {type(exc).__name__}") finally: # Spec signals for the agent: recorded once, at exit. agent_duration.record(time.monotonic() - start, agent_attrs) agent_tool_calls.record(tool_calls, agent_attrs)start_state = {"invoice_text": "ACME LTD ... TOTAL 4,200.00 USD", "fields": {}, "errors": [], "tool_calls": 0}run_agent(start_state, watch_at=200)snapshot("after exit")Output, from a run against LangGraph 1.2.18, langchain-core 1.7.3 and opentelemetry-sdk 1.45.2:
while running, after 200 node executions: finished spans by operation: {'chat': 100, 'execute_tool': 100} spans with error status: 0 input tokens counted: 41,200 invoke_agent.duration points: 0 invoke_agent.tool_calls points: 0stopped by GraphRecursionErrorafter exit: finished spans by operation: {'chat': 5004, 'execute_tool': 5003, 'invoke_agent': 1} spans with error status: 1 input tokens counted: 2,061,648 invoke_agent.duration points: 1 invoke_agent.tool_calls points: 1The 10,007-step stop is LangGraph's default recursion limit in the installed 1.2.18 package. The documentation says 1,000, and early versions used 25, so check the number your own version and configuration actually apply.
As a timeline, with one row per signal and one column per point in the run, it looks like this:
grid-rows: 5
grid-columns: 5
grid-gap: 8
classes: {
head: {style: {fill: "#95A5A6"; stroke: "#6E7B7C"; font-color: "#FFFFFF"; bold: true}}
row: {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
live: {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
empty: {style: {fill: "#F4F6F6"; stroke: "#95A5A6"; font-color: "#2C2C2A"; stroke-dash: 4}}
exit: {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
h0: "Node executions" {class: head}
h1: "1" {class: head}
h2: "200" {class: head}
h3: "5,000" {class: head}
h4: "10,007 (exit)" {class: head}
a0: "chat + execute_tool\nspans" {class: row}
a1: "1 exported" {class: live}
a2: "200 exported, all OK" {class: live}
a3: "exported, all OK" {class: live}
a4: "10,007 exported" {class: live}
b0: "token counters" {class: row}
b1: "412" {class: live}
b2: "41,200" {class: live}
b3: "rising" {class: live}
b4: "2,061,648" {class: live}
c0: "invoke_agent span" {class: row}
c1: "open, not exported" {class: empty}
c2: "open, not exported" {class: empty}
c3: "open, not exported" {class: empty}
c4: "exported, ERROR" {class: exit}
d0: "invoke_agent.duration\n+ .tool_calls" {class: row}
d1: "no data" {class: empty}
d2: "no data" {class: empty}
d3: "no data" {class: empty}
d4: "first data point" {class: exit}
Both bottom rows stay empty for the first 10,006 node executions and fill in at the exception. That is the whole problem in one picture: the signals that describe the agent, as opposed to its individual steps, are recorded at exit, and the failure is a run that does not exit. Raising the recursion limit, as teams often do for legitimately long agents, makes the empty stretch longer.
Two objections deserve a direct answer. The first is that a bound would have stopped this loop. A cap of three repair attempts, or a recursion limit of 8, ends this run in seconds, and the exit-recorded signals then arrive almost at once. That is true, and the guard in Block 5 includes a step budget for exactly that reason. But agents that are legitimately long-running need generous bounds, and inside a generous bound the only agent-level signals the spec gives you are still recorded at exit. A bound is also only visible to the fleet if it emits a signal when it fires.
The second objection is that the child spans are in flight. After 200 node executions, 200 of them had been exported, all with one trace ID. A backend that can count spans per trace over a time window can see the loop. So the precise claim is narrower than "traces cannot see it". The agent-level span and the spec's agent metrics cannot see it, and most fleet alerting runs on metrics. Metrics cannot isolate one session without the cardinality problem in the next section.
This turns into an audit you can run on your own alerts. For each agent-level signal, ask one question: when does its value come into existence? If the answer is "when the run ends", the signal is exit-recorded, and it needs an in-flight partner:
| Agent-level signal | When its value exists | In-flight partner |
|---|---|---|
gen_ai.invoke_agent.duration | at exit | fleet.agent.oldest_open_session_age, a gauge per agent (Block 4) |
gen_ai.invoke_agent.tool_calls | at exit | fleet.agent.steps, counted per node execution (Block 5) |
| Goal completion | at exit | repeated rejections of the same kind (Block 5) |
| LLM-as-judge quality score | after the final output | judge every Nth intermediate output with the same rubric |
| Total spend on the root span | at exit | per-call token counters are already in flight; add a per-session budget check |
The oldest-open-session gauge is the direct replacement for the duration histogram. It reports one number per agent, the age of the longest-running invocation that has not exited, so it stays bounded and it rises during exactly the failure the histogram cannot see. The per-session budget check is the subject of Part 5.
A fleet-wide spend alert may eventually fire on the token counter. It cannot tell you which session is responsible, and the next section explains why adding that information to the metric is the wrong fix.
Wrong Way: Session IDs on Metrics and One Project per Agent
After a run like that, the first instinct is to make the Tier 2 metrics more specific. If the cost counter carried the session ID, the dashboard could show cost per session and the runaway would stand out. A second instinct, common in teams that started with LangSmith, is to give each agent its own tracing project so its traces stay readable.
Both make the fleet harder to see. Separate projects per agent - LANGSMITH_PROJECT=invoice-agent in one service and approval-agent in the next - mean no query crosses an agent handoff. To answer "what did one invoice cost across all three agents?", someone has to join trace IDs across projects by hand.
You can measure how the session-ID idea fails. By default, the OpenTelemetry metrics SDK caps each instrument at 2,000 attribute sets, and folds the rest into one overflow point:
# Block 3 (wrong way): put the session on every metric point so a# dashboard can "group by session".wrong_reader = InMemoryMetricReader()wrong_meter = MeterProvider(metric_readers=[wrong_reader]).get_meter("wrong")session_cost = wrong_meter.create_counter("fleet.agent.token_cost", unit="{USD}")for i in range(2500): session_cost.add(0.0129, {"gen_ai.agent.name": "invoice_extractor", "session.id": f"sess-{i:05d}"})cost_points = (wrong_reader.get_metrics_data().resource_metrics[0] .scope_metrics[0].metrics[0].data.data_points)overflow = [p for p in cost_points if p.attributes.get("otel.metric.overflow")]print(f"series kept: {len(cost_points) - len(overflow)}")print(f"overflow point attributes: {dict(overflow[0].attributes)}")print(f"sessions folded into it: {round(overflow[0].value / 0.0129)}")series kept: 2000overflow point attributes: {'otel.metric.overflow': True}sessions folded into it: 500direction: down
in: "2,500 sessions\neach a new session.id value" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
sdk: "OTel SDK aggregation\ncardinality limit: 2,000 per instrument" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
kept: "2,000 series\none per early session" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
lost: "1 overflow point: 500 sessions\notel.metric.overflow=true\nagent name dropped" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
in -> sdk: "1. add(cost, session.id=...)"
sdk -> kept: "2. first 2,000 sets"
sdk -> lost: "3. everything after"
Look at what the overflow point keeps: its only attribute is otel.metric.overflow, so the 500 newest sessions lost both their session ID and gen_ai.agent.name. The total is still correct. What is lost is attribution. With cumulative temporality, which is the default for the Python OTLP exporter, the limit holds for the life of the process, so on a busy service every session after the first 2,000 lands in that point. With delta temporality the SDK forgets old series after each export, so overflow needs 2,000 new sessions inside one interval. The newest sessions, which are the ones an incident is about, are exactly the ones you can no longer break down.
The SDK limit is the smaller cost. Every session ID that does reach the backend becomes a new time series, and on Prometheus that turns into memory and cost growth on every replica of every agent service, whatever the SDK does. Enforce the rule where it cannot be skipped: the Collector config later in this article deletes session attributes from every metric data point before export.
Right Way: Sessions on Spans, Bounded Metrics, Exemplars
Fix it by putting each attribute where its cardinality belongs. Session and identity go on spans, which are stored one per operation and are indexed for search. Metrics carry only attributes with a small, known set of values: agent name, agent version, fleet. To link a metric back to a session, use an exemplar, which the SDK attaches when a measurement is recorded inside a sampled span. The same block adds the gauge from the audit table: a small registry of open sessions, read by a callback each time metrics are collected.
# Block 4 (right way): identity and session on spans, bounded attributes# on metrics, and signals that are recorded while the agent runs.from dataclasses import dataclassfrom opentelemetry.metrics import Observation@dataclass(frozen=True)class AgentIdentity: name: str # gen_ai.agent.name version: str # fleet.agent.version (not on the spec's internal span) fleet: str # fleet.id - the spec has no fleet conceptdef span_attributes(agent: AgentIdentity, session_id: str) -> dict: """Everything a trace query needs, including the session.""" return { "gen_ai.operation.name": "invoke_agent", "gen_ai.agent.name": agent.name, "gen_ai.conversation.id": session_id, # spec's session key "fleet.agent.version": agent.version, "fleet.id": agent.fleet, }def metric_attributes(agent: AgentIdentity) -> dict: """Bounded on purpose: no session, no request, no user.""" return {"gen_ai.agent.name": agent.name, "fleet.agent.version": agent.version, "fleet.id": agent.fleet}# In-flight signals: recorded per node execution, while the agent runs.agent_steps = meter.create_counter("fleet.agent.steps", unit="{step}")reasoning_loops = meter.create_counter( "fleet.agent.reasoning_loops", unit="{repeat}")loop_halts = meter.create_counter("fleet.agent.loop_halts", unit="{halt}")# The in-flight partner of gen_ai.invoke_agent.duration: the age of the# oldest invocation that has not exited yet, one value per agent.open_sessions: dict[str, tuple[AgentIdentity, float]] = {}def oldest_open_session(options): oldest: dict[AgentIdentity, float] = {} now = time.monotonic() for agent, started in list(open_sessions.values()): oldest[agent] = max(oldest.get(agent, 0.0), now - started) for agent, age in oldest.items(): yield Observation(age, metric_attributes(agent))meter.create_observable_gauge( "fleet.agent.oldest_open_session_age", callbacks=[oldest_open_session], unit="s",)Three naming decisions in that block are deliberate.
For the session key, the block uses gen_ai.conversation.id, which the spec defines as the identifier "for a conversation (session, thread)". It is conditionally required on the in-process invoke_agent span. Your pipeline probably already carries a session_id in its state, and the later parts of this series do. That value is what goes into this attribute.
Agent version lives in fleet.agent.version. The spec has gen_ai.agent.version, but only on the client span for a remote agent, and a LangGraph agent running in your process gets the internal span, which does not list it. Using the spec name there would claim a conformance the span does not have.
Every metric the spec does not define lives under fleet.agent.*. A future spec release cannot collide with a namespace you own.
The gauge reads open_sessions at collection time, so it reports a value only while an invocation is open and reports nothing once every invocation has exited. Block 5 registers each session on entry and removes it in a finally clause.
grid-rows: 1
grid-gap: 60
span: "Span: one per session (high cardinality is fine)" {
label.near: top-center
grid-columns: 1
style: {fill: "#EAF2FB"; stroke: "#2C6FB0"}
s1: "gen_ai.agent.name = invoice_extractor" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
s2: "gen_ai.conversation.id = sess-00043" {style: {fill: "#C2185B"; stroke: "#8E1243"; font-color: "#FFFFFF"}}
s3: "fleet.agent.version = 1.4.0" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
s4: "fleet.id = invoice_processing" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
s5: "trace_id of sess-00043" {style: {fill: "#955D37"; stroke: "#6B4228"; font-color: "#FFFFFF"}}
}
metric: "Metric point: one per agent version (bounded)" {
label.near: top-center
grid-columns: 1
style: {fill: "#EEFAF0"; stroke: "#3F9E52"}
m1: "fleet.agent.reasoning_loops = 4" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
m2: "gen_ai.agent.name, fleet.agent.version,\nfleet.id" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
m3: "no session attribute" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
m4: "exemplar: a sampled trace,\nhere sess-00043's" {style: {fill: "#955D37"; stroke: "#6B4228"; font-color: "#FFFFFF"}}
}
metric.m4 -> span.s5: "exemplar link"
Those brown cells are the bridge. However many sessions run, the metric point stays bounded at three attributes, and its exemplar names a sampled trace that contributed to the measurement. The SDK keeps a small sample per collection interval. With many sessions, the exemplar points at whichever one was sampled, which may be a different session from the one you are investigating. Block 5's output shows exactly that. Exemplars also exist only for sampled spans, and they reach Grafana only if Prometheus scrapes OpenMetrics with exemplar storage turned on. For the reliable path from an alert to a session, query spans by fleet.agent.outcome or gen_ai.conversation.id. That is the session replay the rest of this series relies on: an agent run is a sequence of state transitions, and the unit you debug is the whole session.
Detecting Agent Reasoning Loops In Flight
fleet.agent.reasoning_loops is the in-flight Tier 3 signal that the looping run needed, and the guard below records it from inside the graph. It applies two rules before every model call. A step budget halts any run that reaches 40 node executions. A repeat rule hashes the validator's verdict, counts how often the same rejection comes back, and halts on the third. Both counters live in graph state, so each session carries its own. To make the test harder, this block swaps in a model that rewords the amount on every retry:
# Block 5: in-flight guards that run inside the graph, with their# counters kept in graph state so every session has its own.import hashlibimport itertoolsclass GuardedState(InvoiceState): steps: int verdicts: dict[str, int] halted: strdef counted(node_fn, agent: AgentIdentity): """Count every node execution, in state and as a metric.""" def node(state: dict) -> dict: agent_steps.add(1, metric_attributes(agent)) return {**node_fn(state), "steps": state["steps"] + 1} return nodedef verdict_key(errors: list[str]) -> str: # Key on the validator's verdict, not on the model's output: a real # model rewords its answer on every retry, the rejection stays the same. return hashlib.sha256(json.dumps(sorted(errors)).encode()).hexdigest()[:12]def loop_guard(node_fn, agent: AgentIdentity, max_repeats: int = 3, max_steps: int = 40): def node(state: dict) -> dict: attrs = metric_attributes(agent) if state["steps"] >= max_steps: loop_halts.add(1, {**attrs, "fleet.agent.halt_reason": "step_budget"}) return {"halted": f"step budget of {max_steps} reached"} verdicts = dict(state["verdicts"]) if state["errors"]: key = verdict_key(state["errors"]) verdicts[key] = verdicts.get(key, 0) + 1 if verdicts[key] > 1: reasoning_loops.add(1, attrs) if verdicts[key] >= max_repeats: loop_halts.add(1, {**attrs, "fleet.agent.halt_reason": "repeat"}) return {"halted": f"same rejection {verdicts[key]} times", "verdicts": verdicts} return {**node_fn(state), "verdicts": verdicts} return nodedef after_extract(state: GuardedState) -> str: # Route the halt from the guarded node itself. With `validate` # registered unwrapped, a router after it never saw `halted`: # LangGraph narrowed its input to the InvoiceState annotation. return END if state["halted"] else "validate"extractor = AgentIdentity("invoice_extractor", "1.4.0", "invoice_processing")def build_guarded(max_steps: int = 40): g = StateGraph(GuardedState) g.add_node("extract", counted( loop_guard(extract, extractor, max_steps=max_steps), extractor)) g.add_node("validate", counted(validate, extractor)) g.set_entry_point("extract") g.add_conditional_edges("extract", after_extract, ["validate", END]) g.add_conditional_edges("validate", route, ["extract", END]) return g.compile()def varied_replies(): # A model that rewords the amount on every retry. Each one still fails. for amount in itertools.cycle(["4,200.00", "$4,200.00", "4200.00 USD"]): yield AIMessage( content=json.dumps({"vendor": "Acme", "amount": amount}), usage_metadata={"input_tokens": 412, "output_tokens": 18, "total_tokens": 430}, )llm = GenericFakeChatModel(messages=varied_replies()) # replaces Block 1'sdef gauge_value() -> float: for p in points("fleet.agent.oldest_open_session_age"): return round(p.value, 3) return 0.0def run_session(session_id: str, agent) -> None: spans.clear() open_sessions[session_id] = (extractor, time.monotonic()) ages = [] try: with tracer.start_as_current_span( "invoke_agent invoice_extractor", attributes=span_attributes(extractor, session_id), ) as root: state = {**start_state, "steps": 0, "verdicts": {}, "halted": ""} halted = "" for step, update in enumerate( agent.stream(state, stream_mode="updates"), start=1 ): if step in (1, 3): ages.append(gauge_value()) # read while the run is open halted = update.get("extract", {}).get("halted", halted) root.set_attribute("fleet.agent.outcome", "halted_loop" if halted else "completed") finally: del open_sessions[session_id] chat = [s for s in spans.get_finished_spans() if s.name.startswith("chat")] print(f"{session_id}: halted ({halted}) after {len(chat)} model calls, " f"{sum(s.attributes['gen_ai.usage.input_tokens'] for s in chat):,} " f"input tokens") print(f" oldest_open_session_age rose while open: {0 < ages[0] < ages[1]}") session_traces[root.get_span_context().trace_id] = session_idsession_traces: dict[int, str] = {}guarded_agent = build_guarded()run_session("sess-00042", guarded_agent)run_session("sess-00043", guarded_agent)run_session("sess-00044", build_guarded(max_steps=4)) # budget fires firstdata = reader.get_metrics_data() # one collection, read oncemetrics_now = data.resource_metrics[0].scope_metrics[0].metricsprint("after exit, oldest_open_session_age reported:", any(m.name == "fleet.agent.oldest_open_session_age" for m in metrics_now))for metric in metrics_now: if metric.name in ("fleet.agent.steps", "fleet.agent.reasoning_loops"): p = metric.data.data_points[0] linked = {session_traces.get(e.trace_id) for e in p.exemplars} print(f"{metric.name} = {p.value}") print(f" attributes: {sorted(p.attributes)}") print(f" exemplars link to sessions: {sorted(linked)}")sess-00042: halted (same rejection 3 times) after 3 model calls, 1,236 input tokens oldest_open_session_age rose while open: Truesess-00043: halted (same rejection 3 times) after 3 model calls, 1,236 input tokens oldest_open_session_age rose while open: Truesess-00044: halted (step budget of 4 reached) after 2 model calls, 824 input tokens oldest_open_session_age rose while open: Trueafter exit, oldest_open_session_age reported: Falsefleet.agent.steps = 19 attributes: ['fleet.agent.version', 'fleet.id', 'gen_ai.agent.name'] exemplars link to sessions: ['sess-00044']fleet.agent.reasoning_loops = 4 attributes: ['fleet.agent.version', 'fleet.id', 'gen_ai.agent.name'] exemplars link to sessions: []The first two sessions ran through the same compiled graph and each stopped after three model calls, so the guard's state is per session. The third uses a graph with a step budget of 4. Its budget fires after two model calls, before any rejection repeats, which is the case a step budget exists for. In all three, the gauge rose between the first and third node execution and reported nothing once every session had exited. fleet.agent.steps = 19 is seven node executions for each of the first two sessions and five for the third. The exemplar on fleet.agent.steps points at sess-00044, the session recorded in the latest collection interval. fleet.agent.reasoning_loops has no exemplar at all, because the last session halted before it repeated. That is the sampling behavior described in the previous section.
grid-rows: 1
grid-gap: 40
before: "Exit-recorded only" {
label.near: top-center
direction: down
style: {fill: "#FDEDEC"; stroke: "#B03A2E"}
e: "extract" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
v: "validate" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
x: "GraphRecursionError\nat step 10,007" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
r: "5,004 model calls\n2,061,648 input tokens" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
e -> v: "fields"
v -> e: "errors (loop)"
v -> x: "limit"
x -> r
}
after: "In-flight guard" {
label.near: top-center
direction: down
style: {fill: "#EEFAF0"; stroke: "#3F9E52"}
g: "guard: step budget +\nrepeated rejection" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
e: "extract" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
v: "validate" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
h: "END: halted_loop\non 3rd same rejection" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
r: "3 model calls\n1,236 input tokens" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
g -> e: "new state"
e -> v
v -> g: "errors (loop)"
g -> h: "repeat >= 3"
h -> r
}
Same agent and same validator. The right-hand run even has the harder model, the one that rewords its answer on every retry. On the left, a framework safety limit stops an unbounded run after 5,004 calls. On the right, the agent's own Tier 3 signal stops it after three. Against an unbounded run, that is a token difference of roughly 1,700 to 1.
Four details in the code matter in production. First, the guard keys on the validator's verdict, not on the model's output. My first version hashed the extracted fields, and it worked only because the fake model returned identical bytes every time. A real model at any temperature above zero rewords its answer, every rewording produces a new hash, and that guard never fires. Second, the counters live in graph state rather than in the closure. A closure dict is shared by every session that runs through the compiled graph, and every session starts from the same empty state, so the third invoice processed would halt on its first step. State also survives a checkpoint resume. Third, the halt is routed from the guarded node itself. In an earlier version, validate was registered without the counted wrapper and the halt check sat in the router after it. LangGraph passed that node only the keys of its InvoiceState annotation, which has no halted key, so the router never saw the halt and the run went to 10,007 node executions. Deciding right after the node that sets halted removes the dependence on how later nodes are annotated. Fourth, a halt ends the run normally with span status OK. It is visible as fleet.agent.outcome on the span and as fleet.agent.loop_halts on the metrics. Error-rate alerts will not see it, so alert on the halt counter directly. A halt from inside the graph covers loops the agent can detect itself; stopping an agent from outside, mid-run, is the job of a kill switch.
Three alert rules cover the in-flight signals. The names are the ones the Collector's Prometheus exporter produces with its default suffixes, so check them against your /metrics endpoint before relying on them:
# Any loop halt in the fleet is worth a look.
sum by (gen_ai_agent_name) (increase(fleet_agent_loop_halts_total[10m])) > 0
# An invocation has been open longer than this agent ever should run.
# max, because every replica reports its own oldest session.
max by (gen_ai_agent_name) (fleet_agent_oldest_open_session_age_seconds) > 300
# The token counter disappeared: renamed upstream, or the exporter died.
absent(gen_ai_client_inference_usage_input_tokens_total)
Set the 300-second threshold per agent from its real runtime distribution; a research agent and an invoice extractor need very different numbers.
For loops, the AWS Agentic AI Lens recommends collecting "reasoning iteration counts" over two to four weeks and running anomaly detection on the baseline. That catches drift. It does not catch this loop, because a baseline is built from completed invocations, and the anomaly here is an invocation that never completes. A loop needs a hard per-session rule that is evaluated in flight. Anomaly detection belongs on top of that rule.
Routing Fleet Telemetry Through One OTel Collector
A guard on one agent closes the Telemetry Surface Gap for that agent. A control plane works at fleet scope, and a real fleet is not all LangGraph: some agents use the OpenAI Agents SDK, and others are custom orchestrators. Each speaks OpenTelemetry Protocol (OTLP), and each backend expects a different session key. Those differences get resolved once, in the OpenTelemetry Collector, instead of in every agent:
# otel-collector.yaml - one ingress for every agent framework in the fleetreceivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318processors: # Agents emit the spec's session key once. Each backend gets its own dialect. transform/session_keys: error_mode: ignore trace_statements: - >- set(span.attributes["langsmith.metadata.thread_id"], span.attributes["gen_ai.conversation.id"]) where span.attributes["gen_ai.conversation.id"] != nil - >- set(span.attributes["session.id"], span.attributes["gen_ai.conversation.id"]) where span.attributes["gen_ai.conversation.id"] != nil # Redact, do not drop. A filter processor would remove the whole span # and leave a hole in the session you are trying to replay. # Illustrative pattern: 13-16 digits, optionally split by spaces or dashes. transform/redact: error_mode: ignore trace_statements: - >- replace_pattern(span.attributes["gen_ai.input.messages"], "\\b\\d(?:[ -]?\\d){12,15}\\b", "[card]") - >- replace_pattern(span.attributes["gen_ai.output.messages"], "\\b\\d(?:[ -]?\\d){12,15}\\b", "[card]") # Sessions belong on spans. Strip them from metrics, whatever an agent sends. transform/bounded_metrics: error_mode: ignore metric_statements: - delete_key(datapoint.attributes, "session.id") - delete_key(datapoint.attributes, "gen_ai.conversation.id") batch: {}exporters: otlphttp/langsmith: traces_endpoint: https://api.smith.langchain.com/otel/v1/traces headers: x-api-key: ${env:LANGSMITH_API_KEY} Langsmith-Project: invoice-processing-fleet prometheus: endpoint: 0.0.0.0:8889 # Exemplars are exported only in OpenMetrics format. Without this, # the metric-to-session link from the agents is dropped here. enable_open_metrics: trueservice: pipelines: traces: receivers: [otlp] processors: [transform/session_keys, transform/redact, batch] exporters: [otlphttp/langsmith] metrics: receivers: [otlp] processors: [transform/bounded_metrics, batch] exporters: [prometheus]direction: down
agents: "Agent fleet" {
label.near: top-center
grid-rows: 1
grid-gap: 16
style: {fill: "#EAF2FB"; stroke: "#2C6FB0"}
a1: "LangGraph\ninvoice_extractor" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
a2: "OpenAI Agents SDK\napproval_router" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
a3: "custom orchestrator\nledger_sync" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
}
col: "OTel Collector" {
label.near: top-center
grid-rows: 1
grid-gap: 16
style: {fill: "#F3EFFD"; stroke: "#5A4BC4"}
p1: "1. transform/session_keys\ngen_ai.conversation.id ->\nthread_id, session.id" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
p2: "2. transform/redact\ncard numbers -> [card]" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
p3: "metrics: delete\nsession attributes" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}
back: "Backends" {
label.near: top-center
grid-rows: 1
grid-gap: 16
style: {fill: "#EEFAF0"; stroke: "#3F9E52"}
b1: "LangSmith\ntraces, threads, replay" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
b2: "Prometheus + Grafana\nTier 2 and Tier 3 alerts" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
}
agents -> col: "OTLP spans + metrics"
col -> back: "traces to LangSmith, metrics to Prometheus"
Each agent emits one session key, the spec's, and the Collector writes it again in the forms the backends read. LangSmith groups traces into threads by a thread_id or session_id metadata key, which over OTel is a langsmith.metadata.* span attribute. Langfuse reads session.id. Without the translation step, every agent team ends up hard-coding a backend's dialect, and changing backends means changing every agent.
Redaction uses the transform processor's replace_pattern on purpose. The filter processor is often suggested for removing Personally Identifiable Information (PII), but its documented behavior is that matching telemetry "is dropped". A filter rule that matches a card number removes the whole chat span from the session, so the replay you need during the incident has a hole exactly where the sensitive input was. The pattern itself is illustrative. It catches card numbers written with spaces or dashes, and it also redacts any other 13 to 16 digit number, including millisecond timestamps and long order IDs. A real policy needs a Luhn check in the SDK or a dedicated redaction step.
The metrics pipeline has its own transform. It deletes session.id and gen_ai.conversation.id from every metric data point, so one agent that ignores the rule cannot turn the fleet's metrics into one series per session.
This config passes otelcol-contrib validate on Collector contrib 0.164.0, which also parses every OTTL statement. Two caveats remain. replace_pattern works on string attributes, so it redacts message content only if your instrumentation records gen_ai.input.messages as a JSON string; if it records structured values, redact in the SDK before export. Finally, the LangSmith exporter uses the Langsmith-Project header, which is the one for project-based workspaces. Newer agent-based workspaces use Langsmith-Agent-Id plus Langsmith-Agent-Environment instead, and LangSmith rejects a request that mixes the two styles.
Session Retention for Agent Replay Is Something You Build
Session replay is only as good as how long the session is kept. A common assumption is that hosted trace platforms handle hot, warm and cold storage for you. LangSmith does not. A project chooses base retention of 14 days or extended retention of up to 180 days, which LangSmith's data purging documentation gives as the SaaS maximum from 14 September 2026. In a project on base retention, individual traces are upgraded to extended retention when they get feedback, match an automation rule, or enter an annotation queue. The diagram shows that case.
direction: down
d0: "Base-retention project\nDay 0-14: all traces" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
d1: "Day 14-180\nextended retention\nonly traces with feedback,\na rule match or annotation" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
d2: "After retention\ntrace ages out\nof LangSmith" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
d3: "Your object storage\nParquet via bulk export\nor a Collector fan-out" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
d0 -> d1: "upgrade"
d1 -> d2
d0 -> d3: "export"
Only the green box is a long-term tier, and you own it. LangSmith's bulk export writes Parquet files to an S3-compatible bucket, and for accounts created after 3 August 2026 it is an Enterprise plan feature. On any plan, you can instead add a second exporter to the same Collector that writes traces to your own object storage. That choice belongs in the architecture before an auditor or a quarterly regression review asks for a session from last spring, and the control decisions inside it deserve a harness-owned log on its own retention. Part 6 of this series picks up the compliance side of the same question.
OpenTelemetry GenAI Metric Renames Break Dashboards Silently
One more source of Telemetry Surface Gap does not come from your agents at all. OpenTelemetry's GenAI semantic conventions now live in their own repository, open-telemetry/semantic-conventions-genai. Every metric in them still has Development status, and there is no tagged release; the conventions ship from the main branch. On 22 September 2026 the spec replaced the gen_ai.client.token.usage histogram with the gen_ai.client.inference.usage.* counters used in Block 1. On 6 October 2026 inference latency moved to gen_ai.client.inference.duration. As of 30 October 2026, the reference Python instrumentation in opentelemetry-util-genai still documents the old names.
direction: down
emit: "Instrumentation library\nemits gen_ai.client.token.usage" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
dash: "Cost dashboard\nqueries gen_ai.client.inference.usage.input_tokens" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
bad: "Empty panel\nno error, no alert" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
good: "Alert on absence:\nabsent(series) for 15m" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
emit -> dash: "name mismatch"
dash -> bad: "without a guard"
dash -> good: "with a guard"
A dashboard keyed on one name while the emitter uses the other does not throw an error. It shows an empty panel, and an empty cost panel looks like a quiet day. Two habits prevent this. Put an absence alert on every metric a Tier 2 or Tier 3 alert depends on, for example Prometheus absent() over 15 minutes. Write it against the exported Prometheus name. The exporter turns dots into underscores and adds a _total suffix to counters. And when you upgrade either the instrumentation or the dashboards, treat metric names as a versioned interface: a transform processor in the Collector can rename the old metric to the new one for the length of the migration.
Agent Fleet Observability Instrumentation Checklist
Before your first agent goes to production:
- Every agent emits an
invoke_agentspan withgen_ai.agent.name,gen_ai.conversation.id,fleet.agent.versionandfleet.id - Token cost comes from the provider's reported usage (
usage_metadata), not an estimate from reply length - Session and request IDs appear on spans only, never as metric attributes
- Every Tier 2 and Tier 3 metric carries only bounded attributes: agent name, version, fleet
- Exemplars are on (the SDK default is trace-based), the Collector exports OpenMetrics, and the backend stores exemplars; alerts still link to sessions through a span query, not only through an exemplar
- Every agent has in-flight signals: a per-node-execution step counter, a step budget, and the oldest-open-session gauge
- Loop guards key on the validator's verdict, not on the model's output, and keep their counters in graph state, not in a closure
- Halts are alerted on through
fleet.agent.loop_halts, because a halted run ends with status OK - Collector redaction uses
transformandreplace_pattern, neverfilter - Collector config maps
gen_ai.conversation.idto each backend's session key, and deletes session attributes from metrics - Trace retention beyond 14 days is a decision on record: extended retention, bulk export, or your own object storage
Before adding the second agent type to the fleet:
- All agents share one
fleet.idvalue and one tracing project or workspace - Fleet dashboards query by
fleet.id, not by project - "Total cost of one pipeline run across every agent" is answerable from one trace query
- Absence alerts exist on every metric that an alert depends on
- Alert thresholds are defined at fleet scope, with per-agent overrides only where needed
How the Rest of the AI Control Plane Series Builds on This Layer
Back to the opening belief: tracing every agent is not observability. The agent-level span and the spec's agent metrics are exit-recorded, and an agent failing by not finishing produces neither. Observability for a fleet is three tiers with bounded metrics, sessions carried on spans and linked by exemplars, and at least one behavioral signal measured in flight. That is what closes the Telemetry Surface Gap.
Every later part of this series assumes this layer exists. Part 2, Agent Policy Enforcement: Why Copied Guardrails Fail, moves fleet-wide rules (egress allowlist, agent registry, model allowlist) to one decision point at a tool gateway, keeps per-agent gate logic in the Gated Execution layer, and feeds its policy decisions into the Tier 2 metrics defined here. Part 3 needs session replay to trace a cascade back to its source. Part 4 gates canary promotion on the Tier 3 quality score. Part 5 attributes cost through the span attributes, and Part 6 turns the same spans into an audit trail.
References
- OpenTelemetry. Semantic Conventions for Generative AI (repository; main branch, Development status, checked 30 October 2026). https://github.com/open-telemetry/semantic-conventions-genai
- OpenTelemetry. Semantic conventions for generative AI metrics. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md
- OpenTelemetry. Semantic conventions for GenAI agent spans. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md
- OpenTelemetry. (22 September 2026). Pull request #374: usage metrics renamed to gen_ai.client.inference.usage.*. https://github.com/open-telemetry/semantic-conventions-genai/pull/374
- OpenTelemetry. (6 October 2026). Pull request #521: gen_ai.client.inference.duration. https://github.com/open-telemetry/semantic-conventions-genai/pull/521
- OpenTelemetry. Metrics SDK specification: cardinality limits and exemplars. https://opentelemetry.io/docs/specs/otel/metrics/sdk/
- OpenTelemetry. (2026). Cardinality limits in OpenTelemetry. https://opentelemetry.io/blog/2026/cardinality-limits-in-opentelemetry/
- OpenTelemetry Python. opentelemetry.sdk.metrics.export (InMemoryMetricReader). https://opentelemetry-python.readthedocs.io/en/latest/sdk/metrics.export.html
- OpenTelemetry Python Contrib. opentelemetry-util-genai. https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/util/opentelemetry-util-genai
- OpenTelemetry Collector Contrib. Transform processor. https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/transformprocessor/README.md
- OpenTelemetry Collector Contrib. Prometheus exporter (
enable_open_metricsand exemplars). https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusexporter/README.md - OpenTelemetry Collector Contrib. Filter processor. https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md
- Jaeger. Discussion #3628: in-flight spans for long-running jobs. https://github.com/orgs/jaegertracing/discussions/3628
- Prometheus. Query functions: absent(). https://prometheus.io/docs/prometheus/latest/querying/functions/#absent
- OpenTelemetry. (2025). AI Agent Observability - Evolving Standards and Best Practices. https://opentelemetry.io/blog/2025/ai-agent-observability/
- LangChain. LangGraph Graph API: recursion limit. https://docs.langchain.com/oss/python/langgraph/graph-api
- LangChain. UsageMetadata (langchain-core reference). https://reference.langchain.com/python/langchain-core/messages/ai/UsageMetadata
- LangChain. Trace with OpenTelemetry (LangSmith). https://docs.langchain.com/langsmith/trace-with-opentelemetry
- LangChain. Threads (LangSmith). https://docs.langchain.com/langsmith/threads
- LangChain. Data purging and compliance (LangSmith retention). https://docs.langchain.com/langsmith/data-purging-compliance
- LangChain. Bulk data export (LangSmith). https://docs.langchain.com/langsmith/data-export
- Langfuse. OpenTelemetry integration. https://langfuse.com/integrations/native/opentelemetry
- Galileo. Sessions overview. https://docs.galileo.ai/concepts/logging/sessions/sessions-overview
- AWS. (June 2026). AGENTOPS05-BP02 Monitor agent behavior patterns and detect anomalies, Agentic AI Lens, AWS Well-Architected Framework. https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentops05-bp02.html
- Cemri, M., Pan, M. Z., Yang, S., et al. (2025). Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657, version 3. https://arxiv.org/abs/2503.13657
- Ranjan Kumar. (April 2026). Harness Engineering: The Missing Layer Between LLMs and Production Systems. https://ranjankumar.in/harness-engineering-the-missing-layer-between-llms-and-production-systems
- Ranjan Kumar. (April 2026). Gated Execution: Why Your Agent Should Never Act Without Permission. https://ranjankumar.in/harness-engineering-gated-execution-llm-agents-policy-safety
- Ranjan Kumar. (April 2026). Validation Layer Design: Building the Reflex That Catches What the Model Gets Wrong. https://ranjankumar.in/harness-engineering-validation-layer-design-llm-output-repair
- Ranjan Kumar. (April 2026). Retry, Fallback, and Circuit Breaking: Building LLM Infrastructure That Survives Outages. https://ranjankumar.in/harness-engineering-retry-fallback-circuit-breaking-llm-resilience
Related Articles
- Agent Versioning and Deployment Strategies: Shipping Agent Updates Without Breaking Running Pipelines
- Compliance, Audit Trails, and Regulatory Requirements for Agentic Systems
- Cost Governance and Budget Allocation Across Agent Types: Token Spend Is Infrastructure Spend






