← Back to Guides
4

Series

Federated AI Agents· Part 4

Companion to the book Federated AI Agents · this part maps to Chapter 12

GuideFor: AI Engineers, ML Engineers, Platform Engineers, AI Systems Architects

A2A Idempotency: Your Retry Key Lives in Partner Memory

A resend under the same messageId is safe only while the partner reads the key and still remembers it. The duplicate it lets through leaves your own ledger clean, and nothing you read back from the partner except its task list shows it.

#a2a-protocol#idempotency#retries#reconciliation#langgraph#multi-agent-systems

This series grew into a book: Federated AI Agents. Chapter 12 covers this part. Where the chapter goes further is at the end of this article.

At 02:10 a claims agent at a motor insurer sent a payment of 3,000 to the Finance team's A2A agent. Finance opened a task and paid. No answer came back, because Finance was in the middle of a deploy that drained connections. Twenty seconds later the claims agent sent the payment again, under the same messageId, as its retry code had been written to do. Is it safe to retry an A2A SendMessage under the same messageId? Only if the partner implements idempotency, reads the key, and still remembers it, and the spec requires none of that. Finance opened a second task and paid 3,000 again.

Claims recorded one entry in its ledger: the second task, completed, with a payment reference. Every check the claims team ran agreed with Finance about that task. Eleven days later the first payment, the one the ledger never names, showed up in Finance's bank reconciliation.

The insurer is this series' fictional running example; the mechanism is real. On a2a-sdk 1.3.1 (current as of 2026-11-19), the SDK's own request handler and a reused messageId gave two tasks, two payments, one ledger entry, and an inward check that says everything agrees.

For A2A retries, the usual advice is to reuse the messageId as an idempotency key. Even the spec suggests it: "Agents may utilize the messageId to detect duplicate messages." Gateways build on it. At Clix, the A2A gateway's idempotency header "Falls back to message.messageId", and its API reference gives no period for which a key is kept (Clix). On a gateway like that the advice fails quietly. A caller cannot tell whether a resend after the gateway's next restart, or an hour later, still reaches a key the gateway remembers.

That advice treats the key as a property of your request, and the key protects nothing on its own. An idempotency key is a claim on the partner's memory, and so is every reconciliation read. A partner keeps two memories a caller can use: a key index, which makes a resend safe, and a task store, which ListTasks reads. Both can forget the same payment. After that, nothing the caller can read back through GetTask or ListTasks shows the duplicate. So the caller's two numbers, its retry deadline and its reconciliation cadence, have to be checked against lifetimes the partner states on its card as dated terms, the same way Part 3 checked the rest of the card.

When the partner's memory does fail, the duplicate it lets through has a shape I call a clean-ledger duplicate: two or more partner tasks opened by one key, while the caller's ledger names exactly one of them and agrees with the partner about it. Nothing on the caller's side is wrong, so no check that reads only the tasks the caller's records name can find it. Finding it means reading the partner's task list first. That list is a second memory of the partner's, with four limits of its own. I measured two of them; the spec and the SDK's source document the other two.

The worst case is the one I have not seen written down. A partner that restores from backup between a lost answer and the resend pays twice, and nothing the caller can read back through GetTask or ListTasks shows it. The rest of what this guide adds is specific to A2A too. The key window and the task retention become per-skill terms on the agent card, and the caller's code checks both at connect. LangGraph's default node retry is measured against a partner that honours keys. The limits of ListTasks as the counterparty's record are measured, one by one.

Some of this is old, and I credit it where it comes up. Stripe's documentation, the IETF's Idempotency-Key draft and OneUptime's 2026 guides already say that a key's retention must cover the retry window. Payments teams already reconcile against the counterparty's records.

A2A idempotency: what the spec promises a retried SendMessage

On duplicates, the A2A specification (v1.0.1) gives a send one sentence. Section 3.3.1 says "Get operations (Get Task, List Tasks, Get Extended Agent Card) are naturally idempotent", and then: "Send Message operations MAY be idempotent. Agents may utilize the messageId to detect duplicate messages." In the proto, the message id "is created by the message creator", which is why a caller can choose it and use it as a key.

MAY means a partner can ignore it and break no rule. DefaultRequestHandler, the Python SDK's handler, does ignore it. In a2a-sdk 1.3.1 the string message_id does not occur anywhere in default_request_handler.py. It looks only at task_id (line 301, task_id = params.message.task_id or None), and when that is empty, every send becomes a fresh task. A public reproduction on a2a-sdk 1.1.5 got "2 external effects for 1 logical operation" the same way, and its author calls it, correctly, not a protocol violation (arjun2075, 2026).

So a caller whose send timed out is looking at four cases it cannot tell apart from its side of the seam (any A2A boundary between two agents, in Part 1's terms):

d2
direction: down
send: "Claims sends pay_claim under key k, and the request times out" {
  style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}
}
cases: "What happened on Finance's side" {
  grid-columns: 1
  grid-gap: 14
  label.near: top-center
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  never: "The send never arrived. A resend is safe." {
    style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
  }
  failed: "It arrived; Finance failed before opening a task. A resend is safe." {
    style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
  }
  lost: "Finance opened a task and paid; the answer was lost. A resend pays again." {
    style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}
  }
  slow: "Finance is still working on the first task. A resend opens a second one." {
    style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}
  }
}
send -> cases: "the caller sees the same timeout in all four"

The last two cases are the expensive ones. A partner's deploy produces the third every time connections are drained, and a slow partner produces the fourth. A resend is safe in it only if the partner reads the key. A2A's maintainers know this. Issue #928, "Idempotency hole when sending first message", was opened in July 2025. It is now consolidated into the open epic #1987, "Idempotency & safe retries", labelled v1.1-candidate. Until a spec revision closes the hole, the partner's half of a key is whatever that partner's code does, and the caller cannot see that code.

Duplicate A2A tasks the caller's ledger cannot see: the clean-ledger duplicate

Here is the resend the incident's team wrote. It is a reasonable piece of code: the key is minted before the first send, checkpointed, and reused on every attempt, which is everything the caller's half of a key can do:

python
async def send(client: Any, claim: dict[str, Any], key: str) -> Task:    """One SendMessage under `key`. A lost answer raises the SDK's timeout error."""    message = new_data_message(claim, role=Role.ROLE_USER)    message.message_id = key  # the sender chooses a message's id: ours is the key    message.metadata.update({"skill": "pay_claim"})    task = None    async for event in client.send_message(SendMessageRequest(message=message),                                           context=ClientCallContext(timeout=5)):        if event.HasField("task"):            task = event.task    assert task is not None    return taskasync def resend_on_loss(client: Any, claim: dict[str, Any], key: str,                         attempts: int = 3) -> Task:    """The caller's resend: the same key every time, as the incident's team wrote it."""    for attempt in range(attempts):        try:            return await send(client, claim, key)        except Exception:            if attempt == attempts - 1:                raise    raise AssertionError

Snippets in this guide leave out imports: Any, Callable and TypedDict come from typing, dataclass from dataclasses, date from datetime, defaultdict from collections, asyncio, random, time and uuid from the standard library, and the SDK names from a2a-sdk 1.3.1's a2a.types, a2a.client, a2a.helpers, a2a.server and a2a.utils.errors packages. Later snippets reuse send and the names defined before them.

I ran it against a payments partner served by the SDK's DefaultRequestHandler, with an in-process transport that delivers every send and loses the first answer. Then I ran two reconciliation passes. In the inward pass, the job reads the task each ledger entry names, which is what a sweep or a status poll does. Its counterpart, the outward pass, lists the partner's tasks and groups them by the id of each task's first message. The minimal inward pass the experiments use is a few lines. A production one also skips resolved entries, looks a key up when an entry names no task, and treats TaskNotFoundError as a lost task:

python
async def inward(client: Any, ledger: list[Entry]) -> dict[str, str]:    """From each entry to the task it names: does the partner agree?"""    verdicts = {}    for entry in ledger:        task = await client.get_task(GetTaskRequest(id=entry["task_id"]))        done = task.status.state == TaskState.TASK_STATE_COMPLETED        state = "completed" if done else TaskState.Name(task.status.state)        verdicts[entry["key"]] = "agrees" if state == entry["state"] else "behind"    return verdicts

A ledger entry here holds the key, the task id and the state; the graph further down defines it in full as Entry. The outward pass is printed with the reconciliation job below. Both ran on the same partner:

text
plain      paid=2 amounts=[3000.0, 3000.0] listed=2 ledger_names=567a4256           inward={'k-CLM-1001-pay': 'agrees'}           outward={'k-CLM-1001-pay': ['567a4256', '9ac3232f']}

Two payments of 3,000. Claims' ledger names task 567a4256, the id the resend's answer carried (ledger_names prints what the caller recorded, not a list position), and the inward pass says it agrees, because the entry says completed and so does the partner. Task 9ac3232f is the other one. Because the answer that came back belonged to the resend, 9ac3232f is the task Finance opened at 02:10 for the lost answer, and nothing the caller holds names it.

Put the two records side by side and read across a row: every task the caller knows about, the partner agrees with.

d2
grid-rows: 1
grid-gap: 40
claims: "Claims' ledger (the caller)" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  style: {fill: "#EAF2FB"; stroke: "#2C6FB0"; font-color: "#2C2C2A"}
  e1: "entry: key k, task B, completed" {
    style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}
  }
  in: "inward pass: read task B. Agrees." {
    style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
  }
}
finance: "Finance's tasks (the partner)" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 14
  style: {fill: "#FDF6E3"; stroke: "#C9A227"; font-color: "#2C2C2A"}
  ta: "task A: first message k, completed, paid 3,000" {
    style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}
  }
  tb: "task B: first message k, completed, paid 3,000" {
    style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}
  }
  out: "outward pass: key k opened two tasks" {
    style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}
  }
}

That is the clean-ledger duplicate. Claims' ledger is complete for everything it knows. Polling GetTask for the ids you hold keeps reporting agreement, and the usual ListTasks lookup by your own key runs only for entries that are still unresolved, which this one is not. That inward view is what the A2A maintainers' own answer on issue #928, quoted below, describes. What detects it is a group-by over the partner's list: one key, more than one task. A pass has to begin at the partner's records to compute it. That rule is old in payments, where a merchant's books are reconciled against the processor's settlement file and not the other way round. What A2A changes is that the settlement file is ListTasks, and that file is weaker than a bank statement in ways the last sections measure.

A2A idempotency key retention: three ways the partner forgets a key

Everyone reaches first for a fix on the partner's side, and it is the right first fix: the partner keeps a key index. A repeat of a key inside a stated window gets the task the first send opened, and the executor runs nothing:

python
@dataclassclass Kept:    task_id: str    at: floatclass KeyIndexHandler(DefaultRequestHandler):    """The partner's half of a key: a repeat inside `window` seconds gets the first task.    The index lives in `keys`. Whatever empties it - the window passing, a release that    starts with a fresh index, a restore from backup - turns the key back into a label.    """    def __init__(self, *args: Any, window: float, keys: dict[str, Kept],                 now: Callable[[], float], **kwargs: Any) -> None:        super().__init__(*args, **kwargs)        self.window, self.keys, self.now = window, keys, now    async def on_message_send(self, params: SendMessageRequest,                              context: ServerCallContext) -> Any:        message = params.message        if message.task_id:  # a follow-up on a task that exists            return await super().on_message_send(params, context)        kept = self.keys.get(message.message_id)        if kept and self.now() - kept.at < self.window:  # a repeat: the first task            task = await self.task_store.get(kept.task_id, context)            if task is None:  # purged: refuse rather than run the effect again                raise InvalidParamsError(message="key names a task no longer held")            return task        result = await super().on_message_send(params, context)        if isinstance(result, Task):            self.keys[message.message_id] = Kept(result.id, self.now())        return result

This handler is deliberately short, and it leaves out three things a production one needs. It does not scope the key to the caller, so one caller's key could collide with another's. It does not compare the request, so a corrected amount sent under an old key would get the old task back. And it records the key only after the first send has finished, so a repeat that arrives while the first send is still running, which is what a slow partner's timeout produces, finds no key and runs again. In my runs the answer was always lost after the partner had finished, so the 24-hour row below measures the easy case. If the key is still held but its task was purged, the handler refuses instead of paying again. All three are in the book's version. The A2A idempotency epic, #1987, lists the same requirements in its discussion, from keys scoped to the caller to one attempt for concurrent duplicates.

Two alternatives come up whenever I show this. The first is a non-blocking send. With return_immediately set, the proto says the operation "returns immediately after creating the task, even if processing is still in progress", so the caller can record the task id before the effect runs. A lost answer becomes a lost acknowledgement. That shrinks the window without closing it, because the acknowledgement can be lost too, and then the caller is back to resending under a key. The second is to deduplicate on a business key, one payment per claim_id, which is what most payments teams do. It survives a release that starts with a fresh key index. It is still the partner's memory, though, and a restore rewinds it with everything else. Both alternatives move the problem without moving it to the caller's side. The random suffix in record_key, further down, exists because one claim can carry more than one legitimate payment, a supplement after a reinspection for example, so the claim id alone cannot be the key.

What matters is the docstring. Whether your resend is safe depends on the contents of the index, and the index is storage the partner owns. I ran the same caller, the same key and the same lost answer against five partners:

PartnerWhat happened between the lost answer and the resendPaymentsTasks listedInwardOutward
Plain SDK handlernothing22agreeskey opened 2 tasks
Honours keys, 24 h20 s passed11agreesnothing
Honours keys, 24 hthe same resend, 24 h and 1 s later (the clock is moved forward to stand in for a next-day re-run under the same key)22agreeskey opened 2 tasks
Honours keys, 24 ha release shipped with a fresh key index22agreeskey opened 2 tasks
Honours keys, 24 hlast night's backup restored: both in-memory stores, tasks and key index, reset to a copy taken before the first send21agreesnothing

One row out of five pays once. In the other four, the caller sent the same key and the partner had forgotten it. Grouped by what the partner forgot, the rows look like this.

d2
direction: down
key: "Claims sends again under the same key k" {
  style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}
}
index: "Does Finance's key index still hold k?" {
  style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}
}
outcomes: "What the resend gets" {
  label.near: top-center
  grid-rows: 2
  grid-columns: 2
  grid-gap: 20
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  yes: "Yes: task A comes back. One payment." {
    style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
  }
  expired: "Window passed: new task, second payment" {
    style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}
  }
  release: "Fresh index on a release: new task, second payment" {
    style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}
  }
  restore: "Backup restored: new task, second payment, task A gone from the list" {
    style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}
  }
}
key -> index
index -> outcomes

None of the three red cases is exotic. OneUptime's 2026 guides state the general rule, that key retention must cover the longest period in which a client can retry or reconcile (OneUptime, 2026-08-14), and warn that cache eviction and restarts can let duplicates through (OneUptime, 2026-07-29). A window that runs out is the design: Stripe's documentation says keys can be removed "after they're at least 24 hours old" and that "We generate a new request if a key is reused after the original is pruned" (Stripe). A key index that does not survive a release is an implementation detail the caller has no way to see. And restores happen to databases that matter. GitLab's 2017 outage ended with a restore from "the LVM snapshot created 6 hours before the outage" (GitLab postmortem). Restored that way, a key index forgets every key issued in those hours. Of real billing failures, the closest I know of is Twilio's in 2013. A failed recovery of its Redis master lost balance data, and auto-recharge then "charged customer credit cards to increase account balances without being able to update the balances themselves" (Twilio postmortem). That was balance state, not a key index. Still, the shape is the same: the record that stops a repeat was lost, and the repeats went through.

Of the five rows, the last matters most. After the restore, Finance had paid twice and listed one task. Both passes missed it: the inward pass agreed and the outward pass found nothing. Now the duplicate is invisible to every read the caller can make through GetTask or ListTasks, and I come back to it at the end.

A2A agent card key window: checking retry deadlines at connect

If a key's protection lasts as long as the partner's memory, the caller needs to know how long that is, and it needs to know in a form its code can check. Everyone who has written about idempotency keys agrees on the first half, OneUptime's second 2026 guide among them. According to the IETF's Idempotency-Key header draft, a resource "SHOULD define such expiration policy and publish it in the documentation" (draft-ietf-httpapi-idempotency-key-header-08). Among A2A proposals the closest, #1893, says "Servers SHOULD retain idempotency records for a minimum of 24 hours" and advertises support on the card as "capabilities": {"idempotency": true}.

A boolean tells a caller that keys are read. It does not say for how long, and how long is the only number a retry deadline can be checked against. Documentation is readable by a person and by no code. Both leave the caller's deadline unchecked.

In Part 3 the partner's terms became data on the card, one revision per release with an effective date, and the caller checked its own expectations against every revision at connect. Per skill, the key window and the task retention are two more terms in that structure:

python
@dataclass(frozen=True)class KeyTerms:    """The partner's half, published per skill on its card with the revision's date."""    skill: str    key_seconds: int | None  # None: the partner reads no key    effective: date    task_seconds: int | None = None  # how long a finished task stays on the list@dataclass(frozen=True)class RetryExpectation:    """The caller's half, kept in the caller's own repository."""    skill: str    attempts: int  # the first send included    deadline_seconds: int  # no resend later than this after the first send    skew_seconds: int = 30  # the clock difference both teams budget for    reconcile_every_seconds: int = 3600  # how often the outward pass runsdef window_breach(terms: KeyTerms, wants: RetryExpectation) -> str:    """Checked at connect, against every revision on the card, announced ones too."""    found = []    # key_seconds None: nothing promised, and may_resend never resends on this skill    if (terms.key_seconds is not None            and wants.deadline_seconds > terms.key_seconds - wants.skew_seconds):        found.append(f"from {terms.effective} the partner keeps a key "                     f"{terms.key_seconds} s; this seam resends for "                     f"{wants.deadline_seconds} s plus {wants.skew_seconds} s of skew")    if (terms.task_seconds is not None            and wants.reconcile_every_seconds >= terms.task_seconds):        found.append(f"from {terms.effective} a finished task is listed for "                     f"{terms.task_seconds} s; the outward pass runs every "                     f"{wants.reconcile_every_seconds} s")    return f"{terms.skill}: " + "; ".join(found) if found else ""

On the payments seam, the caller resends for at most 600 seconds and runs its outward pass every hour. Finance's current revision keeps keys for 24 hours and lists finished tasks for a week. Then Finance announces two revisions. One, effective in March, keeps keys for five minutes. The other, effective in April, lists finished tasks for an hour:

text
2026.14 (key 24 h, tasks 7 d)  -> ''2026.15 (key 300 s, tasks 7 d) -> 'pay_claim: from 2027-03-01 the partner keeps a key                                  300 s; this seam resends for 600 s plus 30 s of                                  skew'2026.16 (key 24 h, tasks 1 h)  -> 'pay_claim: from 2027-04-05 a finished task is                                  listed for 3600 s; the outward pass runs every 3600                                  s'2026.13 (no key window)        -> ''

Part 3 calls the second and third lines a dated breach: found on the day Finance announces the revision, logged with its effective date, while the seam keeps running on the revision in force. That gives the Claims team until March to shorten its deadline or negotiate the window, and until April to run its job more often or negotiate the retention. Finding out from a bank statement is worse.

Empty output on the last line is deliberate. A partner that states no window has promised nothing, so the caller's policy never resends on that skill, and the seam is safe without retries. A seam that works should not stop because the caller would like to retry.

d2
direction: down
clock: "One clock, two owners' numbers" {
  label.near: top-center
  grid-rows: 2
  grid-columns: 2
  grid-gap: 16
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  first: "02:10:00 first send under key k" {
    style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}
  }
  resend: "02:10:20 resend under k: task A returned" {
    style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
  }
  deadline: "02:20:00 Claims' retry deadline (600 s)" {
    style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}
  }
  window: "02:09:30 next day: Finance's window (24 h) less 30 s skew" {
    style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}
  }
}
announced: "Announced revision: window 300 s from 2027-03-01. The deadline no longer fits, so the breach is logged today." {
  style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}
}
clock -> announced: "check at connect, every revision"

LangGraph RetryPolicy on an A2A send node: three payments from two lost answers

A LangGraph caller gets a second retry, one it may not know it wrote. add_node takes a retry_policy, and RetryPolicy() with no arguments looks like a safe default for a node that calls a remote service. Its defaults in langgraph 1.2.23 are three attempts, an initial interval of 0.5 s, a backoff factor of 2, and jitter. Its retry_on is default_retry_on:

python
def default_retry_on(exc: Exception) -> bool:    import httpx    import requests    if isinstance(exc, ConnectionError):        return True    if isinstance(exc, httpx.HTTPStatusError):        return 500 <= exc.response.status_code < 600    if isinstance(exc, requests.HTTPError):        return 500 <= exc.response.status_code < 600 if exc.response else True    if isinstance(        exc,        (            ValueError,            TypeError,            ArithmeticError,            ImportError,            LookupError,            NameError,            SyntaxError,            RuntimeError,            ReferenceError,            StopIteration,            StopAsyncIteration,            OSError,        ),    ):        return False    return True

Everything turns on the last line. a2a-sdk raises A2AClientTimeoutError when an answer is lost, a plain Exception subclass, so default_retry_on returns True for exactly the case that pays twice. Because the jitter is additive, interval + random.uniform(0, 1), the backoff between the three attempts totals at most three and a half seconds. That spreads nothing: every worker that lost an answer in the same deploy comes back inside the same few seconds.

LangGraph's way of retrying makes it worse. Each attempt clears the node's writes and runs the node again from the same input. A key the node mints itself is minted again on every attempt, and a partner that honours keys sees three different keys. The wrong way takes one argument to add_node. Here State has one field, task_id, and CLAIM is the 3,000 payment:

python
async def pay(state: State) -> State:    key = f"k-{uuid.uuid4()}"  # minted inside the node: a new key on every attempt    task = await send(client, CLAIM, key)    return {"task_id": task.id}graph.add_node("pay", pay, retry_policy=RetryPolicy())

Against a partner that honours keys for 24 hours, with the first two answers lost, it printed payments=3 tasks=3 on each of three runs. Then I compared it with a key checkpointed one superstep earlier and with no node retry, three runs per variant:

Send nodePartnerPaymentsTasksSeconds (3 runs)
Mints its key inside the nodehonours keys, 24 h332.95, 2.70, 2.12
Reads a key checkpointed one superstep earlierhonours keys, 24 h112.94, 2.36, 2.58
Reads a key checkpointed one superstep earlierplain SDK handler332.61, 3.05, 1.91
No node retry (control)honours keys, 24 h11raised A2AClientTimeoutError

The seconds are backoff alone. The simulated loss is instant, and a real timeout adds the client's own wait (5 seconds in send) to every attempt.

In the first row the partner did everything right and still paid three times, because the caller's framework turned one key into three.

d2
grid-rows: 1
grid-gap: 40
wrong: "Key minted inside the retried node" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 12
  style: {fill: "#FDEDEC"; stroke: "#B03A2E"; font-color: "#2C2C2A"}
  a1: "attempt 1: key k1, answer lost" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  a2: "attempt 2: key k2, answer lost" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  a3: "attempt 3: key k3, answer arrives" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  w: "Finance sees 3 keys: 3 tasks, 3 payments" {style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}}
}
right: "Key checkpointed one superstep earlier" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 12
  style: {fill: "#EAFAF1"; stroke: "#3F9E52"; font-color: "#2C2C2A"}
  b0: "record_key node: k in state, checkpointed" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  b1: "attempts 1-3 all send under k" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  b2: "Finance sees 1 key: 1 task, 1 payment" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  b3: "still 3 sends, under 3.5 s of backoff: safe only if Finance honours k" {style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}}
}

Moving the key into its own superstep fixes the key and leaves the policy. Against a partner that reads no key, the same graph pays three times, as the third row shows. Nothing in the default retry_on tells a lost answer from a refusal, and a custom one would still need the partner's terms to know whether a resend is safe. So give send nodes no node-level retry, and let a policy that knows the partner's terms decide. Part 2 made the same move for the resume re-run. Its thread link, the caller's record of the partner's contextId and taskId, is checkpointed by a node that returns before anyone is asked. The key needs the same treatment one step earlier, in a node that returns before the send. Unlike the thread link's ids, which the partner mints, the key has to exist before the partner has seen anything.

Which A2A errors to retry: no answer from the agent, nothing else

A send node needs a decision with four checks, in this order, and then a wait:

python
import httpxfrom a2a.client import A2AClientError, A2AClientTimeoutErrorNO_ANSWER = {429, 502, 503, 504}  # a gateway or a busy partner, not the agent decidingdef answer_lost(error: BaseException) -> bool:    """True when the agent gave no answer. A JSON-RPC error is the partner deciding."""    if isinstance(error, A2AClientTimeoutError):        return True    if not isinstance(error, A2AClientError):        return False    cause = error.__cause__    if isinstance(cause, httpx.HTTPStatusError):        return cause.response.status_code in NO_ANSWER    return isinstance(cause, httpx.TransportError)def wait_before_resend(error: BaseException, attempt: int, base: float = 0.5,                       cap: float = 30.0) -> float:    """Full-jitter backoff, at least any Retry-After the partner or gateway sent."""    wait = random.uniform(0, min(cap, base * 2 ** attempt))    cause = error.__cause__    if isinstance(cause, httpx.HTTPStatusError):        header = cause.response.headers.get("retry-after", "")        if header.isdigit():            wait = max(wait, float(header))    return waitdef may_resend(terms: KeyTerms, wants: RetryExpectation, error: BaseException,               attempt: int, first_sent: float, now: float) -> str:    """Why a send may not be made again now, or "" if it may."""    if not answer_lost(error):        return f"{type(error).__name__} is an answer"    if attempt >= wants.attempts:        return f"{wants.attempts} attempts made"    if terms.key_seconds is None:        return "the partner states no key window: look the key up, never resend"    window = min(wants.deadline_seconds, terms.key_seconds - wants.skew_seconds)    if now - first_sent >= window:        return f"{now - first_sent:.0f} s since the first send, past {window} s"    return ""

Most of the work is done by the first check, and it has two kinds of no-answer. A timeout or a broken connection means the answer was lost. A 502, 503 or 504 from a load balancer draining Finance's connections usually means the agent never answered. A 502 or a 504 can still hide a send that ran, so each gets the same key and window checks as a timeout. A 429 is different: the agent may have rate-limited the call itself, which is an answer of a kind, but it asks for a later attempt, so it is treated as a no-answer that must respect Retry-After. In a2a-sdk 1.3.1 those status codes arrive as A2AClientError caused by httpx.HTTPStatusError, and a partner's JSON-RPC refusal arrives as its own class. A rejection, a revoked delegation, a breach hold or a malformed artifact is a decision, made by the partner or by your own seam. Sending the request again buys the same decision a second time, and the partner pays for the work. When Finance's handler refuses a send over the wire, the client raises InvalidParamsError, and may_resend returns 'InvalidParamsError is an answer' and never sends again. Even for a no-answer, the remaining checks hold the resend to the partner's promise, and wait_before_resend never waits less than a Retry-After the partner or the gateway sent.

d2
grid-rows: 5
grid-columns: 2
grid-gap: 22
q1: "1. Did the agent answer?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a1: "Yes, even a refusal: stop. Timeout, 502-504, 429: go on" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
q2: "2. Attempts left?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a2: "No: record unknown. Stop." {style: {fill: "#95A5A6"; stroke: "#6B7B7C"; font-color: "#2C2C2A"}}
q3: "3. Does the card state a key window?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a3: "No: look the key up, never resend" {style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}}
q4: "4. Inside deadline and window less skew?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a4: "No: record unknown; reconciliation takes it" {style: {fill: "#95A5A6"; stroke: "#6B7B7C"; font-color: "#2C2C2A"}}
q5: "Wait (Retry-After at least), resend under the same key" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
b5: "" {style: {opacity: 0}}
q1 -> q2
q2 -> q3
q3 -> q4
q4 -> q5

Wired into the payments node, with no retry_policy on it:

python
import timeimport uuidfrom langgraph.checkpoint.memory import InMemorySaverfrom langgraph.graph import END, START, StateGraphPAYMENTS = RetryExpectation("pay_claim", attempts=3, deadline_seconds=600)class Entry(TypedDict):    key: str    task_id: str    state: str  # intended -> completed | unknown    minted_at: float  # no later than the first send, so the deadline errs early    resent_after_loss: bool  # the one fact every clean-ledger duplicate needs    why: strclass Claim(TypedDict, total=False):    claim_id: str    amount: float    payment: Entrydef build(client: Any, terms: KeyTerms) -> Any:    def record_key(claim: Claim) -> Claim:  # its own superstep: checkpointed first        return {"payment": {"key": f"pay-{claim['claim_id']}-{uuid.uuid4()}",                            "task_id": "", "state": "intended",                            "minted_at": time.time(), "resent_after_loss": False,                            "why": ""}}    async def pay(claim: Claim) -> Claim:  # no retry_policy on this node        entry = claim["payment"]        age, limit = time.time() - entry["minted_at"], PAYMENTS.deadline_seconds        if terms.key_seconds is not None:            limit = min(limit, terms.key_seconds - PAYMENTS.skew_seconds)        if age > limit:  # a re-entry: the first send may already have left            return {"payment": {**entry, "state": "unknown",                                "why": f"re-entered {age:.0f} s after the key"}}        data = {"claim_id": claim["claim_id"], "amount": claim["amount"]}        attempt = 0        while True:            attempt += 1            try:                task = await send(client, data, key=entry["key"])            except Exception as error:                wait = wait_before_resend(error, attempt)                why = may_resend(terms, PAYMENTS, error, attempt,  # judged at landing                                 entry["minted_at"], time.time() + wait)                if why:  # record what we know; the reconciliation job takes it                    return {"payment": {**entry, "state": "unknown", "why": why}}                entry = {**entry, "resent_after_loss": True}                await asyncio.sleep(wait)                continue            return {"payment": {**entry, "task_id": task.id, "state": "completed"}}    graph = StateGraph(Claim)    graph.add_node("record_key", record_key)    graph.add_node("pay", pay)    graph.add_edge(START, "record_key")    graph.add_edge("record_key", "pay")    graph.add_edge("pay", END)    return graph.compile(checkpointer=InMemorySaver())

Run it with durability="sync", as Part 2 does for every seam graph. LangGraph's default writes a step's checkpoint while the next step runs, so a crash during pay could lose record_key's checkpoint, and the restart would mint a second key. The age check at the top of pay covers a re-entry: if the key was minted longer ago than the smaller of the seam's deadline and the key window less the skew, the first send may already have left, so the node records unknown instead of sending. A crash inside the deadline is the window this graph does not close. The book closes it with a lookup before the first send.

Against a partner that honours keys for 24 hours, with the first two answers lost; against one that states no window, with the first answer lost; and after a crash just after Finance had the send, resumed 700 seconds later:

text
paid=1 state=completed resent_after_loss=True  why=''paid=1 state=unknown resent_after_loss=False  why='the partner states no key window: look the key up, never resend'paid=1 state=unknown resent_after_loss=False  why='re-entered 700 s after the key'

The second and third results are the honest ones. Finance paid once, Claims does not know it, and each entry says so in words. The resends here land a few seconds after the first send, so they never reach the deadline check. Called directly 601 seconds after the first send, may_resend returns '601 s since the first send, past 600 s'. The spec allows a partner to ask for a longer wait: "Servers MAY include retry guidance (e.g., Retry-After header in HTTP)". may_resend judges a resend at the moment it would land, so a 503 whose Retry-After is 900 seconds is recorded unknown at once, with '900 s since the first send, past 600 s', and nothing is sent. An unknown entry is not a failure to hide. It is the input to the next section.

Reconciling A2A tasks in both directions

A reconciliation job compares the caller's ledger with what the partner says, on a schedule. It runs two passes, and they find different things.

An inward pass starts from the caller's records. For each unresolved entry it reads the task the entry names, or, when the entry names none, it lists the partner's tasks and looks for one opened by the entry's key. That lookup is the A2A maintainers' own answer to the lost-answer case: "With the new ListTasks method a client can now recover from this kind of failure" (darrelmiller on #928, November 2025; the same maintainer reopened it in April 2026 because "there is a larger issue around idempotency that is not limited to List Tasks"). The recovery work I found for agent systems starts from the caller's records too. Garg's A2A and MCP crash study reconciles "the original durable task", a task the system already names, after a partner crash (Garg, 2026). Inward passes close the crash window and adopt tasks the caller failed to record. They do not find a clean-ledger duplicate, because the entry is resolved and agrees. A key lookup for every entry, resolved ones included, would find it. But ListTasks has no filter by message id, so that lookup has to list all of the partner's tasks and match each one. At that point it is an outward pass.

The outward pass starts from the partner's records:

python
async def list_all(client: Any) -> list[Task]:    tasks, token = [], ""    while True:        # historyLength left unset: the server's default, which "may be all history"        page = await client.list_tasks(ListTasksRequest(page_size=50, page_token=token))        tasks += list(page.tasks)        token = page.next_page_token        if not token:            return tasksasync def outward(client: Any, ledger: list[Entry],                  skill: str = "pay_claim") -> dict[str, list[str]]:    """From the partner's list to the ledger: each key with a task no entry names.    Tasks kept with no history have no key and no skill; they group under "<no key>"."""    named = {entry["task_id"] for entry in ledger}    by_key: dict[str, list[str]] = defaultdict(list)    for task in await list_all(client):        first = task.history[0] if task.history else None        if first is not None and ("skill" not in first.metadata                                  or first.metadata["skill"] != skill):            continue  # another seam's task, reconciled against another ledger        by_key[first.message_id if first is not None else "<no key>"].append(task.id)    return {key: ids for key, ids in by_key.items() if not set(ids) <= named}

On the incident, it returns {'k-CLM-1001-pay': ['567a4256', '9ac3232f']}: one key, two tasks, one of them unnamed. That finding goes to a person, never to a resend, and the action belongs to the partner. Undoing a payment another owner already made can only be asked for, and asking is the subject of Part 5.

d2
grid-rows: 1
grid-gap: 36
inward: "Inward: from Claims' ledger" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 12
  style: {fill: "#EAF2FB"; stroke: "#2C6FB0"; font-color: "#2C2C2A"}
  i1: "for each unresolved entry" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  i2: "read its task, or look its key up" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  i3: "finds: a crash after the send, a moved task, a lost task" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  i4: "misses: the clean-ledger duplicate (entry resolved, agrees)" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
outward: "Outward: from Finance's task list" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 12
  style: {fill: "#FEF1EA"; stroke: "#D9785A"; font-color: "#2C2C2A"}
  o1: "list every task Finance shows Claims" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  o2: "group by first message id (the key)" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  o3: "finds: one key with two tasks; a task no entry names" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  o4: "action: a finding for a person, never a resend" {style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}}
}

Each of the two passes needs its own authority. The inward pass reads one claim's task, so it runs under the delegation that claim's own run used. The outward pass lists every task Finance holds for Claims, and a list returns whole tasks. With historyLength unset, the server returns its own default amount of history, which may include the first message and the claim data in it; artifacts come back only if the request sets include_artifacts, which defaults to false. So the outward pass's credential reads task contents through the list. Scope it to Claims as the caller and to the list call, leave artifacts off, and keep only the key and the task id from what it reads. Run both passes from one worker on one schedule. Two workers would file every finding twice.

A2A ListTasks limits: purge, history and pagination

Through ListTasks, the outward pass reads a second memory of the partner's: its task store. That memory has limits of its own, and each one hides the duplicate a different way. I ran the outward pass on the incident's duplicate against three versions of the partner's store:

Partner's task storeOutward finding
Both tasks held, first messages kept{'k-CLM-1001-pay': ['74f73187', '1a818ce2']}
Task A purged after its retention ran out{}
No message history kept on tasks{'<no key>': ['a3003e47', '53cb3fa7']}

Purge is the quiet one. Under the spec a task can disappear: TaskNotFoundError covers a task that "might be invalid, expired, or already completed and purged". Real stores do purge. Redis's A2A task store for Agent Mesh defaults to a one-day TTL for terminal tasks (Redis docs). A job that runs less often than the partner's retention can miss any duplicate whose first task was purged between two runs.

History is the second limit. Each key lives in the first message, and the spec leaves keeping that message to the partner: "The agent is responsible to determine which Messages are persisted in the Task History." Without history the pass still sees tasks no entry names, but it cannot tie them to a key or tell a duplicate from an unrelated claim. An unset historyLength is defined by the spec as the server's default amount, which "may be all history", and a positive one as "Return at most this many recent messages". So leave it unset, as list_all does, and confirm with the partner that its default keeps the first message. On a partner that appends replies to the history, a positive historyLength such as 1 returns the most recent message, so the first message and its key are not in what comes back. This partner keeps only the opening message, so I could not show that failure here and the table does not claim it.

Two more limits come from the list itself. In the spec's words, the operation "MUST return only tasks visible to the authenticated client", which is what makes an outward pass safe to run at all. Confirm your partner implements that rule. If its list includes tasks from other callers, the pass will report each of them as a task no entry names, and you will be reading another company's traffic. And the SDK's in-memory store, in 1.3.1, warns in a source comment that "A task updated mid-listing can move above the cursor and be skipped." So an empty finding is never proof, and the job must not resend on the strength of one.

How a job pages also depends on the cursor. Listing every task a partner holds on every run gets expensive, so pass the time of the last run as status_timestamp_after, which the proto defines as returning only tasks whose status changed at or after that time. Subtract a wide margin from it, because a task updated mid-listing can be skipped. Paging by time does not need both tasks of a duplicate in one page. Task A is a task no entry names, so outward flags it in whichever run lists it, even if task B came back in an earlier window.

d2
direction: down
pass: "The outward pass reads Finance's task store through ListTasks" {
  style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}
}
limits: "Four ways that store forgets or hides a duplicate" {
  label.near: top-center
  grid-rows: 2
  grid-columns: 2
  grid-gap: 18
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  purge: "Purge: task A deleted after its retention. Nothing found." {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  hist: "No history: tasks found, key unknown" {style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}}
  scope: "Scope: another caller's tasks mixed in" {style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}}
  cursor: "Cursor: a task updated mid-listing is skipped" {style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}}
}
rule: "Run the job more often than the shortest retention; never resend on an empty finding" {
  style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}
}
pass -> limits
limits -> rule

That is why window_breach checks the job's cadence against the partner's stated task retention, the same way it checks the retry deadline against the key window. A partner that states no retention gives the job no bound at all, which belongs in the contract review, not in the code.

A2A reconciliation after a partner restores from backup

Go back to the last row of the five-partner table. Finance restored last night's backup between the lost answer and the resend. Its key index went back, so the resend opened a new task and paid. Because the task store went back too, task A, the one Finance paid at 02:10, is no longer on the list. The inward pass agrees with task B, and the outward pass finds one task under one key. Finance paid twice. It is the clean-ledger duplicate at its limit: the partner's list is clean too. Under this guide's own policy that needs a fast restore: the resend has to land inside the 600-second deadline, after the restore has finished. A slower restore gets no resend at all, the entry stays unknown, and the reconciliation job raises it. The worst case is mainly a risk for naive resend code like the incident's, and for snapshot restores that take minutes.

d2
grid-rows: 1
grid-gap: 24
kept: "What the restore put back" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 12
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  idx: "key index: forgot k" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  store: "task store: forgot task A" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
bank: "What the restore did not put back" {
  label.near: top-center
  grid-columns: 1
  grid-gap: 12
  style: {fill: "#F4F6F7"; stroke: "#95A5A6"; font-color: "#2C2C2A"}
  paid: "Finance's payments: 3,000 twice" {style: {fill: "#FFD93D"; stroke: "#C9A227"; font-color: "#2C2C2A"}}
  who: "only Finance can read it" {style: {fill: "#95A5A6"; stroke: "#6B7B7C"; font-color: "#2C2C2A"}}
}

Finance's records went back and its money did not. No read the caller can make through GetTask or ListTasks shows this, because both of the partner's memories behind those calls now agree with the caller. What remains is the partner's effect ledger, its record of payments actually made, which only the partner (and the payee's bank) can read.

Two traces can survive on the caller's side. If the caller registered a push notification config on the send, Finance may have posted task A's status updates to the caller's webhook before the restore. That is the one A2A-borne record of task A, and it exists only if the receiver was registered and up. The other trace is the caller's own: the entry knows that a resend followed a lost answer. That is why Entry carries resent_after_loss. It cannot confirm a duplicate, but every clean-ledger duplicate has it set, so it lists exactly which entries are suspect.

Those two traces decide what to do about a restore. A wider outward pass helps only when the restore rewound tasks the caller already names: they come back as TaskNotFoundError on the inward pass. It cannot find task A, which is gone from the list. So the seam's terms should oblige the partner to report a restore and the window it rewound. On that report, the caller takes every entry first sent inside the window with resent_after_loss set, sends that list of keys to the partner, and asks it to check them against its effect ledger. The effect ledger is evidence only the partner holds. Turning a partner's word into a record the caller can cite is what the book calls a seam receipt, in Chapter 17. Testing all of this against a partner you cannot deploy, by replaying lost answers, repeated keys and restores from a stub, is Part 6.

A key works only while the partner remembers it, and a reconciliation pass finds only what the partner still remembers. When both memories forget the same payment, no GetTask or ListTasks read will show it, and the caller's best evidence is its own list of resends after a lost answer. That is why both lifetimes belong on the card, checked at connect against the retry deadline and the job's cadence, and why a restore belongs in the seam's terms.

Checklist: A2A retries and reconciliation across an owner seam

  • Every SendMessage with a side effect at the partner (money moved, a record created) carries a key minted in an earlier superstep and checkpointed before the send.
  • No send node has a LangGraph retry_policy. On langgraph 1.2.23, RetryPolicy() sends three times after a lost answer, with at most 3.5 seconds of backoff between them, and re-runs the node from its input.
  • Resend only after a lost answer, never after an error the partner returned, and only when the card states a key window for that skill, and only if the retry deadline plus the clock skew fits inside that window. With no window, record unknown and let the job look the key up.
  • At connect, check the retry deadline and the job's cadence against every revision on the card. A shorter announced key window or task retention is a dated breach on the day it is announced.
  • A reconciliation job runs an inward pass over unresolved entries and an outward pass over the partner's task list, grouped by first-message key.
  • That job runs more often than the shortest stated task retention, leaves historyLength unset (and confirms the partner's default keeps the first message), and pages from the last run with a wide status_timestamp_after margin.
  • Every outward finding goes to a person. Nothing resends because a list came back empty.
  • Every entry records resent_after_loss. Your seam's terms oblige the partner to report a restore and how far back it went, and on that report you send the partner the keys of every resent entry in the rewound window to check against its effect ledger.

Going deeper: Chapter 12 of the book

This is Part 4 of the Federated AI Agents guide, a series that follows my book Federated AI Agents one chapter at a time. Chapter 12, "Retries, Idempotency, and Reconciliation," is where this part comes from. This post adds the five-partner memory experiment, the LangGraph node-retry measurement and the limits of the outward pass, and you can build the retry policy and the job without the book. Code for the chapter is the ch12-retries-idempotency-reconciliation tag of the companion repository. Its tests run with no network and no model key. I ran them at the book's pins, a2a-sdk 1.2.2 and langgraph 1.2.15, and again on a2a-sdk 1.3.1 and langgraph 1.2.23, and they pass on both.

Federated AI Agents: LangGraph and A2A for Agents You Don't Own, by Ranjan Kumar

References


Agentic AI

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:


Books by Ranjan Kumar

Federated AI Agents cover

Federated AI Agents

Right-Sized AI cover

Right-Sized AI

Harness Engineering for Production AI Systems cover

Harness Engineering

The 7 GenAI Architectures cover

The 7 GenAI Architectures

Building Real-World Agentic AI Systems with LangGraph cover

Building Real-World Agentic AI Systems

The ChatML Handbook cover

The ChatML Handbook

The Chat Templates Handbook cover

The Chat Templates Handbook

Comments