Idempotency, Retries and Durable Execution Cheatsheet for AI Agent Backends

An agent that books a meeting, sends an email or issues a refund is not a chatbot anymore. It is a distributed system that happens to have a language model in the middle. It calls APIs that time out, runs on workers that get restarted, waits on rate limits, and sometimes runs for hours. Every one of those steps can fail halfway.

The failure that hurts is not the crash. It is what happens after the crash. The agent restarts and sends the email again. The retry logic issues the refund twice. A 30-step research task fails at step 28 and starts over from step 1, paying for every token again. None of these are model problems. They are the oldest problems in backend engineering, showing up in a new place.

The fixes are well known: idempotency, disciplined retries, and durable execution. This cheatsheet explains each one, how they apply to agent tool calls specifically, and gives small Python helpers you can adapt.

1. Why agents need this

Two words to keep straight:

• At-least-once: every step eventually happens, possibly more than once. This is what retries give you.
• Exactly-once effect: every real-world effect happens once, even if the step ran twice. This is what retries plus idempotency give you.

True exactly-once delivery across a network is not achievable in general. Exactly-once effects are, if every side-effecting operation is idempotent.

2. Idempotency: the same request twice, the same result once

An operation is idempotent if doing it twice has the same effect as doing it once.

HTTP semantics (RFC 9110) define GET, HEAD, PUT, DELETE, OPTIONS and TRACE as idempotent. POST and PATCH are not, which is exactly where most side effects live.

Idempotency keys

The standard fix for non-idempotent operations: the client generates a unique key for each logical operation and sends it with every attempt. The server stores the key with the result of the first attempt, and returns the stored result for any later request with the same key.

Stripe popularized the pattern with its Idempotency-Key header, and an IETF HTTP APIs working group draft describes it as a standard header. The draft's rules are good defaults even if you do not follow it exactly:

• The key must be unique per operation and must not be reused with a different payload. Reusing it with a different payload should get a 422 error.
• If a request with the same key is still in progress, the server should answer 409 Conflict instead of running it again.
• Keys can expire; publish the expiry policy. Stripe, for example, says keys can be removed after they are at least 24 hours old.

Where the key comes from matters

For agents, deterministic keys are usually the right choice: task_id + step_index + tool_name + hash(arguments). If the agent is restarted and replays step 7, it produces the same key, and the tool returns the stored result instead of acting again.

3. Retries: necessary, and dangerous

Retries handle transient failures. Done carelessly, they turn small outages into large ones and single actions into duplicates.

What to retry

How to retry: exponential backoff with jitter

• Exponential backoff: wait longer after each failure (for example, 0.5s, 1s, 2s, 4s, capped at 30s).
• Jitter: randomize each wait. Without jitter, many clients that failed together retry together, hitting the recovering service in synchronized waves. AWS's analysis of backoff strategies found that "full jitter" (a random wait between zero and the exponential cap) spreads load well and reduces total work.
• Limit attempts and total time. A task should fail clearly after a few attempts, not retry forever.
• Retry at one layer only. If the HTTP client, the SDK, the tool wrapper and the agent loop each make up to three attempts, one failing call becomes 81 attempts (3 x 3 x 3 x 3). Pick the layer that has the context to decide; disable the others.
• Respect Retry-After. Providers tell you when to come back. Listen.

Retry budgets and circuit breakers

When a dependency is really down, retries only add load. Two standard guards:

• Retry budget: allow retries only while they are a small fraction of total requests (for example, 10%). Beyond that, fail fast.
• Circuit breaker: after repeated failures, stop calling the dependency for a cool-down period and fail immediately. Then let a few test requests through.

For agents, failing fast has a bonus: the model can be told "the calendar API is unavailable" and choose another plan, instead of hanging.

4. Checkpoints and durable execution

Idempotency makes a single step safe to repeat. Durable execution makes a whole multi-step task safe to resume.

The problem

A naive agent loop keeps all its state in memory: conversation, plan, tool results. If the process dies at step 28, everything is gone. You can restart from scratch (expensive, and side effects may repeat) or give up.

Level 1: checkpoint the state

After each step, persist the agent's state (messages, tool results, step counter) to a database. On restart, load the latest checkpoint and continue. Agent frameworks increasingly ship this: LangGraph's checkpointers persist graph state after each step, for example.

Checkpointing alone has a gap: if the process dies after a tool performed its side effect but before the checkpoint was written, the step runs again on resume. That is why checkpointing and idempotent tools go together.

Level 2: durable execution engines

Durable execution engines, with Temporal as the best-known example, take this further. You write the task as ordinary code (a workflow). Every side-effecting or non-deterministic call (an activity: an LLM call, an API request, a database write) is recorded in an event history. If a worker crashes, another worker replays the workflow code, uses the recorded results for completed activities instead of re-running them, and continues from the first unfinished one.

The key rule: workflow code must be deterministic. All randomness, clocks, network and LLM calls go into activities. That fits agents well, because the non-deterministic part (the model's output) gets recorded once and replayed faithfully.

In July 2025, Temporal and OpenAI announced an integration that runs agents built with the OpenAI Agents SDK as Temporal workflows, with every model invocation executed as an activity. It went from public preview to general availability in March 2026. Similar ideas show up across the ecosystem: workflow engines adding agent support, and agent frameworks adding durable state.

Choosing a level

5. Side effects and human approval

Some actions should not just be idempotent. They should be deliberate.

• Separate "decide" from "do". Let the model propose an action (a structured intent: "refund 40 EUR on order 123"), persist it, and have deterministic code execute it with an idempotency key.
• Two-phase actions. For high-impact effects (payments, deletions, external emails), create a pending action, wait for approval (a human or a policy check), then commit. In a durable workflow, waiting for approval is just a signal; the workflow can wait for days at no cost.
• Outbox pattern. When an action must happen together with a database change, write both the change and an "action to perform" row in the same transaction; a separate worker performs the action with an idempotency key. No more "the database says it happened but the email never went", or the reverse.
• Compensation, not rollback. You cannot un-send an email. Design compensating actions (cancel, refund, follow-up correction) for steps that can fail after earlier ones succeeded. This is the saga pattern.
• Tools should be honest about effects. A tool description that says "creates a ticket (idempotent with key)" helps the model and the reviewer reason about what a retry does.

6. Python helpers

Two small, standard-library helpers: retries with full jitter, and an idempotent tool wrapper backed by SQLite.

import functools
import hashlib
import json
import random
import sqlite3
import time
from typing import Any, Callable


class RetryableError(Exception):
    """Raise for transient failures: timeouts, 429, 5xx."""


def retry(max_attempts: int = 5, base: float = 0.5, cap: float = 30.0):
    """Exponential backoff with full jitter. Retries only RetryableError."""
    def decorator(fn: Callable) -> Callable:
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            for attempt in range(1, max_attempts + 1):
                try:
                    return fn(*args, **kwargs)
                except RetryableError:
                    if attempt == max_attempts:
                        raise
                    time.sleep(random.uniform(0, min(cap, base * 2 ** attempt)))
        return wrapper
    return decorator


class IdempotentTools:
    """Stores the result of each side-effecting tool call under a deterministic key."""

    def __init__(self, path: str = "tool_results.db"):
        self.db = sqlite3.connect(path)
        self.db.execute(
            "CREATE TABLE IF NOT EXISTS results ("
            " key TEXT PRIMARY KEY, status TEXT NOT NULL, result TEXT)"
        )

    @staticmethod
    def key(task_id: str, step: int, tool: str, args: dict[str, Any]) -> str:
        payload = json.dumps(args, sort_keys=True, separators=(",", ":"))
        digest = hashlib.sha256(payload.encode()).hexdigest()[:16]
        return f"{task_id}:{step}:{tool}:{digest}"

    def call(self, task_id: str, step: int, tool: str,
             fn: Callable[..., Any], **args: Any) -> Any:
        k = self.key(task_id, step, tool, args)
        row = self.db.execute(
            "SELECT status, result FROM results WHERE key = ?", (k,)
        ).fetchone()
        if row and row[0] == "done":
            return json.loads(row[1])                 # replay: no second side effect
        if row and row[0] == "started":
            raise RuntimeError(f"{k} started earlier with unknown outcome; reconcile first")
        with self.db:
            self.db.execute("INSERT INTO results VALUES (?, 'started', NULL)", (k,))
        result = fn(idempotency_key=k, **args)        # pass the key downstream too
        with self.db:
            self.db.execute(
                "UPDATE results SET status = 'done', result = ? WHERE key = ?",
                (json.dumps(result), k),
            )
        return result


@retry()
def create_ticket(title: str, idempotency_key: str) -> dict:
    # Call the real ticketing API here, sending idempotency_key as a header.
    return {"ticket_id": "T-1001", "title": title}


tools = IdempotentTools(":memory:")
first = tools.call("task-42", 3, "create_ticket", create_ticket, title="Fix login")
again = tools.call("task-42", 3, "create_ticket", create_ticket, title="Fix login")
assert first == again   # second call returned the stored result

Two details are deliberate:

• The started state catches the dangerous case: the process died after starting a side effect but before recording the result. Instead of guessing, the wrapper stops and asks for reconciliation (check the external system, or rely on the downstream API honoring the same idempotency key).
• The key is passed downstream. Local deduplication protects against your own replays; the downstream API's idempotency support protects against the gap between "request sent" and "result stored".

7. Checklist for an agent backend

Why this matters

• Agents are distributed systems. Every lesson about networks, retries and partial failure applies.
• Retries without idempotency create duplicates. Retries with idempotency create reliability.
• Use deterministic keys for agent tool calls so that a replayed step finds its earlier result.
• Back off, add jitter, and retry in one place. Stacked retries and synchronized retry waves turn small failures into outages.
• Durable execution turns crashes into pauses. For long tasks with real side effects, it is the difference between "resume" and "start over, and hope nothing ran twice".

Final Thoughts

The models behind agents get most of the attention, but the incidents that hurt usually come from the plumbing: a double refund, a duplicated email campaign, a long task that restarted from zero three times overnight. These failures are predictable, and the solutions are decades old.

Make every side effect idempotent. Retry carefully, in one place, with backoff and jitter. Persist progress so that a crash is a pause, not a reset. And test it the only way that counts: kill the worker in the middle of a task and see what happens.

Sources

• IETF HTTP APIs working group, The Idempotency-Key HTTP Header Field (Internet-Draft): key uniqueness, expiry, 409 for in-progress duplicates, 422 for reuse with a different payload.
• IETF, RFC 9110: HTTP Semantics (2022), section 9.2.2: idempotent methods.
• Stripe, Idempotent requests: Idempotency-Key header, stored results, key removal after at least 24 hours.
• Brandur Leach, Implementing Stripe-like Idempotency Keys in Postgres (2017): recovery points and atomic phases for idempotent APIs.
• Marc Brooker (AWS), Exponential Backoff And Jitter (2015), and AWS Builders' Library, Timeouts, retries, and backoff with jitter: full jitter, retry amplification across layers, retry budgets.
• Temporal, Durable Execution documentation: workflows, activities, event history, replay, determinism.
• Temporal, Production-ready agents with the OpenAI Agents SDK + Temporal (July 30, 2025; GA update March 2026).
• LangChain, LangGraph persistence: checkpointers that save graph state after each step.
• Chris Richardson, Pattern: Saga and Pattern: Transactional outbox.