Error Handling and Retries — The Retry Rodeo for Agentic Workflows

"If at first you don't succeed, add backoff and a unique operation ID. Also breath." — Your future SRE

You're already cozy with planner–executor architectures and grounding via external tools. Now it's time for the part of the job where reality bites: networks flap, APIs sleep, and users invent corner cases just to keep you humble. This lesson teaches you how to classify failures, design safe retry strategies, and integrate error-handling into agentic planners and tool wrappers — without turning your system into a machine that keeps sending invoices to the same person until they scream.

Why this matters (quick reminder)

• The planner decides what to do. The executor (and tool wrappers) actually do it. Failures can happen at either stage.
• Grounding via external tools means your model often depends on flaky third parties. That makes robust error handling essential.
• Safety & ethics: retries can re-trigger harmful actions (duplicate emails, duplicated payments, repeated disclosure of PII). We must avoid that.

1) Classify the enemy: Types of errors

Before retrying, know what you're facing.

• Transient errors — Temporary, likely to succeed if retried (e.g., 502/503, timeouts, rate limit spikes).
• Persistent errors — Bad request, malformed input, authentication failure, policy violations. Retrying won't help.
• Ambiguous errors — Unknown state: call timed out but server might have processed the request (danger zone for side effects).
• User-caused errors — Bad user input, missing consent, or policy-safety issues.

Ask: "If we retry, will we cause duplication or harm?" If yes, proceed with caution.

2) Core principles (golden rules)

1. Prefer idempotency: Make operations repeatable without extra effect (use idempotent endpoints, operation IDs, or check-before-write).
2. Differentiate retry logic by error class: Retry on transient errors; fail fast on persistent ones.
3. Use backoff + jitter: Prevent thundering herds and cascade failures.
4. Cap retries (max attempts, timeouts) and escalate to human or planner when limits reached.
5. Log, alert, and surface context: Include operation ID, user ID, tool name so you can debug.
6. Safety first: Avoid blind retries for actions that have safety implications — instead, consult the planner or a human.

3) Retry patterns (with pseudocode)

Simple retry (for pure transient reads)

attempts = 0
max_attempts = 3
while attempts < max_attempts:
  try:
    return call_tool()
  except TransientError:
    attempts += 1
    sleep(2 ** attempts)
raise PermanentFailure()

Exponential backoff with jitter (real-world staple)

base = 0.5  # seconds
max_backoff = 30
for attempt in range(1, max_attempts+1):
  try:
    return call_tool()
  except TransientError:
    backoff = min(max_backoff, base * (2 ** attempt))
    sleep(random.uniform(0, backoff))

Circuit breaker (when downstream is flailing)

• If failures exceed a threshold in a window, open the circuit and stop retries for a cool-down period.
• This prevents wasting cycles and lets the downstream recover.

4) Agentic workflows: who handles what?

• Tool wrapper / executor

  • Handles low-level retries for transient, safe-to-repeat operations (GETs, idempotent reads).
  • Adds operation IDs, logs, traces. Validates tool output against schema (grounding checkpoint!).

• Planner

  • Handles strategy-level failures: ambiguous outcomes, repeated executor failures, or safety-related problems.
  • Replans: choose alternative tools, ask clarifying questions, or escalate to humans.

Flow example:

1. Planner issues: WriteInvoice(user_id, amount, idempotency_key)
2. Executor calls billing API with that idempotency_key.
3. If transient failure -> executor retries using backoff.
4. If ambiguous (timeout but unknown state) -> executor returns AmbiguousResult to planner.
5. Planner decides: "Check invoice status" (via a read tool) rather than blind re-issue.

5) Design patterns and templates

• Idempotent wrapper: attach an operation ID and store status in a durable store before performing the action.
• Check-then-act: for sensitive writes, read current state first and only write if needed.
• Two-phase commit / confirmatory step: executor performs an action then asks planner to confirm finalization.

Code-ish sketch:

# Tool wrapper pseudocode
def safe_call(tool, op_id, max_retries=3):
  if tool.is_idempotent:
    for i in range(max_retries):
      result = tool.call(op_id)
      if result.success: return result
      if result.ambiguous: return 'AMBIGUOUS'
      sleep(backoff(i))
  else:
    return 'ESCALATE_TO_PLANNER'

6) Safety & ethics checklist (builds on previous topic)

• Use consent checks before retries that could reveal PII.
• Avoid retries that resend messages/payments without idempotency or explicit confirmation.
• Rate-limit retries to avoid amplifying harm (e.g., spam).
• Maintain an auditable trail: operation IDs, user identity, timestamps, and decision rationale.
• If policy triggers (safety filter hit), do not retry automatically — escalate to a human or safe fallback.

7) Observability, testing, and failure drills

• Simulate failures: timeouts, 5xx spikes, malformed responses.
• Monitor metrics: retry rates, success-after-retry, ambiguous outcomes, circuit opens.
• Alerts: when certain thresholds are breached (e.g., high ambiguity or retries for paid actions).

Table: Quick Strategy Cheat Sheet

Closing: Practical checklist (copy-paste into your brain)

1. Classify the error.
2. If retrying, ensure idempotency or use a confirm step.
3. Use exponential backoff + jitter; cap attempts.
4. For ambiguous or safety-relevant cases, escalate to the planner or human.
5. Log everything with operation IDs. Monitor and drill.

Final mic drop: Retries are not a magic wand — they're a policy. Treat each retry like a tiny contract: what are you allowed to do again, and who pays the cost if it goes wrong?

Version note: This builds on planner–executor architectures and grounding via external tools. If the planner keeps getting ambiguous signals, adjust tool validation, idempotency guarantees, or add explicit status-check steps to the plan.

Version metadata:

• Helpful sample prompts and code stubs available on request (e.g., idempotency middleware for REST, GraphQL, or tool wrappers for LLM agents).
• Next up: "Observability and Recovery — teaching agents to ask for help before they start a forest fire."