> tpl_air_041
Agent Failure, Fallback and Recovery Plan
Operational resilience plan and deterministic recovery runbook standardizing agent error classification, recursive loop detection, context-window overflow recovery, model degradation fallbacks, and human-in-the-loop dead-letter queue (DLQ) replay workflows.
Agent resilience plan codifying error taxonomy, loop detection, circuit breakers, and dead-letter queue replay.
Important Tech Document Template & Operational Notice
TinyCTO.tv Tech Document Template Notice: This template is a general educational and operational starting point. It is not legal, tax, accounting, investment, procurement, regulatory, security or certification advice. Requirements vary by jurisdiction, organization, contract and risk. Review and adapt it with qualified professionals before relying on it.
Problem Solved
Autonomous agents enter infinite recursive tool-calling loops, crash on unhandled API rate limits, or silently hallucinate corrupted outputs when primary models experience degraded performance, with zero automated recovery.
When to Use
- •Establishing production SRE resilience standards for mission-critical customer-facing autonomous agents
- •Configuring multi-provider fallback routing when primary LLM endpoints experience latency spikes or 5xx outages
- •Designing Dead Letter Queue (DLQ) triage workflows for failed agent runs requiring human remediation
When NOT to Use
- •For overarching physical datacenter and cloud facility disaster recovery planning (use TPL-OPS-004)
- •For software application bug defect triage and sprint backlog management (use TPL-QAV-010)
5 Template Sections & Structural Outline
Classifying failures: Severity 1 (Infinite recursion loops / tool starvation), Severity 2 (Model provider 5xx outages / rate limits), Severity 3 (Context overflow / unparseable JSON output), Severity 4 (Semantic hallucination / refusal).
Implementing deterministic loop tripwires: Monitoring identical tool call repetitions, duplicate parameter payloads, and step counter limits. Tripping circuit breakers after 3 identical consecutive actions.
Engineering dynamic routing cascades: If Primary Model (e.g. Claude 3.5 Sonnet) fails or p95 latency exceeds 5,000ms, failover transparently to Secondary Model (e.g. GPT-4o) with prompt adaptation.
Routing irrevocably failed agent runs to a persistent DLQ (SQS/Kafka). Providing an internal operations dashboard for support engineers to inspect the trace, fix state variables, and replay from the failure point.
Converting production agent failures into permanent automated evaluation test cases. Running synthetic simulations against historical failure scenarios before promoting agent updates.
Completion Instructions
Independent Review Checklist
- All mandatory sections completed
- No secrets or passwords included
- Executive sponsor sign-off obtained
Agent Failure, Fallback and Recovery Plan - Worked Case Study
Fictional Entity: Autonomous Financial Reconciliation Agent System (Executing 50,000 Complex Multi-Step Invocations Daily)
Real-world production case study demonstrating complete operational adoption for Autonomous Financial Reconciliation Agent System (Executing 50,000 Complex Multi-Step Invocations Daily).
- •Reduced catastrophic agent execution crashes by 91% through automated multi-provider fallback cascades
- •Recovered 100% of failed high-value invoice batches via Temporal durable execution checkpoints and DLQ replays
- •Blocked 420+ potential infinite recursive loops using strict 3-strike execution circuit breakers
Frequently Asked Questions
How does a loop-detection circuit breaker identify an agent trapped in a recursive cycle?
The runtime orchestrator calculates a sliding-window cryptographic hash of each tool call name and argument payload. If the exact same hash sequence appears 3 times consecutively, or if the agent executes 10 steps without progressing the global state, the circuit breaker trips, halting execution and emitting a Sev-1 alert.
Why is durable execution (e.g. Temporal) vital for autonomous agent recovery?
Without durable execution, an agent failure on step 8 of a 10-step multi-turn workflow forces the system to re-run steps 1 through 7, incurring massive token costs and risking duplicate mutating external API actions. Durable execution persists step checkpoints, allowing the agent to resume immediately from step 8 upon recovery.
What should an automated fallback model do if the secondary LLM expects a different tool schema?
The LLM gateway layer must maintain bidirectional schema translation adapters. When routing from Anthropic tool use syntax to OpenAI function calling syntax, the gateway transforms the parameters and system instructions transparently before calling the secondary provider.
Download Tech Document Pack
Auth RequiredDownload all blank templates, worked scenarios, and verification manifests in a single verified archive.
Authoritative Sources
- Temporal.io: Durable Execution Architecture for AI AgentsTemporal Technologies • OFFICIAL REQUIREMENT
- OpenTelemetry Semantic Conventions for Generative AI OperationsOpenTelemetry • OFFICIAL REQUIREMENT
- Martin Fowler: Circuit Breaker Software Architectural PatternMartin Fowler • OFFICIAL REQUIREMENT