The AI DR Runbook Executives Keep Forgetting

Most AI features are shipped like they live outside the laws of production software.

We talk about prompts, evals, and model choice. Then something predictable happens: the vendor rate-limits, the policy layer starts blocking a category you rely on, the model behavior shifts, or your own legal team changes what “allowed” means. Suddenly the AI path is gone and the product has no posture except: “Sorry, try again later.”

The uncomfortable truth is simple. If AI sits inside a revenue process, a support process, a regulated process, or anything that has humans waiting, you owe it the same kind of continuity thinking you already apply to payments, identity, or your database. Not identical methods, but the same seriousness.

This piece is the missing runbook: how to define RTO and RPO for AI capabilities, how to design graceful degradation, and when to hard-disable an agent without bricking the workflow. It is prompted by “Building resilient AI-powered software, and planning for when things go wrong”, which rightly pulls disaster recovery thinking into the AI conversation.

Start by naming the real dependency: the “AI supply chain”

When an AI feature fails, the failure rarely sits in one place. It can be any link in a chain:

• Model runtime: latency spikes, timeouts, capacity limits, region outage.
• Vendor contract surface: rate limits, quota exhaustion, pricing changes, deprecations.
• Policy layer: moderation, safety filters, internal governance gates, data residency constraints.
• Context layer: RAG index stale, embeddings drift, permissions wrong, missing documents.
• Tooling layer: an agent can call the CRM, ticketing, email, code repo. Any one of those being down turns “smart” into “stuck”.

If you treat “AI” as one dependency, you will write a runbook that looks tidy and fails in reality. The runbook must map failure modes to user-visible outcomes. That mapping is the whole job.

Make RTO and RPO explicit for AI, even if you hate the words

In classical DR, RTO is how long you can tolerate downtime. RPO is how much data loss you can tolerate. For AI, the equivalents exist, but teams avoid stating them because they expose product ambiguity.

Here is the translation that works in practice:

• AI-RTO: how long can the AI-assisted path be unavailable before you must switch users to an alternate path that still completes the job.
• AI-RPO: how much AI-produced work can you afford to lose or invalidate. Think drafts, classifications, summaries, auto-filled fields, suggested replies, proposed code changes.

Two rules make this usable.

• RTO is owned by the business process, not the engineering team. If the AI helps close deals, who can accept a four hour outage. If it helps triage incidents, who can accept five minutes.
• RPO is mostly about trust, not storage. If the model output disappears, can the user reconstruct it. If the model output later becomes non-compliant, can you retract it.

Write these per AI capability, not per product. Your “email reply assistant” and your “refund decision agent” should not share a continuity target.

Design degradation paths, not “downtime pages”

Graceful degradation is not a banner that says “AI is temporarily unavailable.” It is a pre-built alternate route that still finishes the workflow.

I use three fallback modes. Pick one per capability, and pre-wire the switch.

Mode 1: Assist only, never block

This is the simplest posture. The AI produces suggestions, but the user can always complete the task manually. If AI fails, the product still works, only slower.

• Example: suggested ticket tags, draft release notes, call summary.
• Design requirement: the UI must not depend on the suggestion to unlock the next step.
• Runbook action: on failure, hide AI affordances, preserve user input, log the incident.

Mode 2: Cached or replayable results

Some tasks are too expensive or too slow to regenerate, and users will not accept losing the output. Here you treat AI output like a computed artifact that must be persisted and replayed.

• Example: a long document transformation, an analysis report attached to a case, a set of extracted fields that drive downstream work.
• Design requirement: store the output with provenance (model, prompt template version, retrieval sources). That is your “replay handle.”
• Runbook action: if the vendor is down, serve the last good artifact with a freshness marker, and queue regeneration.

Mode 3: Deterministic fallback

For certain capabilities, you can replace “intelligence” with simpler deterministic logic that is worse, but safe.

• Example: routing rules, keyword-based classification, templates, static decision trees.
• Design requirement: define the minimum acceptable behavior that preserves compliance and customer promises.
• Runbook action: switch to deterministic mode when confidence drops, not only when the model is unreachable.

The key executive question is not “do we have a fallback.” It is: what is the degraded experience we are willing to ship on purpose.

The hard part: when the model is up, but wrong

Traditional DR is comfortable because outages are binary. AI failure is often “the service is running, but the answers are subtly broken.” That is the incident class that hurts brand and creates regulatory exposure.

You need triggers that are not uptime-based:

• Quality regression trigger: eval scores drop, human review rejection rises, or downstream metrics swing.
• Policy regression trigger: moderation starts blocking valid work, or stops blocking things you cannot ship.
• Cost anomaly trigger: token usage per task jumps, usually because prompts or tool calls balloon.

When those triggers fire, your runbook should not say “investigate.” It should say “switch mode.” A quality incident needs the same decisiveness as a security incident: contain first, analyze second.

A kill switch that does not kill the business

Every team now says they have an AI kill switch. Many of them are only API key revocation. That is not a kill switch. That is an outage you caused.

A real kill switch has three properties:

• Scoped: you can disable one capability, one tenant, one geography, one channel, or one tool integration, without turning off the entire product.
• Reversible: you can re-enable safely, with a ramp-up (percentage rollout, cohort-based gating).
• Workflow-preserving: the user path still completes, using one of the degradation modes above.

This is where feature flags, policy configuration, and UI design meet. It is also where many “agentic” projects fall apart. They wire an agent into a critical path, then discover there is no safe intermediate state between “agent runs” and “workflow dead.”

The simplest way to avoid that is to require one sentence during design review: “If we disable this agent at 10:00, how does a user finish the job at 10:01?” If the answer is vague, you do not have a product. You have a demo.

The runbook template I wish more teams started with

You do not need a hundred page binder. You need a page per AI capability, written in plain language, that an on-call person can execute under pressure.

1. Capability: what the AI does (one verb). Drafts replies. Extracts fields. Suggests actions. Executes actions.
2. Business owner: who decides what “acceptable degradation” means.
3. AI-RTO and AI-RPO: time-to-fallback, and what output loss is tolerable.
4. Dependencies: model vendor, vector store, moderation, tool integrations.
5. Failure modes: vendor down, latency, quota, policy block, quality drift, tool failure.
6. Fallback mode: assist only, cached, deterministic.
7. Kill switch scope: by tenant, by feature, by tool, by geography.
8. User messaging: what the product says when degraded. No apologies, only instruction and status.
9. Data handling: what gets stored, for how long, and how you retract outputs if needed.
10. Observability: the three signals you watch (quality, policy, cost) plus latency and error rate.

One capability, one page. If you cannot fit it on a page, the design is not ready.

One lived lesson: continuity beats cleverness

I have watched an R&D organization ship “smart” features that looked impressive in a sprint review, then quietly created support load because the product had no degraded mode. The surprising part was not the model misbehaving. It was how quickly the team normalized manual heroics to keep customers unblocked.

That is what an AI DR runbook prevents. Not incidents. Incidents will happen. It prevents the slow acceptance of chaos as normal work.

Where this connects to cost and ownership

Resilience is not only about uptime. It is also about predictable spend and clear accountability when things go wrong. If you want the finance side of this conversation, it pairs naturally with The CFO-Grade Playbook to Prevent Agentic COGS Shock. If you want the governance angle, read AI Sprawl in Danish SMEs Is Not an AI Problem. It’s a Money-and-Risk Ownership Problem.

The opinion: treat AI like a tiered dependency, not a magic layer

AI is not special because it is intelligent. It is special because it fails in more ways than most teams are used to, and many of those failures look like success until a customer complains.

If you do one thing this quarter, do this. Pick your top three AI capabilities and write their one page runbooks. Put an explicit AI-RTO and AI-RPO on each. Choose a degradation mode that still completes the job. Wire a scoped kill switch that does not brick the workflow.

That is what makes AI safe to put into the middle of the business, not just on the edge.

If you want a second set of eyes on your AI DR posture, reach out via the contact page and tell me what workflows you cannot afford to break.