API outage: impact, safe-retry playbook, and backfill checklist

Applies to: Admins, CRM/IT, Developers
Outcome: Know what breaks when MyOperator APIs are unavailable, how to keep operations running, and how to reconcile once service returns.
Time: 5–10 minutes

Prerequisites (before you test/escalate)

• Account/Client ID
• Example request (endpoint & HTTP method), timestamp (IST), and correlation/request ID if you log one
• Your API key/token location (env/secret manager), and the base URL your app calls

Do this now (numbered response plan)

1. Confirm scope
• Try 1–2 read calls (e.g., your normal list/fetch endpoint) and 1 write call.
• Note HTTP status: 5xx/timeout = provider issue; 401/403 = auth/permission; 429 = rate limit.
2. Enable safe retries (temporary failures)
• Use exponential backoff + jitter; cap total retry window (e.g., 2–10 minutes).
• For POST/PUT, make them idempotent: include a unique client-generated key in a header (e.g., X-Idempotency-Key) or payload field so duplicates don’t create multiple actions.
3. Queue writes; degrade reads
• Push failed write requests to a durable queue for replay.
• Serve cached or “last known” data for read endpoints in dashboards, with a stale indicator.
4. Operational fallbacks
• Call initiation: temporarily switch to manual dialing / web panel if available.
• Recordings: retry fetch later; avoid marking “missing” permanently during outage.
• User sync: allow local logins/roles; replay diffs after recovery.
5. Escalate if still failing
• Email support@myoperator.co with the template below.

Verify recovery & backfill (after service returns)

1. Probe health: one read + one write should return 2xx within normal latency.
2. Replay queue: drain queued writes using your idempotency keys; monitor for 2xx only.
3. Reconcile counts: compare API totals vs. CRM/BI for the outage window; export and import missing rows.
4. Spot-check recordings: recent calls should return URLs; retry any prior 404s.
5. Close the incident: document start/end times, error codes seen, backlog processed.

Troubleshooting guide (by status)

• 401/403 (Unauthorized/Forbidden): expired token, wrong key, or role missing → rotate credentials; confirm header names & scopes.
• 404 (Not found): item not committed yet; retry with backoff; verify ID correctness.
• 429 (Rate limited): honor Retry-After header; slow down globally; implement circuit breaker.
• 5xx / Timeout: treat as transient; retry with backoff; switch to fallbacks.
• TLS/DNS failures: check proxy/firewall, system CA store, or egress rules.

Escalation (email template)

To: support@myoperator.co
Subject: API outage suspected — [Account/Client ID]
Body:

• Endpoint(s): [e.g., /v1/calls, /v1/logs]
• Time window (IST): [start → end]
• Sample requests (method + exact time): [ ]
• HTTP responses seen: [timeouts / 5xx / 429 / 401]
• Impacted feature(s): [click-to-call, recordings fetch, reports]
• Current mitigations: [queued writes, manual dialing, cache]
• Attach: request IDs, logs, screenshots