CDN Failover Testing Guide: Build and Rehearse an Origin-Outage Runbook
CDN failoverorigin redundancyCDN health checkshigh availabilitydisaster recoveryweb operations

CDN Failover Testing Guide: Build and Rehearse an Origin-Outage Runbook

CCache Cloud Editorial Team
2026-11-12
7 min read

Learn how to test CDN origin failover, health checks, cache fallback, DNS dependencies, alerts, rollback, and recurring resilience drills.

CDN Failover Testing Guide: Build and Rehearse an Origin-Outage Runbook

A CDN can only protect availability during an origin outage if its health checks, origin pool, cache behavior, DNS, alerts, and rollback steps have been tested together. This guide provides a repeatable failover workflow, a tracking checklist, and a practical incident worksheet for verifying that visitors receive the right response when an origin becomes slow or unavailable.

Overview

Origin failover is the process of moving requests from an unhealthy primary server to a secondary origin, backup region, or alternate hosting environment. Depending on the CDN, this may be controlled by active health checks, passive error detection, an origin pool, load-balancing rules, or a combination of these features.

Failover testing is different from checking whether both origins respond in normal conditions. The useful question is whether the complete delivery path behaves correctly under pressure: does the CDN identify the failure, stop sending traffic to the unhealthy origin, select the backup, preserve valid cached content where appropriate, and notify the team without creating a second incident?

Begin by documenting the intended design. Record the primary and secondary origins, the hostnames and paths used for health checks, expected status codes, timeout thresholds, retry behavior, cache rules, DNS dependencies, alert destinations, and the person authorized to restore normal routing. Provider dashboards use different names for these controls, so keep an internal runbook that describes the desired behavior rather than relying only on screenshots.

For broader context on selecting an edge platform and managed caching arrangement, see the CDN and managed caching comparison guide. The failover test should validate the configuration you actually operate, not an assumed feature set.

What to track

1. Health-check configuration

Track the health-check URL, method, protocol, port, expected response code, and any required headers. Use a lightweight endpoint that confirms the application can serve traffic, rather than a static file that remains available while the database, application process, or critical dependency is failing. At the same time, avoid making the check so complex that a minor nonessential dependency removes a healthy origin from service.

Record the check interval, timeout, failure threshold, recovery threshold, and whether checks run from multiple network locations. These values determine how quickly failover occurs and how easily a temporary network problem can trigger a route change. The correct settings depend on the application and provider; test them under realistic conditions instead of choosing values solely for the fastest possible switch.

2. Origin pool and routing behavior

Document the order or weight assigned to each origin, the conditions that remove an origin from rotation, and how long an origin must remain healthy before it is reintroduced. Confirm whether the CDN retries a failed request against another origin and whether a retry can safely repeat a request. A retry of a read-only GET is generally easier to reason about than an automatic retry of a payment, account update, or other state-changing request.

Check that the backup origin has the required TLS certificate, host-header handling, application configuration, assets, database access, and secrets. An origin that passes a shallow health check but cannot render a real page is not a dependable failover target.

3. Cache and fallback behavior

List which responses may be served from edge cache during an origin outage. Public, versioned static assets are usually easier to retain than personalized HTML, account pages, carts, or administrative routes. Confirm how your CDN treats stale objects, origin errors, cache revalidation, and cache purges. Some platforms support serving stale content under defined conditions; others require explicit configuration or do not provide the same fallback behavior.

Review safe caching rules for HTML, assets, APIs, and WordPress before testing. A failover drill should never depend on accidentally cached private or outdated data.

4. Observability and user impact

Track the signals that show both infrastructure behavior and visitor experience:

  • Health-check state and time of each origin transition.
  • Requests sent to the primary and backup origins.
  • CDN response status codes, especially 5xx, 502, 503, and 504 responses.
  • Cache hit ratio, origin response time, and time to first byte.
  • Availability from external probes and a real browser transaction.
  • Alerts received, their delivery time, and the person who acknowledged them.
  • Application errors, queue delays, database pressure, and duplicate or incomplete writes.

These measures help distinguish a successful routing change from a successful customer outcome. The CDN analytics guide can help organize the metrics used for ongoing review.

Cadence and checkpoints

Run a controlled failover drill before a major launch, after changing origins, when modifying CDN routing or cache-control headers, after a DNS or TLS change, and whenever the recovery architecture changes. For a stable service, a monthly review of configuration and alerts plus a more complete quarterly exercise is a practical starting point. Adjust the cadence to the impact of downtime and the frequency of infrastructure changes.

Prepare the drill

  1. Define the test window, owner, observer, success criteria, and rollback authority.
  2. Notify stakeholders whose systems may interpret the test as an outage. Do not publish an incident notice unless the exercise requires it.
  3. Capture a baseline from the primary origin: representative URLs, response codes, cache headers, TTFB, application transactions, and current origin traffic.
  4. Confirm that the backup contains current code, content, configuration, certificates, and data needed for the selected test paths.
  5. Verify that monitoring, logs, and alert routing are working before inducing failure.

Execute the test

  1. Start with a low-risk path or controlled test hostname if the platform supports one.
  2. Induce a failure that matches the failure mode you need to understand, such as blocking the health-check path, stopping the application process, or making the origin intentionally unavailable. Use an approved method that can be reversed.
  3. Observe the health-check transition and record the time it takes for the CDN to mark the origin unhealthy.
  4. Request uncached HTML, static assets, API endpoints, and any critical user journey that is in scope. Test from more than one network location where possible.
  5. Confirm that traffic reaches the backup, cache headers remain appropriate, and private or state-changing requests are not incorrectly served from cache.
  6. Check alerts, logs, origin load, and application behavior. Capture evidence rather than relying on a dashboard status alone.
  7. Restore the primary, confirm its health, and observe whether traffic returns according to the intended recovery policy.

DNS considerations

DNS is often part of the availability design, but it is not a substitute for CDN origin failover. If the CDN hostname remains in use, the CDN may switch origins without changing public DNS. If the design requires DNS traffic steering, record the resolver and TTL assumptions, recognize that cached DNS answers may persist beyond the configured TTL, and test both the normal and fallback records. Keep DNS rollback instructions separate from CDN rollback instructions so operators do not change both systems unnecessarily during an incident.

How to interpret changes

Compare the drill with the baseline, not with an idealized target. A short failover time is not automatically better if it creates false removals, incomplete transactions, or repeated switching between origins. Look for these patterns:

  • Health check passes, but pages fail: the check is too shallow or does not use the same host, protocol, dependency, or authentication conditions as real traffic.
  • Traffic reaches the backup, but visitors see errors: inspect configuration parity, database connectivity, TLS, host headers, missing assets, and application secrets.
  • Cached pages work while dynamic actions fail: this may be expected. Document which journeys are unavailable and provide an intentional maintenance or degraded-service response rather than broadening cache rules unsafely.
  • Failover is slow: review check intervals, thresholds, connection timeouts, retry behavior, and any DNS layer involved. Change one variable at a time.
  • Origins flap between healthy and unhealthy: investigate intermittent network or dependency failures, then consider whether thresholds and recovery delays need adjustment.
  • Cache hit ratio drops sharply: check whether the outage caused mass revalidation, a purge, changed cache keys, or responses that became uncacheable.

For each finding, record the observation, likely cause, evidence, owner, corrective action, and retest date. Avoid treating a green dashboard as proof that every route works. Pair synthetic checks with a browser journey and, where safe, a representative API or transaction test.

When to revisit

Keep the runbook as a living operational document. Review it monthly or quarterly according to your risk level, and perform a full drill after any change to the CDN provider, origin host, load balancer, DNS, TLS certificate, deployment process, cache policy, authentication flow, database topology, or monitoring system. Provider-specific health-check names, failover controls, and dashboard locations can change, so verify those details at least every six months and after platform upgrades.

Use this compact incident worksheet for each review:

  • Test date and version: record the runbook revision and infrastructure change being validated.
  • Failure introduced: state exactly what was made unavailable and how it was reversed.
  • Observed transition: record detection time, failover time, recovery time, and affected routes.
  • Visitor result: note status codes, cache behavior, TTFB, dynamic journey results, and any stale or missing content.
  • Alerts: record which notifications arrived, when they arrived, and who acknowledged them.
  • Decision: mark the test passed, passed with follow-up, or failed.
  • Next action: assign an owner and due date for every corrective change, then schedule a retest.

Before the next launch or resilience exercise, open this worksheet, confirm the current origins and health-check paths, test one representative cached resource and one dynamic journey, and verify that rollback is understood by someone other than the person who made the configuration. Rehearsal turns origin redundancy from a configuration claim into a measured recovery capability.

Related Topics

#CDN failover#origin redundancy#CDN health checks#high availability#disaster recovery#web operations
C

Cache Cloud Editorial Team

Technical SEO Editors

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.