Disaster recovery sites fail their real test at the worst possible moment – during an actual failover, when the DR environment turns out to have been quietly drifting out of sync with production for months and nobody noticed because nobody was watching it. Planning monitoring coverage for a DR site means treating it as a live piece of infrastructure with its own SLAs, not a cold shelf you dust off once a year during the compliance audit.
Why DR sites get monitored less than production, and why that’s backwards
The instinct is understandable. Production gets the alerts, the dashboards, the on-call rotation, because it’s serving customers right now. The DR site – whether it’s a warm standby in a second AWS region, a cold set of VMs in a colo facility, or a replicated database cluster three states away – doesn’t generate revenue on a normal Tuesday, so it slides down the priority list.
That’s exactly the problem. A DR site with no monitoring coverage is a DR site with an unknown recovery time objective. You can write RTO: 15 minutes into a runbook, but if the standby database has been silently failing replication since a patch three weeks ago, your actual RTO is however long it takes to notice the failure, diagnose it, and rebuild the replica from scratch – which could be six hours, not fifteen minutes.
A common failure pattern: a mid-size retailer runs a warm standby in a secondary region, tests failover twice a year during scheduled DR drills, and passes both times. Eight months after the last successful drill, the primary site goes down during a Black Friday traffic spike. The failover script runs clean, but the standby database is 11 days behind on replication because a firewall rule change in the secondary VPC silently blocked the replication port and nobody was alerting on replication lag there. The DR site was “up” the whole time – ping worked, the service was listening – but it wasn’t actually ready.
What to monitor at a DR site, and how it differs from production
DR monitoring isn’t just a copy of your production monitoring config pointed at different IPs. The metrics that matter shift because the DR site’s job isn’t to serve traffic, it’s to be ready to serve traffic on short notice.
Core coverage should include:
Replication health and lag – for database DR setups (MySQL, PostgreSQL, MongoDB replica sets), lag in seconds or bytes matters more than uptime. A replica that’s technically running but 40 minutes behind gives you a false sense of security. See the guidance in Monitoring MariaDB and Galera Cluster Replication Health for the specific thresholds and gotchas around cluster replication.
Storage and capacity parity – the DR site needs enough disk, IOPS, and memory headroom to actually absorb production load, not just enough to sit idle. Capacity drift between primary and DR is one of the most common silent failures; track it the same way you’d track capacity planning with historical monitoring data on the primary side.
Network path and latency to the DR site – VPN tunnels, dedicated circuits, or Direct Connect/ExpressRoute links between sites need continuous latency and packet loss checks, not just a green/red status. Review How to Monitor VPN Tunnels and Remote Access Infrastructure if the DR link runs over a VPN rather than a dedicated circuit.
Service and process parity – every critical service running in production should have a matching health check at the DR site, even if it’s currently idle or in standby mode. A service that crashes on the DR box and restarts in a loop won’t show up unless you’re checking process state there too.
Backup integrity feeding the DR environment – if DR relies on restoring from backup rather than live replication, the backup jobs themselves need verification, not just a “completed successfully” log line. Corrupted or incomplete backups are a leading cause of failed recoveries.
Building the actual monitoring plan
Start by mapping RTO and RPO commitments to specific, measurable signals. If the RPO is 5 minutes, that’s not a policy statement – it needs an actual alert on replication lag exceeding 5 minutes, checked at an interval short enough to catch a breach before it compounds (checking every 15 minutes on a 5-minute RPO defeats the purpose).
Next, decide whether the DR site gets its own dashboard or a merged view. Most teams do better with a merged view that shows primary and DR side by side, so a widening gap is visually obvious rather than buried in a separate tab nobody opens. A multi-server dashboard that shows everything in one view makes this comparison a glance instead of a chore.
Then set alert thresholds that reflect DR-specific tolerances, which are often tighter than production tolerances in one dimension: staleness. Production might tolerate a CPU spike for ten minutes without paging anyone. A DR replica falling behind by ten minutes, if it happens during an actual outage, could mean losing ten minutes of customer orders.
Finally, treat DR monitoring alerts with the same escalation rigor as production incidents – not lower priority, just different urgency. A replication lag alert on the DR site at 2am shouldn’t wake anyone up the way a production outage would, but it also shouldn’t sit unacknowledged for three days. Build it into the same rotation structure described in How to Build an Effective On-Call Rotation for IT Teams, with a lower-severity queue rather than no queue at all.
Common mistakes in DR monitoring coverage
The biggest one is testing failover on a schedule but monitoring replication continuously never — teams run a DR drill every six months, confirm it works that day, then assume the gap between drills is fine to leave dark. Configuration drift, patches, and firewall changes don’t wait for the calendar.
The second is alerting on the wrong layer. Pinging the DR host and confirming the OS is up tells you almost nothing about whether the application stack, the database replica, and the network path are all actually in a failover-ready state. A green ping check has given false confidence in more DR failures than any other single metric.
The third is forgetting that monitoring itself needs to survive the disaster. If your monitoring platform’s only presence is in the primary region, you lose visibility into the DR site at exactly the moment you need it most – during the actual failover. Wherever possible, run monitoring checks from a location independent of both primary and DR, so external checks against the DR site’s endpoints keep working even if the primary region is the one that’s down.
FAQ
How often should DR site monitoring alerts be reviewed if the site is rarely used?
Review the alert thresholds quarterly and the actual metrics continuously – “rarely used” applies to the failover event, not to whether the site should be watched. A replica or standby instance that’s silently broken for months provides zero protection when it’s finally called on.
Should DR monitoring use the same tool as production monitoring?
Generally yes, for consistency in thresholds and easier side-by-side comparison, but the monitoring agent or checks running against the DR site need to keep working independently of production’s health – a single centralized monitoring dashboard covering both sites, as long as the platform itself isn’t hosted only in the primary region, is a difference-maker for catching drift early.
What’s the minimum viable monitoring coverage for a small DR setup with a single standby server?
At minimum: replication or sync status, disk space, service/process checks matching production, and a network reachability check from outside both the primary and DR networks. Even a single-server DR setup with no budget for elaborate tooling benefits from these four signals over nothing at all.
Treat the DR site’s monitoring as proof of readiness, not paperwork. A DR plan that’s only validated during scheduled drills is a plan with an unknown number of unmonitored days between those drills – and that number is where recovery objectives quietly go to die.