Most infrastructure monitoring setups don’t fail because nobody installed a tool – they fail because nobody revisited the tool after the first six months. If you’re running a monitoring audit for the first time, or it’s been a year since anyone touched the alert thresholds, this is the process for finding out what’s actually broken versus what just looks fine on a dashboard.
Why monitoring setups quietly rot
A monitoring stack degrades the same way a garden does – not through one dramatic event, but through neglect that compounds. A server gets decommissioned and nobody removes its checks. A junior admin sets a CPU alert at 90% because that’s what the tutorial said, and it’s still there three years later generating 40 pages a week for a database server that runs hot by design. New services get added without corresponding checks because “we’ll get to it after launch.”
By the time someone runs an audit, it’s common to find monitoring coverage that’s 60% relevant, 25% stale, and 15% completely missing for systems that matter most. That’s not a hypothetical – it’s the typical shape of a setup left unaudited for 18-24 months.
Step 1: Inventory what you’re actually monitoring
Before touching thresholds or dashboards, build a list of every host, service, and check currently configured. This sounds tedious but it surfaces problems immediately.
Pull the list from your monitoring platform’s config or API rather than relying on the dashboard UI – dashboards often hide disabled or muted checks. Cross-reference against your actual asset inventory: CMDB, cloud provider console, or even a spreadsheet if that’s what you’ve got. Anything monitored that no longer exists gets flagged for removal. Anything that exists but isn’t monitored gets flagged for addition.
A common finding here: production databases monitored fine, but the read replicas or the connection pooler (PgBouncer, ProxySQL) sitting in front of them have zero visibility. See database connection pool monitoring and optimization for what that gap typically costs you during a traffic spike.
Step 2: Check alert-to-incident ratio
Pull 90 days of alert history and count how many alerts fired versus how many represented an actual incident that required action. A healthy ratio is somewhere around 1 real incident per 3-5 alerts. If you’re seeing ratios like 1:50 or worse, alert fatigue has already set in, and the team is very likely ignoring pages, including the ones that matter.
Look specifically for:
– Alerts that fired and auto-resolved within 2 minutes (usually a threshold set too tight, or a check interval too short for a transient blip)
– Alerts with no documented response or runbook link
– The same alert firing more than 10 times in a week – that’s not a monitoring signal anymore, it’s background noise
This is the single highest-leverage fix in most audits. Reworking thresholds and consolidating duplicate alerts can cut page volume by half without losing detection coverage. The deeper mechanics of this are covered in reducing alert fatigue through smarter thresholds and notification rules.
Step 3: Verify your escalation paths actually work
An alert that fires correctly but pages the wrong person, or nobody, is functionally the same as no monitoring at all. Pull up the on-call schedule and trace it against the actual escalation policy configured in the monitoring tool. It’s not unusual to find an engineer who left the company eight months ago still first in the escalation chain, with everyone else assuming someone else got the page.
Run a live test: trigger a low-severity synthetic alert during business hours and time how long it takes to reach a human, and through how many channels. If it takes more than 10-15 minutes to reach someone who can act, the escalation policy needs rework, not just the monitoring config.
Step 4: Validate that thresholds reflect actual baselines, not defaults
Most monitoring tools ship with generic defaults – 80% CPU, 85% disk, 500ms response time. These are starting points, not truth. A seasoned SRE pulls 30-60 days of historical metrics per host and sets thresholds relative to that host’s actual behavior, not a number copied from a blog post.
A batch processing server that regularly hits 95% CPU during its nightly job is not having an incident at 10pm – it’s working as intended. A web frontend that never exceeds 40% CPU under normal load but suddenly sits at 75% for six hours is a much more meaningful signal, even though it’s below the generic “critical” line. This is where performance baselines and knowing your normal pays off directly during an audit.
Step 5: Check coverage gaps beyond CPU and disk
Teams tend to monitor the metrics that are easiest to collect – CPU, memory, disk space – and skip the ones that actually predict outages. During an audit, verify coverage of: SSL certificate expiry dates, DNS resolution times, backup job completion and integrity, and SLA compliance tracking against contractual commitments. These get missed constantly because they don’t fail gracefully in a dashboard widget the way a CPU graph does.
Common mistakes found during audits
Three patterns show up in nearly every audit performed after a year of no review. First, monitoring gets added when a service launches but never gets removed or updated when that service is deprecated or migrated – checks keep running against IPs that now belong to something else entirely. Second, teams trust a single data source without cross-verification; a load balancer reports healthy while the backend pool behind it is failing over repeatedly, and nobody notices because only the LB’s own health check is wired to an alert. Third, capacity-related metrics get treated as point-in-time snapshots instead of trends – a disk at 70% looks fine today, but nobody’s tracking that it grew from 40% to 70% in six weeks.
Myth: more monitoring data means better visibility
This is backwards in practice. Teams that add every available metric without curating alerts end up with dashboards nobody opens and alert channels everyone mutes. Visibility comes from having the right 20 signals tuned correctly, not 200 signals dumped into a channel. An audit’s real output isn’t a longer list of monitored things – it’s a shorter, sharper list of things that actually predict problems, correlated in a way that supports fast troubleshooting. See correlating metrics across servers for faster troubleshooting for how that correlation work actually gets done once the noise is cut.
FAQ
How often should a monitoring setup be audited?
Every 6 months for stable environments, and every 3 months for infrastructure under active growth or frequent architecture changes. Any major migration – new cloud provider, database engine change, containerization effort – should trigger an unscheduled audit immediately after cutover.
What’s a realistic timeline for a full monitoring audit?
For an environment with 50-100 hosts, budget 2-3 days for inventory and alert history review, plus another 1-2 days for threshold rework and escalation testing. Larger or multi-site environments with SNMP devices and cloud integrations can take 1-2 weeks.
Should old alerts just be deleted or disabled first?
Disable, don’t delete, for at least one full monitoring cycle (30 days minimum). This gives you a rollback path if a check turns out to matter for a reason that wasn’t obvious during the audit, without losing the historical configuration.
Run the audit as a recurring calendar item, not a one-time cleanup project. A monitoring setup that gets reviewed quarterly stays sharp; one that gets reviewed only after an outage stays reactive by definition.