Alert fatigue is the silent killer of good incident response – it doesn’t announce itself, it just slowly trains your team to ignore the monitoring system you spent weeks setting up. If your phone buzzes 200 times a night and only 3 of those pings actually matter, you’ve got a threshold problem, not a monitoring problem. This article covers how to fix it: smarter thresholds, better notification rules, and the mindset shift that separates noisy dashboards from ones people actually trust.
What alert fatigue actually looks like in practice
Picture a mid-sized IT team running 40 servers. Someone set CPU alerts at 80% “to be safe” years ago. Every night, three or four batch jobs spike CPU to 85% for six minutes, trigger an alert, resolve themselves, and nobody ever looks twice. Fast forward a year and the on-call engineer has a Slack channel with 4,000 unread notifications. Then one night a database connection pool actually exhausts itself and takes down a customer-facing app for 40 minutes – and the alert was sitting in that same channel, indistinguishable from the noise.
This is not a hypothetical. It’s the default outcome of static, one-size-fits-all thresholds applied without ever revisiting them. The fix isn’t fewer alerts for the sake of fewer alerts – it’s alerts that mean something when they fire.
Myth: more alerts equal better coverage
A lot of teams operate under the assumption that if something is worth monitoring, it’s worth alerting on immediately. That’s backwards. Monitoring and alerting are different jobs. You want to monitor almost everything – collect the metric, keep the history, graph it. You only want to alert on the subset of conditions that require a human to act right now. Conflating the two is exactly how teams end up with alert channels nobody reads. A metric can be extremely important and still not deserve a page at 3am.
Start with baselines, not arbitrary numbers
The single biggest improvement most teams can make is replacing guessed thresholds with data-driven ones. If you don’t know what “normal” looks like for a given server or service, any static number you pick is a guess dressed up as a rule. This is covered in more depth in this piece on performance baselines, but the short version:
1. Collect at least 1-2 weeks of metric history before setting a threshold, ideally covering a full business cycle (including month-end batch jobs, backup windows, etc.).
2. Look at the 95th or 99th percentile of “normal” behavior, not the average – averages hide spikes that happen every single day.
3. Set your warning threshold just above that percentile, and your critical threshold well above it, with enough separation that a warning doesn’t tip into critical every time.
4. Revisit baselines quarterly – infrastructure changes, and a threshold that made sense with 20 servers might be wrong at 60.
Use duration, not just a single data point
One of the fastest wins for cutting alert noise: stop alerting on a single sample crossing a line. Most transient spikes – a cron job, a garbage collection pause, a brief network hiccup – resolve within a minute or two on their own. Requiring a threshold breach to persist for, say, 5 consecutive minutes before firing eliminates a huge chunk of “flapping” alerts without meaningfully delaying detection of real problems. The trade-off is a small delay in notification, which is almost always worth it compared to training your team to ignore alerts.
Build tiered severity instead of one flat alert level
Not every threshold breach deserves the same response. A practical structure that works well for most teams:
– Info – logged, visible on the dashboard, no notification sent.
– Warning – sent to a low-priority channel (Slack, email digest), reviewed during business hours.
– Critical – triggers a page, requires acknowledgment, escalates if unacknowledged.
This tiering only works if escalation rules are actually enforced – a critical alert that just sits in an inbox is functionally the same as no alert. If you haven’t set up escalation paths yet, this guide on automated alert escalation walks through building rules that notify the right person, then the backup person, if nobody responds in time.
Route alerts by context, not by metric type
Another common mistake is routing all alerts from all systems to the same channel regardless of who owns what. A database alert should reach the DBA on-call, not the network team. Grouping notification rules by service ownership, not just by metric type, cuts down on the “not my problem, someone else will handle it” effect that lets real issues sit unacknowledged. If your team doesn’t have a formal on-call structure yet, it’s worth reading through how to build an effective on-call rotation before layering smarter alerting on top – rotation and routing need to work together.
Suppress known noise deliberately, not accidentally
Maintenance windows, planned deploys, and known batch jobs should have explicit suppression rules – not get silently tolerated because everyone’s learned to ignore that particular alert. The difference matters: a deliberate suppression window is documented, time-bound, and automatically lifts. An “ignored” alert is a permanent blind spot that nobody remembers creating. Every time you suppress something, put an expiration on it and a note explaining why.
FAQ
How do I know if my thresholds are too sensitive?
If more than a small fraction of your alerts get closed without any action taken, or the same alert fires and resolves itself repeatedly within the same day, that’s a strong sign the threshold is too tight or missing a duration requirement. Track your alert-to-action ratio for a couple of weeks and use it as a baseline metric in its own right.
Should every server have identical thresholds?
No. A database server under constant heavy load has a completely different “normal” than a lightly used file server. Group servers by role and workload pattern, and set thresholds per group based on their own baseline rather than a single global number.
Will reducing alerts mean I miss real incidents?
Done correctly, no – you’re not lowering coverage, you’re raising the signal-to-noise ratio. Real incidents almost always involve a sustained, significant deviation from baseline, which is exactly what duration-based, baseline-driven thresholds are built to catch. What you lose is the flood of transient noise that was masking those real signals in the first place.
Getting it right takes iteration, not a one-time fix
Alert tuning isn’t a project you finish – it’s a habit. Every incident postmortem should ask two questions: did we get alerted in time, and did we get alerted too much beforehand. Adjust thresholds, escalation paths, and routing rules based on the answers, and revisit them on a schedule rather than waiting for the next 3am false alarm to remind you. The goal isn’t a quiet dashboard – it’s a dashboard where every notification earns the attention it gets.