Building an effective on-call rotation is one of those challenges every IT team eventually faces – and most teams get it wrong the first time. Done poorly, on-call schedules burn out engineers, create single points of failure, and leave incidents unresolved for too long. Done well, an on-call rotation distributes the load fairly, ensures someone qualified responds quickly, and keeps your infrastructure reliable without destroying morale.
Why Most On-Call Rotations Fail
The most common failure mode is treating on-call as an afterthought – assigning it to whoever is available rather than building a proper system. Teams start with one or two people covering everything, those people stop sleeping well, and then they quit.
Another failure: alerts without context. Getting paged at 2 AM with “CPU high on server-07” is nearly useless if you don’t know what server-07 runs, what normal looks like, or what to do next. The engineer on call spends the first 10 minutes just orienting themselves instead of fixing the problem.
Rotation coverage gaps are also common. Teams schedule a weekly rotation but forget about handoffs, holidays, and time zones. One missed handoff and you have no one on call for 12 hours.
Defining the Scope Before Building Any Schedule
Before building any schedule, define what “on call” actually means for your team. What systems require 24/7 coverage? What constitutes an incident that warrants waking someone up? Not every alert needs an on-call response – that distinction matters enormously.
A practical starting point: separate your services into tiers. Tier 1 is revenue-critical or user-facing infrastructure that requires immediate response. Tier 2 is internal tools or non-critical services that can wait until business hours. On-call coverage should focus almost entirely on Tier 1.
Write this down and share it with the whole team. Ambiguity about what requires a response is a major source of on-call stress.
Structuring the On-Call Rotation
A well-structured rotation needs at minimum three layers. Primary on-call is the first responder for any incident. Secondary on-call is the backup if the primary doesn’t acknowledge within a defined window – typically 5 to 10 minutes. Escalation goes to a manager or senior engineer for incidents that exceed the on-call engineer’s authority or expertise.
Weekly rotations are the most common and generally work well for teams of four or more people. Shorter rotations (3–4 days) reduce burnout but increase handoff overhead. For small teams, consider a follow-the-sun model if you have engineers in different time zones – it dramatically reduces out-of-hours alerts for everyone.
Schedule rotations at least four weeks in advance and use a tool that handles swaps and vacation coverage automatically. Manual spreadsheets break down fast.
Alert Quality Is the Real Foundation
No rotation schedule saves you from bad alerts. If your monitoring system sends 50 alerts a night and half of them are noise, your on-call engineers will start ignoring pages – including the ones that matter.
Good real-time alert quality depends on setting meaningful thresholds, not just default ones. CPU at 90% for 30 seconds might be normal for a batch job. CPU at 90% for five minutes might indicate a real problem. The distinction matters enormously at 3 AM.
Alert fatigue is the single biggest predictor of on-call burnout. Audit your alerts regularly – if an alert fires more than twice a week without resulting in any action, either fix the underlying issue or raise the threshold.
Runbooks and Escalation Paths
Every alert that can wake someone up should have a runbook. A runbook doesn’t need to be complicated – even a short document explaining what the service does, what this alert means, and the first three steps to investigate is infinitely better than nothing.
Incident response playbooks take this further by covering multi-system failures and coordination between teams. For on-call purposes, focus first on single-service runbooks and build toward broader playbooks as your team matures.
Escalation paths should be documented and automated where possible. Automated alert escalation ensures that if the primary on-call doesn’t acknowledge within a defined window, the alert routes automatically to the secondary – without requiring manual intervention at 3 AM.
Compensating and Supporting the On-Call Engineer
This is the section most technical guides skip, but it’s critical. Engineers who feel undervalued during on-call will leave.
Compensation should be explicit – whether that’s additional pay, time off in lieu, or a combination. Many teams offer a flat weekly stipend plus per-incident compensation for out-of-hours responses. What matters most is that it’s transparent and consistent.
Give on-call engineers protected time the day after a difficult night. A team that expects full productivity after a night of incidents will burn through engineers quickly.
Measuring On-Call Health Over Time
Track these metrics consistently. Mean time to acknowledge (MTTA) shows how quickly the on-call engineer responds to an alert. Mean time to resolve (MTTR) shows how long incidents take to close. Out-of-hours alert rate measures how many pages fire outside business hours each week. Incident repeat rate reveals recurring problems that aren’t being permanently fixed.
Review these in monthly retrospectives. Rising MTTA often indicates alert fatigue. High repeat rates indicate your team is fighting fires rather than preventing them.
A common misconception is that a low MTTA means your on-call system is healthy. An MTTA of two minutes means nothing if MTTR is four hours. Fast acknowledgement without fast resolution suggests your runbooks or tooling need work – not your schedule.
Frequently Asked Questions
How many people do you need before on-call becomes sustainable?
Most teams find that four to six engineers is the minimum for a healthy weekly rotation. Below that, the frequency of shifts is too high and burnout is nearly inevitable. If your team is smaller, focus on reducing alert volume before expanding rotation coverage.
Should junior engineers be on call?
Yes, but with guardrails. Junior engineers benefit from on-call exposure and can handle many incidents independently. The key is pairing them with an accessible senior engineer as secondary or escalation contact, and ensuring runbooks cover the most common scenarios in enough detail to act without escalating every alert.
How do you handle on-call during public holidays?
Holiday coverage should be agreed in advance and compensated separately. Many teams use a volunteer-first approach with extra compensation, then assign coverage if no one volunteers. Never assume someone will cover a holiday – it needs to be in the schedule explicitly.
Building Something That Lasts
An effective on-call rotation isn’t built once – it’s maintained. Start simple: define your service tiers, build a rotation with a real escalation path, clean up your noisiest alerts, and write a runbook for your five most common incidents. That foundation will carry you further than any scheduling tool or platform.
Review and adjust quarterly. Your infrastructure changes, your team changes, and your rotation should evolve with both. The teams that treat on-call as a living system – not a one-time setup – are the ones that avoid the burnout and turnover that make on-call feel impossible.