Cron jobs are the backbone of most Linux infrastructure – log rotation, database backups, certificate renewals, report generation, cache warming – and yet they’re almost universally the least-monitored part of the stack. Monitoring cron jobs reliably means knowing not just that a job ran, but that it finished, finished on time, and did what it was supposed to do – which is a very different problem than checking a server is up.
The reason this gets neglected is structural. A web server that goes down triggers an alert within seconds because something is actively polling it. A cron job that silently fails at 2 AM triggers nothing, because by design cron doesn’t care whether the command it ran succeeded. It fires the process and moves on. If that process throws an exception, hangs, or gets OOM-killed, cron’s own logging (usually a line in /var/log/syslog or /var/log/cron on RHEL-based systems) just shows that the job started. That’s it.
The myth: “no cron errors means the job worked”
This is the misconception worth killing first. Cron will happily log “CMD (/opt/scripts/backup.sh)” every single night whether the script backed up 40GB of data or exited immediately because a mounted volume wasn’t there. Exit codes only get captured if something downstream is checking for them – cron itself doesn’t evaluate them, and by default it only emails output if MAILTO is configured and mail delivery actually works on that box (which, on most modern cloud instances, it doesn’t, because nobody set up a local MTA).
A specific case that comes up constantly: a nightly pg_dump job that’s been “running fine” for eight months. Someone rotates the database credentials during a routine security pass and updates the app’s .env file but forgets the cron script pulls credentials from a separate file. The job starts every night at 1 AM, fails auth immediately, exits in under a second, and writes nothing anywhere useful. Nobody notices until a restore is needed three months later and the last valid backup turns out to be from before the rotation. This is precisely the scenario covered in How to Monitor Backup Jobs and Verify Data Integrity – the failure mode isn’t the backup process crashing loudly, it’s the process succeeding at “running” while failing at its actual job.
What reliable cron monitoring actually requires
There are three separate things to verify, and most setups only catch the first one, if that:
– Did the job start at the expected time (or within an acceptable window)?
– Did the job finish, and with a zero exit code?
– Did the job produce the expected result – a file of the right size, a row count that isn’t zero, a checksum that matches?
Catching only #1 gives false confidence. A job that starts on schedule and hangs forever (a stuck lock on a MySQL table, a network call with no timeout) looks identical to a healthy job from a “did it start” perspective.
The dead man’s switch pattern
The most reliable architecture for this is inverted from how people usually think about monitoring. Instead of a monitoring system asking “did this run,” the job itself reports “I ran, and here’s how it went” via an HTTP call at the end of its script, and the monitoring system alerts if that check-in doesn’t arrive within the expected window.
A minimal version looks like this at the end of a bash script:
curl -fsS -m 10 –retry 3 “https://your-monitor/ping/backup-job?status=$?”
The key detail: the ping needs to fire based on the actual exit status of the previous command, not unconditionally. A script that does `command; curl …/ping` will report success even if `command` failed, because it’s the curl call that succeeded. Capture `$?` immediately after the real work and pass that through.
For jobs with variable duration – a report generator that normally takes 4 minutes but occasionally 25 when the dataset is large – set the alert window generously (say, 45 minutes) rather than tightly, or a job that’s merely slow starts paging someone at 3 AM for no reason. This ties directly into Reducing Alert Fatigue – Smarter Thresholds and Notification Rules – a threshold set to the average runtime instead of the 95th percentile is one of the fastest ways to train a team to ignore pages.
Step-by-step setup for an existing cron job
Start by listing every cron entry across every host with `crontab -l` for each user plus `/etc/cron.d/*` – it’s common to find jobs nobody remembers writing, some of which are load-bearing. For each job worth keeping, wrap the command so the exit code and duration get captured, not just “it started.” Log stdout and stderr to a file with a timestamp in the name rather than /dev/null, even if you never plan to read it manually – it’s the difference between debugging a failure in five minutes versus reconstructing it from memory. Add the check-in ping as the last line of the script, gated on the real exit code. Set the expected window based on observed historical runtime, not a guess, and finally verify the alert actually fires by breaking the job intentionally once (comment out a step, or point it at a bad path) and confirming a notification arrives. Skipping that last step is how teams discover their “monitoring” was misconfigured only during the actual incident it was supposed to catch.
For jobs running inside containers or Kubernetes CronJobs, the same principle applies but the failure surface changes – a CronJob whose pod gets OOMKilled shows as a failed job in `kubectl get jobs`, but if nothing is watching that API, it’s invisible in exactly the same way a bare cron failure is. Process-level visibility into what’s actually running (and what silently died) matters here too, which overlaps with Process Monitoring: Track What’s Running on Your Servers.
Common mistakes
The most frequent mistake is monitoring the cron daemon instead of the jobs – confirming crond itself is running tells you nothing about whether any individual task succeeded. A close second is alerting only on failure and never on absence; a job that gets accidentally deleted from the crontab during a server migration produces zero errors because there’s nothing left to fail, so a check-in-based system that alerts on missing pings is the only thing that catches this. Third, teams often test the happy path once during setup and never test the failure path, so the alerting pipeline itself – webhook, email relay, whatever – goes stale and silently breaks months later without anyone noticing until the job it was protecting actually fails.
FAQ
How do I monitor a cron job without modifying the script itself?
Wrap the crontab entry rather than editing the script: `0 1 * * * /opt/scripts/backup.sh; curl -fsS “https://monitor/ping/backup?status=$?”`. This keeps the monitoring logic out of the application code entirely, at the cost of losing per-step granularity inside the script.
What’s an acceptable check-in window for a daily job?
A common baseline is the job’s typical runtime plus 50%, with a hard ceiling matched to how much staleness is tolerable downstream – for a nightly backup feeding a disaster recovery plan, that’s usually 2-4 hours past the scheduled time, not 24.
Does systemd timers solve this better than cron?
Systemd timers give you `systemctl status` and journal integration, which is a real improvement for start/stop visibility, but they have the same core gap – a timer unit that runs and exits 1 doesn’t page anyone unless something is watching `journalctl` output or the unit’s `ActiveState`.
Cron failures are quiet by default, and quiet failures are the ones that turn into three-month-old backups and expired certificates nobody caught in time. Building an explicit check-in for every job that matters, with a window based on real historical runtime, closes that gap without requiring a rewrite of the underlying scripts.