Monitoring systemd timers: alert when a timer job fails or stops running

systemd timers are a solid replacement for cron. You get journald logs, dependencies and Persistent=true to catch up on missed runs. But a timer that's disabled, a unit that's masked, or a service that fails every night still won't tell anyone. systemctl list-timers only helps if someone looks at it.

Report success with ExecStartPost

For a Type=oneshot service, ExecStartPost= only runs if ExecStart= succeeded, which is exactly the condition for a success ping:

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup
OnFailure=backup-failed.service

[Service]
Type=oneshot
ExecStart=/opt/jobs/backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 https://cronpup.com/p/YOUR-UUID

Report failure with OnFailure

OnFailure= starts another unit when this one fails. Use it to send the fail signal, so you're alerted right away instead of when the grace period runs out:

# /etc/systemd/system/backup-failed.service
[Service]
Type=oneshot
ExecStart=/usr/bin/curl -fsS -m 10 --retry 3 https://cronpup.com/p/YOUR-UUID/fail

The timer

# /etc/systemd/system/backup.timer
[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

In CronPup, set the monitor's schedule to the matching cron expression (0 3 * * *) in the server's timezone, with a grace period that covers RandomizedDelaySec plus the normal run time (say 30 minutes).

Why not just watch the journal?

Journal-based alerting catches failures that get logged. It can't catch a timer that never fires: disabled after an upgrade, masked by configuration management, or lost when the server was rebuilt. A heartbeat catches all of these, because the alarm is the absence of a signal.

Set up a heartbeat for your timers. 10 monitors are free.