Monitoring Kubernetes CronJobs for missed and failed runs

Kubernetes CronJobs fail in quiet ways. A run can be skipped after too many missed schedules (startingDeadlineSeconds), be blocked by concurrencyPolicy: Forbid while an earlier pod hangs, fail to schedule because of resource quotas, or have the CronJob itself deleted in a Helm upgrade. Most cluster alerting watches for things that exist and are unhealthy. It doesn't watch for things that should exist and don't.

Add a heartbeat to the container command

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 4 * * *"
  timeZone: "Etc/UTC"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: report
            image: myorg/report:1.4
            command: ["/bin/sh", "-c"]
            args:
            - |
              wget -qO- https://cronpup.com/p/YOUR-UUID/start
              /app/report; rc=$?
              wget -qO- https://cronpup.com/p/YOUR-UUID/$rc
              exit $rc

Use the same cron expression and timezone in the CronPup monitor, so it expects each run exactly when the CronJob should fire.

Keep the URL out of the manifest

Put the ping URL in a Secret and load it with envFrom. Anyone who has the URL can send pings to it.

Distroless images

If your image has no shell, curl or wget, ping from the application itself. Any HTTP client works. For example, in Python: urllib.request.urlopen(url, timeout=10). In Go: http.Get(url).

Start monitoring your CronJobs. The free plan covers 10.