Monitoring Go scheduled jobs (robfig/cron and plain cron)

Go services often schedule work in-process with github.com/robfig/cron/v3, or ship a small binary that system cron runs. Either way, a job that stops running looks just like a job that has nothing to do. A heartbeat makes the difference visible: the job pings when it finishes, and you're alerted when the ping is late or reports a failure.

A helper

package heartbeat

import (
	"log"
	"net/http"
	"time"
)

var client = &http.Client{Timeout: 5 * time.Second}

func ping(url string) {
	resp, err := client.Get(url)
	if err != nil {
		log.Printf("heartbeat ping failed: %v", err) // never let monitoring break the job
		return
	}
	resp.Body.Close()
}

// Run reports start, then success or failure (including panics) of fn.
func Run(url string, fn func() error) (err error) {
	ping(url + "/start")
	defer func() {
		if r := recover(); r != nil {
			ping(url + "/fail")
			panic(r)
		}
		if err != nil {
			ping(url + "/fail")
		} else {
			ping(url)
		}
	}()
	return fn()
}

With robfig/cron

loc, _ := time.LoadLocation("Europe/Berlin")
c := cron.New(cron.WithLocation(loc))
c.AddFunc("0 3 * * *", func() {
	if err := heartbeat.Run(os.Getenv("EXPORT_PING_URL"), runExport); err != nil {
		log.Printf("export failed: %v", err)
	}
})
c.Start()

Give the CronPup monitor the same expression, 0 3 * * *, and timezone, Europe/Berlin. Add a one-minute heartbeat too, so you hear about a dead process within minutes.

A binary run by system cron

No code changes needed: wrap the command with cronpup-run, which reports start, the exit code and the last lines of output:

0 3 * * *  cronpup-run https://cronpup.com/p/EXPORT-UUID /usr/local/bin/export --full

Monitor your Go jobs. 10 monitors are free.