Cron Job Monitoring: Dead Man's Switch & Alerting

Cron jobs are the silent workhorses of production infrastructure — running database cleanups, sending invoices, syncing third-party data, expiring sessions. And they fail silently. No exception bubbles up to a user. No HTTP 500 shows up in your error tracker. The scheduler simply moves on to the next minute, and your critical job never ran. You find out three days later when a customer calls about a stale report.

This guide covers the full monitoring stack for cron jobs: why silent failures happen, the dead man's switch pattern that's become the industry standard, practical implementation, and the alerting strategies that won't wake you up at 3 AM for non-issues.

The Silent Failure Problem

Unix cron is a fire-and-forget scheduler. It forks a process, sets the environment, and runs your command. What it does not do is verify the outcome. If your script exits with code 1, cron has no mechanism to notify anyone beyond a local mail to the system user — a feature that's disabled or unconfigured on virtually every modern server.

Silent failures in cron jobs occur for several distinct reasons, and conflating them leads to incomplete monitoring solutions:

• The job never starts. The cron daemon itself crashed, the machine rebooted, or a DST clock change caused a 2 AM job to be skipped entirely. The expression is correct; the scheduler just didn't fire.
• The job starts but exits non-zero. A dependency is down, a file is missing, a database query times out. The job ran; it just failed. Without exit code monitoring, you can't tell.
• The job takes far longer than expected. A job that normally takes 30 seconds is now running for 20 minutes, blocking the next scheduled invocation or holding a lock that starves other processes.
• The job produces wrong output silently. The script exits with code 0 but processed zero records instead of the expected 10,000. Success by the scheduler's definition; silent data corruption by yours.

Standard monitoring tools — uptime monitors, APM agents, error trackers — are blind to all four of these scenarios. You need a fundamentally different approach.

Dead Man's Switch Pattern

A dead man's switch inverts the monitoring model. Instead of watching for failure signals, you watch for the absence of a success signal. The job is expected to check in within a defined window. If the check-in doesn't arrive, the monitoring system raises an alert.

The mechanics are straightforward. After a cron job completes successfully, it sends an HTTP GET or POST request (a "ping" or "heartbeat") to a monitoring endpoint. The endpoint records the timestamp. A background process continuously checks: has any active check expired its expected window? If yes, alert.

Here's what a minimal heartbeat looks like at the end of a shell script:

#!/bin/bash
set -euo pipefail

# ... your actual job logic ...
python3 /opt/scripts/sync_invoices.py

# Signal success. Only reached if the script didn't exit early.
curl --silent --max-time 10 \
  "https://hc-ping.com/your-uuid-here" > /dev/null

The set -euo pipefail at the top is critical — without it, a failed command mid-script won't stop execution, and the heartbeat fires even after a partial failure. The --max-time 10 prevents a slow monitoring service from blocking your job indefinitely.

For jobs where start and end timing both matter, most services support a start ping followed by a completion ping:

# Signal job start
curl --silent "https://hc-ping.com/your-uuid-here/start" > /dev/null

# ... job logic ...

# Signal job end (success)
curl --silent "https://hc-ping.com/your-uuid-here" > /dev/null

The monitoring service can now measure execution duration and alert if the job runs longer than a defined threshold — not just if it fails to start.

For signalling failure explicitly (rather than relying on missing pings), append /fail to the URL and call it in a trap:

#!/bin/bash
set -euo pipefail
UUID="your-uuid-here"
BASE="https://hc-ping.com/${UUID}"

cleanup() {
  local exit_code=$?
  if [ $exit_code -ne 0 ]; then
    curl --silent "${BASE}/fail" > /dev/null
  fi
}
trap cleanup EXIT

curl --silent "${BASE}/start" > /dev/null

# ... job logic ...

curl --silent "${BASE}" > /dev/null

This pattern gives you three signal types — start, success, and explicit failure — with zero changes to the core job logic.

Popular Monitoring Services

Several managed services implement the dead man's switch pattern. Here's an honest comparison of the major options:

For most teams running fewer than 20 cron jobs, Healthchecks.io on its free tier is sufficient. Its open-source nature means you can self-host it inside your VPC if compliance requirements prohibit external heartbeat calls. Cronitor is the right choice when you need execution telemetry and trend analysis rather than simple uptime. Dead Man's Snitch suits teams that want zero configuration overhead and just need email when something stops running.

Building a Custom Dead Man's Switch

If you'd rather not depend on a third-party service — or if you need tight integration with your internal tooling — a custom implementation is surprisingly straightforward. You need three components: a heartbeat receiver, a persistent timestamp store, and a checker that runs on its own schedule.

The schema in Postgres (or any relational database) looks like this:

CREATE TABLE cron_heartbeats (
  id          SERIAL PRIMARY KEY,
  job_name    TEXT NOT NULL UNIQUE,
  last_ping   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  interval_s  INTEGER NOT NULL,   -- expected max seconds between pings
  grace_s     INTEGER NOT NULL DEFAULT 60,  -- tolerance window
  is_active   BOOLEAN NOT NULL DEFAULT TRUE
);

-- Index for the checker query
CREATE INDEX ON cron_heartbeats (is_active, last_ping);

The receiver is a minimal HTTP endpoint. In Node.js with no framework:

import { createServer } from 'node:http';
import { Pool } from 'pg';

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

createServer(async (req, res) => {
  const jobName = new URL(req.url ?? '/', 'http://x').pathname.slice(1);
  if (!jobName) { res.writeHead(400).end('missing job name'); return; }

  await pool.query(
    `INSERT INTO cron_heartbeats (job_name, last_ping)
     VALUES ($1, NOW())
     ON CONFLICT (job_name) DO UPDATE SET last_ping = NOW()`,
    [jobName]
  );

  res.writeHead(200).end('ok');
}).listen(8080);

The checker runs as its own cron job (yes, a cron job monitoring cron jobs) at a high frequency — every minute is appropriate:

-- Find jobs that have missed their expected check-in window
SELECT job_name, last_ping, interval_s
FROM cron_heartbeats
WHERE is_active = TRUE
  AND last_ping < NOW() - INTERVAL '1 second' * (interval_s + grace_s);

Any rows returned by this query represent jobs that have gone silent. Pipe these results into your alerting channel of choice. The checker itself should be monitored by an external service (use Healthchecks.io free tier for this meta-monitoring) to ensure the checker doesn't silently fail too — a common oversight that defeats the entire system.

Alerting Strategies

Detecting a failure is half the problem. Getting the right person notified at the right severity level — without creating alert fatigue — is the other half.

PagerDuty and Opsgenie Integration

For incidents that require immediate human intervention, integrate with an on-call platform. Both PagerDuty and Opsgenie accept events via a simple HTTP POST:

curl --request POST \
  --url https://events.pagerduty.com/v2/enqueue \
  --header 'Content-Type: application/json' \
  --data '{
    "routing_key": "YOUR_INTEGRATION_KEY",
    "event_action": "trigger",
    "payload": {
      "summary": "Cron job invoice-sync has not checked in for 35 minutes",
      "severity": "critical",
      "source": "cron-monitor",
      "custom_details": {
        "job": "invoice-sync",
        "last_ping": "2026-06-13T00:15:00Z",
        "expected_interval": "3600s"
      }
    }
  }'

Define escalation policies that match the actual business impact. A nightly report job missing its window at 2 AM is low urgency — notify the on-call engineer via Slack in the morning. A payment sync job missing its 5-minute window at peak hours is P1 — page immediately and escalate to a second responder after 10 minutes of no acknowledgment.

Slack Webhooks

For non-critical alerts or ambient awareness, Slack webhooks are fast to implement and appropriate for business-hours jobs:

curl --request POST \
  --url "$SLACK_WEBHOOK_URL" \
  --header 'Content-Type: application/json' \
  --data "{
    \"text\": \":warning: *Cron alert*: \`${JOB_NAME}\` has not pinged in ${ELAPSED_MINUTES} minutes.\",
    \"blocks\": [{
      \"type\": \"section\",
      \"text\": {
        \"type\": \"mrkdwn\",
        \"text\": \"*Job:* \`${JOB_NAME}\`\\n*Last seen:* ${LAST_PING}\\n*Expected interval:* ${INTERVAL}\"
      }
    }]
  }"

Preventing Alert Fatigue

Alert fatigue is the silent killer of monitoring systems. When engineers start ignoring alerts because they fire too often or too noisily, you've recreated the silent failure problem at the human layer. Practical mitigations:

• Grace windows are not optional. Network blips, slow DNS resolution, and machine load can delay a heartbeat by 30–90 seconds. A grace period of 5% of the job's interval (minimum 60 seconds) prevents constant false positives for healthy jobs.
• Don't alert on the first missed ping. Alert after two consecutive missed windows for non-critical jobs. One missed ping is a transient issue; two in a row is a real problem.
• Implement alert deduplication. If a job is failing and firing alerts every minute, the on-call engineer should receive one incident, not 60 pages. Both PagerDuty and Opsgenie deduplicate by incident key — use a stable key like the job name.
• Auto-resolve when the job recovers. Send a resolve event when the heartbeat comes back. Leaving a stale incident open trains engineers to close alerts without investigating.
• Separate business-hours vs. after-hours severity. A job that only matters for the Monday morning report should not page a human at Saturday 3 AM. Route alerts to a delayed queue or set a time-based escalation policy.

Monitoring Metrics Beyond Uptime

Binary uptime monitoring — did the job run or not — is table stakes. Production-grade monitoring tracks the full execution profile of each job run.

Execution Duration Tracking

Store the start and end timestamps from your heartbeat pings, then compute duration per run. Alert on two conditions:

• Duration exceeds threshold. A job that normally takes 90 seconds and suddenly takes 20 minutes is either running on degraded infrastructure, processing a pathologically large dataset, or has hit a deadlock. Alert at 3× the 30-day p95 duration.
• Duration trend is increasing. If a job's average duration increases 10% week-over-week, it's a scaling signal, not yet an incident. Track this in a time-series database (Prometheus, InfluxDB, or even a Postgres table with weekly aggregates) and add it to a dashboard.

Exit Code Monitoring

Capturing the exit code at the heartbeat call level gives you structured failure data. Extend the heartbeat URL pattern to include the exit code:

EXIT_CODE=$?
curl --silent \
  "https://your-monitor/ping/${JOB_NAME}?exit=${EXIT_CODE}&duration=${DURATION_MS}" \
  > /dev/null

On the receiver side, store the exit code alongside the timestamp. Non-zero exit codes that still reach the heartbeat endpoint (because the trap sends them) become categorized failures you can query: "Show me all jobs that exited non-zero in the last 7 days, grouped by job name and exit code."

Output Size and Record Count Anomalies

Some of the most damaging silent failures produce a successful exit code but wrong results. A data sync job that processed 0 records when it normally processes 50,000 is a bug, not a success. Instrument your jobs to output a record count and include it in the heartbeat metadata:

RECORDS_PROCESSED=$(python3 sync.py --count-only)
curl --silent \
  "https://your-monitor/ping/${JOB_NAME}" \
  --data "records=${RECORDS_PROCESSED}" \
  > /dev/null

Alert if records_processed falls below a configurable minimum threshold. This catches truncated data pulls, empty API responses from upstream services, and filter bugs that cause zero-row results.

Resource Consumption

Wrap critical jobs in a resource-measuring harness to capture peak memory and CPU time alongside each run:

/usr/bin/time -v python3 sync.py 2> /tmp/job_metrics.txt
WALL_TIME=$(grep "Elapsed (wall clock)" /tmp/job_metrics.txt | awk '{print $NF}')
MAX_RSS=$(grep "Maximum resident set size" /tmp/job_metrics.txt | awk '{print $NF}')

A job whose peak RSS doubles between runs is either processing a larger dataset or has a memory leak. Neither is immediately an incident, but both are actionable trends worth tracking before they become one.

Testing Your Monitoring

A monitoring system you've never tested is not a monitoring system — it's a hope. The only way to trust your alerting pipeline is to deliberately break things and verify the alerts fire correctly, at the right severity, to the right people.

Chaos Engineering for Cron

Schedule deliberate skips in a staging environment. Comment out the heartbeat call in one job and verify an alert fires within the expected window plus grace period. Then uncomment it and verify the alert auto-resolves.

For production, use a canary job — a trivial script whose only purpose is to test the monitoring pipeline end to end:

# canary.sh — runs every 5 minutes, sole purpose is to ping
#!/bin/bash
curl --silent --max-time 5 \
  "https://hc-ping.com/canary-uuid" > /dev/null

If the canary alert fires in production, you know the monitoring infrastructure is healthy. If it doesn't fire when you've disabled the canary, your alerting pipeline is broken.

Simulating Failure Modes

Test each distinct failure mode explicitly:

• Missing heartbeat: Kill the cron process or add a sleep 9999 before the ping call. Verify the alert fires after the window + grace period elapses.
• Explicit failure signal: Have a test job call the /fail endpoint directly. Verify the immediate alert fires without waiting for the window to expire.
• Duration threshold breach: Add sleep 300 to a test job that normally completes in 10 seconds. Verify the duration alert fires before the completion ping arrives.
• Checker outage: Temporarily stop the checker service itself. Verify that your meta-monitor (the external service monitoring the checker) alerts within its own window. This is the most commonly skipped test — and the most important.

Runbook Integration

Every alert should link to a runbook. An alert without a runbook is an alert that gets resolved by rebooting the server and hoping the problem doesn't recur. The runbook for a missed cron heartbeat should cover at minimum: how to check if the cron daemon is running, how to check the last run's logs, how to manually trigger the job safely, and who owns the job if the on-call engineer can't resolve it.

Store runbooks in your internal wiki and include the URL in the alert payload. PagerDuty supports a links field in the event payload specifically for this purpose. An alert body that reads "invoice-sync missed its window. Runbook: https://wiki.internal/runbooks/invoice-sync" is worth ten times more than one that just says "alert: invoice-sync DOWN."

Summary

Cron monitoring requires a different mental model than service monitoring. You're watching for absence rather than presence, which means every component of the monitoring stack — the heartbeat sender, the receiver, the checker, the alerting pipeline — must itself be monitored. Build that outermost layer first, then work inward. A well-instrumented cron infrastructure turns silent failures into explicit, actionable incidents with full execution context attached.