Preventing Cron Job Overlap: Locking Strategies

Cron jobs are assumed to be discrete, isolated events. Schedule a job for every five minutes, and the mental model is clean: it runs, finishes, and the next invocation starts fresh. That model breaks the moment a job takes longer than its interval — and when it breaks, the consequences range from annoying to catastrophic. Duplicate payment charges, corrupted database records, servers grinding to a halt under a pile of zombie processes: all of these are downstream effects of one root cause, the overlap problem. This guide covers every practical strategy for preventing it, from a single Linux server all the way to distributed infrastructure.

The Overlap Problem

Consider a billing job scheduled to run every minute. On most days it finishes in fifteen seconds. On the last day of the month, it processes ten times the normal volume and takes ninety seconds. By the time it is done, the cron daemon has already spawned a second instance at the one-minute mark and a third at the two-minute mark. All three are now reading the same queue of unpaid invoices. Each one charges the same customers independently, and your support queue fills up with furious emails before you finish your morning coffee.

The overlap problem has three distinct failure modes. The first is duplicate work: multiple instances process the same records, leading to double charges, duplicate emails, or redundant API calls that exhaust rate limits. The second is data corruption: two instances read a shared resource, compute divergent results, and write conflicting state back. The third is resource exhaustion: a slow job spawns a chain of children, each one waiting for a database connection or a lock on a file, until the server runs out of file descriptors, memory, or database pool slots.

None of these failures are hypothetical. They are the kind of incident that wakes engineers up at 3 AM. The solution is a locking strategy applied at the right layer of your stack.

File-Based Locking with flock

On a single Linux server, flock is the simplest and most reliable locking primitive available. It is a POSIX advisory lock that the kernel enforces. If a process holds an exclusive lock on a file and a second process tries to acquire the same lock, the second process either blocks or exits immediately, depending on how you call it.

The key flag is -n, which makes flock non-blocking. Instead of waiting, the second invocation exits with a non-zero status code the moment it finds the lock held. This is exactly the behavior you want for cron jobs: skip this run, do not wait for the previous one.

# Wrap any cron command with flock
* * * * * /usr/bin/flock -n /var/lock/billing.lock /opt/app/bin/billing-runner

The lock file path is arbitrary. Using /var/lock/ is conventional because it is cleared on reboot, which prevents stale locks from blocking jobs after an unclean shutdown. The lock is automatically released by the kernel when the process exits, even if it crashes, so there is no manual cleanup required.

You can also wrap flock in a shell script when you need more control over logging or error handling:

#!/usr/bin/env bash
set -euo pipefail

LOCK_FILE=/var/lock/billing.lock
EXEC_CMD=/opt/app/bin/billing-runner

if ! flock -n "${LOCK_FILE}" -c "${EXEC_CMD}"; then
  echo "[$(date --iso-8601=seconds)] billing-runner already running, skipping." >> /var/log/cron-skip.log
  exit 0
fi

An alternative to flock is a PID file. The job writes its process ID to a known file on startup, checks whether a process with that PID already exists before starting, and removes the file on exit. PID files require more application-level code and are susceptible to race conditions in the check-then-write window. Prefer flock on Linux; PID files make sense when flock is unavailable or when you need to inspect the running process from another tool.

flock Limitations

flock only works on a single host. If your cron jobs run across multiple servers — even two — a file lock provides zero protection. The lock file exists on one server's filesystem; the other server cannot see it. The moment your infrastructure scales horizontally, you need a distributed strategy.

Database Advisory Locks

PostgreSQL ships with advisory locks: application-level locks that the database manages but that carry no automatic semantics. They are purely cooperative signals between clients. This makes them a clean fit for distributed cron coordination, because every application server already has a database connection and does not need additional infrastructure.

The core function is pg_try_advisory_lock(key bigint). It returns true if the lock was acquired and false if another session already holds it. Unlike pg_advisory_lock, the try variant is non-blocking — essential for cron overlap prevention.

-- Attempt to acquire a session-level advisory lock with a stable integer key
-- Returns true if acquired, false if already held
SELECT pg_try_advisory_lock(42);

Session-level locks persist until the database connection closes or until you explicitly call pg_advisory_unlock. This is usually the right choice for cron jobs: the lock lives as long as the job runs and releases automatically when the connection drops, even on crash.

In application code, the pattern looks like this:

-- Inside your job's database transaction or connection lifecycle
DO $$
BEGIN
  IF NOT pg_try_advisory_lock(12345) THEN
    RAISE NOTICE 'Job already running, exiting.';
    RETURN;
  END IF;

  -- Your job logic runs here
  PERFORM process_pending_invoices();

  PERFORM pg_advisory_unlock(12345);
END;
$$;

For timeout patterns — where you want to wait briefly in case the existing job finishes quickly — you can use lock_timeout:

SET lock_timeout = '5s';
SELECT pg_advisory_lock(12345); -- blocks up to 5 seconds, then raises an error

Use a consistent hashing scheme to map job names to integer keys. A simple approach: hashtext('billing_runner') returns a stable bigint, eliminating the need to maintain a manual registry of lock IDs.

Transaction-Level Advisory Locks

pg_try_advisory_xact_lock releases the lock automatically at transaction commit or rollback, not at connection close. This can be convenient if your entire job runs inside a single transaction, but it also means the lock vanishes the moment you commit intermediate results. For long-running jobs with multiple commit points, session-level locks are the safer choice.

Redis-Based Distributed Locking

Redis is the most common choice for distributed locking outside of a relational database. The atomic SET ... NX ... EX command sets a key only if it does not exist and assigns an expiry in one operation. If the key is already set, the command returns nil and the calling job exits.

# Acquire the lock: set key "lock:billing" only if not exists, expire in 120 seconds
SET lock:billing ${UNIQUE_TOKEN} NX EX 120

The expiry is critical. It is your safety net against a dead process that never released its lock. Set it to a generous multiple of your expected job duration — long enough that a legitimately slow job does not lose the lock mid-run, but short enough that a crashed job does not block the next scheduled interval for too long.

The {UNIQUE_TOKEN} value matters for safe release. Each job instance generates a random token (UUID or similar) and stores it as the lock value. On release, it checks that the value matches before deleting the key. Without this check, a slow job could release a lock that a different instance already re-acquired after the first instance's token expired.

-- Lua script for safe release (atomic check-and-delete)
if redis.call("get", KEYS[1]) == ARGV[1] then
  return redis.call("del", KEYS[1])
else
  return 0
end

Execute this as a Lua script via EVAL for atomicity. A plain GET followed by DEL has a race condition window between the two commands.

The Redlock Algorithm

Single-node Redis is sufficient for most use cases: if your Redis instance goes down, your jobs do not run anyway, so the lock state is irrelevant. However, if you need fault tolerance — guaranteeing mutual exclusion across a Redis cluster where individual nodes can fail — the Redlock algorithm is the standard approach.

Redlock acquires the lock on N independent Redis nodes (typically five) with a quorum requirement of N/2+1 successful acquisitions. If the job cannot acquire the lock on a majority of nodes within a small timeout window, it treats the attempt as failed and releases any partial acquisitions. This prevents a single Redis node failure from causing split-brain lock state.

Libraries implementing Redlock exist for every major language: redlock-py for Python, redlock-rb for Ruby, node-redlock for Node.js. Unless you have a strong availability requirement for the lock service itself, single-node Redis with a sensible expiry handles the cron overlap problem effectively and with far less operational complexity.

Application-Level Patterns

Infrastructure locks catch the problem at the process boundary. Application-level patterns catch it earlier, inside the job's own logic, and provide richer semantics like retrying, queueing, and state introspection.

Job State Machines

Persist job state in a database table with a status column: idle, running, completed, failed. The job's first action is to atomically transition from idle to running. If the transition fails — because the status is already running — it exits immediately.

-- Atomic claim: only succeeds if status is currently idle
UPDATE cron_jobs
SET status = 'running', started_at = now(), worker_id = '${WORKER_ID}'
WHERE job_name = 'billing_runner'
  AND status = 'idle'
RETURNING id;

If the UPDATE returns zero rows, the job is already running. On completion or error, the job updates the status back to idle or failed. A separate watchdog query identifies jobs stuck in running for longer than their expected maximum duration and resets them — your automated recovery from crashes.

In-Process Semaphores

For multi-threaded applications where cron triggers internal functions rather than launching new processes, use language-level semaphores or atomic flags. In most runtimes, a simple boolean flag protected by a mutex is sufficient:

// Go example: atomic flag prevents re-entrance within the same process
var billingRunning int32

func runBillingJob() {
    if !atomic.CompareAndSwapInt32(&billingRunning, 0, 1) {
        log.Println("billing job already running, skipping")
        return
    }
    defer atomic.StoreInt32(&billingRunning, 0)

    // job logic here
}

This only prevents overlap within a single process instance. Combine it with an infrastructure-level lock for multi-process or distributed deployments.

Monitoring Overlapping Runs

Locks prevent the bad outcome, but monitoring tells you when your jobs are approaching the danger zone. A job that consistently takes 55 seconds on a 60-second interval is one slow query away from overlapping. You want to know that before it happens.

The essential metrics to track for every cron job are: execution duration (wall clock time from start to finish), lock acquisition time (time spent waiting to acquire the lock, which should normally be zero), lock contention count (how many times a job was skipped because the lock was held), and job lag (actual start time minus scheduled start time).

Emit these as structured log events or time-series metrics. A simple shell wrapper captures duration without any application changes:

#!/usr/bin/env bash
JOB_NAME="billing_runner"
START=$(date +%s%3N)

if ! flock -n /var/lock/${JOB_NAME}.lock /opt/app/bin/${JOB_NAME}; then
  echo '{"event":"lock_contention","job":"'${JOB_NAME}'","ts":"'$(date --iso-8601=seconds)'"}'
  exit 0
fi

END=$(date +%s%3N)
DURATION=$((END - START))
echo '{"event":"job_complete","job":"'${JOB_NAME}'","duration_ms":'${DURATION}',"ts":"'$(date --iso-8601=seconds)'"}'

Set alerts on two conditions: lock contention rate rising above zero (any skipped run is worth investigating), and execution duration exceeding 80% of the schedule interval (your early warning threshold). If a 60-second job starts taking 50 seconds regularly, it is time to optimize or increase the interval before overlap becomes inevitable.

For distributed systems, track lock holder identity — which server, which process — in the lock metadata. When contention occurs, you can immediately identify whether the previous instance is genuinely running or is a ghost from a crashed server that never released its lock.

Choosing the Right Strategy

Every strategy involves trade-offs between complexity, infrastructure requirements, and correctness guarantees. Use the following table to match your situation to the right approach.

The decision path is straightforward. Start at infrastructure: if you are on a single Linux server and speed is your priority, flock is the answer — it takes sixty seconds to add to any cron entry and requires zero code changes. If you are already running multiple servers or containers, you need a distributed strategy. If your application already connects to Postgres, advisory locks are the path of least resistance: no new services, no new dependencies. If you have Redis in your stack for caching or queues, extend it to cover locking with SET NX.

Reserve Redlock for situations where the Redis lock service itself must survive node failures independently of your application. For the vast majority of cron overlap problems, single-node Redis is correct and sufficient. The operational cost of a five-node Redlock cluster is rarely justified by the actual failure scenarios it protects against.

Job state machines are worth the additional complexity when you need visibility into job history, the ability to manually trigger retries, or audit trails for compliance. They are not locking primitives in isolation — they should be combined with a database-level lock (advisory lock or row-level lock on the status update) to close the race condition window in the claim step.

Whatever strategy you choose, pair it with duration monitoring and lock contention alerting from day one. A lock prevents the overlap; the metrics tell you when you are trending toward a situation where the lock will be exercised frequently, which is a signal that the underlying job performance problem needs to be addressed at the root rather than papered over with better locking alone.