Python Distributed Locks — Core Concepts

Why Distributed Locks?

When multiple processes or servers access a shared resource, you need coordination. A regular threading.Lock only works within a single process. A distributed lock works across processes, machines, and data centers.

Common use cases:

• Prevent duplicate processing — only one worker processes a given job
• Mutual exclusion — only one process runs a migration or cron job at a time
• Resource protection — limit concurrent access to a rate-limited API
• Leader election — one server becomes the coordinator while others wait

Redis-Based Locks

Redis is the most popular choice for distributed locks in Python because it’s fast, widely deployed, and supports atomic operations.

Basic Pattern with redis-py

The redis-py library includes a built-in lock:

import redis

r = redis.Redis()

# Acquire a lock with 30-second timeout
lock = r.lock('my-resource-lock', timeout=30)

if lock.acquire(blocking=True, blocking_timeout=10):
    try:
        # Critical section — only one process runs this
        process_resource()
    finally:
        lock.release()

Under the hood, this uses SET key value NX EX timeout — an atomic set-if-not-exists with an expiration. The value is a unique token so only the lock holder can release it.

Key Parameters

• timeout — how long the lock lives. After this, it auto-expires even if not released. Prevents deadlocks from crashed processes.
• blocking_timeout — how long to wait to acquire the lock. If None, wait forever.
• sleep — polling interval while waiting for the lock. Default is 0.1 seconds.

Database Advisory Locks

PostgreSQL and MySQL offer advisory locks — lightweight locks managed by the database:

# PostgreSQL advisory lock with SQLAlchemy
from sqlalchemy import text

def with_advisory_lock(session, lock_id, callback):
    # pg_advisory_lock blocks until acquired
    session.execute(text(
        f"SELECT pg_advisory_lock({lock_id})"
    ))
    try:
        return callback()
    finally:
        session.execute(text(
            f"SELECT pg_advisory_unlock({lock_id})"
        ))

Advisory locks are free (no table rows created), fast, and automatically released when the database connection closes. They’re a good choice when you’re already using PostgreSQL and don’t want to add Redis just for locking.

Lock Safety

The Timeout Dilemma

Set the timeout too short: the lock expires while you’re still working, and another process barges in. Set it too long: a crashed process blocks everyone for the full timeout duration.

There’s no perfect answer. The practical approach:

1. Set a generous timeout (2-3x your expected processing time)
2. Extend the lock periodically if the operation is still running
3. Monitor for timeout-related issues

Fencing Tokens

Even with proper timeouts, there’s a subtle race condition. Process A acquires the lock, gets delayed (GC pause, network issue), the lock expires, Process B acquires the lock, and now both think they hold it.

Fencing tokens solve this: each lock acquisition gets a monotonically increasing token. The protected resource rejects requests with tokens older than the last one it saw.

When Not to Use Distributed Locks

• Idempotent operations — if running twice is harmless, you don’t need a lock
• Database transactions — if the contention is purely within a database, use SELECT FOR UPDATE instead
• Single-process systems — use threading.Lock or asyncio.Lock

Common Misconception

“Distributed locks guarantee mutual exclusion.” They don’t — at least, not with single-node Redis. Network partitions, clock skew, and Redis failovers can all cause two processes to hold the same lock simultaneously. Distributed locks provide mutual exclusion most of the time, which is sufficient for most applications. For stronger guarantees, you need consensus protocols (Redlock, ZooKeeper, etcd).

One thing to remember: Always use lock timeouts (auto-expiry) in distributed locks. A lock without a timeout is a deadlock waiting to happen — one crashed process will lock out everyone else permanently.