Python Threading Locks and Semaphores — Core Concepts

The Problem: Race Conditions

When multiple threads access shared data without coordination, you get race conditions — bugs that depend on the timing of thread execution:

import threading

counter = 0

def increment():
    global counter
    for _ in range(100_000):
        counter += 1  # NOT atomic: read → increment → write

threads = [threading.Thread(target=increment) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(counter)  # Expected: 400000, Actual: something less

counter += 1 is actually three operations. Two threads can read the same value, both increment it, and both write back — losing one increment.

Lock: Mutual Exclusion

A Lock ensures only one thread executes a critical section at a time:

lock = threading.Lock()
counter = 0

def safe_increment():
    global counter
    for _ in range(100_000):
        with lock:
            counter += 1  # only one thread at a time

The with statement acquires the lock on entry and releases it on exit — even if an exception occurs. This is always preferable to manual lock.acquire() / lock.release().

RLock: Reentrant Lock

A regular Lock will deadlock if the same thread tries to acquire it twice. An RLock (reentrant lock) allows the same thread to acquire it multiple times:

rlock = threading.RLock()

def outer():
    with rlock:
        inner()  # same thread acquires rlock again — OK with RLock

def inner():
    with rlock:
        print("Inside inner")

The RLock tracks which thread owns it and a recursion count. It only fully releases when the count drops to zero.

When to use: Functions that call other functions which also need the lock. Without RLock, you’d need to restructure your code to avoid double-acquisition.

Semaphore: Limited Concurrency

A Semaphore allows up to N threads to access a resource simultaneously:

# Allow 3 concurrent database connections
db_semaphore = threading.Semaphore(3)

def query_database(query):
    with db_semaphore:
        connection = get_connection()
        try:
            return connection.execute(query)
        finally:
            release_connection(connection)

The semaphore maintains an internal counter. acquire() decrements it (blocking if zero). release() increments it. Think of it as N parking spots — once all are full, new cars wait.

BoundedSemaphore: With Safety Rails

BoundedSemaphore works like Semaphore but raises ValueError if you release() more times than you acquire():

pool = threading.BoundedSemaphore(5)
# pool.release() without acquire → ValueError

Use BoundedSemaphore by default — it catches bugs where releases are accidentally duplicated. Regular Semaphore silently increases the count beyond the initial value, which can mask errors.

Deadlocks

Deadlocks occur when two threads each hold a lock the other needs:

lock_a = threading.Lock()
lock_b = threading.Lock()

def thread_1():
    with lock_a:        # holds A
        with lock_b:    # waits for B
            work()

def thread_2():
    with lock_b:        # holds B
        with lock_a:    # waits for A — DEADLOCK
            work()

Prevention strategies:

1. Lock ordering: Always acquire locks in the same order (A before B, never B before A).
2. Timeout: Use lock.acquire(timeout=5) to detect deadlocks.
3. Minimize scope: Hold locks for the shortest time possible.

Condition Variables

For “wait until something changes” patterns, combine a lock with a Condition:

condition = threading.Condition()
queue = []

def producer():
    with condition:
        queue.append(item)
        condition.notify()  # wake one waiting consumer

def consumer():
    with condition:
        while not queue:
            condition.wait()  # release lock and sleep
        item = queue.pop(0)

wait() releases the lock, sleeps until notified, then re-acquires the lock. This avoids busy-waiting (checking in a loop without sleeping).

Common Misconception

“The GIL makes locks unnecessary.” The GIL prevents crashes from concurrent memory access, but it doesn’t prevent logical race conditions. The counter += 1 example above breaks even with the GIL because the operation spans multiple bytecodes, and the GIL can release between them. Always use locks for shared mutable state.

One thing to remember: Use Lock for one-at-a-time access, Semaphore for limited-concurrency access, and always use with statements to guarantee the lock is released — forgetting to release is the most common source of deadlocks.