Synchronization in Python

What is synchronization?

Synchronization is how you coordinate multiple threads/processes so they:

• don’t corrupt shared data
• don’t run in the wrong order
• can safely communicate

Why it matters

Without synchronization you can get:

• race conditions
• deadlocks
• inconsistent reads

Common synchronization tools

For threads (threading)

• Lock, RLock
• Semaphore, BoundedSemaphore
• Event
• Condition
• Barrier

For processes (multiprocessing)

• multiprocessing.Lock, RLock
• multiprocessing.Semaphore
• multiprocessing.Event
• multiprocessing.Condition

Design tip

Prefer designs that reduce shared mutable state:

• message passing via queue.Queue (threads)
• multiprocessing.Queue (processes)

When you must share state, use synchronization.

Why counter += 1 is three operations

A single statement compiles to a read, an add, and a write. A thread can be suspended between any two of them, and when it resumes it writes back a value computed from stale data:

diagram Diagram mermaid

  sequenceDiagram
  participant A as thread A
  participant M as counter (memory)
  participant B as thread B
  Note over M: counter = 5
  A->>M: read -> 5
  B->>M: read -> 5
  A->>A: add 1 -> 6
  B->>B: add 1 -> 6
  A->>M: write 6
  B->>M: write 6
  Note over M: counter = 6, not 7
one increment vanished

Both threads did their work. One increment is gone, nothing raised, and the total is simply wrong.

The uncomfortable part: you probably cannot reproduce it

Four threads, 200,000 increments each, plain counter += 1 on CPython 3.14.4:

run	result	expected	lost
no lock, plain +=	800,000	800,000	0
no lock, widened read-modify-write, 1 µs switch interval	480,000	480,000	0
no lock, with an explicit yield between read and write	4,000	16,000	12,000 (75%)
with threading.Lock	16,000	16,000	0

The first two rows are the important ones. The race is genuinely present in that code — nothing about it is thread-safe — and it still lost nothing across 800,000 opportunities, because the gap between the read and the write is only a few bytecodes wide and the interpreter rarely switches inside it.

Forcing a yield into that gap loses 75% immediately, and gives the same figure on CPython 3.11.9, so this is not a quirk of one version.

A test that passes proves nothing here

This is why concurrency bugs reach production. They do not fail on your laptop, in CI, or during review; they fail under load, on different hardware, at 3 a.m. Reason about whether shared state is protected — do not conclude it is safe because a test passed.

the_fix.py

import threading
 
counter = 0
lock = threading.Lock()
 
def worker(n):
    global counter
    for _ in range(n):
        with lock:            # the whole read-modify-write is now indivisible
            counter += 1

See it move

Step the two threads by hand and try to lose an increment. With the lock on, the second thread cannot enter until the first has written back.

sketch Losing an increment, one step at a time p5.js Pause Restart

Each thread reads, adds, then writes. Without a lock the interleaving lets one thread overwrite the other's result. With a lock the sequence cannot be split.

The other primitives, measured

Lock is not reentrant. RLock is.

reentrant.py

lk = threading.Lock()
lk.acquire()
lk.acquire(timeout=0.3)      # False — the SAME thread cannot acquire it twice
                             # without a timeout this deadlocks against itself
 
rl = threading.RLock()
rl.acquire()
rl.acquire(timeout=0.3)      # True — counts recursion, needs matching releases

A method that takes a lock and calls another method that takes the same lock will hang on a plain Lock. That is the case RLock exists for.

Semaphore(n) caps how many run at once. 12 jobs of 0.20 s through Semaphore(3): measured peak concurrency exactly 3.

Event releases every waiter at once, not one:

event.py

ev = threading.Event()
# four threads call ev.wait()
# before ev.set(): 0 released
ev.set()
# after  ev.set(): 4 released   <- all of them

Use an Event for “the configuration is loaded, everyone may proceed”. Use a Semaphore when you want to admit a fixed number at a time.

Deadlock needs only two locks and two orders

deadlock.py

def ab():
    with A:
        with B:  ...          # A then B
 
def ba():
    with B:
        with A:  ...          # B then A   <- opposite order

Measured with each thread guaranteed to be holding its first lock: the second acquire timed out, which is what a deadlock looks like when you give it a deadline. Without timeout=, both threads wait forever, consuming no CPU — a process that is perfectly healthy by every metric and doing nothing at all.

The rule that prevents it

Establish one global order for acquiring locks and follow it everywhere. If every thread takes A before B, the cycle cannot form. Where that is impractical, use timeout= and treat failure as a real error rather than retrying immediately.

Check yourself

pch.quizTag pch.quizDefaultTitle

1. Four threads each ran counter += 1 200,000 times with no lock, and the total was exactly 800,000 with nothing lost. What should you conclude?

   A the code is thread-safe, since the result was correct B the GIL makes += atomic, so no lock is needed C the race exists but the window is too narrow to hit reliably; a passing run proves nothing D the threads ran one after another rather than concurrently
   pch.quizShowAnswer

   C — the race exists but the window is too narrow to hit reliably; a passing run proves nothing — Forcing a yield between the read and the write lost 75% immediately, on both 3.14 and 3.11. The unprotected version is unsafe whether or not a given run exposes it — which is why these bugs survive testing.

2. A method holding a threading.Lock calls another method that acquires the same Lock. What happens?

   A it succeeds, since the same thread already owns it B it blocks forever: a plain Lock is not reentrant C it raises RuntimeError immediately D the second acquire is ignored
   pch.quizShowAnswer

   B — it blocks forever: a plain Lock is not reentrant — Lock.acquire from the owning thread returns False with a timeout and blocks without one — the thread deadlocks against itself. RLock counts recursion and is the right tool here.

3. Four threads are waiting on a threading.Event. What happens when one thread calls ev.set()?

   A one waiter is released, like a queue B all four waiters are released C the waiters are released one per subsequent set() call D nothing until clear() is called
   pch.quizShowAnswer

   B — all four waiters are released — An Event is a broadcast flag: set() releases every waiter. Measured 0 released before set and 4 after. Use a Semaphore when you want to admit a fixed number.

4. Thread 1 takes lock A then B; thread 2 takes B then A. What is the standard fix?

   A give both threads a shorter timeout B replace both locks with RLock C acquire the locks in the same global order everywhere D run the threads at different priorities
   pch.quizShowAnswer

   C — acquire the locks in the same global order everywhere — A cycle in the wait graph is what causes the deadlock. One consistent ordering makes the cycle impossible. Timeouts detect the problem rather than preventing it.

🧪 Try It Yourself

Exercise 1 – Basic Lock Usage

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Lock as Context Manager

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Detect Locked State

pch.exerciseTag pch.exerciseTitle