Thread Communication (Queue)

  flowchart LR
  A["producer thread"] --> B["q.put(item)"]
  B --> C{"maxsize reached?"}
  C -->|yes| D["producer BLOCKS -- back-pressure"]
  C -->|no| E["item queued"]
  E --> F["q.get() in a worker"]
  F --> G{"empty?"}
  G -->|yes| H["worker BLOCKS -- no polling, no spinning"]
  G -->|no| I["process it"]
  I --> J["q.task_done()"]
  J --> K["q.join() returns when every put has a matching task_done"]
  L["a sentinel value"] --> M["is how workers are told to stop"]

Why Queue?

queue.Queue is thread-safe.

Use it to:

• pass tasks from producer threads to worker threads
• avoid manual lock management

Producer-consumer example

import threading
import queue
import time
 
q = queue.Queue()
 
 
def producer():
    for i in range(5):
        q.put(i)
        print("produced", i)
    q.put(None)  # sentinel
 
 
def consumer():
    while True:
        item = q.get()
        try:
            if item is None:
                break
            print("consuming", item)
            time.sleep(0.2)
        finally:
            q.task_done()
 
 
t1 = threading.Thread(target=producer)
t2 = threading.Thread(target=consumer)
 
t1.start(); t2.start()
 
t1.join()
q.join()  # wait until all tasks are marked done
print("All tasks processed")

Tips

• Use a sentinel (None) to stop consumers.
• Always call task_done() (often in a finally).

Why threads need a safe channel

Threads share memory, which makes it easy to pass data around and equally easy to corrupt it. Sharing a plain list means guarding every read and write with a lock. queue.Queue does that work for you: put and get are already thread-safe, and get waits politely when the queue is empty, so a consumer never spins in a busy loop.

Following the example

The producer puts five numbers on the queue and then a None sentinel. The consumer loops on q.get(), which blocks until an item is available. For each real item it prints, does some work, and calls task_done() in a finally block. When it receives None it breaks out. In the main thread, t1.join() waits for the producer, and q.join() waits until every item, including the sentinel, has been marked done. The output lines from producer and consumer may interleave differently on each run, which is normal.

Several workers sharing a job list

A queue also spreads work across several consumers. Each item goes to exactly one consumer:

import queue
import threading
 
jobs: queue.Queue = queue.Queue()
results: queue.Queue = queue.Queue()
 
 
def worker() -> None:
    while True:
        n = jobs.get()
        if n is None:
            jobs.task_done()
            return
        results.put(n * n)
        jobs.task_done()
 
 
threads = [threading.Thread(target=worker) for _ in range(3)]
for t in threads:
    t.start()
 
for n in range(10):
    jobs.put(n)
for _ in threads:
    jobs.put(None)  # one stop signal per worker
 
jobs.join()
for t in threads:
    t.join()
 
print(sorted(results.get() for _ in range(results.qsize())))

The output is [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]. Sorting makes it independent of which worker finished first.

Useful variations

• Queue(maxsize=10) makes put wait when full, which applies backpressure to fast producers.
• q.get(timeout=1) raises queue.Empty instead of waiting forever.
• q.get_nowait() and q.put_nowait() never block.
• queue.LifoQueue and queue.PriorityQueue change the order items come out.

Common mistakes

• Forgetting task_done(), so join() never returns.
• Too few sentinels for the number of consumers.
• Using a plain list shared between threads without a lock.
• Using qsize() or empty() to make decisions. Another thread may change the queue right after you check.

When to use it

Use a queue whenever one thread produces work and others consume it: downloaders feeding a parser, a logger thread collecting messages, or a pool of workers processing jobs.

Check yourself

🧪 Try It Yourself

Exercise 1 – Put and Get

Press Run to see what your code prints.

Exercise 2 – Producer-Consumer

Press Run to see what your code prints.

Exercise 3 – Queue Size

Press Run to see what your code prints.