Sharing Data (Queue, Pipe, Manager)

Why sharing is different

Processes do not share memory like threads.

So you can’t safely update a global variable and expect other processes to see it.

multiprocessing.Queue (recommended)

Use a Queue to send messages/results.

mp_queue.py

from multiprocessing import Process, Queue
 
 
def worker(q: Queue, x: int) -> None:
    q.put(x * x)
 
 
if __name__ == "__main__":
    q = Queue()
 
    procs = [Process(target=worker, args=(q, i)) for i in range(5)]
    for p in procs:
        p.start()
 
    results = [q.get() for _ in procs]
 
    for p in procs:
        p.join()
 
    print(sorted(results))

Pipe

Pipe is a two-way connection.

mp_pipe.py

from multiprocessing import Process, Pipe
 
 
def worker(conn):
    conn.send("hello")
    conn.close()
 
 
if __name__ == "__main__":
    parent, child = Pipe()
    p = Process(target=worker, args=(child,))
    p.start()
    print(parent.recv())
    p.join()

Manager (shared dict/list)

Manager provides proxy objects.

mp_manager.py

from multiprocessing import Process, Manager
 
 
def worker(shared, i):
    shared[i] = i * i
 
 
if __name__ == "__main__":
    with Manager() as manager:
        shared = manager.dict()
 
        procs = [Process(target=worker, args=(shared, i)) for i in range(5)]
        for p in procs:
            p.start()
        for p in procs:
            p.join()
 
        print(dict(shared))

Guidance

• Prefer passing data via Queue when possible.
• Use Manager for simple shared state (slower than local memory).

Choosing a channel

Processes do not share memory, so everything they exchange must be copied across a boundary. The four tools differ in shape, not in power:

diagram Diagram mermaid

  flowchart TD
  S["child must send something back"] --> Q{"what shape?"}
  Q -->|"a stream of results,
any number of workers"| QU["Queue
FIFO, many-to-many"]
  Q -->|"a private two-way link
between exactly 2 processes"| PI["Pipe
fastest, 2 ends only"]
  Q -->|"a dict or list that
several processes edit"| MA["Manager
proxied, a server process"]
  Q -->|"a single number or
fixed block of numbers"| VA["Value / Array
true shared memory"]
  VA --> LK["needs an explicit Lock"]

Start with the mistake this all exists to prevent:

not_shared.py

import multiprocessing as mp
 
def add(lst):
    lst.append("from child")
 
if __name__ == "__main__":
    lst = []
    p = mp.Process(target=add, args=(lst,)); p.start(); p.join()
    print(lst)          # []   <- the child appended to ITS OWN copy

The list was pickled into the child. The child’s append succeeded — in the child. The parent’s list is untouched, and no error is raised anywhere.

The four channels, measured

channels.py

q = mp.Queue()                     # FIFO, any number of readers and writers
# child: q.put(i*i) for i in range(5); q.put(None)
# parent receives: [0, 1, 4, 9, 16]      None used as the "done" sentinel
 
parent_conn, child_conn = mp.Pipe()   # exactly two ends
# child: conn.send({"pid": ..., "payload": [1, 2, 3]})
# parent: conn.recv() -> {'pid': 19392, 'payload': [1, 2, 3]}
 
with mp.Manager() as m:            # a real server process holding the objects
    d, l = m.dict(), m.list([1, 2])
    # after the child writes: d -> {'child': 'wrote this'}   l -> [1, 2, 99]

A Pipe carries arbitrary picklable objects, not bytes — the dict above arrived intact. A Manager dict is a proxy: every read and write is a round trip to a separate server process, which is why it is the most convenient and the slowest.

The sentinel is not optional

q.get() blocks forever when the queue is empty. A consumer loop needs a way to know production has stopped — conventionally one None per producer. Forgetting it hangs the program with no error and no CPU use, which is the hardest kind of failure to find.

See it move

Value is genuine shared memory, and that is exactly why it can be corrupted. v.value += 1 is three operations — read, add, write — and a second process can land between them. Run it with the lock off and watch updates disappear.

sketch Lost updates without a Lock p5.js Pause Restart

v.value += 1 is read, add, write. Two processes interleaving those steps overwrite each other, so increments are silently lost.

Two processes, 10,000 increments each, measured:

	result	expected	lost
with mp.Lock()	20,000	20,000	0
without a lock	15,347	20,000	4,653

Nearly a quarter of the work vanished, with no exception and no warning. The number differs on every run, which is the signature of a race — and the reason a test that passes once proves nothing here.

locking.py

v = mp.Value("i", 0)
lock = mp.Lock()
 
def worker(v, lock):
    for _ in range(10_000):
        with lock:          # remove this line and increments are lost
            v.value += 1

What the copying costs

cost.py

# 200,000 ints put on a Queue and taken off again: 33 ms

Every object crossing a process boundary is pickled on one side and unpickled on the other. That is why sending a large DataFrame to a worker can cost more than the work the worker was sent to do — and why it is usually better to send a description of the task, like a range of indices, and let the child load the data itself.

Check yourself

pch.quizTag pch.quizDefaultTitle

1. A parent passes an empty list to a child, which appends to it. What does the parent see afterwards?

   A the appended item, since lists are mutable B an empty list, because the child received a pickled copy C a TypeError, because lists cannot be passed to processes D the appended item, but only after join()
   pch.quizShowAnswer

   B — an empty list, because the child received a pickled copy — Arguments are pickled into the child. The child mutated its own copy successfully, so nothing raises — the parent's list is simply still empty.

2. Two processes each run v.value += 1 ten thousand times on a shared Value with no Lock. What is a realistic result?

   A exactly 20000, because Value is shared memory B exactly 10000, because one process overwrites the other entirely C a varying number below 20000, such as 15347 D a deadlock
   pch.quizShowAnswer

   C — a varying number below 20000, such as 15347 — += is read, add, write. Interleaving those steps loses updates. Measured 15347 of 20000 on one run, and a different figure on the next — that variability is what identifies a race.

3. Why does a consumer loop reading mp.Queue conventionally need a sentinel such as None?

   A to mark which producer sent each item B because q.get() on an empty queue blocks forever, so the consumer needs an explicit end signal C because Queue cannot hold more than a fixed number of items D to force the queue to flush its buffer
   pch.quizShowAnswer

   B — because q.get() on an empty queue blocks forever, so the consumer needs an explicit end signal — An empty queue is indistinguishable from one whose producer is merely slow, so get() waits. Without a sentinel the program hangs using no CPU and raising nothing.

4. Which channel is a proxy backed by a separate server process, making it the most convenient and the slowest?

   A Pipe B Queue C Manager D Array
   pch.quizShowAnswer

   C — Manager — A Manager hosts the real dict or list in its own process; every read and write is a round trip. Value and Array are true shared memory, and Pipe is a direct two-ended link.

🧪 Try It Yourself

Exercise 1 – Start a Process

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Process Pool map()

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Multiprocessing Queue

pch.exerciseTag pch.exerciseTitle