Why sharing is different
Processes do not share memory like threads.
So you can’t safely update a global variable and expect other processes to see it.
multiprocessing.Queue (recommended)
Use a Queue to send messages/results.
from multiprocessing import Process, Queue
def worker(q: Queue, x: int) -> None:
q.put(x * x)
if __name__ == "__main__":
q = Queue()
procs = [Process(target=worker, args=(q, i)) for i in range(5)]
for p in procs:
p.start()
results = [q.get() for _ in procs]
for p in procs:
p.join()
print(sorted(results))Pipe
Pipe is a two-way connection.
from multiprocessing import Process, Pipe
def worker(conn):
conn.send("hello")
conn.close()
if __name__ == "__main__":
parent, child = Pipe()
p = Process(target=worker, args=(child,))
p.start()
print(parent.recv())
p.join()Manager (shared dict/list)
Manager provides proxy objects.
from multiprocessing import Process, Manager
def worker(shared, i):
shared[i] = i * i
if __name__ == "__main__":
with Manager() as manager:
shared = manager.dict()
procs = [Process(target=worker, args=(shared, i)) for i in range(5)]
for p in procs:
p.start()
for p in procs:
p.join()
print(dict(shared))Guidance
- Prefer passing data via Queue when possible.
- Use Manager for simple shared state (slower than local memory).
Choosing a channel
Processes do not share memory, so everything they exchange must be copied across a boundary. The four tools differ in shape, not in power:
flowchart TD
S["child must send something back"] --> Q{"what shape?"}
Q -->|"a stream of results,
any number of workers"| QU["Queue
FIFO, many-to-many"]
Q -->|"a private two-way link
between exactly 2 processes"| PI["Pipe
fastest, 2 ends only"]
Q -->|"a dict or list that
several processes edit"| MA["Manager
proxied, a server process"]
Q -->|"a single number or
fixed block of numbers"| VA["Value / Array
true shared memory"]
VA --> LK["needs an explicit Lock"]
Start with the mistake this all exists to prevent:
import multiprocessing as mp
def add(lst):
lst.append("from child")
if __name__ == "__main__":
lst = []
p = mp.Process(target=add, args=(lst,)); p.start(); p.join()
print(lst) # [] <- the child appended to ITS OWN copyThe list was pickled into the child. The child's append succeeded — in the child. The parent's list is untouched, and no error is raised anywhere.
The four channels, measured
q = mp.Queue() # FIFO, any number of readers and writers
# child: q.put(i*i) for i in range(5); q.put(None)
# parent receives: [0, 1, 4, 9, 16] None used as the "done" sentinel
parent_conn, child_conn = mp.Pipe() # exactly two ends
# child: conn.send({"pid": ..., "payload": [1, 2, 3]})
# parent: conn.recv() -> {'pid': 19392, 'payload': [1, 2, 3]}
with mp.Manager() as m: # a real server process holding the objects
d, l = m.dict(), m.list([1, 2])
# after the child writes: d -> {'child': 'wrote this'} l -> [1, 2, 99]A Pipe carries arbitrary picklable objects, not bytes — the dict above arrived
intact. A Manager dict is a proxy: every read and write is a round trip to a
separate server process, which is why it is the most convenient and the slowest.
See it move
Value is genuine shared memory, and that is exactly why it can be corrupted.
v.value += 1 is three operations — read, add, write — and a second process can land
between them. Run it with the lock off and watch updates disappear.
Two processes, 10,000 increments each, measured:
| result | expected | lost | |
|---|---|---|---|
with mp.Lock() | 20,000 | 20,000 | 0 |
| without a lock | 15,347 | 20,000 | 4,653 |
Nearly a quarter of the work vanished, with no exception and no warning. The number differs on every run, which is the signature of a race — and the reason a test that passes once proves nothing here.
v = mp.Value("i", 0)
lock = mp.Lock()
def worker(v, lock):
for _ in range(10_000):
with lock: # remove this line and increments are lost
v.value += 1What the copying costs
# 200,000 ints put on a Queue and taken off again: 33 msEvery object crossing a process boundary is pickled on one side and unpickled on the other. That is why sending a large DataFrame to a worker can cost more than the work the worker was sent to do — and why it is usually better to send a description of the task, like a range of indices, and let the child load the data itself.
Check yourself
A parent passes an empty list to a child, which appends to it. What does the parent see afterwards?
Arguments are pickled into the child. The child mutated its own copy successfully, so nothing raises — the parent's list is simply still empty.
Show answer
B — an empty list, because the child received a pickled copy — Arguments are pickled into the child. The child mutated its own copy successfully, so nothing raises — the parent's list is simply still empty.
Two processes each run v.value += 1 ten thousand times on a shared Value with no Lock. What is a realistic result?
+= is read, add, write. Interleaving those steps loses updates. Measured 15347 of 20000 on one run, and a different figure on the next — that variability is what identifies a race.
Show answer
C — a varying number below 20000, such as 15347 — += is read, add, write. Interleaving those steps loses updates. Measured 15347 of 20000 on one run, and a different figure on the next — that variability is what identifies a race.
Why does a consumer loop reading mp.Queue conventionally need a sentinel such as None?
An empty queue is indistinguishable from one whose producer is merely slow, so get() waits. Without a sentinel the program hangs using no CPU and raising nothing.
Show answer
B — because q.get() on an empty queue blocks forever, so the consumer needs an explicit end signal — An empty queue is indistinguishable from one whose producer is merely slow, so get() waits. Without a sentinel the program hangs using no CPU and raising nothing.
Which channel is a proxy backed by a separate server process, making it the most convenient and the slowest?
A Manager hosts the real dict or list in its own process; every read and write is a round trip. Value and Array are true shared memory, and Pipe is a direct two-ended link.
Show answer
C — Manager — A Manager hosts the real dict or list in its own process; every read and write is a round trip. Value and Array are true shared memory, and Pipe is a direct two-ended link.
🧪 Try It Yourself
Exercise 1 – Start a Process
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 2 – Process Pool map()
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 3 – Multiprocessing Queue
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.