Multiprocessing in Python

What is multiprocessing?

Multiprocessing runs work in multiple processes.

• Each process has its own Python interpreter and memory space.
• This can use multiple CPU cores.

Why multiprocessing helps CPU-heavy work

In CPython, threads are limited by the GIL for CPU-bound code.

Multiprocessing avoids this because:

• each process has its own GIL

When to use multiprocessing

Best for CPU-bound tasks:

• heavy number crunching
• image processing
• data transformations
• simulations

For I/O-bound tasks, threading or async may be better.

Important note (Windows / notebooks)

When using multiprocessing, always guard code with:

main_guard.py

if __name__ == "__main__":
    # start processes here
    pass

This avoids infinite child-process spawning on some platforms.

Visualize it

Unlike threads, which share one interpreter and memory space, each process gets its own — so they run truly in parallel on separate CPU cores with no GIL to fight over:

diagram Each process runs in parallel, in its own memory mermaid

  flowchart TB
  Parent[Parent process]
  Parent --> P1[Process 1
own memory + own interpreter]
  Parent --> P2[Process 2
own memory + own interpreter]
  Parent --> P3[Process 3
own memory + own interpreter]
  P1 --- C1[CPU core 1]
  P2 --- C2[CPU core 2]
  P3 --- C3[CPU core 3]
  P1 -. pickled data via Queue/Pipe .-> Parent
  P2 -. pickled data via Queue/Pipe .-> Parent
  P3 -. pickled data via Queue/Pipe .-> Parent

A parent process spawns three child processes, each with its own Python interpreter and memory, running in parallel on separate CPU cores. Data must be sent explicitly (pickled) between them, since nothing is shared.

🧪 Try It Yourself

Exercise 1 – Start a Process

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Process Pool map()

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Multiprocessing Queue

pch.exerciseTag pch.exerciseTitle