Common Pitfalls (pickling, __main__)

diagram why a multiprocessing script must guard __main__ mermaid

  flowchart TD
  A["parent: Process(...).start()"] --> B["spawn: a new interpreter"]
  B --> C["it IMPORTS your module"]
  C --> D{"is the start code at module level?"}
  D -->|yes| E["the child starts another child"]
  E --> B
  D -->|"inside if __name__ == '__main__'"| F["__name__ is not '__main__' in the child"]
  F --> G["only the target function runs"]
  H["arguments are PICKLED to reach the child"] --> I["a lambda, a local function or an open file cannot be sent"]
  I --> J["use a module-level function and plain data"]

On Windows and macOS a new process is started by launching a fresh interpreter and importing your module. Anything at module level therefore runs again in the child -- including the code that started the process. Without the guard that recursion is unbounded, which is why the symptom is an explosion of processes rather than an error message.

1) Missing main guard

Always use:

main_guard.py

if __name__ == "__main__":
    pass

Without this, some environments will repeatedly spawn child processes.

2) Pickling errors

Multiprocessing needs to serialize (pickle) functions and data.

Avoid:

• lambdas
• nested functions
• open file handles
• database connections

Prefer:

• top-level functions
• simple data (numbers/strings/lists/dicts)

3) Oversubscribing CPUs

Creating too many processes can slow down your system.

Guidance:

• start with os.cpu_count()
• benchmark for your workloads

4) Returning huge data

Sending massive arrays through Queue can be slow.

Options:

• write output to files
• batch results
• aggregate inside workers

sketch Why a multiprocessing script must guard __main__ p5.js Pause Restart

On Windows and macOS a child process is started by launching a fresh interpreter and IMPORTING your module. So anything at module level runs again in the child -- including the code that started the process. Without the guard that recursion has no bottom, which is why the symptom is an explosion of processes rather than an error.

Check yourself

pch.quizTag pch.quizDefaultTitle

1. Why must `Process(...).start()` sit inside `if __name__ == '__main__':` on Windows?

   A To make the code readable B Because the child starts by importing your module, so module-level start() would recurse C Because Windows needs an entry point D To set the process name
   pch.quizShowAnswer

   B — Because the child starts by importing your module, so module-level start() would recurse — The spawn start method launches a fresh interpreter and imports your module. Anything at module level runs again in the child — including the line that created it.

2. What is the symptom of forgetting the guard?

   A A clear ImportError B An explosion of processes, or a RuntimeError about bootstrapping C The child silently does nothing D A pickling error
   pch.quizShowAnswer

   B — An explosion of processes, or a RuntimeError about bootstrapping — The recursion has no bottom, which is why it presents as the machine filling with processes rather than a single clear error at the offending line.

3. Why can a `lambda` not be passed as a `Process` target on Windows?

   A Lambdas are too slow B Arguments and targets must be PICKLED to reach the child, and a lambda cannot be pickled C Lambdas cannot take arguments D It is allowed
   pch.quizShowAnswer

   B — Arguments and targets must be PICKLED to reach the child, and a lambda cannot be pickled — Use a module-level function and plain data. The same restriction covers locally defined functions, open files and live connections.

4. Code using multiprocessing works on Linux and fails on Windows. What is the usual reason?

   A Windows has no fork, so the default is spawn, which re-imports the module B Windows limits process count C Linux ignores the GIL D Pickle is disabled on Windows
   pch.quizShowAnswer

   A — Windows has no fork, so the default is spawn, which re-imports the module — `fork` copies the process without re-importing, so a missing `__main__` guard is invisible there. The guard costs nothing — write it always.

🧪 Try It Yourself

Exercise 1 – Start a Process

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Process Pool map()

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Multiprocessing Queue

pch.exerciseTag pch.exerciseTitle