Common Pitfalls (pickling, __main__)

  flowchart TD
  A["parent: Process(...).start()"] --> B["spawn: a new interpreter"]
  B --> C["it IMPORTS your module"]
  C --> D{"is the start code at module level?"}
  D -->|yes| E["the child starts another child"]
  E --> B
  D -->|"inside if __name__ == '__main__'"| F["__name__ is not '__main__' in the child"]
  F --> G["only the target function runs"]
  H["arguments are PICKLED to reach the child"] --> I["a lambda, a local function or an open file cannot be sent"]
  I --> J["use a module-level function and plain data"]

1) Missing main guard

Always use:

if __name__ == "__main__":
    pass

Without this, some environments will repeatedly spawn child processes.

2) Pickling errors

Multiprocessing needs to serialize (pickle) functions and data.

Avoid:

• lambdas
• nested functions
• open file handles
• database connections

Prefer:

• top-level functions
• simple data (numbers/strings/lists/dicts)

3) Oversubscribing CPUs

Creating too many processes can slow down your system.

Guidance:

• start with os.cpu_count()
• benchmark for your workloads

4) Returning huge data

Sending massive arrays through Queue can be slow.

Options:

• write output to files
• batch results
• aggregate inside workers

Why these problems happen

Every pitfall on this page comes from one fact: a child process is a separate program with its own memory. Python cannot hand it your variables directly. It has to serialize (pickle) the function and its arguments, send the bytes across, and rebuild them in the child. On Windows and macOS the child also starts from a fresh interpreter (the spawn start method) that imports your script before doing anything else. Linux historically used fork, which copies the parent and hides several of these bugs, so code that works on Linux may fail on Windows.

A worked example of the guard

from multiprocessing import Process
 
 
def work(n: int) -> None:
    print("child got", n)
 
 
if __name__ == "__main__":
    p = Process(target=work, args=(42,))
    p.start()
    p.join()
    print("parent done")

When the child starts, it imports guard_demo.py. Module-level code runs again, but the if __name__ == "__main__": block does not, because inside the child __name__ is not "__main__". Remove the guard and each child would create another child, which is why the symptom is either a flood of processes or a RuntimeError about bootstrapping.

Fixing a pickling error

from multiprocessing import Pool
 
 
def square(x: int) -> int:  # top-level function: picklable
    return x * x
 
 
if __name__ == "__main__":
    with Pool(2) as pool:
        print(pool.map(square, [1, 2, 3]))
        # pool.map(lambda x: x * x, [1, 2, 3])  # fails: a lambda cannot be pickled

Functions are pickled by name, so they must be importable from the top level of a module. Lambdas and functions defined inside other functions have no importable name.

Common mistakes

• Leaving out the guard because "it works on my Linux server".
• Passing open files, sockets or database connections to a worker. Pass a filename or connection settings and open the resource inside the worker.
• Starting more processes than CPU cores for CPU-bound work. Extra processes only add switching overhead.
• Sending huge results back through a Queue. Every result is pickled and copied; aggregate in the worker or write to disk.

A quick checklist

Before running multiprocessing code, check that the entry point is guarded, the target is a top-level function, the arguments are plain data, and the pool size is close to os.cpu_count().

Check yourself

🧪 Try It Yourself

Exercise 1 – Start a Process

Press Run to see what your code prints.

Exercise 2 – Process Pool map()

Press Run to see what your code prints.

Exercise 3 – Multiprocessing Queue

Press Run to see what your code prints.