Process Pool (Pool, map, starmap)

Why use a process pool

Creating many processes manually is not ideal.

A pool:

• reuses worker processes
• makes parallel execution easier

Pool.map

from multiprocessing import Pool
 
 
def square(x: int) -> int:
    return x * x
 
 
if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.map(square, [1, 2, 3, 4, 5])
 
    print(results)

Pool.starmap (multiple arguments)

from multiprocessing import Pool
 
 
def power(x: int, p: int) -> int:
    return x ** p
 
 
if __name__ == "__main__":
    with Pool(processes=2) as pool:
        results = pool.starmap(power, [(2, 3), (3, 2), (4, 2)])
 
    print(results)

Common mistakes

• Forgetting the if __name__ == "__main__" guard.
• Passing non-picklable objects (like lambdas on some platforms).

What a pool does for you

Starting a process is expensive: Python must launch a new interpreter and import your module. A Pool pays that cost once, keeps a fixed number of worker processes alive, and feeds them tasks as they become free. You describe the work, and the pool splits it among the workers and collects the answers.

Reading the examples

pool.map(square, [1, 2, 3, 4, 5]) calls square on each item, spread across the workers. It blocks until everything is done and returns the results in the same order as the inputs, even if a later item finished first. So the result is [1, 4, 9, 16, 25].

map only passes one argument to your function. When the function needs several, use starmap with a list of tuples: each tuple is unpacked into the arguments. (2, 3) becomes power(2, 3), so the example prints [8, 9, 16].

The with Pool(...) as pool: block shuts the workers down cleanly when you leave it. Always use it, otherwise stray processes can be left behind.

Choosing the pool size

Pool() with no argument uses os.cpu_count() workers, which is a good default for CPU-bound work. More workers than cores rarely help, and for tiny tasks the cost of sending data between processes can outweigh the gain. Try it on your own workload:

import time
from multiprocessing import Pool
 
 
def heavy(n: int) -> int:
    return sum(i * i for i in range(n))
 
 
if __name__ == "__main__":
    data = [2_000_000] * 8
 
    start = time.perf_counter()
    serial = [heavy(n) for n in data]
    print("serial  :", round(time.perf_counter() - start, 2), "s")
 
    start = time.perf_counter()
    with Pool() as pool:
        parallel = pool.map(heavy, data)
    print("parallel:", round(time.perf_counter() - start, 2), "s")
 
    print(serial == parallel)  # True

On a multi-core machine the parallel run is noticeably faster. Times will differ on your computer, but the two result lists are identical.

Common mistakes

• Forgetting the if __name__ == "__main__": guard, which causes a crash on Windows and macOS.
• Passing a lambda or a nested function; use a top-level function.
• Using map for a function with two parameters; use starmap.
• Using a pool for very small tasks, where the overhead beats the benefit.
• Expecting workers to update your variables. Each worker has its own copy, so return values instead.

When to use it

Use Pool.map for CPU-bound work applied to many independent items: image processing, number crunching, parsing many files. For I/O-bound work such as downloads, threads or asyncio are usually a better fit.

🧪 Try It Yourself

Exercise 1 – Start a Process

Press Run to see what your code prints.

Exercise 2 – Process Pool map()

Press Run to see what your code prints.

Exercise 3 – Multiprocessing Queue

Press Run to see what your code prints.