Thread Pool with concurrent.futures

Why use ThreadPoolExecutor

Creating threads manually is okay for a few tasks.

For many tasks, prefer:

• concurrent.futures.ThreadPoolExecutor

It provides:

• a thread pool
• easy result collection
• simpler error handling

Example: run many tasks

from concurrent.futures import ThreadPoolExecutor, as_completed
import time
 
 
def fetch(i: int) -> int:
    time.sleep(0.2)
    return i * i
 
 
results = []
with ThreadPoolExecutor(max_workers=5) as ex:
    futures = [ex.submit(fetch, i) for i in range(10)]
 
    for f in as_completed(futures):
        results.append(f.result())
 
print(sorted(results))

Exception handling

If a worker raises an exception, future.result() raises it.

from concurrent.futures import ThreadPoolExecutor
 
 
def work(x: int) -> float:
    return 10 / x
 
with ThreadPoolExecutor() as ex:
    futures = [ex.submit(work, x) for x in [2, 1, 0]]
 
    for f in futures:
        try:
            print(f.result())
        except Exception as e:
            print("Worker failed:", e)

When to prefer a thread pool

• API calls
• scraping
• downloading files
• parallel file operations

From threads to tasks

With raw Thread objects you create each thread, start it, join it, and figure out how to get a result out. A ThreadPoolExecutor changes the question from "run this in a thread" to "run this function and give me its result later". You submit a function and its arguments and get back a Future, an object that stands for a result that may not exist yet. The pool reuses a fixed number of threads, so ten tasks on five workers run in two waves.

Reading the first example

ex.submit(fetch, i) returns immediately with a future. as_completed(futures) yields futures in the order they finish, not the order you submitted them, and f.result() gives the return value. That is why the example sorts the list at the end: completion order varies. Each task sleeps 0.2 seconds, so ten tasks on five workers take about 0.4 seconds in total instead of 2 seconds one after another. The with block waits for everything and shuts the pool down.

map: the simple version

When you just want results in input order, ex.map is shorter:

from concurrent.futures import ThreadPoolExecutor
import time
 
 
def slow_square(n: int) -> int:
    time.sleep(0.2)
    return n * n
 
 
with ThreadPoolExecutor(max_workers=4) as ex:
    results = list(ex.map(slow_square, range(8)))
 
print(results)  # [0, 1, 4, 9, 16, 25, 36, 49]

map returns results in the same order as the inputs, regardless of which finished first.

Errors travel with the future

An exception inside a worker does not crash your program at that moment. It is stored in the future and raised when you call result(), as the second example shows: dividing by zero raises ZeroDivisionError only where you ask for that result, so you can catch it there and handle each task separately.

Common mistakes

• Never calling result(), so errors disappear unnoticed.
• Using a pool for CPU-heavy work. Because of the GIL, threads do not speed up pure Python calculation. Use ProcessPoolExecutor for that.
• Too many workers. Hundreds of threads waste memory; start with 5 to 20 for I/O.
• Sharing mutable data between tasks without a lock. Prefer returning values.

When to use it

A thread pool is the go-to tool for I/O-bound batches: many API calls, downloads, file reads, or database lookups. Where the work is CPU-bound, switch the class name to ProcessPoolExecutor and keep everything else almost the same.

🧪 Try It Yourself

Exercise 1 – Submit a Task

Press Run to see what your code prints.

Exercise 2 – Map Over a List

Press Run to see what your code prints.

Exercise 3 – Check Future Done

Press Run to see what your code prints.