Skip to content

Thread Pool with concurrent.futures

Why use ThreadPoolExecutor

Creating threads manually is okay for a few tasks.

For many tasks, prefer:

  • concurrent.futures.ThreadPoolExecutorconcurrent.futures.ThreadPoolExecutor

It provides:

  • a thread pool
  • easy result collection
  • simpler error handling

Example: run many tasks

thread_pool.py
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
 
 
def fetch(i: int) -> int:
    time.sleep(0.2)
    return i * i
 
 
results = []
with ThreadPoolExecutor(max_workers=5) as ex:
    futures = [ex.submit(fetch, i) for i in range(10)]
 
    for f in as_completed(futures):
        results.append(f.result())
 
print(sorted(results))
thread_pool.py
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
 
 
def fetch(i: int) -> int:
    time.sleep(0.2)
    return i * i
 
 
results = []
with ThreadPoolExecutor(max_workers=5) as ex:
    futures = [ex.submit(fetch, i) for i in range(10)]
 
    for f in as_completed(futures):
        results.append(f.result())
 
print(sorted(results))

Exception handling

If a worker raises an exception, future.result()future.result() raises it.

thread_pool_errors.py
from concurrent.futures import ThreadPoolExecutor
 
 
def work(x: int) -> float:
    return 10 / x
 
with ThreadPoolExecutor() as ex:
    futures = [ex.submit(work, x) for x in [2, 1, 0]]
 
    for f in futures:
        try:
            print(f.result())
        except Exception as e:
            print("Worker failed:", e)
thread_pool_errors.py
from concurrent.futures import ThreadPoolExecutor
 
 
def work(x: int) -> float:
    return 10 / x
 
with ThreadPoolExecutor() as ex:
    futures = [ex.submit(work, x) for x in [2, 1, 0]]
 
    for f in futures:
        try:
            print(f.result())
        except Exception as e:
            print("Worker failed:", e)

When to prefer a thread pool

  • API calls
  • scraping
  • downloading files
  • parallel file operations

๐Ÿงช Try It Yourself

Exercise 1 โ€“ Submit a Task

Exercise 2 โ€“ Map Over a List

Exercise 3 โ€“ Check Future Done

If this helped you, consider buying me a coffee โ˜•

Buy me a coffee

Was this page helpful?

Let us know how we did