Thread Pool with concurrent.futures
Why use ThreadPoolExecutor
Creating threads manually is okay for a few tasks.
For many tasks, prefer:
concurrent.futures.ThreadPoolExecutorconcurrent.futures.ThreadPoolExecutor
It provides:
- a thread pool
- easy result collection
- simpler error handling
Example: run many tasks
thread_pool.py
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
def fetch(i: int) -> int:
time.sleep(0.2)
return i * i
results = []
with ThreadPoolExecutor(max_workers=5) as ex:
futures = [ex.submit(fetch, i) for i in range(10)]
for f in as_completed(futures):
results.append(f.result())
print(sorted(results))thread_pool.py
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
def fetch(i: int) -> int:
time.sleep(0.2)
return i * i
results = []
with ThreadPoolExecutor(max_workers=5) as ex:
futures = [ex.submit(fetch, i) for i in range(10)]
for f in as_completed(futures):
results.append(f.result())
print(sorted(results))Exception handling
If a worker raises an exception, future.result()future.result() raises it.
thread_pool_errors.py
from concurrent.futures import ThreadPoolExecutor
def work(x: int) -> float:
return 10 / x
with ThreadPoolExecutor() as ex:
futures = [ex.submit(work, x) for x in [2, 1, 0]]
for f in futures:
try:
print(f.result())
except Exception as e:
print("Worker failed:", e)thread_pool_errors.py
from concurrent.futures import ThreadPoolExecutor
def work(x: int) -> float:
return 10 / x
with ThreadPoolExecutor() as ex:
futures = [ex.submit(work, x) for x in [2, 1, 0]]
for f in futures:
try:
print(f.result())
except Exception as e:
print("Worker failed:", e)When to prefer a thread pool
- API calls
- scraping
- downloading files
- parallel file operations
๐งช Try It Yourself
Exercise 1 โ Submit a Task
Exercise 2 โ Map Over a List
Exercise 3 โ Check Future Done
If this helped you, consider buying me a coffee โ
Buy me a coffeeWas this page helpful?
Let us know how we did
