Goal
Build a script that:
- checks many URLs
- runs requests concurrently
- returns status codes
- uses timeouts and a concurrency limit
Implementation
import asyncio
import aiohttp
async def fetch(session: aiohttp.ClientSession, sem: asyncio.Semaphore, url: str):
async with sem:
try:
async with session.get(url) as resp:
await resp.read()
return url, resp.status
except Exception as e:
return url, str(e)
async def main():
urls = [
"https://api.github.com",
"https://httpbin.org/status/404",
"https://httpbin.org/delay/2",
"https://example.com",
]
sem = asyncio.Semaphore(5)
timeout = aiohttp.ClientTimeout(total=5)
async with aiohttp.ClientSession(timeout=timeout) as session:
results = await asyncio.gather(*(fetch(session, sem, u) for u in urls))
for url, status in results:
print(url, "->", status)
if __name__ == "__main__":
asyncio.run(main())Improvements
- read URLs from a file
- write results to CSV
- add retries with exponential backoff
What a URL checker must get right
The happy path is four lines. Everything that makes it a tool is the handling of things that go wrong, and each failure has a different shape:
flowchart TD
U["url"] --> R{"request"}
R -->|"connection refused, DNS fails"| CE["ClientError
no status at all"]
R -->|"no reply in time"| TO["TimeoutError
must be bounded"]
R -->|"reply arrives"| ST{"status"}
ST -->|"2xx"| OK["up"]
ST -->|"4xx / 5xx"| BAD["reachable but failing
still a RESULT, not an error"]
CE --> ROW["one row per url, always"]
TO --> ROW
OK --> ROW
BAD --> ROW
The design rule that falls out: every URL produces exactly one row. A checker that crashes on the first bad host is useless, and one that silently drops failures is worse than useless.
import asyncio, aiohttp, time
async def check(session, url, sem):
async with sem:
started = time.perf_counter()
try:
async with session.get(url) as r:
await r.read() # finish the body before timing
return {"url": url, "status": r.status,
"ok": r.status < 400,
"ms": round((time.perf_counter() - started) * 1000)}
except asyncio.TimeoutError:
return {"url": url, "status": None, "ok": False, "error": "timeout"}
except aiohttp.ClientError as e:
return {"url": url, "status": None, "ok": False,
"error": type(e).__name__}
async def check_all(urls, limit=10, timeout=5):
sem = asyncio.Semaphore(limit)
conf = aiohttp.ClientTimeout(total=timeout)
async with aiohttp.ClientSession(timeout=conf) as session:
return await asyncio.gather(*[check(session, u, sem) for u in urls])Three decisions worth naming:
return_exceptionsis not needed because nothing is allowed to escapecheck. Each call returns a row describing success or failure.- A 404 is a result, not an error. The site answered. Conflating "unreachable" with "returned 404" hides the difference that matters when you are debugging.
- The timeout is on the session, so it covers connection, headers and body together rather than only one phase.
See it move
Each row is a URL. Watch how the concurrency limit turns a long queue into a few short rounds — and how failures take their place in the results without stopping anything.
Reporting the result
Sorting by status makes the output usable — the failures are what you opened the report to read:
rows = asyncio.run(check_all(urls, limit=10))
rows.sort(key=lambda r: (r["ok"], r.get("status") or 0))
for r in rows:
status = r.get("status") or r.get("error")
print(f"{str(status):>8} {r['url']}")
up = sum(1 for r in rows if r["ok"])
print(f"\n{up}/{len(rows)} up")Because check never raises, len(rows) == len(urls) always holds — a property worth
asserting in a test.
Extensions worth doing
- Retry only what is worth retrying. A timeout or a 503 may succeed on a second attempt; a 404 never will. Retry with a growing delay, and cap the attempts.
- Follow redirects deliberately. aiohttp follows them by default; pass
allow_redirects=Falsewhen you are auditing what a URL says, not where it ends up. - Record the elapsed time per URL. A site that answers in 4 s is a problem your checker should surface even though its status is 200.
Check yourself
Why should check() catch its own exceptions and return a row rather than letting them propagate to gather?
By default the first exception in gather propagates and the batch is lost. Returning a row per URL keeps len(rows) == len(urls), which is a property a test can assert.
Show answer
B — so every URL yields exactly one result row, and one unreachable host cannot end the run — By default the first exception in gather propagates and the batch is lost. Returning a row per URL keeps len(rows) == len(urls), which is a property a test can assert.
In a URL checker, how should a 404 be treated?
A 404 means the server was reached and replied. Merging it with 'unreachable' destroys the distinction you most need when diagnosing, and retrying it is pointless.
Show answer
B — as a result: the host answered, so record the status and mark it not ok — A 404 means the server was reached and replied. Merging it with 'unreachable' destroys the distinction you most need when diagnosing, and retrying it is pointless.
Checking 12 URLs at 0.2 s each with Semaphore(6) took 0.42 s. What predicts that?
A semaphore of 6 gives ceil(12/6) = 2 rounds, so 0.40 s predicted against 0.42 s measured. The same formula gave 0.80 s against 0.83 s at Semaphore(3).
Show answer
A — 12 x 0.2 divided by 6 workers plus overhead, i.e. ceil(12/6) rounds of 0.2 s — A semaphore of 6 gives ceil(12/6) = 2 rounds, so 0.40 s predicted against 0.42 s measured. The same formula gave 0.80 s against 0.83 s at Semaphore(3).
Which failure is worth retrying with a backoff?
Timeouts and 503s are transient and often succeed on a later attempt. A 404, a 401 and a DNS failure will return the same answer however many times you ask.
Show answer
C — a timeout or a 503 Service Unavailable — Timeouts and 503s are transient and often succeed on a later attempt. A 404, a 401 and a DNS failure will return the same answer however many times you ask.
🧪 Try It Yourself
Exercise 1 – Your First Coroutine
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 2 – await asyncio.sleep
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 3 – Gather Two Coroutines
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.