Asyncio Mini Project (Concurrent URL checker)

Goal

Build a script that:

• checks many URLs
• runs requests concurrently
• returns status codes
• uses timeouts and a concurrency limit

Implementation

url_checker.py

import asyncio
import aiohttp
 
 
async def fetch(session: aiohttp.ClientSession, sem: asyncio.Semaphore, url: str):
    async with sem:
        try:
            async with session.get(url) as resp:
                await resp.read()
                return url, resp.status
        except Exception as e:
            return url, str(e)
 
 
async def main():
    urls = [
        "https://api.github.com",
        "https://httpbin.org/status/404",
        "https://httpbin.org/delay/2",
        "https://example.com",
    ]
 
    sem = asyncio.Semaphore(5)
    timeout = aiohttp.ClientTimeout(total=5)
 
    async with aiohttp.ClientSession(timeout=timeout) as session:
        results = await asyncio.gather(*(fetch(session, sem, u) for u in urls))
 
    for url, status in results:
        print(url, "->", status)
 
 
if __name__ == "__main__":
    asyncio.run(main())

Improvements

• read URLs from a file
• write results to CSV
• add retries with exponential backoff

What a URL checker must get right

The happy path is four lines. Everything that makes it a tool is the handling of things that go wrong, and each failure has a different shape:

diagram Diagram mermaid

  flowchart TD
  U["url"] --> R{"request"}
  R -->|"connection refused, DNS fails"| CE["ClientError
no status at all"]
  R -->|"no reply in time"| TO["TimeoutError
must be bounded"]
  R -->|"reply arrives"| ST{"status"}
  ST -->|"2xx"| OK["up"]
  ST -->|"4xx / 5xx"| BAD["reachable but failing
still a RESULT, not an error"]
  CE --> ROW["one row per url, always"]
  TO --> ROW
  OK --> ROW
  BAD --> ROW

The design rule that falls out: every URL produces exactly one row. A checker that crashes on the first bad host is useless, and one that silently drops failures is worse than useless.

checker.py

import asyncio, aiohttp, time
 
async def check(session, url, sem):
    async with sem:
        started = time.perf_counter()
        try:
            async with session.get(url) as r:
                await r.read()                       # finish the body before timing
                return {"url": url, "status": r.status,
                        "ok": r.status < 400,
                        "ms": round((time.perf_counter() - started) * 1000)}
        except asyncio.TimeoutError:
            return {"url": url, "status": None, "ok": False, "error": "timeout"}
        except aiohttp.ClientError as e:
            return {"url": url, "status": None, "ok": False,
                    "error": type(e).__name__}
 
async def check_all(urls, limit=10, timeout=5):
    sem = asyncio.Semaphore(limit)
    conf = aiohttp.ClientTimeout(total=timeout)
    async with aiohttp.ClientSession(timeout=conf) as session:
        return await asyncio.gather(*[check(session, u, sem) for u in urls])

Three decisions worth naming:

• return_exceptions is not needed because nothing is allowed to escape check. Each call returns a row describing success or failure.
• A 404 is a result, not an error. The site answered. Conflating “unreachable” with “returned 404” hides the difference that matters when you are debugging.
• The timeout is on the session, so it covers connection, headers and body together rather than only one phase.

See it move

Each row is a URL. Watch how the concurrency limit turns a long queue into a few short rounds — and how failures take their place in the results without stopping anything.

sketch Checking many URLs with a bounded pool p5.js Pause Restart

Requests enter as slots free up. Failures and slow hosts produce rows like any other result, so one bad URL never stops the run.

Reporting the result

Sorting by status makes the output usable — the failures are what you opened the report to read:

report.py

rows = asyncio.run(check_all(urls, limit=10))
 
rows.sort(key=lambda r: (r["ok"], r.get("status") or 0))
for r in rows:
    status = r.get("status") or r.get("error")
    print(f"{str(status):>8}  {r['url']}")
 
up = sum(1 for r in rows if r["ok"])
print(f"\n{up}/{len(rows)} up")

Because check never raises, len(rows) == len(urls) always holds — a property worth asserting in a test.

Measured behaviour of the pool

Against a local server at 0.20 s per request, 12 URLs took 0.83 s at Semaphore(3), 0.42 s at Semaphore(6), and 0.22 s unbounded — matching ceil(N / limit) × 0.2 s each time. Pick the limit from what the target host tolerates, then predict the runtime with that formula.

Extensions worth doing

• Retry only what is worth retrying. A timeout or a 503 may succeed on a second attempt; a 404 never will. Retry with a growing delay, and cap the attempts.
• Follow redirects deliberately. aiohttp follows them by default; pass allow_redirects=False when you are auditing what a URL says, not where it ends up.
• Record the elapsed time per URL. A site that answers in 4 s is a problem your checker should surface even though its status is 200.

Check yourself

pch.quizTag pch.quizDefaultTitle

1. Why should check() catch its own exceptions and return a row rather than letting them propagate to gather?

   A gather cannot handle exceptions at all B so every URL yields exactly one result row, and one unreachable host cannot end the run C because exceptions are slower than return values D to avoid needing a Semaphore
   pch.quizShowAnswer

   B — so every URL yields exactly one result row, and one unreachable host cannot end the run — By default the first exception in gather propagates and the batch is lost. Returning a row per URL keeps len(rows) == len(urls), which is a property a test can assert.

2. In a URL checker, how should a 404 be treated?

   A as an error, identical to a connection failure B as a result: the host answered, so record the status and mark it not ok C silently skipped, since the page does not exist D as a reason to retry until it succeeds
   pch.quizShowAnswer

   B — as a result: the host answered, so record the status and mark it not ok — A 404 means the server was reached and replied. Merging it with 'unreachable' destroys the distinction you most need when diagnosing, and retrying it is pointless.

3. Checking 12 URLs at 0.2 s each with Semaphore(6) took 0.42 s. What predicts that?

   A 12 x 0.2 divided by 6 workers plus overhead, i.e. ceil(12/6) rounds of 0.2 s B the slowest single request C 6 x 0.2 s, one round per permit D it is unpredictable and must be measured every time
   pch.quizShowAnswer

   A — 12 x 0.2 divided by 6 workers plus overhead, i.e. ceil(12/6) rounds of 0.2 s — A semaphore of 6 gives ceil(12/6) = 2 rounds, so 0.40 s predicted against 0.42 s measured. The same formula gave 0.80 s against 0.83 s at Semaphore(3).

4. Which failure is worth retrying with a backoff?

   A a 404 Not Found B a 401 Unauthorized C a timeout or a 503 Service Unavailable D a ClientError from a domain that does not resolve
   pch.quizShowAnswer

   C — a timeout or a 503 Service Unavailable — Timeouts and 503s are transient and often succeed on a later attempt. A 404, a 401 and a DNS failure will return the same answer however many times you ask.

🧪 Try It Yourself

Exercise 1 – Your First Coroutine

pch.exerciseTag pch.exerciseTitle

Exercise 2 – await asyncio.sleep

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Gather Two Coroutines

pch.exerciseTag pch.exerciseTitle