Handling Errors in Long-Running Scripts

Principles

• fail fast on programmer errors
• retry transient network failures
• alert on repeated failure

Retry with backoff (simple)

import time
 
 
def retry(fn, tries=3, base_delay=1.0):
    last = None
    for i in range(tries):
        try:
            return fn()
        except Exception as e:
            last = e
            time.sleep(base_delay * (2 ** i))
    raise last

Wrap your main

import logging
 
log = logging.getLogger("job")
 
 
def main():
    ...
 
 
if __name__ == "__main__":
    try:
        main()
    except Exception:
        log.exception("job failed")
        raise

Visualize it

Resilient scripts retry transient failures with backoff and only alert once retries are exhausted.

  flowchart TD
    A["Try the action"] --> B{"Succeeded?"}
    B -- "Yes" --> C["Continue"]
    B -- "No" --> D["Log error"]
    D --> E["Retry with backoff"]
    E --> F{"N failures reached?"}
    F -- "No" --> A
    F -- "Yes" --> G["Give up and alert"]

What goes wrong in a script that runs for days

A script you run by hand fails in front of you. A script that runs on a schedule, or loops for weeks, fails when nobody is looking. The network drops, an API returns 503, a disk fills, one file in ten thousand is corrupt. Good error handling means separating errors into two kinds and treating each properly.

• Transient errors go away if you try again: a timeout, a connection reset, HTTP 429 or 503. Retry these, with a growing delay.
• Permanent errors will happen every time: a KeyError from a bug, a 404, a wrong password. Retrying wastes time. Log them and stop, or skip the item.

The principles on this page follow from that. Fail fast on bugs, retry the transient, and tell a human when retries run out.

A retry helper that only retries what it should

The simple retry above catches every Exception, so it would also retry a TypeError caused by a typo. This version retries only the errors you name, adds a little random jitter, and does not swallow the original traceback.

import logging
import random
import time
 
log = logging.getLogger("job")
 
 
def retry(fn, tries=3, base_delay=1.0, retry_on=(ConnectionError, TimeoutError)):
    for attempt in range(1, tries + 1):
        try:
            return fn()
        except retry_on as err:
            if attempt == tries:
                log.error("giving up after %d attempts: %s", tries, err)
                raise
            delay = base_delay * (2 ** (attempt - 1)) + random.uniform(0, 0.25)
            log.warning("attempt %d failed (%s), retrying in %.1fs", attempt, err, delay)
            time.sleep(delay)
 
 
calls = {"n": 0}
 
 
def flaky():
    calls["n"] += 1
    if calls["n"] < 3:
        raise ConnectionError("network blip")
    return "ok"
 
 
logging.basicConfig(level=logging.INFO)
print(retry(flaky, base_delay=0.01))

The delays double on each attempt (1 s, 2 s, 4 s), which gives a struggling server time to recover. The random jitter stops many scripts from retrying at the same instant. On the last attempt the function re-raises with a bare raise, so the caller sees the real error and the full traceback. With requests, the exceptions to retry on are requests.ConnectionError and requests.Timeout.

Handle one item at a time

In a loop over many items, one bad item should not kill the whole run. Catch the error per item, record it, and continue. At the end, report how many failed.

def process_all(items, handle):
    failures = []
    for item in items:
        try:
            handle(item)
        except Exception as err:            # deliberately broad at this boundary
            failures.append((item, repr(err)))
    return failures

Catching Exception is acceptable at this one boundary, because you record every error. Do not use a bare except: because it also catches KeyboardInterrupt and SystemExit.

Common mistakes

• except Exception: pass. It hides every failure. At the least, log it.
• Retrying forever. Always cap the attempts.
• Retrying non-idempotent actions, such as sending a payment or an email, unless you are sure the first try did not succeed.
• No alert after the last retry. A silent failure is the worst kind. Send a message (see the communication phase).
• Losing the traceback. Use log.exception or a bare raise.

🧪 Try It Yourself

Exercise 1 – Retry a Flaky Function

Press Run to see what your code prints.

Check your understanding