Handling Errors in Long-Running Scripts

Principles

• fail fast on programmer errors
• retry transient network failures
• alert on repeated failure

Retry with backoff (simple)

retry_backoff.py

import time
 
 
def retry(fn, tries=3, base_delay=1.0):
    last = None
    for i in range(tries):
        try:
            return fn()
        except Exception as e:
            last = e
            time.sleep(base_delay * (2 ** i))
    raise last

Wrap your main

main_guard.py

import logging
 
log = logging.getLogger("job")
 
 
def main():
    ...
 
 
if __name__ == "__main__":
    try:
        main()
    except Exception:
        log.exception("job failed")
        raise

Visualize it

Resilient scripts retry transient failures with backoff and only alert once retries are exhausted.

diagram Resilient error handling flow mermaid

  flowchart TD
    A["Try the action"] --> B{"Succeeded?"}
    B -- "Yes" --> C["Continue"]
    B -- "No" --> D["Log error"]
    D --> E["Retry with backoff"]
    E --> F{"N failures reached?"}
    F -- "No" --> A
    F -- "Yes" --> G["Give up and alert"]

How a long-running script retries with backoff and eventually alerts