Logging for Automation Scripts (logging module)

Why logging

Logs help you answer:

• what happened?
• when did it happen?
• why did it fail?

Basic setup

import logging
 
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s %(message)s",
    handlers=[
        logging.FileHandler("job.log", encoding="utf-8"),
        logging.StreamHandler(),
    ],
)
 
log = logging.getLogger("job")
log.info("started")

Rotation

For large logs, use RotatingFileHandler.

Why not just print

print goes to the screen, and when a script runs from cron or Task Scheduler there is no screen. A log is a permanent record with a time, a severity, and the place it came from. When something fails at 3 a.m., the log is the only witness.

Levels

Each message has a level, and the logger discards anything below the level you set.

Set INFO for normal runs and DEBUG when you are hunting a problem. You change one setting, not every line of code.

Loggers, handlers and formatters

logging.getLogger("job") gives you a named logger. A handler decides where messages go (a file, the console), and a formatter decides what they look like. In the example, basicConfig attaches a FileHandler and a StreamHandler so each line goes to both job.log and the terminal. Configure logging once, at the start of the script, and call getLogger(__name__) in each module.

Rotation and exceptions

A log that grows forever will fill the disk. RotatingFileHandler starts a new file when the current one reaches a size, and keeps a fixed number of old files. TimedRotatingFileHandler does the same by time, such as one file per day.

import logging
from logging.handlers import RotatingFileHandler
 
 
def setup_logging(path: str = "job.log") -> logging.Logger:
    handler = RotatingFileHandler(path, maxBytes=1_000_000, backupCount=5, encoding="utf-8")
    handler.setFormatter(
        logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")
    )
    logger = logging.getLogger("job")
    logger.setLevel(logging.INFO)
    if not logger.handlers:                 # avoid duplicate lines if called twice
        logger.addHandler(handler)
        logger.addHandler(logging.StreamHandler())
    return logger
 
 
log = setup_logging()
log.info("started")
 
files = ["a.csv", "b.csv"]
for name in files:
    try:
        if name == "b.csv":
            raise ValueError("bad header")
        log.info("processed %s", name)
    except ValueError:
        log.exception("failed on %s", name)     # records the message and the traceback
 
log.info("finished")

This keeps at most six files of one megabyte each. log.exception may only be used inside an except block, and it records the traceback as well as your message, which is what you need to debug later. Note that the messages use %s placeholders with arguments, not f-strings. The logger only builds the string if the message will actually be written, which is cheaper when logging is disabled.

What to log

• Start and finish, with counts, so you can tell that a run completed.
• Each significant action, such as "moved report.pdf to archive".
• Every caught error, with the traceback.
• The values that identify the run: input path, date, and arguments.

Do not log passwords, tokens, or personal data. A log file is often readable by more people than the source code is.

Common mistakes

• Calling basicConfig after logging has started. It does nothing the second time.
• Adding handlers on every call, which repeats each line two, three, four times.
• Using print for errors in scheduled jobs.
• Logging with error instead of exception and losing the traceback.
• Relative log paths under cron or Task Scheduler, where the working directory differs. Use an absolute path.

🧪 Try It Yourself

Exercise 1 – List Files with os.listdir

Press Run to see what your code prints.

Exercise 2 – Join Paths with os.path.join

Press Run to see what your code prints.

Exercise 3 – Write and Read a File

Press Run to see what your code prints.