Automating Browser Tasks (Headless Mode)

Headless Chrome

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
 
opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--window-size=1280,800")
 
 
driver = webdriver.Chrome(options=opts)
try:
    driver.get("https://example.com")
    print(driver.title)
finally:
    driver.quit()

Tips

• set window size for predictable layout
• take screenshots for debugging

What headless means

A normal browser opens a window. On a server, or in a scheduled job at 3 a.m., there is no screen to open a window on. Headless mode runs the full browser engine, including JavaScript, CSS layout, and cookies, but draws nothing to a display. Your Selenium code does not change. Only the options you pass to webdriver.Chrome do.

Use headless mode when the script runs on a schedule, on a server or in a container, or in a continuous-integration job. While you are writing and debugging a script, run with the window visible so you can see what the browser is doing, and switch to headless when it works.

About the flag

Recent Chrome versions use --headless=new, which is what the example uses. Older tutorials show --headless, which selects the old headless implementation and behaves a little differently. Firefox uses opts.add_argument("-headless") with selenium.webdriver.firefox.options.Options. Selenium 4.6 and later downloads a matching driver for you (Selenium Manager), so you do not need to install chromedriver by hand as older guides say.

A script that is safe to schedule

Headless jobs run with nobody watching, so they need a screenshot on failure and a guaranteed shutdown.

from datetime import datetime
from pathlib import Path
 
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
 
 
def make_driver() -> webdriver.Chrome:
    opts = Options()
    opts.add_argument("--headless=new")
    opts.add_argument("--window-size=1280,800")
    opts.add_argument("--disable-gpu")
    return webdriver.Chrome(options=opts)
 
 
def run() -> None:
    driver = make_driver()
    try:
        driver.get("https://example.com")
        heading = WebDriverWait(driver, 10).until(
            EC.visibility_of_element_located((By.TAG_NAME, "h1"))
        )
        print("heading:", heading.text)
    except Exception:
        stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
        Path("debug").mkdir(exist_ok=True)
        driver.save_screenshot(f"debug/failure_{stamp}.png")
        raise
    finally:
        driver.quit()
 
 
run()

The finally block runs whether the script succeeds or fails, so no browser process is left running. A leaked Chrome process on a server slowly eats memory until the machine stalls. The except block saves a screenshot and then re-raises, so the failure still shows up in your logs.

Common mistakes

• Forgetting the window size. A headless browser has a default size that may be small, so responsive pages hide elements you expect to see. Set --window-size.
• Not calling driver.quit(). Leaves orphaned processes.
• Running as root in a container and getting a sandbox error. The usual fix is --no-sandbox, but it lowers the browser's security, so use it only in a disposable container.
• Different behaviour from the visible browser. Some sites detect headless mode and serve different content or block you. If a page works visibly but not headless, take a screenshot and check what the browser actually received.
• Hiding your identity to evade blocks. Do not disguise the bot to get around a site that has said no. See the ethics lesson.

Check your understanding