Introduction to Selenium WebDriver

When to use Selenium

Use Selenium when:

• content is rendered by JavaScript
• you must interact with forms/buttons

For static pages, prefer requests + BeautifulSoup.

Minimal example

from selenium import webdriver
from selenium.webdriver.common.by import By
 
 
driver = webdriver.Chrome()  # Selenium 4.6+ fetches the driver automatically
try:
    driver.get("https://example.com")
    h1 = driver.find_element(By.TAG_NAME, "h1")
    print(h1.text)
finally:
    driver.quit()

Notes

• Driver setup differs by OS on older Selenium versions; 4.6+ handles it for you
• Consider headless mode for automation jobs

How Selenium works

Selenium controls a real browser. Your Python code sends commands (open this URL, click that button) to a driver program, and the driver passes them to Chrome or Firefox. The browser then loads the page, runs its JavaScript, and builds the DOM as a user would see it. That is why Selenium can read content that requests never sees.

The price is speed and fragility. A browser is heavy, pages take seconds to load, and a small layout change can break a locator. Use it only when the simpler tools cannot do the job.

Setting up

Install the package with pip install selenium. Since Selenium 4.6 a built-in tool called Selenium Manager finds or downloads the right driver for the browser installed on your machine, so webdriver.Chrome() usually works with no extra setup. On older versions you had to download chromedriver by hand and match its version to Chrome. If Chrome is not installed, install it first.

Finding and using elements

driver.find_element(By.X, "value") returns the first match or raises NoSuchElementException. find_elements returns a list, which is empty when nothing matches, so it is the safe way to test whether something is present.

Once you have an element, .text gives its visible text, .get_attribute("href") reads an attribute, .click() clicks it, and .send_keys() types into it.

A complete small script

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
 
driver = webdriver.Chrome()
try:
    driver.get("https://example.com")
    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.TAG_NAME, "h1")))
 
    print("title:", driver.title)
    print("heading:", driver.find_element(By.TAG_NAME, "h1").text)
 
    for link in driver.find_elements(By.TAG_NAME, "a"):
        print(link.text, "->", link.get_attribute("href"))
finally:
    driver.quit()

driver.quit() closes the browser and the driver. Put it in finally so it runs even after an error, otherwise browser windows pile up in the background.

Common mistakes

• Not quitting the driver. Leaves browsers running.
• Reading elements before they exist. Use explicit waits (see the waits lesson) instead of time.sleep.
• Using Selenium for static pages. requests and BeautifulSoup are faster and simpler.
• StaleElementReferenceException. You kept an element after the page changed. Find it again.
• Brittle locators copied from devtools. Prefer id, name, or a meaningful class.
• Ignoring the site's rules. Automated browsing is still scraping. Check robots.txt and the terms.

Adapting it

Try the same script with your own target page. Then add headless mode for running on a server, and an explicit wait for the element you need.

🧪 Try It Yourself

Exercise 1 – List Files with os.listdir

Press Run to see what your code prints.

Exercise 2 – Join Paths with os.path.join

Press Run to see what your code prints.

Exercise 3 – Write and Read a File

Press Run to see what your code prints.