Parsing HTML with BeautifulSoup

Install

Common package name: beautifulsoup4

Parse HTML

from bs4 import BeautifulSoup
 
html = """
<html><body>
  <h1 class='title'>Hello</h1>
  <a href='https://example.com'>Link</a>
</body></html>
"""
 
soup = BeautifulSoup(html, "html.parser")
print(soup.h1.get_text(strip=True))
print(soup.a["href"])

Find elements

from bs4 import BeautifulSoup
 
soup = BeautifulSoup("<p class='x'>A</p><p class='y'>B</p>", "html.parser")
 
print(soup.find("p", class_="y").text)
print([p.text for p in soup.find_all("p")])

Use CSS selectors

from bs4 import BeautifulSoup
 
soup = BeautifulSoup("<div><span class='price'>$10</span></div>", "html.parser")
print(soup.select_one("span.price").text)

How BeautifulSoup fits in

requests gets the HTML as a string. BeautifulSoup turns that string into a tree of objects you can search, so you do not need regular expressions on raw markup. Install it with pip install beautifulsoup4, and import it as bs4. The second argument to BeautifulSoup(html, "html.parser") chooses the parser. html.parser ships with Python. lxml is faster and more forgiving of broken HTML, but needs installing.

Three ways to find things

1. Attribute access, like soup.h1 or soup.a["href"], returns the first matching tag. Simple, but only for the first one.
2. find and find_all filter by tag name and attributes. Note class_ with the underscore, because class is a Python keyword. find returns one tag or None, and find_all returns a list.
3. select and select_one take a CSS selector. This is the most flexible, and the one to use for anything nested.

To read a result, use .get_text(strip=True) for its text and tag["href"] or tag.get("href") for an attribute. .get returns None when the attribute is missing, while [...] raises KeyError.

A full example: table to rows

from bs4 import BeautifulSoup
 
html = """
<table id="scores">
  <tr><th>Name</th><th>Score</th></tr>
  <tr><td>Alice</td><td>95</td></tr>
  <tr><td>Bob</td><td>88</td></tr>
  <tr><td>Cara</td><td>n/a</td></tr>
</table>
"""
 
soup = BeautifulSoup(html, "html.parser")
rows = []
for tr in soup.select("table#scores tr")[1:]:            # skip the header row
    name, score = [td.get_text(strip=True) for td in tr.find_all("td")]
    if score.isdigit():
        rows.append({"name": name, "score": int(score)})
 
print(rows)

It skips the header, reads the cells, and keeps only rows with a numeric score. In a real script, html comes from requests.get(url, timeout=10).text. Guarding with isdigit() means a page with n/a does not crash the run.

Handling missing elements

select_one and find return None when nothing matches, and calling .get_text() on None raises AttributeError. Decide what a missing element means and code for it:

from bs4 import BeautifulSoup
 
soup = BeautifulSoup("<div class='card'><h2>Pen</h2></div>", "html.parser")
price_tag = soup.select_one(".price")
price = price_tag.get_text(strip=True) if price_tag else None
print(price)   # None

Common mistakes

• Forgetting the parser argument. BeautifulSoup then warns and picks one for you, so results may differ between machines.
• Using class instead of class_.
• Assuming the element exists. Check for None.
• Scraping JavaScript-generated content. If the data is not in the HTML that requests gets, BeautifulSoup cannot see it. Look at the network calls or use Selenium.
• Not respecting the site. Check robots.txt and add delays.
• Parsing relative links as if they were full URLs. Use urljoin.

Adapting it

Change the selector to match your page's structure. Find it with your browser's devtools: right-click the element, inspect it, and look for a stable class or id.

🧪 Try It Yourself

Exercise 1 – List Files with os.listdir

Press Run to see what your code prints.

Exercise 2 – Join Paths with os.path.join

Press Run to see what your code prints.

Exercise 3 – Write and Read a File

Press Run to see what your code prints.