Abstract

Web scraping is one of the most marketable Python skills you can pick up — every business eventually needs to extract data from a page that has no API. In this project you build a real scraper against webscraper.io’s test site, pulling name, price, description, review count, rating, and image URL for each phone, then writing the results to a CSV. We start with the bare-bones version, then fix half a dozen real bugs (CSV quoting, encoding, missing fields), add pagination, retries, polite rate-limiting, and finish with a preview of when to graduate from BeautifulSoup to Scrapy.

You will learn:

• The standard requestsrequests + BeautifulSoupBeautifulSoup pipeline.
• How to find the right CSS selectors using browser DevTools.
• Why naïve string concatenation is wrong for CSV — and how to use the csvcsv module.
• How to handle missing fields gracefully.
• Polite-scraping practices: User-Agent, delays, robots.txt.
• When to switch to a real scraping framework.

Prerequisites

• Python 3.6 or above.
• A code editor or IDE.
• Internet connection.
• Familiarity with basic HTML structure (tags, attributes, classes).

Install Dependencies

pip install requests beautifulsoup4

pip install requests beautifulsoup4

Getting Started

Create the project

1. Create folder basicwebscrapperbasicwebscrapper.
2. Inside, create basicwebscrapper.pybasicwebscrapper.py.

Write the code

# Basic Web Scrapper
 
# Importing Libraries
import requests
from bs4 import BeautifulSoup
 
# URL
url = "https://webscraper.io/test-sites/e-commerce/allinone/phones/touch"
 
# Requesting the URL
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")
 
# Finding the phones
phones = soup.find_all("div", class_="card-body")
 
# Creating a CSV file
open_file = open("phones.csv", "a")
headers = "Name, Price, Description, Reviews, Rating, Image\n"
open_file.write(headers)
 
# Looping through the phones
for phone in phones:
    name = phone.find("a", class_="title")
    price = phone.find("h4", class_="price")
    description = phone.find("p", class_="description")
    reviews = phone.find("p", class_="float-end review-count")
    rating = phone.find("p", attrs={"data-rating": True})
    image = phone.find("img", class_="img-responsive")["src"]
    
    # Writing to the CSV file
    open_file.write(f'{name.text}, {price.text}, {description.text}, {reviews.text}, {rating["data-rating"]}, {image}\n')
    print(f'Name: {name.text} \nPrice: {price.text} \nDescription: {description.text} \nReviews: {reviews.text} \nRating: {rating["data-rating"]} \nImage: {image} \n')
 
# Closing the CSV file
open_file.close()   
    

# Basic Web Scrapper
 
# Importing Libraries
import requests
from bs4 import BeautifulSoup
 
# URL
url = "https://webscraper.io/test-sites/e-commerce/allinone/phones/touch"
 
# Requesting the URL
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")
 
# Finding the phones
phones = soup.find_all("div", class_="card-body")
 
# Creating a CSV file
open_file = open("phones.csv", "a")
headers = "Name, Price, Description, Reviews, Rating, Image\n"
open_file.write(headers)
 
# Looping through the phones
for phone in phones:
    name = phone.find("a", class_="title")
    price = phone.find("h4", class_="price")
    description = phone.find("p", class_="description")
    reviews = phone.find("p", class_="float-end review-count")
    rating = phone.find("p", attrs={"data-rating": True})
    image = phone.find("img", class_="img-responsive")["src"]
    
    # Writing to the CSV file
    open_file.write(f'{name.text}, {price.text}, {description.text}, {reviews.text}, {rating["data-rating"]}, {image}\n')
    print(f'Name: {name.text} \nPrice: {price.text} \nDescription: {description.text} \nReviews: {reviews.text} \nRating: {rating["data-rating"]} \nImage: {image} \n')
 
# Closing the CSV file
open_file.close()   
    

Run it

C:\Users\username\basicwebscrapper> python basicwebscrapper.py
Name: Nokia 123       Price: $24.99    Rating: 3
Name: LG Optimus      Price: $57.99    Rating: 3
…
Saved 9 rows to phones.csv

C:\Users\username\basicwebscrapper> python basicwebscrapper.py
Name: Nokia 123       Price: $24.99    Rating: 3
Name: LG Optimus      Price: $57.99    Rating: 3
…
Saved 9 rows to phones.csv

Step-by-Step Explanation

1. Fetch and parse

import requests
from bs4 import BeautifulSoup
 
URL = "https://webscraper.io/test-sites/e-commerce/allinone/phones/touch"
response = requests.get(URL, timeout=15, headers={
    "User-Agent": "PCH-Scraper/1.0 (+contact@example.com)"
})
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

import requests
from bs4 import BeautifulSoup
 
URL = "https://webscraper.io/test-sites/e-commerce/allinone/phones/touch"
response = requests.get(URL, timeout=15, headers={
    "User-Agent": "PCH-Scraper/1.0 (+contact@example.com)"
})
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

• User-AgentUser-Agent identifies you to the server. The default python-requests/2.xpython-requests/2.x is blocked by many real sites.
• raise_for_status()raise_for_status() raises on 4xx/5xx so we do not silently scrape error pages.
• "html.parser""html.parser" is built in. For tolerant parsing of broken markup, install lxmllxml and use "lxml""lxml".

2. Find the right selectors

Open DevTools (F12) → Inspect → click a product → identify the wrapper:

<div class="card thumbnail">
  <div class="card-body">
    <h4 class="pull-right price">$24.99</h4>
    <a class="title" title="Nokia 123">Nokia 123</a>
    <p class="description">7 day battery</p>
  </div>
  <div class="ratings">
    <p class="pull-right">11 reviews</p>
    <p data-rating="3">★★★☆☆</p>
  </div>
  <img src="/images/test-sites/.../cart2.png"/>
</div>

<div class="card thumbnail">
  <div class="card-body">
    <h4 class="pull-right price">$24.99</h4>
    <a class="title" title="Nokia 123">Nokia 123</a>
    <p class="description">7 day battery</p>
  </div>
  <div class="ratings">
    <p class="pull-right">11 reviews</p>
    <p data-rating="3">★★★☆☆</p>
  </div>
  <img src="/images/test-sites/.../cart2.png"/>
</div>

We will find all div.card-bodydiv.card-body, then within each look up the inner elements.

3. Extract per product

phones = soup.select("div.thumbnail")        # one wrapper per phone
 
rows = []
for phone in phones:
    rows.append({
        "name":        phone.select_one("a.title")["title"].strip(),
        "price":       phone.select_one("h4.price").get_text(strip=True),
        "description": phone.select_one("p.description").get_text(strip=True),
        "reviews":     phone.select_one("p.pull-right").get_text(strip=True),
        "rating":      int(phone.select_one("p[data-rating]")["data-rating"]),
        "image":       phone.select_one("img")["src"],
    })

phones = soup.select("div.thumbnail")        # one wrapper per phone
 
rows = []
for phone in phones:
    rows.append({
        "name":        phone.select_one("a.title")["title"].strip(),
        "price":       phone.select_one("h4.price").get_text(strip=True),
        "description": phone.select_one("p.description").get_text(strip=True),
        "reviews":     phone.select_one("p.pull-right").get_text(strip=True),
        "rating":      int(phone.select_one("p[data-rating]")["data-rating"]),
        "image":       phone.select_one("img")["src"],
    })

• select_one(css)select_one(css) returns the first match (or NoneNone); select(css)select(css) returns a list.
• ["title"]["title"] reads an HTML attribute; .get_text(strip=True).get_text(strip=True) reads text content.
• We pull the rating value from the data-ratingdata-rating attribute, not the visible stars — far more reliable.

4. Save as proper CSV

The naïve version did this:

open_file.write(f'{name}, {price}, {description}, ...\n')   # ❌

open_file.write(f'{name}, {price}, {description}, ...\n')   # ❌

This breaks the moment a description contains a comma ("5 mpx, Android 5.0""5 mpx, Android 5.0"). Always use the csvcsv module:

import csv
with open("phones.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

import csv
with open("phones.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

The csvcsv module quotes any field containing commas, quotes, or newlines, producing real RFC-4180-compliant output that loads cleanly into Excel/Pandas.

Defensive Selectors

What happens when a product has no rating? select_one("p[data-rating]")select_one("p[data-rating]") returns NoneNone, and None["data-rating"]None["data-rating"] raises TypeErrorTypeError. Guard every extraction:

def text_of(parent, selector, default=""):
    node = parent.select_one(selector)
    return node.get_text(strip=True) if node else default
 
def attr_of(parent, selector, attr, default=""):
    node = parent.select_one(selector)
    return node[attr] if node and node.has_attr(attr) else default
 
row = {
    "name":  attr_of(phone, "a.title", "title"),
    "price": text_of(phone, "h4.price"),
    "rating": int(attr_of(phone, "p[data-rating]", "data-rating", 0)),
}

def text_of(parent, selector, default=""):
    node = parent.select_one(selector)
    return node.get_text(strip=True) if node else default
 
def attr_of(parent, selector, attr, default=""):
    node = parent.select_one(selector)
    return node[attr] if node and node.has_attr(attr) else default
 
row = {
    "name":  attr_of(phone, "a.title", "title"),
    "price": text_of(phone, "h4.price"),
    "rating": int(attr_of(phone, "p[data-rating]", "data-rating", 0)),
}

The scraper now runs to completion even when some products miss a field.

Pagination

Most e-commerce sites paginate results. Find the “next” link and loop:

def crawl_all(start_url):
    url = start_url
    while url:
        soup = fetch(url)
        yield from extract_products(soup)
        next_link = soup.select_one("a.next, li.next > a")
        url = urljoin(url, next_link["href"]) if next_link else None
        time.sleep(1)               # polite delay

def crawl_all(start_url):
    url = start_url
    while url:
        soup = fetch(url)
        yield from extract_products(soup)
        next_link = soup.select_one("a.next, li.next > a")
        url = urljoin(url, next_link["href"]) if next_link else None
        time.sleep(1)               # polite delay

A generator (yield fromyield from) streams results one at a time so you can stop early or process incrementally.

Polite Scraping

Hammering a server gets your IP banned. Three rules:

1. Sleep between requests. time.sleep(1)time.sleep(1) between pages is friendly.
2. Honor robots.txtrobots.txt.
   robots.py

   from urllib.robotparser import RobotFileParser
   rp = RobotFileParser()
   rp.set_url("https://example.com/robots.txt"); rp.read()
   if not rp.can_fetch("PCH-Scraper/1.0", URL):
       sys.exit("robots.txt forbids this URL.")

   robots.py

   from urllib.robotparser import RobotFileParser
   rp = RobotFileParser()
   rp.set_url("https://example.com/robots.txt"); rp.read()
   if not rp.can_fetch("PCH-Scraper/1.0", URL):
       sys.exit("robots.txt forbids this URL.")

3. Identify yourself. User-Agent: PCH-Scraper/1.0 (+contact@example.com)User-Agent: PCH-Scraper/1.0 (+contact@example.com) lets a sysadmin reach you instead of just banning.

For more on this, see Basic Web Crawler.

Common Mistakes

Variations to Try

1. Scrape laptops or tablets

for cat in ["phones/touch", "computers/laptops", "computers/tablets"]:
    crawl(f"https://webscraper.io/test-sites/e-commerce/allinone/{cat}")

for cat in ["phones/touch", "computers/laptops", "computers/tablets"]:
    crawl(f"https://webscraper.io/test-sites/e-commerce/allinone/{cat}")

2. Save as JSON

import json
json.dump(rows, open("phones.json", "w", encoding="utf-8"), indent=2)

import json
json.dump(rows, open("phones.json", "w", encoding="utf-8"), indent=2)

3. Save to SQLite

import sqlite3
db = sqlite3.connect("phones.db")
db.execute("""CREATE TABLE IF NOT EXISTS phones(
    name TEXT, price TEXT, description TEXT, reviews TEXT, rating INTEGER, image TEXT)""")
db.executemany("INSERT INTO phones VALUES(?,?,?,?,?,?)",
               [(r["name"], r["price"], r["description"], r["reviews"], r["rating"], r["image"]) for r in rows])
db.commit()

import sqlite3
db = sqlite3.connect("phones.db")
db.execute("""CREATE TABLE IF NOT EXISTS phones(
    name TEXT, price TEXT, description TEXT, reviews TEXT, rating INTEGER, image TEXT)""")
db.executemany("INSERT INTO phones VALUES(?,?,?,?,?,?)",
               [(r["name"], r["price"], r["description"], r["reviews"], r["rating"], r["image"]) for r in rows])
db.commit()

4. Concurrent fetching

Use concurrent.futures.ThreadPoolExecutorconcurrent.futures.ThreadPoolExecutor (with a 1-second per-domain throttle).

5. Price-change tracker

Run daily, compare to yesterday’s CSV, alert on drops. See Web Page Scraper with Notifications for the alerting half.

6. GUI / Web UI

• Tkinter form to enter URL → display table.
• Flask app to expose GET /scrape?url=...GET /scrape?url=....

7. Headless browser

If the site renders client-side, requestsrequests returns empty HTML. Switch to Playwright:

from playwright.sync_api import sync_playwright
with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto(url)
    html = page.content()

from playwright.sync_api import sync_playwright
with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto(url)
    html = page.content()

8. Scrapy framework

For multi-spider projects, Scrapy gives you pipelines, item loaders, throttling, retries, and exports out of the box. Worth learning when you cross 5+ scrapers in one repo.

9. Cron job

On Linux: crontab -ecrontab -e → 0 6 * * * /usr/bin/python /path/to/scraper.py0 6 * * * /usr/bin/python /path/to/scraper.py. On Windows: Task Scheduler.

10. Slack/Discord alerts on new products

Diff yesterday’s run vs today’s; push new SKUs to a webhook.

When to Graduate to Scrapy

A few signs you have outgrown a single script:

• More than ~5 spiders.
• You re-implement retry logic and rate limiting in each script.
• You want resumable crawls.
• You want item-pipeline hooks (clean → validate → save).

Scrapy gives you all of that for free, plus a dedicated CLI (scrapy crawl phones -o phones.jsonscrapy crawl phones -o phones.json).

Legal & Ethical Reminder

Scraping is legal in many jurisdictions when:

• The data is publicly accessible.
• You do not violate the site’s Terms of Service.
• You respect robots.txtrobots.txt and rate limits.
• You do not republish copyrighted content.

It is not legal to scrape:

• Behind logins without permission.
• Personal data covered by GDPR / CCPA without lawful basis.
• Content explicitly forbidden by the site’s ToS.

When in doubt, ask the site owner. Many will give you an API or a data dump.

Real-World Applications

• Competitive pricing intelligence.
• Job board aggregation (most platforms forbid this — check first).
• News aggregation (RSS is usually a better path — see RSS Feed Reader).
• Academic research datasets.
• Inventory monitoring — restock alerts, drop-shipping pipelines.

Educational Value

• HTTP fundamentals, headers, status codes.
• HTML/CSS selectors translated into BeautifulSoup queries.
• Defensive coding — every selector can return NoneNone.
• CSV/JSON/SQLite trade-offs as output formats.
• Politeness — User-Agent, delays, robots.txt.

Next Steps

• Add pagination to crawl every page.
• Add csv.DictWritercsv.DictWriter instead of manual writing.
• Set a real User-Agent and respect robots.txtrobots.txt.
• Save to SQLite so each run inserts incrementally.
• Schedule with cron / Task Scheduler.
• Cross-link with Basic Web Crawler and Web Page Scraper with Notifications.

Visualize it

Here’s how data flows from the live page to your saved results.

  flowchart LR
  A["Send HTTP request (requests)"] --> B["Receive HTML"]
  B --> C["Parse with BeautifulSoup"]
  C --> D["Extract target data"]
  D --> E["Save / print results"]

Conclusion

You built a real scraper against a real e-commerce page, fixed the half-dozen subtle bugs that bite every first scraper, and have a road map to a polite, scheduled, alerting data pipeline. Scraping is one of the best applied-Python skills — every commercial scraper in the world is a few hundred lines on top of requestsrequests + BeautifulSoup. Full source on GitHub. Explore more on Python Central Hub.