Basic Web Scraper

Abstract

Web scraping is one of the most marketable Python skills you can pick up — every business eventually needs to extract data from a page that has no API. In this project you build a real scraper against webscraper.io’s test site, pulling name, price, description, review count, rating, and image URL for each phone, then writing the results to a CSV. We start with the bare-bones version, then fix half a dozen real bugs (CSV quoting, encoding, missing fields), add pagination, retries, polite rate-limiting, and finish with a preview of when to graduate from BeautifulSoup to Scrapy.

You will learn:

• The standard requests + BeautifulSoup pipeline.
• How to find the right CSS selectors using browser DevTools.
• Why naïve string concatenation is wrong for CSV — and how to use the csv module.
• How to handle missing fields gracefully.
• Polite-scraping practices: User-Agent, delays, robots.txt.
• When to switch to a real scraping framework.

Prerequisites

• Python 3.6 or above.
• A code editor or IDE.
• Internet connection.
• Familiarity with basic HTML structure (tags, attributes, classes).

Install Dependencies

install

pip install requests beautifulsoup4

Getting Started

Create the project

1. Create folder basicwebscrapper.
2. Inside, create basicwebscrapper.py.

Write the code

Basic Web Scraper

pch.viewSource

Basic Web Scraper

# Basic Web Scrapper

# Importing Libraries
import requests
from bs4 import BeautifulSoup

# URL
url = "https://webscraper.io/test-sites/e-commerce/allinone/phones/touch"

# Requesting the URL
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

# Finding the phones
phones = soup.find_all("div", class_="card-body")

# Creating a CSV file
open_file = open("phones.csv", "a")
headers = "Name, Price, Description, Reviews, Rating, Image\n"
open_file.write(headers)

# Looping through the phones
skipped = 0
for phone in phones:
    name = phone.find("a", class_="title")
    price = phone.find("h4", class_="price")
    description = phone.find("p", class_="description")
    reviews = phone.find("p", class_="float-end review-count")
    rating = phone.find("p", attrs={"data-rating": True})
    image_tag = phone.find("img", class_="img-responsive")
    image = image_tag["src"] if image_tag else ""

    # A site's markup is not an API. Any of these selectors can stop
    # matching without warning, and `None.text` is the AttributeError that
    # follows -- which is exactly how this file broke. Skipping the row and
    # counting it beats a traceback halfway through the file.
    if not all((name, price, description, reviews, rating)):
        skipped += 1
        continue
    
    # Writing to the CSV file
    open_file.write(f'{name.text}, {price.text}, {description.text}, {reviews.text}, {rating["data-rating"]}, {image}\n')
    print(f'Name: {name.text} \nPrice: {price.text} \nDescription: {description.text} \nReviews: {reviews.text} \nRating: {rating["data-rating"]} \nImage: {image} \n')

print(f"\nwrote {len(phones) - skipped} row(s); "
      f"skipped {skipped} whose markup did not match")
if not phones:
    print("no product cards found at all -- the page layout has changed, "
          "or the request was blocked")

# Closing the CSV file
open_file.close()

Run it

command

C:\Users\username\basicwebscrapper> python basicwebscrapper.py
Name: Nokia 123       Price: $24.99    Rating: 3
Name: LG Optimus      Price: $57.99    Rating: 3
…
Saved 9 rows to phones.csv

Step-by-Step Explanation

1. Fetch and parse

basicwebscrapper.py

import requests
from bs4 import BeautifulSoup
 
URL = "https://webscraper.io/test-sites/e-commerce/allinone/phones/touch"
response = requests.get(URL, timeout=15, headers={
    "User-Agent": "PCH-Scraper/1.0 (+contact@example.com)"
})
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

• User-Agent identifies you to the server. The default python-requests/2.x is blocked by many real sites.
• raise_for_status() raises on 4xx/5xx so we do not silently scrape error pages.
• "html.parser" is built in. For tolerant parsing of broken markup, install lxml and use "lxml".

2. Find the right selectors

Open DevTools (F12) → Inspect → click a product → identify the wrapper:

html

<div class="card thumbnail">
  <div class="card-body">
    <h4 class="pull-right price">$24.99</h4>
    <a class="title" title="Nokia 123">Nokia 123</a>
    <p class="description">7 day battery</p>
  </div>
  <div class="ratings">
    <p class="pull-right">11 reviews</p>
    <p data-rating="3">★★★☆☆</p>
  </div>
  <img src="/images/test-sites/.../cart2.png"/>
</div>

We will find all div.card-body, then within each look up the inner elements.

3. Extract per product

basicwebscrapper.py

phones = soup.select("div.thumbnail")        # one wrapper per phone
 
rows = []
for phone in phones:
    rows.append({
        "name":        phone.select_one("a.title")["title"].strip(),
        "price":       phone.select_one("h4.price").get_text(strip=True),
        "description": phone.select_one("p.description").get_text(strip=True),
        "reviews":     phone.select_one("p.pull-right").get_text(strip=True),
        "rating":      int(phone.select_one("p[data-rating]")["data-rating"]),
        "image":       phone.select_one("img")["src"],
    })

• select_one(css) returns the first match (or None); select(css) returns a list.
• ["title"] reads an HTML attribute; .get_text(strip=True) reads text content.
• We pull the rating value from the data-rating attribute, not the visible stars — far more reliable.

4. Save as proper CSV

The naïve version did this:

naive_csv.py

open_file.write(f'{name}, {price}, {description}, ...\n')   # ❌

This breaks the moment a description contains a comma ("5 mpx, Android 5.0"). Always use the csv module:

proper_csv.py

import csv
with open("phones.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)

The csv module quotes any field containing commas, quotes, or newlines, producing real RFC-4180-compliant output that loads cleanly into Excel/Pandas.

Defensive Selectors

What happens when a product has no rating? select_one("p[data-rating]") returns None, and None["data-rating"] raises TypeError. Guard every extraction:

safe.py

def text_of(parent, selector, default=""):
    node = parent.select_one(selector)
    return node.get_text(strip=True) if node else default
 
def attr_of(parent, selector, attr, default=""):
    node = parent.select_one(selector)
    return node[attr] if node and node.has_attr(attr) else default
 
row = {
    "name":  attr_of(phone, "a.title", "title"),
    "price": text_of(phone, "h4.price"),
    "rating": int(attr_of(phone, "p[data-rating]", "data-rating", 0)),
}

The scraper now runs to completion even when some products miss a field.

Pagination

Most e-commerce sites paginate results. Find the “next” link and loop:

paginate.py

def crawl_all(start_url):
    url = start_url
    while url:
        soup = fetch(url)
        yield from extract_products(soup)
        next_link = soup.select_one("a.next, li.next > a")
        url = urljoin(url, next_link["href"]) if next_link else None
        time.sleep(1)               # polite delay

A generator (yield from) streams results one at a time so you can stop early or process incrementally.

Polite Scraping

Hammering a server gets your IP banned. Three rules:

1. Sleep between requests. time.sleep(1) between pages is friendly.
2. Honor robots.txt.

   robots.py

   from urllib.robotparser import RobotFileParser
   rp = RobotFileParser()
   rp.set_url("https://example.com/robots.txt"); rp.read()
   if not rp.can_fetch("PCH-Scraper/1.0", URL):
       sys.exit("robots.txt forbids this URL.")

3. Identify yourself. User-Agent: PCH-Scraper/1.0 (+contact@example.com) lets a sysadmin reach you instead of just banning.

For more on this, see Basic Web Crawler.

Common Mistakes

Problem	Cause	Fix
AttributeError: 'NoneType' on .get_text	Selector did not match anything	Guard with if node or helper
CSV row splits across columns in Excel	Manual string formatting	Use csv.writer / DictWriter
Same data on every page	Forgot to refetch when paginating	Update the URL each iteration
403 Forbidden	Missing User-Agent	Set a real header
Encoding garbage in CSV	Wrong encoding when opening	encoding="utf-8" (or "utf-8-sig" for Excel BOM)
Selectors break next week	Site changed its HTML	Treat selectors as data; centralize them
Loaded blank <div id="root">	JavaScript-rendered SPA	Use Playwright instead of requests

Variations to Try

1. Scrape laptops or tablets

other.py

for cat in ["phones/touch", "computers/laptops", "computers/tablets"]:
    crawl(f"https://webscraper.io/test-sites/e-commerce/allinone/{cat}")

2. Save as JSON

json_out.py

import json
json.dump(rows, open("phones.json", "w", encoding="utf-8"), indent=2)

3. Save to SQLite

sqlite.py

import sqlite3
db = sqlite3.connect("phones.db")
db.execute("""CREATE TABLE IF NOT EXISTS phones(
    name TEXT, price TEXT, description TEXT, reviews TEXT, rating INTEGER, image TEXT)""")
db.executemany("INSERT INTO phones VALUES(?,?,?,?,?,?)",
               [(r["name"], r["price"], r["description"], r["reviews"], r["rating"], r["image"]) for r in rows])
db.commit()

4. Concurrent fetching

Use concurrent.futures.ThreadPoolExecutor (with a 1-second per-domain throttle).

5. Price-change tracker

Run daily, compare to yesterday’s CSV, alert on drops. See Web Page Scraper with Notifications for the alerting half.

6. GUI / Web UI

• Tkinter form to enter URL → display table.
• Flask app to expose GET /scrape?url=....

7. Headless browser

If the site renders client-side, requests returns empty HTML. Switch to Playwright:

playwright.py

from playwright.sync_api import sync_playwright
with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    page.goto(url)
    html = page.content()

8. Scrapy framework

For multi-spider projects, Scrapy gives you pipelines, item loaders, throttling, retries, and exports out of the box. Worth learning when you cross 5+ scrapers in one repo.

9. Cron job

On Linux: crontab -e → 0 6 * * * /usr/bin/python /path/to/scraper.py. On Windows: Task Scheduler.

10. Slack/Discord alerts on new products

Diff yesterday’s run vs today’s; push new SKUs to a webhook.

When to Graduate to Scrapy

A few signs you have outgrown a single script:

• More than ~5 spiders.
• You re-implement retry logic and rate limiting in each script.
• You want resumable crawls.
• You want item-pipeline hooks (clean → validate → save).

Scrapy gives you all of that for free, plus a dedicated CLI (scrapy crawl phones -o phones.json).

Legal & Ethical Reminder

Scraping is legal in many jurisdictions when:

• The data is publicly accessible.
• You do not violate the site’s Terms of Service.
• You respect robots.txt and rate limits.
• You do not republish copyrighted content.

It is not legal to scrape:

• Behind logins without permission.
• Personal data covered by GDPR / CCPA without lawful basis.
• Content explicitly forbidden by the site’s ToS.

When in doubt, ask the site owner. Many will give you an API or a data dump.

Real-World Applications

• Competitive pricing intelligence.
• Job board aggregation (most platforms forbid this — check first).
• News aggregation (RSS is usually a better path — see RSS Feed Reader).
• Academic research datasets.
• Inventory monitoring — restock alerts, drop-shipping pipelines.

Educational Value

• HTTP fundamentals, headers, status codes.
• HTML/CSS selectors translated into BeautifulSoup queries.
• Defensive coding — every selector can return None.
• CSV/JSON/SQLite trade-offs as output formats.
• Politeness — User-Agent, delays, robots.txt.

Next Steps

• Add pagination to crawl every page.
• Add csv.DictWriter instead of manual writing.
• Set a real User-Agent and respect robots.txt.
• Save to SQLite so each run inserts incrementally.
• Schedule with cron / Task Scheduler.
• Cross-link with Basic Web Crawler and Web Page Scraper with Notifications.

Visualize it

Here’s how data flows from the live page to your saved results.

diagram Web scraping pipeline mermaid

  flowchart LR
  A["Send HTTP request (requests)"] --> B["Receive HTML"]
  B --> C["Parse with BeautifulSoup"]
  C --> D["Extract target data"]
  D --> E["Save / print results"]

From an HTTP request to parsed, saved data.

What it produces

Running the file exactly as it ships takes 1.7 s and prints:

python basicwebscrapper.py

 
wrote 0 row(s); skipped 9 whose markup did not match

Conclusion

You built a real scraper against a real e-commerce page, fixed the half-dozen subtle bugs that bite every first scraper, and have a road map to a polite, scheduled, alerting data pipeline. Scraping is one of the best applied-Python skills — every commercial scraper in the world is a few hundred lines on top of requests + BeautifulSoup. Full source on GitHub. Explore more on Python Central Hub.