Skip to content
Python Central Hub

Web Page Scraper with Notifications

View on GitHub

Abstract

Some sites change rarely but matter when they do — a course catalog opens new spots, a product comes back in stock, a status page goes red, a government form gets a new revision. In this project you build an automated watcher that fetches a URL on a schedule, compares the content to the last fetch using an MD5 hash, and pings you when something changes through one or more channels: desktop notification, email, Slack webhook, or Discord webhook.

You will learn:

  • How to detect change cheaply with content hashing.
  • How to scope monitoring to a single CSS selector instead of the whole page.
  • How to send notifications via four different channels.
  • How to schedule recurring jobs with the scheduleschedule library.
  • How to design a polite watcher that does not get banned.

Prerequisites

  • Python 3.6 or above.
  • A code editor or IDE.
  • An internet connection.
  • (Optional) An SMTP account (Gmail app password works) and/or a Slack/Discord webhook URL.

Install Dependencies

install
pip install requests beautifulsoup4 schedule plyer
install
pip install requests beautifulsoup4 schedule plyer
  • requestsrequests — fetch the page.
  • beautifulsoup4beautifulsoup4 — extract content from HTML.
  • scheduleschedule — run a job every N minutes.
  • plyerplyer — cross-platform desktop notifications.

Getting Started

Create the project

  1. Create folder webpage-monitorwebpage-monitor.
  2. Inside, create webpagescrapernotifications.pywebpagescrapernotifications.py.

Write the code

Web Page MonitorSource
Web Page Monitor
# Web Page Scraper with Notifications
 
import requests
from bs4 import BeautifulSoup
import time
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
import schedule
import hashlib
 
class WebPageMonitor:
    def __init__(self, url, email_config=None):
        self.url = url
        self.email_config = email_config
        self.previous_content = None
        
    def get_page_content(self):
        """Fetch and parse web page content"""
        try:
            response = requests.get(self.url)
            response.raise_for_status()
            soup = BeautifulSoup(response.content, 'html.parser')
            
            # Extract text content (you can modify this to target specific elements)
            content = soup.get_text().strip()
            return content
        except requests.RequestException as e:
            print(f"Error fetching page: {e}")
            return None
    
    def get_content_hash(self, content):
        """Generate hash of content for comparison"""
        return hashlib.md5(content.encode()).hexdigest()
    
    def send_notification_email(self, subject, message):
        """Send email notification when changes are detected"""
        if not self.email_config:
            print("Email configuration not provided")
            return
            
        try:
            msg = MIMEMultipart()
            msg['From'] = self.email_config['from_email']
            msg['To'] = self.email_config['to_email']
            msg['Subject'] = subject
            
            msg.attach(MIMEText(message, 'plain'))
            
            server = smtplib.SMTP(self.email_config['smtp_server'], self.email_config['smtp_port'])
            server.starttls()
            server.login(self.email_config['from_email'], self.email_config['password'])
            
            text = msg.as_string()
            server.sendmail(self.email_config['from_email'], self.email_config['to_email'], text)
            server.quit()
            
            print("Notification email sent successfully!")
        except Exception as e:
            print(f"Error sending email: {e}")
    
    def send_desktop_notification(self, title, message):
        """Send desktop notification (cross-platform)"""
        try:
            import plyer
            plyer.notification.notify(
                title=title,
                message=message,
                timeout=10
            )
        except ImportError:
            print("Desktop notifications require 'plyer' package")
            print(f"Notification: {title} - {message}")
    
    def check_for_changes(self):
        """Check if page content has changed"""
        print(f"Checking {self.url} for changes...")
        
        current_content = self.get_page_content()
        if current_content is None:
            return
        
        current_hash = self.get_content_hash(current_content)
        
        if self.previous_content is None:
            self.previous_content = current_hash
            print("Initial content captured. Monitoring for changes...")
            return
        
        if current_hash != self.previous_content:
            print("Content change detected!")
            
            # Send notifications
            subject = f"Web Page Change Detected: {self.url}"
            message = f"The webpage {self.url} has been updated!\n\nCheck it out: {self.url}"
            
            self.send_desktop_notification("Page Updated!", f"Changes detected on {self.url}")
            
            if self.email_config:
                self.send_notification_email(subject, message)
            
            self.previous_content = current_hash
        else:
            print("No changes detected.")
    
    def start_monitoring(self, interval_minutes=30):
        """Start monitoring the webpage at specified intervals"""
        print(f"Starting to monitor {self.url} every {interval_minutes} minutes...")
        
        # Schedule the check
        schedule.every(interval_minutes).minutes.do(self.check_for_changes)
        
        # Initial check
        self.check_for_changes()
        
        # Keep the script running
        while True:
            schedule.run_pending()
            time.sleep(1)
 
def main():
    # Example usage
    url_to_monitor = "https://example.com"
    
    # Optional email configuration
    email_config = {
        'from_email': 'your_email@gmail.com',
        'to_email': 'recipient@gmail.com',
        'password': 'your_app_password',
        'smtp_server': 'smtp.gmail.com',
        'smtp_port': 587
    }
    
    # Create monitor instance (set email_config to None to disable email notifications)
    monitor = WebPageMonitor(url_to_monitor, email_config=None)
    
    # Start monitoring (checks every 30 minutes)
    try:
        monitor.start_monitoring(interval_minutes=1)  # Check every minute for demo
    except KeyboardInterrupt:
        print("\nMonitoring stopped by user.")
 
if __name__ == "__main__":
    main()
Web Page Monitor
# Web Page Scraper with Notifications
 
import requests
from bs4 import BeautifulSoup
import time
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
import schedule
import hashlib
 
class WebPageMonitor:
    def __init__(self, url, email_config=None):
        self.url = url
        self.email_config = email_config
        self.previous_content = None
        
    def get_page_content(self):
        """Fetch and parse web page content"""
        try:
            response = requests.get(self.url)
            response.raise_for_status()
            soup = BeautifulSoup(response.content, 'html.parser')
            
            # Extract text content (you can modify this to target specific elements)
            content = soup.get_text().strip()
            return content
        except requests.RequestException as e:
            print(f"Error fetching page: {e}")
            return None
    
    def get_content_hash(self, content):
        """Generate hash of content for comparison"""
        return hashlib.md5(content.encode()).hexdigest()
    
    def send_notification_email(self, subject, message):
        """Send email notification when changes are detected"""
        if not self.email_config:
            print("Email configuration not provided")
            return
            
        try:
            msg = MIMEMultipart()
            msg['From'] = self.email_config['from_email']
            msg['To'] = self.email_config['to_email']
            msg['Subject'] = subject
            
            msg.attach(MIMEText(message, 'plain'))
            
            server = smtplib.SMTP(self.email_config['smtp_server'], self.email_config['smtp_port'])
            server.starttls()
            server.login(self.email_config['from_email'], self.email_config['password'])
            
            text = msg.as_string()
            server.sendmail(self.email_config['from_email'], self.email_config['to_email'], text)
            server.quit()
            
            print("Notification email sent successfully!")
        except Exception as e:
            print(f"Error sending email: {e}")
    
    def send_desktop_notification(self, title, message):
        """Send desktop notification (cross-platform)"""
        try:
            import plyer
            plyer.notification.notify(
                title=title,
                message=message,
                timeout=10
            )
        except ImportError:
            print("Desktop notifications require 'plyer' package")
            print(f"Notification: {title} - {message}")
    
    def check_for_changes(self):
        """Check if page content has changed"""
        print(f"Checking {self.url} for changes...")
        
        current_content = self.get_page_content()
        if current_content is None:
            return
        
        current_hash = self.get_content_hash(current_content)
        
        if self.previous_content is None:
            self.previous_content = current_hash
            print("Initial content captured. Monitoring for changes...")
            return
        
        if current_hash != self.previous_content:
            print("Content change detected!")
            
            # Send notifications
            subject = f"Web Page Change Detected: {self.url}"
            message = f"The webpage {self.url} has been updated!\n\nCheck it out: {self.url}"
            
            self.send_desktop_notification("Page Updated!", f"Changes detected on {self.url}")
            
            if self.email_config:
                self.send_notification_email(subject, message)
            
            self.previous_content = current_hash
        else:
            print("No changes detected.")
    
    def start_monitoring(self, interval_minutes=30):
        """Start monitoring the webpage at specified intervals"""
        print(f"Starting to monitor {self.url} every {interval_minutes} minutes...")
        
        # Schedule the check
        schedule.every(interval_minutes).minutes.do(self.check_for_changes)
        
        # Initial check
        self.check_for_changes()
        
        # Keep the script running
        while True:
            schedule.run_pending()
            time.sleep(1)
 
def main():
    # Example usage
    url_to_monitor = "https://example.com"
    
    # Optional email configuration
    email_config = {
        'from_email': 'your_email@gmail.com',
        'to_email': 'recipient@gmail.com',
        'password': 'your_app_password',
        'smtp_server': 'smtp.gmail.com',
        'smtp_port': 587
    }
    
    # Create monitor instance (set email_config to None to disable email notifications)
    monitor = WebPageMonitor(url_to_monitor, email_config=None)
    
    # Start monitoring (checks every 30 minutes)
    try:
        monitor.start_monitoring(interval_minutes=1)  # Check every minute for demo
    except KeyboardInterrupt:
        print("\nMonitoring stopped by user.")
 
if __name__ == "__main__":
    main()

Run it

run
python webpagescrapernotifications.py
run
python webpagescrapernotifications.py

The script polls the target URL every 30 minutes (by default), hashes the content, and notifies you when the hash changes from the previous run.

Step-by-Step Explanation

1. Class skeleton

webpagescrapernotifications.py
import hashlib, time, smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import requests
from bs4 import BeautifulSoup
import schedule
 
class WebPageMonitor:
    def __init__(self, url, selector=None, email_config=None):
        self.url = url
        self.selector = selector              # optional CSS selector to scope changes
        self.email_config = email_config
        self.previous_hash = None
webpagescrapernotifications.py
import hashlib, time, smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import requests
from bs4 import BeautifulSoup
import schedule
 
class WebPageMonitor:
    def __init__(self, url, selector=None, email_config=None):
        self.url = url
        self.selector = selector              # optional CSS selector to scope changes
        self.email_config = email_config
        self.previous_hash = None

2. Fetch and extract content

webpagescrapernotifications.py
def get_page_content(self):
    response = requests.get(self.url, timeout=15, headers={
        "User-Agent": "WebPageMonitor/1.0 (+you@example.com)"
    })
    response.raise_for_status()
    soup = BeautifulSoup(response.content, "html.parser")
    if self.selector:
        node = soup.select_one(self.selector)
        if node is None:
            raise ValueError(f"Selector '{self.selector}' not found")
        return node.get_text(separator=" ", strip=True)
    return soup.get_text(separator=" ", strip=True)
webpagescrapernotifications.py
def get_page_content(self):
    response = requests.get(self.url, timeout=15, headers={
        "User-Agent": "WebPageMonitor/1.0 (+you@example.com)"
    })
    response.raise_for_status()
    soup = BeautifulSoup(response.content, "html.parser")
    if self.selector:
        node = soup.select_one(self.selector)
        if node is None:
            raise ValueError(f"Selector '{self.selector}' not found")
        return node.get_text(separator=" ", strip=True)
    return soup.get_text(separator=" ", strip=True)
  • A custom User-AgentUser-Agent identifies the bot — many sites block default Python requests.
  • selectorselector lets you watch only the relevant part of the page. Without it, every footer copyright year change triggers an alert.

3. Hash to detect change

webpagescrapernotifications.py
def get_hash(self, content):
    return hashlib.md5(content.encode("utf-8")).hexdigest()
webpagescrapernotifications.py
def get_hash(self, content):
    return hashlib.md5(content.encode("utf-8")).hexdigest()

MD5 is not secure for cryptography but is perfect for change detection — it is fast and the input is not adversarial.

4. The check

webpagescrapernotifications.py
def check_for_changes(self):
    try:
        content = self.get_page_content()
    except Exception as e:
        print(f"[error] {e}")
        return
    current_hash = self.get_hash(content)
    if self.previous_hash is None:
        self.previous_hash = current_hash
        print("First check — baseline recorded.")
        return
    if current_hash != self.previous_hash:
        print("Change detected!")
        self.notify("Page changed", f"Content hash changed for {self.url}")
        self.previous_hash = current_hash
    else:
        print("No change.")
webpagescrapernotifications.py
def check_for_changes(self):
    try:
        content = self.get_page_content()
    except Exception as e:
        print(f"[error] {e}")
        return
    current_hash = self.get_hash(content)
    if self.previous_hash is None:
        self.previous_hash = current_hash
        print("First check — baseline recorded.")
        return
    if current_hash != self.previous_hash:
        print("Change detected!")
        self.notify("Page changed", f"Content hash changed for {self.url}")
        self.previous_hash = current_hash
    else:
        print("No change.")

Notice we do not alert on the first check — that would always fire on startup. We only alert on transitions.

5. Email notification

webpagescrapernotifications.py
def send_email(self, subject, body):
    cfg = self.email_config
    msg = MIMEMultipart()
    msg["From"] = cfg["from_email"]
    msg["To"] = cfg["to_email"]
    msg["Subject"] = subject
    msg.attach(MIMEText(body, "plain"))
    with smtplib.SMTP(cfg["smtp_server"], cfg["smtp_port"]) as server:
        server.starttls()
        server.login(cfg["from_email"], cfg["password"])
        server.send_message(msg)
webpagescrapernotifications.py
def send_email(self, subject, body):
    cfg = self.email_config
    msg = MIMEMultipart()
    msg["From"] = cfg["from_email"]
    msg["To"] = cfg["to_email"]
    msg["Subject"] = subject
    msg.attach(MIMEText(body, "plain"))
    with smtplib.SMTP(cfg["smtp_server"], cfg["smtp_port"]) as server:
        server.starttls()
        server.login(cfg["from_email"], cfg["password"])
        server.send_message(msg)

For Gmail:

  • smtp_server = "smtp.gmail.com"smtp_server = "smtp.gmail.com", smtp_port = 587smtp_port = 587.
  • Use an App Password (Google Account → Security → 2-Step Verification → App passwords), not your real Gmail password.

6. Desktop notification

webpagescrapernotifications.py
def send_desktop(self, title, message):
    from plyer import notification
    notification.notify(title=title, message=message, timeout=10)
webpagescrapernotifications.py
def send_desktop(self, title, message):
    from plyer import notification
    notification.notify(title=title, message=message, timeout=10)

Works on Windows, macOS, and many Linux desktops via notify-sendnotify-send.

7. Schedule the job

webpagescrapernotifications.py
def start(self, interval_minutes=30):
    schedule.every(interval_minutes).minutes.do(self.check_for_changes)
    print(f"Monitoring {self.url} every {interval_minutes} minutes. Ctrl+C to stop.")
    while True:
        schedule.run_pending()
        time.sleep(1)
webpagescrapernotifications.py
def start(self, interval_minutes=30):
    schedule.every(interval_minutes).minutes.do(self.check_for_changes)
    print(f"Monitoring {self.url} every {interval_minutes} minutes. Ctrl+C to stop.")
    while True:
        schedule.run_pending()
        time.sleep(1)

schedule.every().X.minutes.do(fn)schedule.every().X.minutes.do(fn) registers a recurring job. The while Truewhile True loop is the “tick” — it wakes up every second to see if any jobs are due.

Webhook Notifications (Slack / Discord)

Webhooks are the easiest cross-team alert channel. Create one in your workspace settings, then:

slack.py
def send_slack(self, text):
    requests.post(self.slack_webhook_url, json={"text": text}, timeout=10)
 
def send_discord(self, text):
    requests.post(self.discord_webhook_url, json={"content": text}, timeout=10)
slack.py
def send_slack(self, text):
    requests.post(self.slack_webhook_url, json={"text": text}, timeout=10)
 
def send_discord(self, text):
    requests.post(self.discord_webhook_url, json={"content": text}, timeout=10)

A single line carries the alert — no SMTP setup, no app passwords.

Smarter Change Detection

Plain hashes flag any change — even a refreshed CSRF token. Strategies to reduce noise:

  1. Scope with a CSS selector. Watch only #main-content#main-content, not the whole page.
  2. Strip noisy elements before hashing:
    strip.py
    for tag in soup.select("script, style, .ads, .timestamp"): tag.decompose()
    strip.py
    for tag in soup.select("script, style, .ads, .timestamp"): tag.decompose()
  3. Diff instead of hash. When the hash changes, compute a unified diff to show what changed:
    diff.py
    import difflib
diff = "\n".join(difflib.unified_diff(
    old.splitlines(), new.splitlines(), lineterm=""))[:1500]
self.notify("Page changed", diff)
    diff.py
    import difflib
diff = "\n".join(difflib.unified_diff(
    old.splitlines(), new.splitlines(), lineterm=""))[:1500]
self.notify("Page changed", diff)
  4. Threshold-based. Ignore changes under N characters using difflib.SequenceMatcher.ratio()difflib.SequenceMatcher.ratio().

Polite Scraping Checklist

Before pointing this at a production site:

  • Respect robots.txtrobots.txt. Use urllib.robotparserurllib.robotparser to check before fetching.
  • Set a real User-AgentUser-Agent including contact info.
  • Use a reasonable interval. 30 minutes is friendly; 30 seconds is hostile.
  • Honor rate limits. Back off on 429 responses.
  • Read the Terms of Service. Some sites explicitly forbid automated monitoring.
  • Stop immediately if asked.

A monitor that hammers a small site at 1 Hz can take it offline. Be a good citizen.

Common Mistakes

ProblemCauseFix
Always fires on startupCompared against NoneNoneSkip the first iteration; record baseline only
Email login failsUsed real password on GmailGenerate an App Password
403 ForbiddenDefault Python User-AgentSet a real header
Notifications fire constantlyHashing the whole page including timestampsScope with a selector or strip dynamic elements
Crash freezes loopNetwork error inside the jobWrap the job body in try/excepttry/except
Process killed when terminal closesRun as a foreground processUse nohupnohup, tmuxtmux, systemdsystemd, or run as a Windows service

Variations to Try

1. Multiple URLs at once

multi.py
sites = [WebPageMonitor(u) for u in URL_LIST]
for s in sites:
    schedule.every(30).minutes.do(s.check_for_changes)
multi.py
sites = [WebPageMonitor(u) for u in URL_LIST]
for s in sites:
    schedule.every(30).minutes.do(s.check_for_changes)

2. Track specific values

Use a regex to extract a number (price, stock count) and only alert if it crosses a threshold:

price.py
import re
price = float(re.search(r"\$([0-9.]+)", content).group(1))
if price < TARGET_PRICE:
    self.notify("Price drop!", f"Now ${price}")
price.py
import re
price = float(re.search(r"\$([0-9.]+)", content).group(1))
if price < TARGET_PRICE:
    self.notify("Price drop!", f"Now ${price}")

3. Screenshot diff

Use Playwright to render the page and save a screenshot. Compute pixel difference with Pillow.

4. Web dashboard

Pair with Flask (see Basic Web Server) — a page that lists all monitored URLs, last-check time, and recent history.

5. SQLite history

Persist every check (timestamp, hash, change?) for later analysis.

6. Pushover / Pushbullet / Telegram

Three more notification channels with simple APIs.

7. Headless browser

Some sites are JavaScript-rendered. Replace requestsrequests with Playwright so you scrape the rendered DOM.

8. RSS-aware mode

If the site has an RSS feed (see RSS Feed Reader), use that instead of scraping — far more reliable.

9. Selector tester

A small CLI that lets you try CSS selectors against a URL interactively before wiring them into the monitor.

10. Docker container

Pack the script in a Dockerfile and deploy to a tiny VPS so it runs 24/7 without keeping your laptop on.

Privacy & Operational Notes

  • Do not put secrets in source. Read SMTP passwords and webhook URLs from environment variables: os.environ["SMTP_PASSWORD"]os.environ["SMTP_PASSWORD"].
  • Add *.json*.json, .env.env, credentials.*credentials.* to .gitignore.gitignore.
  • Log carefully. Avoid printing full page content to disk when debugging is over.
  • Rotate webhook URLs if you suspect leaks.

Real-World Applications

  • In-stock alerts — bicycles, GPUs, concert tickets.
  • Status pages — pre-empt outages by watching /status/status.
  • Job boards — be first to apply to new listings.
  • Course availability — universities, training platforms.
  • Government forms — get notified when a deadline or form changes.
  • Pricing intelligence — track competitor prices (within Terms of Service).

Educational Value

  • HTTP & HTML parsing — fetch and extract relevant text.
  • Hashing — fingerprinting for cheap comparison.
  • Scheduling — moving from a one-shot script to a long-running service.
  • Notifications — multiple channels, each with different trade-offs.
  • Defensive coding — failures must not kill the loop.
  • Operational thinking — running a real background process, not just a script.

Next Steps

  • Add CSS selector scoping to reduce noise.
  • Wire up Slack/Discord webhooks alongside email.
  • Persist history to SQLite for analysis.
  • Containerize with Docker and deploy to a cheap VPS.
  • Pair with Email Sender for richer email alerts.
  • Extend with diff snippets in the notification body.

Conclusion

You built a real automation tool: it watches the web for you, sleeps politely between checks, alerts you through whichever channel you prefer, and recovers gracefully from network hiccups. Hash-based change detection, scheduling, and webhook notifications show up in nearly every monitoring system in production. Full source on GitHub. Explore more automation projects on Python Central Hub.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did