Web Page Content Downloader

Abstract

Saving a web page to disk is the simplest “real” web project: one HTTP request, one file write. But the gap between “works on example.com” and “works on any URL” hides every web-programming concept in one place — encoding, redirects, retries, timeouts, errors, streaming large files, and politeness toward servers. In this tutorial you build the trivial 3-line version with urllib, then evolve to a robust downloader using requests, streaming for large responses, retry policies, progress bars, and a mirror mode that downloads HTML plus its referenced assets.

You will leave understanding:

• The HTTP request/response cycle.
• Why default urllib is fine for toys and requests is the standard for real code.
• How content encoding (gzip, br), character encoding, and redirects all bite naïve scripts.
• How to stream large responses without loading them into memory.
• The ethics and legality of downloading — robots.txt, rate limits, copyright.

Prerequisites

• Python 3.6 or above.
• A text editor or IDE.
• Internet connection.
• Comfort with file I/O.

Part 1 — The Trivial Version

urllib ships with Python so no install is needed.

Create the project

1. Create folder web-content-downloader.
2. Inside, create webpagecontentdownloader.py.

Write the code

Web Page Downloader

pch.viewSource

Web Page Downloader

"""Download a web page to a file -- the naive way, then the way that works.

The three-line version at the bottom of every tutorial reads the whole
response into memory, decodes it as UTF-8 whatever the server said, and
crashes on any HTTP error. Each of those is fine until it is not:

* a 404 raises `HTTPError`, which is not caught, so the script ends in a
  traceback instead of a message;
* `decode("utf-8")` on a page served as ISO-8859-1 either raises or produces
  mojibake;
* reading a 2 GB file into a string does exactly what it says.

`download` fixes the first two. `download_large` fixes the third by streaming.

    python webpagecontentdownloader.py            # downloads example.com
    python webpagecontentdownloader.py --test     # no network
"""

import os
import shutil
import sys
import urllib.error
import urllib.parse
import urllib.request

USER_AGENT = "Mozilla/5.0 (compatible; python-central-hub-demo/1.0)"


def ask(prompt="", default=""):
    """Read a line, or fall back to `default` when nobody is there to type."""
    try:
        return input(prompt).strip() or default
    except EOFError:
        print(f"{default}   (no input available, using the default)")
        return default


def guess_encoding(response, fallback="utf-8") -> str:
    """Take the encoding from the response, not from hope.

    `Content-Type: text/html; charset=iso-8859-1` is the server telling you
    exactly how to decode the bytes. Ignoring it and assuming UTF-8 is the
    single most common cause of a page full of question marks.
    """
    charset = response.headers.get_content_charset()
    return charset or fallback


def download(url: str, filename: str, timeout: int = 20) -> int | None:
    """Fetch `url` into `filename`. Returns bytes written, or None on failure.

    Everything the naive version leaves out is here: a User-Agent (many sites
    reject the default `Python-urllib`), a timeout (without one the script can
    hang indefinitely), the server's own encoding, and an error path that
    reports rather than raises.
    """
    request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response:
            encoding = guess_encoding(response)
            text = response.read().decode(encoding, errors="replace")
    except urllib.error.HTTPError as exc:
        print(f"{url} returned HTTP {exc.code} ({exc.reason})")
        return None
    except urllib.error.URLError as exc:
        print(f"could not reach {url}: {exc.reason}")
        return None

    with open(filename, "w", encoding="utf-8", newline="") as handle:
        handle.write(text)
    print(f"{url} -> {filename}  ({len(text):,} characters, "
          f"decoded as {encoding})")
    return len(text)


def download_large(url: str, filename: str, chunk_size: int = 64 * 1024,
                   timeout: int = 20) -> int | None:
    """Stream to disk instead of reading the whole body into memory.

    `shutil.copyfileobj` moves the response to the file `chunk_size` bytes at
    a time, so peak memory is the chunk, not the file. For a page this is
    pointless; for the 700 MB ISO someone eventually points this at, it is the
    difference between working and not.
    """
    request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
    try:
        with urllib.request.urlopen(request, timeout=timeout) as response, \
                open(filename, "wb") as handle:
            shutil.copyfileobj(response, handle, chunk_size)
    except (urllib.error.HTTPError, urllib.error.URLError) as exc:
        print(f"could not download {url}: {exc}")
        return None
    size = os.path.getsize(filename)
    print(f"{url} -> {filename}  ({size:,} bytes, streamed in "
          f"{chunk_size // 1024} KB chunks)")
    return size


def safe_name(url: str, default: str = "page.html") -> str:
    """A filename derived from the URL, without the path traversal.

    `os.path.basename` on an attacker-supplied URL is what stops
    `../../etc/passwd` from being a valid destination.
    """
    path = urllib.parse.urlparse(url).path
    name = os.path.basename(path.rstrip("/"))
    return name or default


def main():
    url = ask("Enter the URL: ", "https://example.com")
    filename = ask("Enter the file name: ", safe_name(url, "example.html"))
    if download(url, filename) is None:
        return 1
    download_large(url, "streamed-" + filename)

    # What the error path looks like when the page is not there.
    download("https://example.com/definitely-not-a-real-page", "missing.html")
    return 0


if __name__ == "__main__":
    if "--test" in sys.argv:
        import unittest

        class TestNaming(unittest.TestCase):
            def test_basename_from_url(self):
                self.assertEqual(
                    safe_name("https://example.com/a/b/page.html"),
                    "page.html")

            def test_empty_path_uses_default(self):
                self.assertEqual(safe_name("https://example.com"),
                                 "page.html")

            def test_traversal_stripped(self):
                self.assertEqual(
                    safe_name("https://example.com/../../etc/passwd"),
                    "passwd")

            def test_trailing_slash(self):
                self.assertEqual(safe_name("https://example.com/docs/"),
                                 "docs")

        unittest.main(argv=sys.argv[:1], exit=False)
    else:
        raise SystemExit(main())

Run it

command

C:\Users\Your Name\web-content-downloader> python webpagecontentdownloader.py
Enter the URL: https://www.example.com
Enter the file name: example.html
# Page content saved to example.html

What it produces

Running the file exactly as it ships takes 0.5 s and prints:

python webpagecontentdownloader.py

Enter the URL: https://example.com   (no input available, using the default)
Enter the file name: example.html   (no input available, using the default)
https://example.com -> example.html  (559 characters, decoded as utf-8)
https://example.com -> streamed-example.html  (559 bytes, streamed in 64 KB chunks)
https://example.com/definitely-not-a-real-page returned HTTP 404 (Not Found)

How it fits together

Read from the top: this is what runs when you execute the file, and which function calls which. It is generated from the code, so it cannot drift from it.

diagram Diagram mermaid

  flowchart TD
  RUN(["python webpagecontentdownloader.py"])
  ask("ask")
  RUN --> ask

Step-by-Step Explanation (Trivial Version)

trivial.py

import urllib.request
 
url = input("URL: ")
filename = input("Filename: ")
response = urllib.request.urlopen(url)
data = response.read().decode("utf-8")
with open(filename, "w", encoding="utf-8") as f:
    f.write(data)

• urlopen sends a GET and returns a file-like object.
• read() slurps the entire response into memory.
• .decode("utf-8") turns the bytes into a Python string — fails for non-UTF-8 pages, which is most of the legacy web.

This version works only when the URL is reachable, the page is UTF-8, smaller than your RAM, and does not redirect.

Part 2 — Robust Version with requests

install

pip install requests

robust.py

import requests, sys
from urllib.parse import urlparse
 
def download(url: str, filename: str | None = None) -> str:
    if not url.startswith(("http://", "https://")):
        url = "https://" + url
    try:
        r = requests.get(url, timeout=30, headers={"User-Agent": "PCH-Downloader/1.0"})
        r.raise_for_status()
    except requests.HTTPError as e:
        print(f"HTTP {e.response.status_code}: {e.response.reason}")
        sys.exit(1)
    except requests.RequestException as e:
        print(f"Network error: {e}")
        sys.exit(1)
 
    filename = filename or (urlparse(url).path.rsplit("/", 1)[-1] or "page.html")
    encoding = r.encoding or r.apparent_encoding or "utf-8"
    with open(filename, "w", encoding=encoding) as f:
        f.write(r.text)
    print(f"Saved {len(r.text):,} chars to {filename}")
    return filename

What this fixes:

• Custom User-Agent. Default python-requests/... is widely blocked.
• Timeout. No hanging on slow servers.
• raise_for_status() turns 4xx/5xx into exceptions.
• r.encoding + r.apparent_encoding. First trusts the server’s Content-Type header; falls back to chardet-based guess.
• URL auto-completion. example.com → https://example.com.
• Smart filename. Derives from URL path if not provided.

Stream Large Files

requests.get(...).text loads everything into RAM. For PDFs, ZIPs, or images, stream instead:

stream.py

def download_large(url: str, filename: str, chunk: int = 8192):
    with requests.get(url, stream=True, timeout=30) as r:
        r.raise_for_status()
        total = int(r.headers.get("Content-Length", 0))
        done = 0
        with open(filename, "wb") as f:
            for piece in r.iter_content(chunk_size=chunk):
                if not piece: continue
                f.write(piece)
                done += len(piece)
                if total:
                    pct = done * 100 // total
                    print(f"\r{pct}%  ({done:,}/{total:,} bytes)", end="", flush=True)
        print()

stream=True defers the body download. iter_content yields it in chunks so memory stays flat regardless of file size.

For a real progress bar, use tqdm:

tqdm.py

from tqdm import tqdm
with requests.get(url, stream=True) as r, open(filename, "wb") as f:
    bar = tqdm(total=int(r.headers["Content-Length"]), unit="B", unit_scale=True)
    for chunk in r.iter_content(8192):
        f.write(chunk); bar.update(len(chunk))
    bar.close()

Retries with Back-off

Transient errors (5xx, network blips) should retry; permanent ones (404) should not:

retry.py

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
 
retry = Retry(total=5, backoff_factor=1,
              status_forcelist=[502, 503, 504], allowed_methods=["GET"])
adapter = HTTPAdapter(max_retries=retry)
session = requests.Session()
session.mount("https://", adapter)
session.mount("http://", adapter)
r = session.get(url, timeout=30)

backoff_factor=1 means waits of 1, 2, 4, 8, 16 seconds between retries.

Common Mistakes

Problem	Cause	Fix
UnicodeDecodeError	Forced utf-8 on non-UTF-8 page	Use r.encoding / r.apparent_encoding
403 Forbidden	Default User-Agent blocked	Send a real header
Hangs indefinitely	No timeout	timeout=30
MemoryError on 4 GB ZIP	r.text loads everything	stream=True + iter_content
File not saved cross-platform	\ in filename	Use urllib.parse.quote or Path
Directory traversal	User-supplied filename "../../etc/passwd"	Sanitize with os.path.basename

Variations to Try

1. Mirror mode

Pull the HTML, parse with BeautifulSoup, then download every <img>, <script>, <link rel=stylesheet> and rewrite the HTML to point at the local copies. Result: a fully self-contained index.html plus an assets/ folder.

2. Recursive crawl

Follow internal <a> links up to depth N. See Basic Web Crawler for the proper queue + visited-set pattern.

3. Authentication

requests.get(url, auth=("user", "pass")) for HTTP Basic; or use session cookies for login forms.

4. Robots.txt compliance

Use urllib.robotparser to honor crawl directives before downloading:

robots.py

from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url(urlparse(url)._replace(path="/robots.txt").geturl())
rp.read()
if not rp.can_fetch("PCH-Downloader/1.0", url):
    print("robots.txt forbids this URL"); return

5. JSON-aware mode

If Content-Type is application/json, pretty-print with json.dumps(r.json(), indent=2).

6. PDF extraction

pip install pypdf. After download, extract text and save side-by-side as .txt.

7. Headless browser

Some pages are JavaScript-rendered and arrive empty via requests. Switch to Playwright to get the rendered DOM.

8. Resume support

For broken downloads, use HTTP Range headers:

resume.py

existing = os.path.getsize(filename) if os.path.exists(filename) else 0
r = requests.get(url, headers={"Range": f"bytes={existing}-"}, stream=True)
mode = "ab" if r.status_code == 206 else "wb"

9. Batch from a file

Read URLs from urls.txt and download all (with rate limiting).

10. GUI version

Tkinter window with URL entry, save-as button, and progress bar. See Currency Exchange Rate Calculator GUI for the pattern.

Ethical & Legal Notes

• Read the ToS. Many sites forbid bulk downloading.
• Respect robots.txt. It is the standard signal.
• Identify yourself with a real User-Agent plus contact info.
• Rate-limit aggressively when crawling many pages — 1 request per second is friendly.
• Copyright applies to downloaded content; for personal study most uses are fine, redistribution is not.
• Never bypass paywalls or login walls without explicit permission.

Security Considerations

• Validate URLs — refuse file:, javascript:, data: schemes if users supply URLs.
• Sanitize filenames with os.path.basename to block directory traversal.
• Scan downloaded executables before running them — requests does not check for malware.
• Pin TLS for sensitive endpoints with verify= and a CA bundle.

Real-World Applications

• Personal-page archive (“save for offline reading”).
• Migration tools — copying content from a CMS to a static site.
• SEO audits — pull pages locally for parsing.
• Dataset construction for ML — but check licenses.
• One-off data extraction for research.

Educational Value

• HTTP fundamentals — GET, status codes, headers.
• Encoding awareness — content vs. character encoding.
• Streaming — when memory is finite and the file is not.
• Error taxonomy — HTTPError, Timeout, ConnectionError mean different things.
• Politeness — the principle that scales when you scale up.

Next Steps

• Replace urllib → requests for everything.
• Add streaming + progress bar.
• Add retries with exponential backoff.
• Build the mirror mode that downloads referenced assets.
• Compare with Basic Web Crawler for multi-page traversal.

Conclusion

You upgraded a trivial download script into a robust, polite, streaming, resumable, retrying downloader. Every web-aware program you ever write reuses these patterns — fail safely, respect the server, never load more than you need. Full source on GitHub. Explore more web projects on Python Central Hub.

Pitfalls

• decode("utf-8") regardless of what the server said. The response carries a charset in its Content-Type header, and it is not always UTF-8. Assuming is how a page becomes Café. response.headers.get_content_charset() is the answer the server already gave you.
• No error handling at all. urlopen raises HTTPError on a 404 and URLError when DNS fails, and the original caught neither, so any missing page ended in a traceback. The measured run above shows the fixed path: https://example.com/definitely-not-a-real-page returned HTTP 404.
• No timeout. Without one, a server that accepts the connection and never answers hangs the script indefinitely. There is no default.
• response.read() loads the whole body into memory. Fine for a 559-byte page, fatal for the large file someone eventually points this at. download_large streams with shutil.copyfileobj so peak memory is one 64 KB chunk.
• A filename taken from a URL is attacker-controlled. safe_name runs os.path.basename, so https://example.com/../../etc/passwd writes to passwd in the current directory rather than wherever the path pointed.
• errors="ignore" destroys data. It makes the exception go away by deleting the bytes it could not decode — 6 characters lost in the exercise below, unrecoverably.

Recap

• Measured run: 559 characters decoded as utf-8, the same page streamed as 559 bytes in 64 KB chunks, and a 404 reported rather than raised.
• Four things separate download from the three-line version: a User-Agent, a timeout, the server’s own encoding, and an error path.
• Bytes have no encoding. A decoding is a claim about what they mean, and the wrong claim either raises or silently produces the wrong text.
• Mojibake is recoverable when the wrong codec mapped every byte to something (cp1252 does); it is not recoverable after errors="ignore".

pch.quizTag pch.quizDefaultTitle

1. The same UTF-8 bytes decoded as cp1252 give 'Café' instead of 'Café'. Can that be undone?

   A No, the data is lost B Yes — cp1252 maps every byte to some character, so re-encoding as cp1252 recovers the original bytes and they can be decoded again as UTF-8 C Only if the text is ASCII D Only with a third-party library
   pch.quizShowAnswer

   B — Yes — cp1252 maps every byte to some character, so re-encoding as cp1252 recovers the original bytes and they can be decoded again as UTF-8

2. Why does download_large use shutil.copyfileobj instead of response.read()?

   A It is faster B It moves the body to disk in fixed-size chunks, so peak memory is the chunk rather than the whole file C read() does not work on HTTP responses D It handles encoding automatically
   pch.quizShowAnswer

   B — It moves the body to disk in fixed-size chunks, so peak memory is the chunk rather than the whole file

3. What does errors='ignore' do to text it cannot decode?

   A Replaces it with a placeholder character B Deletes it — the exception goes away because the data does, and no later re-encoding brings it back C Logs a warning and keeps the bytes D Falls back to latin-1
   pch.quizShowAnswer

   B — Deletes it — the exception goes away because the data does, and no later re-encoding brings it back

Try it yourself

pch.exerciseTag pch.exerciseTitle