Abstract

Saving a web page to disk is the simplest “real” web project: one HTTP request, one file write. But the gap between “works on example.comexample.com” and “works on any URL” hides every web-programming concept in one place — encoding, redirects, retries, timeouts, errors, streaming large files, and politeness toward servers. In this tutorial you build the trivial 3-line version with urlliburllib, then evolve to a robust downloader using requestsrequests, streaming for large responses, retry policies, progress bars, and a mirror mode that downloads HTML plus its referenced assets.

You will leave understanding:

• The HTTP request/response cycle.
• Why default urlliburllib is fine for toys and requestsrequests is the standard for real code.
• How content encoding (gzipgzip, brbr), character encoding, and redirects all bite naïve scripts.
• How to stream large responses without loading them into memory.
• The ethics and legality of downloading — robots.txt, rate limits, copyright.

Prerequisites

• Python 3.6 or above.
• A text editor or IDE.
• Internet connection.
• Comfort with file I/O.

Part 1 — The Trivial Version

urlliburllib ships with Python so no install is needed.

Create the project

1. Create folder web-content-downloaderweb-content-downloader.
2. Inside, create webpagecontentdownloader.pywebpagecontentdownloader.py.

Write the code

# Web Page Content Downloader
 
import urllib.request, urllib.error, urllib.parse
 
url = input('Enter the URL: ')
fileName = input('Enter the file name: ')
 
response = urllib.request.urlopen(url)
webContent = response.read().decode('utf-8')
 
 
f = open(fileName, 'w')
f.write(webContent)
f.close()

# Web Page Content Downloader
 
import urllib.request, urllib.error, urllib.parse
 
url = input('Enter the URL: ')
fileName = input('Enter the file name: ')
 
response = urllib.request.urlopen(url)
webContent = response.read().decode('utf-8')
 
 
f = open(fileName, 'w')
f.write(webContent)
f.close()

Run it

C:\Users\Your Name\web-content-downloader> python webpagecontentdownloader.py
Enter the URL: https://www.example.com
Enter the file name: example.html
# Page content saved to example.html

C:\Users\Your Name\web-content-downloader> python webpagecontentdownloader.py
Enter the URL: https://www.example.com
Enter the file name: example.html
# Page content saved to example.html

Step-by-Step Explanation (Trivial Version)

import urllib.request
 
url = input("URL: ")
filename = input("Filename: ")
response = urllib.request.urlopen(url)
data = response.read().decode("utf-8")
with open(filename, "w", encoding="utf-8") as f:
    f.write(data)

import urllib.request
 
url = input("URL: ")
filename = input("Filename: ")
response = urllib.request.urlopen(url)
data = response.read().decode("utf-8")
with open(filename, "w", encoding="utf-8") as f:
    f.write(data)

• urlopenurlopen sends a GETGET and returns a file-like object.
• read()read() slurps the entire response into memory.
• .decode("utf-8").decode("utf-8") turns the bytes into a Python string — fails for non-UTF-8 pages, which is most of the legacy web.

This version works only when the URL is reachable, the page is UTF-8, smaller than your RAM, and does not redirect.

Part 2 — Robust Version with requestsrequests

pip install requests

pip install requests

import requests, sys
from urllib.parse import urlparse
 
def download(url: str, filename: str | None = None) -> str:
    if not url.startswith(("http://", "https://")):
        url = "https://" + url
    try:
        r = requests.get(url, timeout=30, headers={"User-Agent": "PCH-Downloader/1.0"})
        r.raise_for_status()
    except requests.HTTPError as e:
        print(f"HTTP {e.response.status_code}: {e.response.reason}")
        sys.exit(1)
    except requests.RequestException as e:
        print(f"Network error: {e}")
        sys.exit(1)
 
    filename = filename or (urlparse(url).path.rsplit("/", 1)[-1] or "page.html")
    encoding = r.encoding or r.apparent_encoding or "utf-8"
    with open(filename, "w", encoding=encoding) as f:
        f.write(r.text)
    print(f"Saved {len(r.text):,} chars to {filename}")
    return filename

import requests, sys
from urllib.parse import urlparse
 
def download(url: str, filename: str | None = None) -> str:
    if not url.startswith(("http://", "https://")):
        url = "https://" + url
    try:
        r = requests.get(url, timeout=30, headers={"User-Agent": "PCH-Downloader/1.0"})
        r.raise_for_status()
    except requests.HTTPError as e:
        print(f"HTTP {e.response.status_code}: {e.response.reason}")
        sys.exit(1)
    except requests.RequestException as e:
        print(f"Network error: {e}")
        sys.exit(1)
 
    filename = filename or (urlparse(url).path.rsplit("/", 1)[-1] or "page.html")
    encoding = r.encoding or r.apparent_encoding or "utf-8"
    with open(filename, "w", encoding=encoding) as f:
        f.write(r.text)
    print(f"Saved {len(r.text):,} chars to {filename}")
    return filename

What this fixes:

• Custom User-Agent. Default python-requests/...python-requests/... is widely blocked.
• Timeout. No hanging on slow servers.
• raise_for_status()raise_for_status() turns 4xx/5xx into exceptions.
• r.encodingr.encoding + r.apparent_encodingr.apparent_encoding. First trusts the server’s Content-TypeContent-Type header; falls back to chardetchardet-based guess.
• URL auto-completion. example.comexample.com → https://example.comhttps://example.com.
• Smart filename. Derives from URL path if not provided.

Stream Large Files

requests.get(...).textrequests.get(...).text loads everything into RAM. For PDFs, ZIPs, or images, stream instead:

def download_large(url: str, filename: str, chunk: int = 8192):
    with requests.get(url, stream=True, timeout=30) as r:
        r.raise_for_status()
        total = int(r.headers.get("Content-Length", 0))
        done = 0
        with open(filename, "wb") as f:
            for piece in r.iter_content(chunk_size=chunk):
                if not piece: continue
                f.write(piece)
                done += len(piece)
                if total:
                    pct = done * 100 // total
                    print(f"\r{pct}%  ({done:,}/{total:,} bytes)", end="", flush=True)
        print()

def download_large(url: str, filename: str, chunk: int = 8192):
    with requests.get(url, stream=True, timeout=30) as r:
        r.raise_for_status()
        total = int(r.headers.get("Content-Length", 0))
        done = 0
        with open(filename, "wb") as f:
            for piece in r.iter_content(chunk_size=chunk):
                if not piece: continue
                f.write(piece)
                done += len(piece)
                if total:
                    pct = done * 100 // total
                    print(f"\r{pct}%  ({done:,}/{total:,} bytes)", end="", flush=True)
        print()

stream=Truestream=True defers the body download. iter_contentiter_content yields it in chunks so memory stays flat regardless of file size.

For a real progress bar, use tqdmtqdm:

from tqdm import tqdm
with requests.get(url, stream=True) as r, open(filename, "wb") as f:
    bar = tqdm(total=int(r.headers["Content-Length"]), unit="B", unit_scale=True)
    for chunk in r.iter_content(8192):
        f.write(chunk); bar.update(len(chunk))
    bar.close()

from tqdm import tqdm
with requests.get(url, stream=True) as r, open(filename, "wb") as f:
    bar = tqdm(total=int(r.headers["Content-Length"]), unit="B", unit_scale=True)
    for chunk in r.iter_content(8192):
        f.write(chunk); bar.update(len(chunk))
    bar.close()

Retries with Back-off

Transient errors (5xx, network blips) should retry; permanent ones (404) should not:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
 
retry = Retry(total=5, backoff_factor=1,
              status_forcelist=[502, 503, 504], allowed_methods=["GET"])
adapter = HTTPAdapter(max_retries=retry)
session = requests.Session()
session.mount("https://", adapter)
session.mount("http://", adapter)
r = session.get(url, timeout=30)

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
 
retry = Retry(total=5, backoff_factor=1,
              status_forcelist=[502, 503, 504], allowed_methods=["GET"])
adapter = HTTPAdapter(max_retries=retry)
session = requests.Session()
session.mount("https://", adapter)
session.mount("http://", adapter)
r = session.get(url, timeout=30)

backoff_factor=1backoff_factor=1 means waits of 1, 2, 4, 8, 16 seconds between retries.

Common Mistakes

Variations to Try

1. Mirror mode

Pull the HTML, parse with BeautifulSoup, then download every <img><img>, <script><script>, <link rel=stylesheet><link rel=stylesheet> and rewrite the HTML to point at the local copies. Result: a fully self-contained index.htmlindex.html plus an assets/assets/ folder.

2. Recursive crawl

Follow internal <a><a> links up to depth N. See Basic Web Crawler for the proper queue + visited-set pattern.

3. Authentication

requests.get(url, auth=("user", "pass"))requests.get(url, auth=("user", "pass")) for HTTP Basic; or use session cookies for login forms.

4. Robots.txt compliance

Use urllib.robotparserurllib.robotparser to honor crawl directives before downloading:

from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url(urlparse(url)._replace(path="/robots.txt").geturl())
rp.read()
if not rp.can_fetch("PCH-Downloader/1.0", url):
    print("robots.txt forbids this URL"); return

from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url(urlparse(url)._replace(path="/robots.txt").geturl())
rp.read()
if not rp.can_fetch("PCH-Downloader/1.0", url):
    print("robots.txt forbids this URL"); return

5. JSON-aware mode

If Content-TypeContent-Type is application/jsonapplication/json, pretty-print with json.dumps(r.json(), indent=2)json.dumps(r.json(), indent=2).

6. PDF extraction

pip install pypdfpip install pypdf. After download, extract text and save side-by-side as .txt.txt.

7. Headless browser

Some pages are JavaScript-rendered and arrive empty via requestsrequests. Switch to Playwright to get the rendered DOM.

8. Resume support

For broken downloads, use HTTP RangeRange headers:

existing = os.path.getsize(filename) if os.path.exists(filename) else 0
r = requests.get(url, headers={"Range": f"bytes={existing}-"}, stream=True)
mode = "ab" if r.status_code == 206 else "wb"

existing = os.path.getsize(filename) if os.path.exists(filename) else 0
r = requests.get(url, headers={"Range": f"bytes={existing}-"}, stream=True)
mode = "ab" if r.status_code == 206 else "wb"

9. Batch from a file

Read URLs from urls.txturls.txt and download all (with rate limiting).

10. GUI version

Tkinter window with URL entry, save-as button, and progress bar. See Currency Exchange Rate Calculator GUI for the pattern.

Ethical & Legal Notes

• Read the ToS. Many sites forbid bulk downloading.
• Respect robots.txtrobots.txt. It is the standard signal.
• Identify yourself with a real User-AgentUser-Agent plus contact info.
• Rate-limit aggressively when crawling many pages — 1 request per second is friendly.
• Copyright applies to downloaded content; for personal study most uses are fine, redistribution is not.
• Never bypass paywalls or login walls without explicit permission.

Security Considerations

• Validate URLs — refuse file:file:, javascript:javascript:, data:data: schemes if users supply URLs.
• Sanitize filenames with os.path.basenameos.path.basename to block directory traversal.
• Scan downloaded executables before running them — requestsrequests does not check for malware.
• Pin TLS for sensitive endpoints with verify=verify= and a CA bundle.

Real-World Applications

• Personal-page archive (“save for offline reading”).
• Migration tools — copying content from a CMS to a static site.
• SEO audits — pull pages locally for parsing.
• Dataset construction for ML — but check licenses.
• One-off data extraction for research.

Educational Value

• HTTP fundamentals — GET, status codes, headers.
• Encoding awareness — content vs. character encoding.
• Streaming — when memory is finite and the file is not.
• Error taxonomy — HTTPErrorHTTPError, TimeoutTimeout, ConnectionErrorConnectionError mean different things.
• Politeness — the principle that scales when you scale up.

Next Steps

• Replace urlliburllib → requestsrequests for everything.
• Add streaming + progress bar.
• Add retries with exponential backoff.
• Build the mirror mode that downloads referenced assets.
• Compare with Basic Web Crawler for multi-page traversal.

Conclusion

You upgraded a trivial download script into a robust, polite, streaming, resumable, retrying downloader. Every web-aware program you ever write reuses these patterns — fail safely, respect the server, never load more than you need. Full source on GitHub. Explore more web projects on Python Central Hub.