Python Networking — requests, urllib & sockets

Networking in Python spans two levels: high-level HTTP (talking to web APIs and sites) and low-level sockets (raw TCP/UDP connections). Most apps live at the HTTP level.

Tool	Level	Notes
requests	High (HTTP)	Third-party, the de-facto standard. pip install requests.
urllib.request	High (HTTP)	Built in; more verbose, no install needed.
urllib.parse	Helper	Build and parse URLs (pure, no network).
socket	Low (TCP/UDP)	Raw connections; build your own protocols.

requests — the friendly HTTP client

requests makes HTTP calls readable. Install it first (pip install requests).

requests_get.py

import requests
 
resp = requests.get("https://api.example.com/users", params={"page": 2})
print(resp.status_code)     # 200
print(resp.headers["Content-Type"])
data = resp.json()          # parse a JSON body into Python
print(data)

requests_post.py

import requests
 
# Send JSON in the body
resp = requests.post(
    "https://api.example.com/users",
    json={"name": "Ada", "role": "admin"},
    headers={"Authorization": "Bearer TOKEN"},
    timeout=10,
)
resp.raise_for_status()     # raise if status is 4xx/5xx
print(resp.json())

requests piece	Purpose
requests.get/post/put/delete	The HTTP verbs.
params={...}	Query-string parameters.
json={...}	Send a JSON body.
headers={...}	Custom request headers.
resp.status_code	The HTTP status (200, 404, …).
resp.json() / resp.text	Parsed JSON / raw text.
resp.raise_for_status()	Turn error statuses into exceptions.
timeout=	Fail fast instead of hanging forever.

urllib — the standard-library option

When you can’t add dependencies, urllib.request does the same job with more ceremony.

urllib_get.py

import urllib.request
import json
 
req = urllib.request.Request(
    "https://api.example.com/data",
    headers={"User-Agent": "my-app"},
)
with urllib.request.urlopen(req, timeout=10) as resp:
    body = resp.read().decode("utf-8")
    data = json.loads(body)
print(data)

urllib.parse — build and parse URLs

This part of urllib is pure (no network) and extremely useful for assembling query strings and dissecting URLs.

urllib_parse.py

from urllib.parse import urlencode, urlparse, parse_qs
 
# Build a query string from a dict
qs = urlencode({"q": "python", "page": 2})
print(qs)                     # q=python&page=2
 
# Dissect a URL
parts = urlparse("https://example.com/search?q=python&page=2")
print(parts.scheme)           # https
print(parts.netloc)           # example.com
print(parts.path)             # /search
print(parse_qs(parts.query))  # {'q': ['python'], 'page': ['2']}

socket — low-level TCP

For custom protocols or learning how the network really works, use raw sockets. Here’s a minimal echo server and client.

tcp_server.py

import socket
 
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("127.0.0.1", 9000))
server.listen()
print("listening on 9000")
 
conn, addr = server.accept()      # blocks until a client connects
with conn:
    data = conn.recv(1024)        # read up to 1024 bytes
    conn.sendall(data)            # echo it back

tcp_client.py

import socket
 
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect(("127.0.0.1", 9000))
    s.sendall(b"hello")
    reply = s.recv(1024)
print(reply)                      # b'hello'

socket call	Purpose
socket(AF_INET, SOCK_STREAM)	Create a TCP socket.
bind((host, port))	Attach a server to an address.
listen() / accept()	Wait for and accept a connection.
connect((host, port))	Client connects to a server.
sendall(bytes) / recv(n)	Send / receive raw bytes.

Sockets speak bytes, not strings — encode with .encode() and decode with .decode().

Choosing a level

decision.txt

Talking to a web API or website?   -> requests (or urllib if no installs)
Just building/parsing a URL?       -> urllib.parse
Custom protocol / raw TCP/UDP?     -> socket

Common pitfalls

• No timeout — network calls can hang forever; always pass timeout=.
• Ignoring status codes — check resp.status_code or call raise_for_status().
• Sending strings over sockets — sockets need bytes; encode/decode explicitly.
• requests isn’t built in — it needs pip install requests; urllib does not.

Practice Exercises

These exercises use urllib.parse, which runs without any network access.

Exercise 1 – Build a query string

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Extract the host from a URL

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Parse query parameters

pch.exerciseTag pch.exerciseTitle

Three levels, and how much they do for you

The same GET can be written at three levels. Each one below hands you more raw material and more responsibility:

diagram Diagram mermaid

  flowchart TD
  R["requests
sessions, redirects, decoding, retries"] --> U["urllib.request
stdlib, no install, manual everything"]
  U --> S["socket
you write the protocol yourself"]
  R -.->|"choose when"| RC["you are talking to an HTTP API"]
  U -.->|"choose when"| UC["a dependency is unacceptable"]
  S -.->|"choose when"| SC["the protocol is not HTTP"]

Reach for requests unless you have a reason not to. Drop to urllib.request when you cannot add a dependency. Drop to socket only when you are speaking a protocol that has no library — at that level you are responsible for framing, encoding, retries and timeouts yourself.

urllib.parse is the part you should use every time

Even with requests, URL manipulation belongs here rather than in an f-string:

parse.py

from urllib.parse import urlparse, parse_qs, urljoin, urlencode
 
u = "https://user:pw@api.example.com:8443/v1/items?q=a+b&tag=x&tag=y#frag"
p = urlparse(u)
 
p.scheme      # 'https'
p.hostname    # 'api.example.com'
p.port        # 8443
p.path        # '/v1/items'
p.query       # 'q=a+b&tag=x&tag=y'
p.fragment    # 'frag'
 
parse_qs(p.query)     # {'q': ['a b'], 'tag': ['x', 'y']}   <- values are LISTS

parse_qs returns lists because a query string may repeat a key — ?tag=x&tag=y is legal and meaningful. Code that assumes a single value silently drops data.

urlparse splits, it does not validate

urlparse("not a url") returns happily with scheme='' and path='not a url'. It tells you nothing about whether the URL is usable.

urljoin is not string concatenation

One trailing slash changes the answer:

base	relative	result
https://x.com/a/b/	c	https://x.com/a/b/c
https://x.com/a/b	c	https://x.com/a/c
https://x.com/a/b/	/c	https://x.com/c
https://x.com/a/b/	../c	https://x.com/a/c
https://x.com/a/	https://y.com/z	https://y.com/z

Without the trailing slash, b is treated as a file and replaced. A leading slash means “from the root”. And a relative reference that is actually absolute replaces the host entirely — which is worth remembering when the relative part comes from user input or from a redirect you did not write.

See it move

TCP is a stream of bytes. It has no idea where your messages begin or end — that is your job. Send some messages and watch what the receiver actually gets.

sketch TCP has no message boundaries p5.js Pause Restart

Four send() calls arrived as 28 recv() chunks. One send is not one recv, so any protocol must carry its own framing.

Measured over a real loopback socket: three 5-byte sends followed by one 100,000-byte send arrived as 28 separate recv() chunks — the first three of 5 bytes, then repeated 4,096-byte reads. Every byte arrived, in order, and recv(4096) never returned more than it was asked for.

framing.py

def recv_exactly(sock, n):
    # recv() returns UP TO n bytes. Loop until you have all of them.
    buf = bytearray()
    while len(buf) < n:
        chunk = sock.recv(n - len(buf))
        if not chunk:                       # peer closed early
            raise ConnectionError(f"wanted {n} bytes, got {len(buf)}")
        buf += chunk
    return bytes(buf)

The bug this prevents

sock.recv(1024) reads up to 1024 bytes. Treating one recv as one message works perfectly on localhost with small payloads and fails the moment the message grows or the network is real. Every TCP protocol needs framing — a length prefix, a delimiter, or fixed-size records.

Check yourself

pch.quizTag pch.quizDefaultTitle

1. Three 5-byte send() calls and one 100,000-byte send() arrived as 28 recv() chunks. What does that show?

   A bytes were lost and had to be retransmitted B TCP is a byte stream with no message boundaries, so one send is not one recv C the socket was misconfigured D recv always returns exactly 4096 bytes
   pch.quizShowAnswer

   B — TCP is a byte stream with no message boundaries, so one send is not one recv — Every byte arrived in order, but the chunking is decided by the network stack. recv(n) returns up to n bytes, so a protocol must carry its own framing.

2. What is urljoin('https://x.com/a/b', 'c')?

   A https://x.com/a/b/c B https://x.com/a/c C https://x.com/c D https://x.com/a/b/c/
   pch.quizShowAnswer

   B — https://x.com/a/c — Without a trailing slash, 'b' is treated as a file name and replaced. With the slash, urljoin('https://x.com/a/b/', 'c') gives .../a/b/c. One character changes the target.

3. parse_qs('q=a+b&tag=x&tag=y') returns values as lists. Why?

   A to keep the return type consistent with urlencode B because a query string may legally repeat a key, and both values matter C because all query values are arrays in HTTP D to preserve the original ordering
   pch.quizShowAnswer

   B — because a query string may legally repeat a key, and both values matter — ?tag=x&tag=y is valid and carries two values. Returning {'tag': ['x','y']} keeps both; code that assumes a single value silently discards data.

4. urljoin('https://x.com/a/', 'https://y.com/z') returns https://y.com/z. Why does that matter?

   A it does not, since the result is obviously correct B an absolute reference replaces the host entirely, so a value from user input or a redirect can send the request somewhere else C it means urljoin ignores its base argument D it indicates the base URL was malformed
   pch.quizShowAnswer

   B — an absolute reference replaces the host entirely, so a value from user input or a redirect can send the request somewhere else — urljoin follows the RFC: an absolute reference wins. If the relative part comes from outside your program, the resulting request may target a host you did not intend.

Summary

• For HTTP, reach for requests (clean API) or urllib.request (built in, verbose).
• Always set a timeout and check the status code.
• urllib.parse builds and dissects URLs with urlencode, urlparse, and parse_qs.
• socket gives raw TCP/UDP for custom protocols — remember it speaks bytes.
• Pick the level that matches the task: APIs → requests, URLs → urllib.parse, raw connections → socket.