Scraping Dynamic JavaScript Websites

First try: inspect network calls

Many “dynamic” sites load data from JSON endpoints.

Use browser devtools Network tab and try:

• call the JSON endpoint with requests
• avoid full browser automation

Selenium approach

• open page
• wait for element
• extract HTML

from selenium import webdriver
 
 
driver = webdriver.Chrome()
try:
    driver.get("https://example.com")
    html = driver.page_source
    print(html[:500])
finally:
    driver.quit()

Risks

• more fragile
• slower
• easier to get blocked

Why some pages look empty to requests

When you fetch a page with requests, you get the HTML the server sends. Many modern sites send an almost empty page plus JavaScript. The browser runs the JavaScript, which calls another URL for the data and builds the page. So "view source" shows nothing useful, while the browser shows a full list. BeautifulSoup only parses what requests received, so it finds nothing.

You have three options, in this order of preference.

1. Call the data endpoint directly. Open devtools, go to the Network tab, filter by Fetch/XHR, and reload. You will often see a request that returns JSON. Copy its URL and headers and call it with requests. It is faster, more stable, and easier to parse than HTML.
2. Use Selenium (or Playwright) to run the JavaScript, wait for the content, then read the HTML.
3. Use an official API, if one exists. Check for this first.

Calling the JSON endpoint

import requests
 
url = "https://httpbin.org/json"          # stands in for an endpoint found in devtools
r = requests.get(url, headers={"User-Agent": "Python Central HubBot/1.0"}, timeout=10)
r.raise_for_status()
 
data = r.json()
for slide in data["slideshow"]["slides"]:
    print(slide["title"])

The result is already structured, so there is no HTML parsing and no selector to break. Some endpoints need a parameter such as ?page=2, or a header copied from the browser. Look at what the browser sends, and send the minimum you need.

The Selenium approach done properly

The example on this page reads page_source immediately after driver.get. On a JavaScript-heavy page that can be empty, because the content has not loaded yet. Wait for the element you need first, then read:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
 
driver = webdriver.Chrome()
try:
    driver.get("https://example.com")
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "h1"))
    )
    html = driver.page_source          # now safe to read
    print(len(html))
finally:
    driver.quit()

You can then hand html to BeautifulSoup, so the parsing code stays the same as for static pages.

Infinite scroll

Some pages load more content as you scroll. With Selenium you can run driver.execute_script("window.scrollTo(0, document.body.scrollHeight)"), wait, and repeat until the height stops growing. Often the same content is available from a paged JSON endpoint, which is again the better route.

Risks and limits

Browser automation is slower, uses more memory, and breaks when the page changes. Sites also detect and block it more often. The ethical rules are the same as for static pages: check robots.txt and the terms, rate-limit, and do not go around a login or CAPTCHA that you should not be using.

Check your understanding