Python Central HubSign in

Automation and Scripting / Phase 3: Web Automation & Scraping

Wait Times - Implicit vs Explicit Waits

Avoid flaky scrapers by using Selenium explicit waits (WebDriverWait) instead of sleeps.

diagram implicit and explicit waits solve different problems mermaid
An implicit wait is a global setting that retries element lookups for up to N seconds. An explicit wait targets one condition and can wait for things that are not just presence -- clickable, visible, text changed. Mixing the two is the classic mistake, because the waits compound in ways that are hard to predict.

Why waits matter

Dynamic pages load elements later.

If you try to locate elements too early, you get:

  • NoSuchElementException
  • ElementNotInteractableException

Implicit wait

Python
from selenium import webdriver
 
 
driver = webdriver.Chrome()
driver.implicitly_wait(5)  # seconds

It applies to element lookups globally.

Python
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
 
 
driver = webdriver.Chrome()
try:
    driver.get("https://example.com")
 
    h1 = WebDriverWait(driver, 10).until(
        EC.visibility_of_element_located((By.TAG_NAME, "h1"))
    )
    print(h1.text)
finally:
    driver.quit()

Tip

Prefer explicit waits for specific elements.

The problem waits solve

When driver.get() returns, the page has loaded, but the JavaScript may still be adding elements. If you look for one too early, Selenium raises NoSuchElementException. If the element exists but is hidden or covered, you get ElementNotInteractableException. The naive fix is time.sleep(5). It is always slower than needed, and still fails on a slow day.

The three kinds of waiting

  • time.sleep(n) waits exactly n seconds. Use it only for polite delays between requests, never to wait for the page.
  • Implicit wait (driver.implicitly_wait(5)) is set once. Every find_element then retries for up to 5 seconds before failing. It only knows about presence in the DOM.
  • Explicit wait (WebDriverWait(driver, 10).until(condition)) waits for one named condition on one element. It polls every half second and returns as soon as the condition is true, or raises TimeoutException after the limit.

Use explicit waits. They say what you are waiting for, and they can wait for more than presence.

Useful conditions

ConditionUse it when
presence_of_element_locatedthe element must exist in the DOM
visibility_of_element_locatedyou need to read visible text
element_to_be_clickableyou are about to click
invisibility_of_element_locatedwaiting for a spinner to go away
text_to_be_present_in_elementwaiting for a value to change
staleness_ofwaiting for the old page to be replaced

Each takes a locator tuple such as (By.ID, "result"). Note the double brackets: one pair for the call, one for the tuple.

A reusable helper

Python
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
 
 
def click_when_ready(driver, css: str, timeout: int = 10) -> None:
    try:
        element = WebDriverWait(driver, timeout).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, css))
        )
    except TimeoutException:
        driver.save_screenshot("timeout.png")
        raise RuntimeError(f"{css} was not clickable after {timeout}s")
    element.click()
 
 
def wait_for_text(driver, css: str, expected: str, timeout: int = 10):
    return WebDriverWait(driver, timeout).until(
        EC.text_to_be_present_in_element((By.CSS_SELECTOR, css), expected)
    )

A clear error message and a screenshot on timeout turn "the script failed sometimes" into something you can debug.

Why not mix implicit and explicit

The Selenium documentation warns against setting both. The timeouts can add up in ways that are hard to predict: an explicit wait that polls for a missing element may spend the implicit wait on each poll. Pick explicit waits and leave the implicit wait at its default of zero.

Common mistakes

  • Wrapping everything in time.sleep.
  • Passing a locator without the tuple, as in EC.presence_of_element_located(By.ID, "x"). It needs (By.ID, "x").
  • Waiting for presence, then clicking. Wait for clickable.
  • A huge timeout that hides a real problem. Ten to twenty seconds is plenty.
  • Catching TimeoutException and continuing as though it succeeded.

Check your understanding

quizCheck yourself
  1. Why is WebDriverWait preferred over time.sleep(5)?

    Show answer

    A — It continues as soon as the condition is true and fails clearly if it never is — sleep always waits the full time, and still fails when the page is slower. An explicit wait polls and returns the moment the condition holds.

  2. Which is the correct way to call the condition?

    Show answer

    B — EC.presence_of_element_located((By.ID, "result")) — The condition takes a single locator, which is a tuple of the strategy and the value. Hence the double brackets.

  3. What happens if you set both an implicit wait and explicit waits?

    Show answer

    A — The waits can compound in unpredictable ways, so pick one, preferably explicit — The documentation advises against mixing them. Explicit waits are clearer because they name the condition you are waiting for.

Was this page helpful?

Let us know how we did