Screen Recognition (Locating Images on Screen)

Concept

You take a small screenshot of a button (template image), then search for it on screen.

Example

import pyautogui
 
# button_pos = pyautogui.locateCenterOnScreen("button.png", confidence=0.9)
# if button_pos:
#     pyautogui.click(button_pos)

Notes

• confidence requires OpenCV support
• matching can fail due to scaling/themes

Why find things by image

Fixed coordinates break as soon as a window moves. Image recognition is more flexible: you save a small screenshot of the button (the template), and pyautogui searches the current screen for a region that looks the same. It returns where it found it, and you click there. The button can then be anywhere.

This works by comparing pixels. It is exact, not intelligent, so the template must look nearly the same as the screen.

The locate functions

• locateOnScreen("button.png") returns a Box(left, top, width, height), or raises ImageNotFoundException (in recent versions) or returns None (in older versions) when nothing matches.
• locateCenterOnScreen("button.png") returns the centre point, which is what you want to click.
• locateAllOnScreen yields every match, such as all checkboxes.
• confidence=0.9 allows a 90 percent match instead of an exact one. It needs the opencv-python package. Without it the confidence argument raises an error.
• grayscale=True speeds up matching and ignores colour differences.
• region=(left, top, width, height) limits the search to part of the screen. It is much faster and avoids false matches elsewhere.

Wait for it, then click

Buttons appear after a delay, so search repeatedly until a timeout, and stop when it never shows up.

import time
 
import pyautogui
 
pyautogui.FAILSAFE = True
 
 
def find_on_screen(image: str, timeout: float = 10.0, confidence: float = 0.9):
    deadline = time.time() + timeout
    while time.time() < deadline:
        try:
            point = pyautogui.locateCenterOnScreen(image, confidence=confidence, grayscale=True)
        except pyautogui.ImageNotFoundException:
            point = None
        if point is not None:
            return point
        time.sleep(0.5)
    return None
 
 
def click_image(image: str) -> None:
    point = find_on_screen(image)
    if point is None:
        pyautogui.screenshot("not_found.png")           # keep evidence for debugging
        raise RuntimeError(f"{image} not found on screen, not clicking")
    pyautogui.click(point)
 
 
# click_image("button.png")

When the image is not found, the script stops and saves a screenshot instead of clicking somewhere else. That is the rule from the intro: never click blind. In older pyautogui versions locateCenterOnScreen returns None instead of raising, and this function handles both.

Making good template images

• Crop tightly, but include some distinctive detail, not just a flat colour.
• Capture on the same screen scale, theme, and resolution that the script will run on. A 125 percent display scale changes the pixel size of everything.
• Avoid text that changes, such as a clock or a filename.
• Save as PNG. JPEG compression changes pixels and breaks matching.
• Keep several images if the button has states (hover, disabled).

Common mistakes

• Using confidence without OpenCV.
• Templates made at a different scale or theme than the run.
• Searching the full screen when a region would do.
• Clicking the top-left of the box instead of the centre.
• Not handling not-found. The script then continues with a missing target.

Check your understanding