Automating Desktop Applications

Strategy

1. open app
2. focus window
3. navigate with hotkeys
4. use image locate for buttons
5. add safe delays

Skeleton script (safe)

import time
import pyautogui
 
 
def main():
    pyautogui.alert("Move mouse to top-left anytime to STOP", title="Safety")
 
    # TODO: open your app manually first
    time.sleep(2)
 
    # Example actions:
    # pyautogui.hotkey("ctrl", "l")
    # pyautogui.write("hello")
    # pyautogui.press("enter")
 
 
if __name__ == "__main__":
    main()

Tip

If an app provides an API or CLI, use that instead.

When desktop automation is the right tool

Some programs have no API and no command line: an old accounting package, a vendor tool, a game. The only way in is the screen. pyautogui can drive them by moving the real mouse and pressing real keys. It is the last resort, because it has no idea what it is clicking. If the program has an API, a file import, a command-line option, or a COM interface (see the Outlook lesson), use that. It is faster and does not break when a window moves.

The strategy, in practice

The five steps in the list above are the whole method. Two of them decide whether the script works.

• Focus the window. Keystrokes go to whichever window is active. If a notification steals focus, your text goes into the wrong place. On Windows you can check the active window title with the pygetwindow package (pygetwindow.getActiveWindowTitle()), and stop when it is not the one you expect.
• Prefer keyboard over mouse. A hotkey such as ctrl+s works wherever the window is. A click at pixel (412, 300) breaks when the window moves, the resolution changes, or the theme differs.

A guarded skeleton

This version sets the safety options, pauses after every action, and checks the window before typing.

import time
 
import pyautogui
 
pyautogui.FAILSAFE = True     # mouse to the top-left corner aborts the script
pyautogui.PAUSE = 0.5         # half a second after every pyautogui call
 
EXPECTED_TITLE = "Untitled - Notepad"
 
 
def active_title() -> str:
    try:
        import pygetwindow
        return pygetwindow.getActiveWindowTitle() or ""
    except Exception:
        return ""
 
 
def type_into_app(text: str) -> None:
    if EXPECTED_TITLE not in active_title():
        raise RuntimeError("Wrong window is focused, stopping before typing")
    pyautogui.write(text, interval=0.05)
    pyautogui.hotkey("ctrl", "s")
 
 
def main() -> None:
    pyautogui.alert("Open Notepad now. Move the mouse to the top-left corner to STOP.", title="Safety")
    for remaining in (3, 2, 1):
        print("starting in", remaining)
        time.sleep(1)
    type_into_app("Report generated automatically")
 
 
if __name__ == "__main__":
    main()

The countdown gives you time to focus the app, and the title check refuses to type anywhere else. Raising an error is better than pressing keys in an unknown window.

Safety

A desktop script controls your whole machine. A wrong click can delete files, send a message, or approve a payment. Test on a machine or a user account you can afford to disturb, and never leave a script running unattended against real data until it has run correctly many times. Keep FAILSAFE on. Note that the failsafe needs your hands free, so do not run a long script and walk away from it.

Common mistakes

• Hard-coded coordinates. Use keyboard shortcuts or image recognition instead.
• No pauses. The app cannot keep up, so keys get lost.
• Assuming focus. Check the window title.
• Running at a different resolution or scale factor than the one you built it on.
• Locked screens. Input is not sent while the screen is locked or the session is disconnected.

Check your understanding