Slack Webhooks for System Alerts

Incoming webhooks

Create an incoming webhook URL in your Slack workspace.

Store it as an environment variable:

• SLACK_WEBHOOK_URL

Example

import os
import requests
 
 
def slack_alert(text: str):
    url = os.environ["SLACK_WEBHOOK_URL"]
    r = requests.post(url, json={"text": text}, timeout=10)
    r.raise_for_status()
 
 
# slack_alert("Job succeeded")

What an incoming webhook is

An incoming webhook is a URL that Slack gives you for one channel. When your script sends a POST request with a JSON body to that URL, Slack posts the message in the channel. There is no bot to run, and no login. You create one in your Slack workspace by making an app (at api.slack.com/apps), turning on Incoming Webhooks, and choosing a channel. The URL contains a secret, so anyone who has it can post as your app. Store it in the SLACK_WEBHOOK_URL environment variable, never in the code or in git.

The simplest body is {"text": "Job succeeded"}. Slack replies with the text ok and status 200 when it accepted the message. A wrong or revoked URL gives a 4xx status, which raise_for_status() turns into an exception.

Formatting and structure

Slack uses its own markup called mrkdwn: *bold*, _italic_, and <https://example.com|link text>. You can use \n for new lines. For richer layouts, send blocks, a list of structured sections. An alert with a header, a detail line, and a link looks like this:

import os
import socket
from datetime import datetime
 
import requests
 
 
def build_payload(job: str, ok: bool, detail: str) -> dict:
    icon = ":white_check_mark:" if ok else ":rotating_light:"
    status = "succeeded" if ok else "FAILED"
    return {
        "text": f"{job} {status}",                         # shown in notifications
        "blocks": [
            {"type": "header", "text": {"type": "plain_text", "text": f"{job} {status}"}},
            {"type": "section", "text": {"type": "mrkdwn", "text": f"{icon} {detail}"}},
            {
                "type": "context",
                "elements": [
                    {"type": "mrkdwn", "text": f"host `{socket.gethostname()}` at {datetime.now():%H:%M:%S}"}
                ],
            },
        ],
    }
 
 
def slack_alert(job: str, ok: bool, detail: str) -> None:
    url = os.environ["SLACK_WEBHOOK_URL"]
    r = requests.post(url, json=build_payload(job, ok, detail), timeout=10)
    r.raise_for_status()
 
 
print(build_payload("Nightly backup", False, "disk is 97% full"))
# slack_alert("Nightly backup", False, "disk is 97% full")

build_payload only builds a dictionary, so you can print it and check it without sending anything, which is the way to test. Always include a plain text value even when you use blocks, because it is what shows in push notifications and in clients that cannot render blocks.

Using it well for system alerts

An alert channel is only useful if people keep reading it.

• Alert on failure, and send one short success summary at most. A message on every successful run trains people to mute the channel.
• Include what, where, and when: the job name, the machine, the time, and the error text.
• Avoid floods. If a check runs every minute and the disk stays full, do not send 1,440 messages a day. Remember the last alert and send again after an hour.
• Never include secrets or customer data in a message.
• Handle 429. Slack limits webhooks to about one message a second. Wait for the Retry-After header value.

Common mistakes

• Committing the URL. Revoke it in the Slack app settings if it leaks.
• No timeout, so a network problem freezes the job you wanted to report on.
• Alerting inside the failing thing. If the network is down, the alert cannot send either. Log to a file too.

Check your understanding