Types
Slack
Deliver incident and monitor alerts to a Slack channel via an incoming webhook URL.
Receive incident, recovery, and maintenance notifications in a Slack channel using an Incoming Webhook URL.
Create a Slack Integration (Webhook Method)
- In Slack (desktop or web) open: Settings & administration → Manage apps (or visit
https://your-workspace.slack.com/apps). - Search for Incoming WebHooks and click Add to Slack (or Configure if already added).
- Click Add New Webhook to Workspace and select the destination channel (e.g.
#status-updates). - Slack generates a Webhook URL (looks like
https://hooks.slack.com/services/T000/B000/XYZ...). Copy it. - In Garmingo → Add a new Slack integration → Paste the Webhook URL.
- Save. (Optional) Trigger a test notification or temporarily toggle a non‑critical monitor.
Message Types
| Event | Trigger |
|---|---|
| Incident Created | New incident opened |
| Incident Updated | Status / message change |
| Incident Resolved | Marked resolved |
| Monitor Down | After retries fail |
| Monitor Up | First succeeding check after downtime |
| Maintenance Upcoming | Optional reminder |
| Maintenance Started / Ended | Window transitions |
Payload Format (Simplified)
Messages are sent as Slack Block / basic attachments (future enhancements may enrich formatting). Core fields:
| Field | Purpose |
|---|---|
| text / blocks | Human readable summary (first line concise) |
| link | Direct to status page / incident detail |
| severity emoji (planned) | Quick visual scan |
Best Practices
| Area | Recommendation |
|---|---|
| Channel Choice | Use a dedicated #status-updates (avoid general chatter) |
| Noise Control | Start with critical events only (Monitor Down / Incident Created / Resolved) then add more if needed |
| Naming | Prefix integration name with environment, e.g. [PROD] Slack Webhook |
| Redundancy | Pair with a second channel (e.g. SMS) for Sev‑1 paging |
| Formatting | Keep incident titles short; details live on the incident page |
| Rate Spikes | If many monitors fail at once, rely on incident aggregation to reduce noise |
Security
- Treat the Webhook URL as a secret; anyone with it can post to your channel.
- Rotate (create new webhook) if exposed; update the Garmingo integration accordingly.
- Limit channel to need‑to‑know participants for sensitive infrastructure signals.
- Avoid embedding confidential data (API keys, internal IPs) in incident titles.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| No messages | Wrong / revoked webhook URL | Recreate webhook and update integration |
| 410 / 404 response | Webhook deleted in Slack | Generate a new one |
| Duplicated alerts | Multiple Slack integrations targeting same channel | Consolidate to one |
| Too noisy | Overly broad event selection | Disable low priority events |
| Formatting odd | Slack markdown differences | Simplify body / limit special characters |
Rotating the Webhook URL
- Create a new webhook in Slack and copy the URL.
- Edit the existing Garmingo Slack integration, paste new URL, save.
- Immediately send test; if successful, delete the old webhook in Slack.
Security: Only grant channel access needed. Remove integration or rotate webhook when decommissioned.