Phone calls, retries and escalation for Grafana alerts.
--:--:--
your time, wherever the pager is
Someonehas towake up.
When a Grafana alert fires, WakeOps calls the engineer who owns that service. If they sleep through it, it calls again, then calls their senior.
WakeOps
checkout-api is failing in production
Error rate is 8.7% on ip-10-0-3-17. You are on call for this service.
How a call unfolds
INC-4D8K29QF checkout-api production
03:07:01
Step 1 of 6
Grafana fires
checkout-api crosses a 5% error rate in production. Grafana posts the alert to your WakeOps webhook, which answers in milliseconds.
Riya
On call
Arjun
Senior
- 03:07:01POST /webhooks/grafana 202 41ms
- 03:07:02mapped service=checkout-api env=production owner=riya
- 03:07:04call 1 to riya ringing slack + email queued
- 03:08:34call 1 no-answer call 2 to riya ringing
- 03:10:05call 2 no-answer escalate call 3 to arjun ringing
- 03:10:19acknowledged by arjun 3m 18s after the alert
- 03:07:0107:01
- 03:07:0207:02
- 03:07:0407:04
- 03:08:3408:34
- 03:10:0510:05
- 03:10:1910:19
Built for the
worst hour
of the night.
WakeOps does one job: get a firing Grafana alert to a person who can fix it, and keep a clear record of how that happened.
- Calls that wake people
A real phone call to the engineer assigned to the service, reading out the environment, host and alert name.
Voice - Retries, then a senior
Unanswered calls are tried again, then handed to the senior engineer for that service.
Escalation - Slack and email, side by side
Each channel is delivered on its own queue, so one provider going down never blocks another.
Channels - One alert, one incident
Repeat deliveries of a problem that is still active are recognised and ignored.
Dedupe - Every attempt on record
Each incident has its own page with calls, messages, acknowledgement and an audit trail.
Timeline
Your labels already say who to call.
Add one webhook as a Grafana contact point. WakeOps reads the service, environment and instance labels on every alert and matches them to the engineers you assigned.
- Per-organization secret, stored only as a hash
- Test the connection before you rely on it
- Firing and recovered events both handled
- Webhook answers fast, the call runs in the background
{"status": "firing","receiver": "wakeops","alerts": [{"labels": {"alertname": "High 5xx rate","service": "checkout-api","environment": "production","instance": "ip-10-0-3-17","severity": "critical",}}]}WakeOps reads
- Alert
- High 5xx rate
- Service
- checkout-api
- Environment
- production
- Host
- ip-10-0-3-17
- Severity
- Critical
Calling first
Riya, on call
Arjun if she misses it