Phone calls, retries and escalation for Grafana alerts.

Someonehas towake up.

When a Grafana alert fires, WakeOps calls the engineer who owns that service. If they sleep through it, it calls again, then calls their senior.

Incoming callCritical
W

WakeOps

checkout-api is failing in production

Error rate is 8.7% on ip-10-0-3-17. You are on call for this service.

How a call unfolds

INC-4D8K29QF

03:07:01

Step 1 of 6

Grafana fires

checkout-api crosses a 5% error rate in production. Grafana posts the alert to your WakeOps webhook, which answers in milliseconds.

RAsleep

Riya

On call

AAsleep

Arjun

Senior

  1. 07:01
  2. 07:02
  3. 07:04
  4. 08:34
  5. 10:05
  6. 10:19

Built for the
worst hour
of the night.

WakeOps does one job: get a firing Grafana alert to a person who can fix it, and keep a clear record of how that happened.

  • Calls that wake people

    A real phone call to the engineer assigned to the service, reading out the environment, host and alert name.

    Voice
  • Retries, then a senior

    Unanswered calls are tried again, then handed to the senior engineer for that service.

    Escalation
  • Slack and email, side by side

    Each channel is delivered on its own queue, so one provider going down never blocks another.

    Channels
  • One alert, one incident

    Repeat deliveries of a problem that is still active are recognised and ignored.

    Dedupe
  • Every attempt on record

    Each incident has its own page with calls, messages, acknowledgement and an audit trail.

    Timeline

Your labels already say who to call.

Add one webhook as a Grafana contact point. WakeOps reads the service, environment and instance labels on every alert and matches them to the engineers you assigned.

  • Per-organization secret, stored only as a hash
  • Test the connection before you rely on it
  • Firing and recovered events both handled
  • Webhook answers fast, the call runs in the background
POST /webhooks/grafana202 Accepted
{"status": "firing","receiver": "wakeops","alerts": [{"labels": {"alertname": "High 5xx rate","service": "checkout-api","environment": "production","instance": "ip-10-0-3-17","severity": "critical",}}]}

WakeOps reads

Alert
High 5xx rate
Service
checkout-api
Environment
production
Host
ip-10-0-3-17
Severity
Critical

Calling first

Riya, on call

Arjun if she misses it

Set it up before tonight's alert, not after.

  1. 1Create your organization
  2. 2Assign engineers to each service
  3. 3Add the Grafana webhook
  4. 4Send a test alert
Get started