AutoGineer
Generated banner for Alerting Irm
Generated logo for Alerting Irm

Alerting Irm

Grafana LabsOfficialPublished by Grafana Labs

  • Skill
  • Tool integration
  • Cloud and DevOps
  • Apache-2.0
  • Medium risk
Download ZIP

From the author

Configure Grafana Alerting, Incident Response Management (IRM), and SLOs end-to-end — provisions Grafana-managed and data-source-managed alert rules, contact points (Slack/PagerDuty/email/webhook), notification policies with hierarchical matchers, silences, mute timings, on-call schedules and escalation chains, incident-management integrations, and SLOs with multi-window burn-rate alerts. Use when configuring alerts, debugging notification routing, setting up on-call rotations, declaring or managing incidents, defining SLOs, provisioning alerting via YAML or API, picking matchers for a notification policy, building a PagerDuty/Slack webhook receiver, or troubleshooting why an alert isn't firing — even when the user says "page me on errors", "alert me when X happens", "route this to the platform team", or "set up an SLO" without naming Alerting or IRM.
— the description by Grafana Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a comprehensive guide to configuring Grafana Alerting and Incident Response Management (IRM). It covers the full lifecycle of alert management, including the provisioning of alert rules, contact points, notification policies, and on-call schedules. It also details how to define Service Level Objectives (SLOs) with multi-window burn-rate alerts to reduce noise, and provides methods for verifying routing and troubleshooting alert delivery.

What it does

  • Provisions alert rules for Grafana, Prometheus, Mimir, and Loki.
  • Configures notification routing via hierarchical matchers and contact points.
  • Sets up on-call rotations, escalation chains, and incident management integrations.
  • Defines SLOs with multi-window burn-rate alerts to prevent alert fatigue.
  • Provides API and YAML-based provisioning patterns for infrastructure-as-code workflows.
  • Includes troubleshooting steps for verifying alert routing and IRM integration.

When to use it

  • When you need to set up automated alerting and incident response workflows.
  • When configuring notification policies to route alerts to specific teams.
  • When defining SLOs and burn-rate alerts for service reliability.
  • When troubleshooting why an alert is not firing or reaching the correct destination.
  • When automating Grafana configuration via YAML or API.

Advantages

  • Enables infrastructure-as-code for alerting and IRM configurations.
  • Reduces alert noise through multi-window burn-rate SLO alerts.
  • Provides clear, repeatable patterns for complex notification routing.
  • Supports native integration between Grafana Alerting and IRM.

Usage examples

“How do I route critical alerts to PagerDuty?”
The skill provides the YAML structure for defining a PagerDuty contact point and a notification policy with a severity matcher to route critical alerts to that receiver.
“Set up an SLO for my service.”
The skill guides you through creating an SLO that automatically generates recording rules and multi-window burn-rate alerts to monitor reliability.
“Why is my alert not reaching the IRM team?”
The skill provides a debugging checklist, including verifying notification policy matchers, checking webhook URLs, and inspecting Grafana logs for HTTP errors.
“How can I provision alert rules using the API?”
The skill provides the specific API endpoints and headers, including the X-Disable-Provenance header, to provision rules while maintaining UI editability.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/grafana/alerting-irm/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip alerting-irm.zip -d .agents/skills

Documentation by Grafana Labs

Grafana Alerting & IRM

Docs: https://grafana.com/docs/grafana/latest/alerting.md

Common Workflows

Provisioning a new alert end-to-end

  1. Create contact points (where notifications go):

    curl -X POST https://grafana.example.com/api/v1/provisioning/contact-points \
  -H 'Authorization: Bearer <token>' -H 'Content-Type: application/json' \
  -d @contact-points.json

    Verify:

    curl https://grafana.example.com/api/v1/provisioning/contact-points \
  -H 'Authorization: Bearer <token>' | jq '.[].name'

  2. Add notification policies (which alerts go where) — see § Notification policies below for the matchers pattern.

  3. Write the alert rule — pick the type:

  4. Verify routing before going live:

    # Force-fire a test alert from the rule's UI, then check Alertmanager's view
curl https://grafana.example.com/api/alertmanager/grafana/api/v2/alerts \
  -H 'Authorization: Bearer <token>' | jq '.[] | {alertname: .labels.alertname, receiver: .receivers}'

    The expected receiver should appear. If the wrong receiver appears, re-check the policy's matchers.

Routing alerts to IRM / on-call

  1. In IRM, create an Integration of type "Grafana Alerting webhook" → copy the integration URL

  2. Add a webhook contact point in Grafana Alerting pointing at that URL (full YAML in references/irm.md § Routing)

  3. Add a notification policy matcher routing the right severity to the new contact point

  4. Verify: trigger a test alert; it should appear in IRM within ~30s. Full debug procedure in references/irm.md § Verifying the IRM integration.

Defining an SLO

  1. Create the SLO via UI or API → Grafana auto-generates recording rules, dashboards, and burn-rate alerts (the generated YAML is in references/slo.md)

  2. Use multi-window burn-rate alerts, not single-window — see references/slo.md § Multi-window burn-rate alerts for why single-window fires on noise

  3. Verify with the 4-step pattern in references/slo.md § Validating SLO config

Contact Points (YAML provisioning)

# provisioning/alerting/contact_points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: pagerduty-critical
    receivers:
      - uid: pd-receiver
        type: pagerduty
        settings:
          integrationKey: YOUR_PAGERDUTY_KEY
          severity: critical

  - orgId: 1
    name: slack-alerts
    receivers:
      - uid: slack-receiver
        type: slack
        settings:
          url: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
          channel: '#alerts'

For email, webhook, Teams, Telegram, OnCall, and other receiver types, see references/alerting.md § Contact point receiver types.

Notification policies

Hierarchical routing tree with label matchers:

# provisioning/alerting/notification_policies.yaml
apiVersion: 1
policies:
  - orgId: 1
    receiver: default-receiver
    group_by: ['alertname', 'cluster', 'service']
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h
    routes:
      # Critical alerts → PagerDuty
      - receiver: pagerduty-critical
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 4h

      # Platform team → Slack, but page on critical
      - receiver: slack-alerts
        matchers:
          - team = platform
        routes:
          - receiver: pagerduty-critical
            matchers:
              - severity = critical

      # Everything else → email
      - receiver: email-alerts
        matchers:
          - severity =~ "warning|info"

Silences

Suppress notifications for matching alerts without stopping evaluation:

curl -X POST https://grafana.example.com/api/alertmanager/grafana/api/v2/silences \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "matchers": [
      {"name": "alertname", "value": "HighErrorRate", "isRegex": false},
      {"name": "env", "value": "staging", "isRegex": false}
    ],
    "startsAt": "2024-01-01T00:00:00Z",
    "endsAt": "2024-01-01T02:00:00Z",
    "comment": "Maintenance window",
    "createdBy": "admin"
  }'

# Verify it was created
curl https://grafana.example.com/api/alertmanager/grafana/api/v2/silences \
  -H 'Authorization: Bearer <token>' | jq '.[] | select(.status.state == "active")'

Alert rule states

StateDescription
NormalCondition not met
PendingCondition met, waiting for for duration
FiringCondition met for full for duration
NoDataQuery returned no data
ErrorQuery/evaluation error
RecoveringWas firing, condition no longer met

Provisioning directory layout

provisioning/alerting/
├── alert_rules.yaml          # Alert and recording rules
├── contact_points.yaml       # Notification destinations
├── notification_policies.yaml  # Routing tree
├── templates.yaml            # Message templates
└── mute_timings.yaml         # Recurring mute windows

API provisioning (keeps UI editable)

Add X-Disable-Provenance: true to keep resources editable in the UI after API provisioning:

curl -X PUT https://grafana.example.com/api/v1/provisioning/policies \
  -H 'Authorization: Bearer <token>' \
  -H 'X-Disable-Provenance: true' \
  -H 'Content-Type: application/json' \
  -d @policy.json

curl -X POST https://grafana.example.com/api/v1/provisioning/alert-rules \
  -H 'Authorization: Bearer <token>' \
  -H 'X-Disable-Provenance: true' \
  -H 'Content-Type: application/json' \
  -d @rule.json

References

  • references/alerting.md — full alert rule YAML (Grafana-managed / Prometheus / Loki) + notification templates

  • references/slo.md — generated SLO recording rules + multi-window burn-rate alert pattern + validation steps

  • references/irm.md — IRM capabilities, integration sources, Alerting → IRM routing + verification + common failure modes

Files

SKILL.md

Grafana Alerting & IRM

Docs: https://grafana.com/docs/grafana/latest/alerting.md

Common Workflows

Provisioning a new alert end-to-end

  1. Create contact points (where notifications go):

    curl -X POST https://grafana.example.com/api/v1/provisioning/contact-points \
  -H 'Authorization: Bearer <token>' -H 'Content-Type: application/json' \
  -d @contact-points.json

    Verify:

    curl https://grafana.example.com/api/v1/provisioning/contact-points \
  -H 'Authorization: Bearer <token>' | jq '.[].name'

  2. Add notification policies (which alerts go where) — see § Notification policies below for the matchers pattern.

  3. Write the alert rule — pick the type:

  4. Verify routing before going live:

    # Force-fire a test alert from the rule's UI, then check Alertmanager's view
curl https://grafana.example.com/api/alertmanager/grafana/api/v2/alerts \
  -H 'Authorization: Bearer <token>' | jq '.[] | {alertname: .labels.alertname, receiver: .receivers}'

    The expected receiver should appear. If the wrong receiver appears, re-check the policy's matchers.

Routing alerts to IRM / on-call

  1. In IRM, create an Integration of type "Grafana Alerting webhook" → copy the integration URL

  2. Add a webhook contact point in Grafana Alerting pointing at that URL (full YAML in references/irm.md § Routing)

  3. Add a notification policy matcher routing the right severity to the new contact point

  4. Verify: trigger a test alert; it should appear in IRM within ~30s. Full debug procedure in references/irm.md § Verifying the IRM integration.

Defining an SLO

  1. Create the SLO via UI or API → Grafana auto-generates recording rules, dashboards, and burn-rate alerts (the generated YAML is in references/slo.md)

  2. Use multi-window burn-rate alerts, not single-window — see references/slo.md § Multi-window burn-rate alerts for why single-window fires on noise

  3. Verify with the 4-step pattern in references/slo.md § Validating SLO config

Contact Points (YAML provisioning)

# provisioning/alerting/contact_points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: pagerduty-critical
    receivers:
      - uid: pd-receiver
        type: pagerduty
        settings:
          integrationKey: YOUR_PAGERDUTY_KEY
          severity: critical

  - orgId: 1
    name: slack-alerts
    receivers:
      - uid: slack-receiver
        type: slack
        settings:
          url: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
          channel: '#alerts'

For email, webhook, Teams, Telegram, OnCall, and other receiver types, see references/alerting.md § Contact point receiver types.

Notification policies

Hierarchical routing tree with label matchers:

# provisioning/alerting/notification_policies.yaml
apiVersion: 1
policies:
  - orgId: 1
    receiver: default-receiver
    group_by: ['alertname', 'cluster', 'service']
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h
    routes:
      # Critical alerts → PagerDuty
      - receiver: pagerduty-critical
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 4h

      # Platform team → Slack, but page on critical
      - receiver: slack-alerts
        matchers:
          - team = platform
        routes:
          - receiver: pagerduty-critical
            matchers:
              - severity = critical

      # Everything else → email
      - receiver: email-alerts
        matchers:
          - severity =~ "warning|info"

Silences

Suppress notifications for matching alerts without stopping evaluation:

curl -X POST https://grafana.example.com/api/alertmanager/grafana/api/v2/silences \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "matchers": [
      {"name": "alertname", "value": "HighErrorRate", "isRegex": false},
      {"name": "env", "value": "staging", "isRegex": false}
    ],
    "startsAt": "2024-01-01T00:00:00Z",
    "endsAt": "2024-01-01T02:00:00Z",
    "comment": "Maintenance window",
    "createdBy": "admin"
  }'

# Verify it was created
curl https://grafana.example.com/api/alertmanager/grafana/api/v2/silences \
  -H 'Authorization: Bearer <token>' | jq '.[] | select(.status.state == "active")'

Alert rule states

StateDescription
NormalCondition not met
PendingCondition met, waiting for for duration
FiringCondition met for full for duration
NoDataQuery returned no data
ErrorQuery/evaluation error
RecoveringWas firing, condition no longer met

Provisioning directory layout

provisioning/alerting/
├── alert_rules.yaml          # Alert and recording rules
├── contact_points.yaml       # Notification destinations
├── notification_policies.yaml  # Routing tree
├── templates.yaml            # Message templates
└── mute_timings.yaml         # Recurring mute windows

API provisioning (keeps UI editable)

Add X-Disable-Provenance: true to keep resources editable in the UI after API provisioning:

curl -X PUT https://grafana.example.com/api/v1/provisioning/policies \
  -H 'Authorization: Bearer <token>' \
  -H 'X-Disable-Provenance: true' \
  -H 'Content-Type: application/json' \
  -d @policy.json

curl -X POST https://grafana.example.com/api/v1/provisioning/alert-rules \
  -H 'Authorization: Bearer <token>' \
  -H 'X-Disable-Provenance: true' \
  -H 'Content-Type: application/json' \
  -d @rule.json

References

  • references/alerting.md — full alert rule YAML (Grafana-managed / Prometheus / Loki) + notification templates

  • references/slo.md — generated SLO recording rules + multi-window burn-rate alert pattern + validation steps

  • references/irm.md — IRM capabilities, integration sources, Alerting → IRM routing + verification + common failure modes

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs network access

Security

What our scanner noticed· Medium risk

  • Possible data-exfiltration endpoints
  • Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
Grafana Labs
Publisher
Grafana Labs (Official)
Source
grafana/skills · skills/grafana-core/alerting-irm · commit 8fb72a5
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Grafana Labs or Grafana Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download alerting-irm.zip