1. Overview — Grafana Alerting Architecture

Grafana Unified Alerting (since Grafana 8+) provides a single, consistent alerting experience across all datasources.

Key Concepts

Alert Rule — query + condition that defines when to alert
Alert Instance — one alert per label combination
Alert States: Normal → Pending → Firing → Error
Notification Policy — routes alerts to contact points
Contact Point — destination (Slack, Email, PagerDuty...)
Silences — temporarily suppress notifications
Mute Timings — recurring suppression windows

Alert Flow

Alert Rule (query + condition)
  → Evaluation Group (interval, e.g. every 1m)
    → Pending (for duration, e.g. 5m)
      → Firing
        → Notification Policy
          → Contact Point (Email, Slack, Teams,
             PagerDuty, Webhook)
            → Template (optional custom message)

2. Elasticsearch Datasource Setup in Grafana

# Grafana datasource provisioning (YAML)
apiVersion: 1
datasources:
  - name: Elasticsearch-Logs
    type: elasticsearch
    access: proxy
    url: http://elasticsearch:9200
    database: "app-logs-*"
    jsonData:
      timeField: "@timestamp"
      esVersion: "8.0+"
      maxConcurrentShardRequests: 5
      logMessageField: message
      logLevelField: level
    isDefault: false

Index Pattern Best Practices

app-logs-* — application logs
filebeat-* — Filebeat indices
metricbeat-* — system metrics

Configuration Keys

URL: Elasticsearch endpoint
Time field: always @timestamp
Version: match your ES version
Max concurrent shard requests: tune for perf

3. Alert Rule — Error Count in Time Window

The most common use case: "Alert if more than X errors in Y minutes"

# Grafana Alert Rule Configuration:
Query A (Elasticsearch):
  Index: app-logs-*
  Query: level:ERROR OR level:error OR log_level:ERROR
  Metric: Count
  Group by: Terms → service.name (optional)
  Time range: now-10m to now

Expression B (Reduce):
  Function: Last
  Input: A
  Mode: Strict

Expression C (Threshold):
  Input: B
  Condition: IS ABOVE 50

# Evaluation:
  Evaluate every: 1m
  For: 5m (pending duration before firing)

# Labels:
  severity: critical
  team: backend
  environment: production

Elasticsearch Query DSL (generated by Grafana)

{
  "query": {
    "bool": {
      "filter": [
        { "range": { "@timestamp": { "gte": "now-10m", "lte": "now" } } },
        { "query_string": { "query": "level:ERROR" } }
      ]
    }
  },
  "aggs": {
    "count": { "value_count": { "field": "_id" } }
  },
  "size": 0
}

4. Common Alert Patterns with Elasticsearch

Pattern 1: Error spike detection

Query: level:ERROR

Metric: Count

Condition: > 100 in 10 minutes

Use case: Application throwing unexpected errors

Pattern 2: No logs received (dead service)

Query: * filtered by service.name:my-service

Metric: Count

Condition: IS BELOW 1 in last 5 minutes

Use case: Service stopped logging = probably crashed

Pattern 3: Slow response times from logs

Query: *

Metric: Average on field response_time_ms

Condition: > 2000 (2 seconds)

Use case: Performance degradation

Pattern 4: Specific error message detection

Query: message:"OutOfMemoryError" OR message:"StackOverflowError"

Metric: Count

Condition: > 0 | For: 0s (fire immediately)

Use case: Critical JVM errors

Pattern 5: Error rate percentage

Query A: level:ERROR → Count

Query B: * → Count (total)

Expression C: Math → $A / $B * 100

Condition: > 5 (more than 5% errors)

Use case: Error ratio monitoring

Pattern 6: Log volume anomaly

Query: *

Metric: Count (current 10m vs previous 10m)

Condition: > 300% increase

Use case: Log flood detection (possible attack or loop)

Pattern 7: 4xx/5xx HTTP status codes

Query: response_code:[400 TO 599] or status_code:>=400

Metric: Count, grouped by status_code

Condition: > 50 in 5 minutes

Use case: API error monitoring

Pattern 8: Disk space from logs (Metricbeat)

Query: metricbeat-* index

Metric: Last value of system.filesystem.used.pct

Condition: > 0.85 (85%)

Use case: Disk full prevention

5. Notification Policies & Routing

# Notification policy tree
Root Policy:
  Contact Point: default-email
  Group by: [alertname, severity]
  Group wait: 30s
  Group interval: 5m
  Repeat interval: 4h

  Child Policies:
    - Matcher: severity=critical
      Contact Point: pagerduty-oncall
      Group wait: 10s
      Repeat interval: 1h

    - Matcher: team=frontend
      Contact Point: slack-frontend

    - Matcher: environment=staging
      Contact Point: slack-staging
      Mute timings: weekends

Contact Point Examples

# Slack
- name: slack-alerts
  type: slack
  settings:
    url: https://hooks.slack.com/services/XXX/YYY/ZZZ
    channel: "#alerts"
    title: |
      {{ .Status | toUpper }} {{ .CommonLabels.alertname }}
    text: |
      {{ range .Alerts }}
      *Service:* {{ .Labels.service_name }}
      *Severity:* {{ .Labels.severity }}
      *Message:* {{ .Annotations.summary }}
      *Value:* {{ .Values.B }}
      {{ end }}

# Microsoft Teams (Webhook)
- name: teams-alerts
  type: teams
  settings:
    url: https://outlook.office.com/webhook/XXX

# Email
- name: email-ops
  type: email
  settings:
    addresses: [email protected];[email protected]
    singleEmail: true

# PagerDuty
- name: pagerduty
  type: pagerduty
  settings:
    integrationKey: "xxxxx"
    severity: "{{ .CommonLabels.severity }}"

# Generic Webhook
- name: custom-webhook
  type: webhook
  settings:
    url: https://api.company.com/alerts
    httpMethod: POST
    maxAlerts: 10

6. Alert Templates (Go Templating)

{{ define "custom.alert.message" }}
🚨 *{{ .Status | toUpper }}* — {{ .CommonLabels.alertname }}

{{ range .Alerts }}
• *Service:* {{ .Labels.service_name | default "unknown" }}
• *Environment:* {{ .Labels.environment }}
• *Severity:* {{ .Labels.severity }}
• *Summary:* {{ .Annotations.summary }}
• *Description:* {{ .Annotations.description }}
• *Value:* {{ index .Values "B" }}
• *Started:* {{ .StartsAt.Format "2006-01-02 15:04:05 MST" }}
{{ if .EndsAt }}• *Resolved:* {{ .EndsAt.Format "2006-01-02 15:04:05 MST" }}{{ end }}
• *Dashboard:* {{ .DashboardURL }}
• *Panel:* {{ .PanelURL }}
• *Silence:* {{ .SilenceURL }}
{{ end }}
{{ end }}

{{ define "custom.alert.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}

7. Silences & Mute Timings

# Mute Timing — suppress alerts during maintenance
- name: maintenance-window
  time_intervals:
    - times:
        - start_time: "02:00"
          end_time: "04:00"
      weekdays: ["sunday"]

- name: weekends
  time_intervals:
    - weekdays: ["saturday", "sunday"]

- name: business-hours-only
  time_intervals:
    - times:
        - start_time: "08:00"
          end_time: "18:00"
      weekdays: ["monday:friday"]

Create Silence via API

# Create silence via Grafana API
curl -X POST http://grafana:3000/api/alertmanager/grafana/api/v2/silences \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [
      { "name": "alertname", "value": "HighErrorRate", "isRegex": false },
      { "name": "environment", "value": "staging", "isRegex": false }
    ],
    "startsAt": "2026-03-12T06:00:00Z",
    "endsAt": "2026-03-12T08:00:00Z",
    "createdBy": "markus",
    "comment": "Deploying v2.1.0"
  }'

8. Grafana Alert Rule Provisioning (IaC)

File Provisioning (YAML)

# /etc/grafana/provisioning/alerting/rules.yaml
apiVersion: 1
groups:
  - orgId: 1
    name: log-alerts
    folder: Infrastructure
    interval: 1m
    rules:
      - uid: error-count-alert
        title: "High Error Count"
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600  # 10 minutes
              to: 0
            datasourceUid: elasticsearch-logs
            model:
              query: "level:ERROR"
              metrics:
                - type: count
                  id: "1"
              bucketAggs:
                - type: date_histogram
                  field: "@timestamp"
                  id: "2"
                  settings:
                    interval: "1m"
          - refId: B
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: "-100"
            model:
              type: reduce
              expression: A
              reducer: last
          - refId: C
            datasourceUid: "-100"
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator:
                    type: gt
                    params: [50]
        for: 5m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "Error count exceeded 50 in 10 minutes"
          description: "Current value: {{ $values.B }}"
          runbook_url: "https://wiki.company.com/runbooks/high-error-rate"

Terraform Provisioning

resource "grafana_rule_group" "log_alerts" {
  org_id           = 1
  name             = "log-alerts"
  folder_uid       = grafana_folder.infra.uid
  interval_seconds = 60

  rule {
    name      = "High Error Count"
    condition = "C"
    for       = "5m"

    labels = {
      severity = "warning"
      team     = "backend"
    }

    annotations = {
      summary     = "Error count exceeded threshold"
      description = "Current value: {{ $values.B }}"
    }

    data {
      ref_id         = "A"
      datasource_uid = grafana_data_source.elasticsearch.uid
      relative_time_range {
        from = 600
        to   = 0
      }
      model = jsonencode({
        query   = "level:ERROR"
        metrics = [{ type = "count", id = "1" }]
      })
    }
    # ... reduce + threshold expressions
  }
}

9. Best Practices & Anti-Patterns

✅ Do

Set meaningful for duration (5m+ for non-critical)
Add runbook_url annotation to every alert
Group alerts by alertname + severity
Use labels for routing: severity, team, environment
Test alerts in staging before production
Set repeat interval ≥ 1h for non-critical
Use mute timings for maintenance windows
Include current value: {{ $values.B }}

❌ Don't

Alert on every single error (set sensible thresholds)
Use for: 0s unless truly critical
Create alerts without contact points
Alert on metrics you won't act on (alert fatigue)
Mix severity levels in same channel
Forget to set group_interval (floods channels)

Severity Levels Guideline

Level	Response Time	Example	Channel
critical	< 15 min	Service down, data loss, security	PagerDuty + Slack
warning	< 1 hour	Error rate high, disk 85%	Slack
info	Next business day	Performance degradation, log anomaly	Email

10. Debugging Grafana Alerts

# Check alert rule state
curl -s http://grafana:3000/api/alertmanager/grafana/api/v2/alerts \
  -H "Authorization: Bearer $TOKEN" | jq '.'

# Check alert rule evaluation
curl -s http://grafana:3000/api/v1/provisioning/alert-rules \
  -H "Authorization: Bearer $TOKEN" | jq '.[].title'

# Test notification
curl -X POST http://grafana:3000/api/alertmanager/grafana/api/v2/alerts \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": { "alertname": "TestAlert", "severity": "info" },
    "annotations": { "summary": "This is a test alert" },
    "startsAt": "2026-03-12T06:00:00Z"
  }]'

# Grafana server logs for alert evaluation
docker logs grafana 2>&1 | grep -i "alert\|eval"

Common Issues

DatasourceError

Check ES connectivity, index exists

NoData

Index pattern wrong, time field mismatch, no data in range

Alert stuck in "Pending"

`for` duration not elapsed yet

Alert never fires

Threshold too high, wrong query, reduce function