Prometheus / Alertmanager

Receive alerts from Prometheus Alertmanager and create incidents automatically.

Endpoint

http

POST /api/integrations/prometheus?integrationId=YOUR_INTEGRATION_ID&integrationKey=YOUR_INTEGRATION_KEY

Setup

Step 1: Create Integration in OpsKnight

Go to Services and select your service
Click Integrations tab
Click Add Integration
Select Prometheus
Copy the Integration ID

Step 2: Configure Alertmanager

Add OpsKnight as a receiver in your alertmanager.yml:

yaml

receivers:
  - name: 'opsknight'
    webhook_configs:
      - url: 'https://YOUR_OPSKNIGHT_URL/api/integrations/prometheus?integrationId=YOUR_INTEGRATION_ID&integrationKey=YOUR_INTEGRATION_KEY'
        send_resolved: true

route:
  receiver: 'opsknight'
  routes:
    - match:
        severity: critical
      receiver: 'opsknight'

Replace YOUR_OPSKNIGHT_URL with your OpsKnight instance URL and YOUR_INTEGRATION_ID with the ID from Step 1.

Payload Format

OpsKnight accepts the standard Alertmanager webhook payload:

json

{
  "version": "4",
  "groupKey": "{}:{alertname=\"HighMemoryUsage\"}",
  "status": "firing",
  "receiver": "opsknight",
  "groupLabels": {
    "alertname": "HighMemoryUsage"
  },
  "commonLabels": {
    "alertname": "HighMemoryUsage",
    "severity": "critical"
  },
  "commonAnnotations": {
    "summary": "Memory usage is above 90%"
  },
  "externalURL": "http://alertmanager:9093",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "HighMemoryUsage",
        "severity": "critical",
        "instance": "web-01:9090"
      },
      "annotations": {
        "summary": "Memory usage is above 90%",
        "description": "Memory usage on web-01 is 95%"
      },
      "startsAt": "2024-01-15T10:00:00Z",
      "generatorURL": "http://prometheus:9090/graph",
      "fingerprint": "abc123def456"
    }
  ]
}

Event Mapping

Alertmanager Status	OpsKnight Action
`firing`	Trigger incident
`resolved`	Resolve incident

The integration checks both status at the payload level and individual alert status. If an alert has endsAt set, it's treated as resolved.

Severity Mapping

OpsKnight reads the severity label from alerts:

Prometheus Severity Label	OpsKnight Severity
`critical`, `page`	critical
`error`	error
`warning`	warning
(default)	warning

Setting Severity in Alert Rules

yaml

groups:
  - name: example
    rules:
      - alert: HighCPUUsage
        expr: cpu_usage > 90
        labels:
          severity: critical
        annotations:
          summary: 'CPU usage is above 90%'

Incident Title

The incident title is extracted in this order:

annotations.summary
annotations.description
labels.alertname
Fallback: "Prometheus Alert"

Deduplication

Dedup keys are generated as follows:

Fingerprint (preferred): Uses Alertmanager's fingerprint field
Label hash: If no fingerprint, creates SHA-256 hash from sorted labels

An alert with no fingerprint or labels falls back to a normalized alert name or summary. Trigger and resolved payloads must produce the same key; verify this with a real Alertmanager lifecycle test.

The v1.4 Prometheus route authenticates with the integration ID and integration key. It does not verify the integration's optional signature secret or an X-Signature header. Keep the generated URL/key secret and place a trusted signing/authentication gateway in front of the route if your policy requires another sender-verification layer.

Testing

Using Alertmanager API

Manually fire a test alert:

bash

curl -X POST http://alertmanager:9093/api/v2/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "instance": "test-01"
    },
    "annotations": {
      "summary": "Test alert from curl"
    }
  }]'

Using cURL Directly

Send a test payload directly to OpsKnight:

bash

curl -X POST "https://YOUR_OPSKNIGHT_URL/api/integrations/prometheus?integrationId=YOUR_ID&integrationKey=YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "version": "4",
    "groupKey": "test",
    "status": "firing",
    "receiver": "test",
    "groupLabels": {},
    "commonLabels": {"alertname": "TestAlert", "severity": "warning"},
    "commonAnnotations": {"summary": "Test alert"},
    "externalURL": "http://test",
    "alerts": [{
      "status": "firing",
      "labels": {"alertname": "TestAlert", "severity": "warning"},
      "annotations": {"summary": "Test alert"},
      "startsAt": "2024-01-15T10:00:00Z",
      "generatorURL": "http://test",
      "fingerprint": "test123"
    }]
  }'

Troubleshooting

Alerts Not Appearing

Check Alertmanager logs for webhook errors
Verify URL is correct and accessible from Alertmanager
Check integration ID is valid
Test with curl to isolate network issues

Duplicate Incidents

Verify fingerprint is being sent by Alertmanager
Check labels are consistent across alerts
Ensure send_resolved: true to properly close incidents

Wrong Severity

Add severity label to your alert rules
Use lowercase values: critical, error, warning, info

Grafana Integration — Grafana can also send Alertmanager-format webhooks
Events API — Programmatic event submission
Integrations Overview — All integrations