Prometheus Alertmanager

What it does

The Prometheus Alertmanager adapter accepts inbound webhook events at /api/integrations/prometheus, validates them through its custom handler, normalizes provider payloads, and submits lifecycle events to the configured service.

Prerequisites

An OpsKnight service and enabled integration record.
The integration identifier and generated integration key.
Permission to configure webhooks in Prometheus Alertmanager.
A network path from the provider to the OpsKnight web runtime.

Setup and configuration

In OpsKnight, open Services → your service → Integrations.
Select Add integration → Prometheus Alertmanager, then save the integration.
Copy the webhook URL and integration key shown by OpsKnight.
Add a receiver with webhook_configs to Alertmanager and set url to the complete OpsKnight URL and send_resolved: true.
Route a dedicated non-production matcher to that receiver first. Preserve the standard Alertmanager body, especially alert fingerprint, alertname, instance, status, annotations, and timestamps.
Run amtool check-config, reload Alertmanager, then fire and clear a test rule before widening the route.

Provider console labels can change independently of OpsKnight. Use the webhook or notification configuration area in the provider rather than copying a URL from another service. Treat keys and signature secrets as credentials; never place them in logs or source control.

Authentication and request verification

The endpoint requires the integration identifier. Signature verification is none using the prometheus verification contract. The exact payload schema is defined by src/app/api/integrations/prometheus/route.ts and src/lib/integrations/prometheus.ts.

Method: POST
Integration identifier: query parameter
Integration key transports: none; this route uses the authentication contract above
Schema: shared provider schema
Body limit: 1048576 bytes (1 MiB)
Rate limit: 100 requests per 60 seconds, per integration

Event mapping and incident lifecycle

The adapter emits the lifecycle actions found in its current source:

trigger
acknowledge
resolve Correlation contract: adapter EventPayload.dedup_key. Recovery contract: adapter emits resolve for its recovery state.

Recovery and deduplication

This route has no provider delivery-ID fence; incident convergence relies on the adapter deduplication key. Incident convergence still depends on the adapter correlation key. Failed deliveries are recorded for operational inspection without exposing secrets.

Limits and testing

Per-integration rate limiting protects the ingestion path. Send a representative trigger and recovery pair in a non-production service, verify that one incident is created, and confirm that recovery updates that incident rather than creating another.

Verify the connection

After the test alert, confirm all of the following:

One incident appears for the selected OpsKnight service.
The incident source identifies Prometheus Alertmanager.
A repeat event updates or correlates according to the adapter identity.
A recovery event resolves the correlated incident.

Error reference

400 — Invalid request or payload validation failed.
401 — Configured authentication or signature validation failed.
403 — Integration is disabled.
404 — Integration record was not found.
413 — Payload exceeds the one MiB body limit.
429 — Per-integration request rate exceeded.
500 — Provider event processing failed.

Troubleshooting

Confirm the integration is enabled and belongs to the intended service.
Verify the integration ID in the URL and rotate any key that may have been exposed.
Inspect Settings → Integrations → Failures for validation or signature errors.
Check for 413 before changing payload templates and 429 before retrying rapidly.
Confirm the provider sends a state supported by the event mapping above.
Preserve the provider delivery identifier and timestamp when escalating.

Security

Use HTTPS, rotate exposed keys at both systems, configure signature verification when supported, and restrict provider egress or ingress controls without blocking legitimate retries.