MCP monitoring and capacity
Every MCP metric, the shipped alerts, the per-request log line and how MCP capacity scales with replicas.
Tool failures reach assistants as HTTP 200 JSON-RPC results, so the generic HTTP 5xx alert never fires when MCP breaks. Scrape /api/metrics and load the opsknight.mcp groups from monitoring/prometheus/opsknight-alerts.yaml and opsknight-recording-rules.yaml; CI runs their promtool unit tests (monitoring/prometheus/mcp.test.yml).
Metrics
No metric carries a user, client, grant or incident identifier. tool is a bounded label (named tools, other for anything else, none off tools/call).
| Metric | Kind | Labels | Scope | Normal | When it is not |
|---|---|---|---|---|---|
opsknight_mcp_requests_total |
counter | outcome, tool |
Counter (sum across replicas) | Mostly served; some tool_error and invalid_request from assistants guessing arguments. |
Alert on error/unavailable share (OpsKnightMcpServerErrorRateHigh). Group mcp.request.completed log lines by outcome, tool and grantId to find the cause. |
opsknight_mcp_in_flight |
gauge | — | Per process (gauge) | Below opsknight_mcp_in_flight_limit most of the time. |
Pinned at the limit with opsknight_mcp_queued above zero: raise the limit or add web replicas. |
opsknight_mcp_queued |
gauge | — | Per process (gauge) | Zero, with short spikes during bursts of parallel tool calls. | Never empties for five minutes: OpsKnightMcpQueueSaturated. |
opsknight_mcp_busy_total |
counter | — | Counter (sum across replicas) | Flat. Each increment is one request shed with MCP_BUSY (work or authentication slot). | Sustained growth: OpsKnightMcpShedding. Raise the limit, add replicas, or find the busy connection. |
opsknight_mcp_call_log_dropped_total |
counter | — | Counter (sum across replicas) | Zero. Each increment is one activity-log row (the Connections tab drawer) not written because the queue was full or a batch write failed; requests themselves are unaffected. | Any growth: search the log for mcp.call_log.write_failed and check database health. Audit rows for changes are written separately and are not affected. |
opsknight_mcp_subscriptions_active |
gauge | — | Per process (gauge) | Open listen streams on this process, at most 32. | At the cap, new streams get MCP_SUBSCRIPTIONS_BUSY (counted as outcome busy). |
opsknight_mcp_in_flight_limit |
gauge | — | Per process (gauge) | The policy value Concurrent requests per process, identical on every replica. | Cluster capacity is this value times the web replicas. |
opsknight_mcp_request_duration_seconds |
histogram | outcome, tool |
Counter (sum across replicas) | Only calls that reached domain work are timed, so rejections never hide slow calls. | p95 above 2.5 s for a tool: OpsKnightMcpP95High. Check the MCP database pool and slow queries. |
opsknight_mcp_oauth_requests_total |
counter | endpoint, outcome |
Counter (sum across replicas) | Mostly issued; invalid_grant when assistants retry stale refresh tokens. |
Token server_error/unavailable: OpsKnightMcpTokenEndpointFailing. A high invalid_grant share: OpsKnightMcpInvalidGrantSpike. |
opsknight_mcp_maintenance_runs_total |
counter | task, result |
Counter (sum across replicas) | One success per task per scheduler maintenance pass. |
Failures: OpsKnightMcpMaintenanceFailing. No success in 15 minutes: OpsKnightMcpMaintenanceStalled. |
outcome on opsknight_mcp_requests_total: served, task_created, tool_error, notification, protocol_rejected, subscription_opened, subscription_rejected, unauthenticated, forbidden, rate_limited, busy, result_too_large, disabled, unavailable, invalid_request, error. Every HTTP 429 counts as rate_limited. The duration histogram observes only served, tool_error, task_created, result_too_large.
opsknight_mcp_oauth_requests_total: endpoint is authorize, token, register, revoke, device_authorization; outcome is issued, invalid_request, invalid_grant, invalid_client, invalid_scope, rate_limited, pending, denied, expired, unavailable, server_error, read from the status and error code actually sent (issued covers any accepted request, including a redirect to sign-in or consent).
Alerts
| Alert | Severity | For | Summary |
|---|---|---|---|
OpsKnightMcpServerErrorRateHigh |
critical | 10m | More than 5% of MCP requests fail on the server |
OpsKnightMcpToolErrorRateHigh |
warning | 15m | More than a quarter of MCP calls return tool errors |
OpsKnightMcpShedding |
warning | 10m | MCP is shedding requests with MCP_BUSY |
OpsKnightMcpQueueSaturated |
warning | immediately | MCP requests have waited for a work slot for five minutes |
OpsKnightMcpAuthFailuresSpike |
warning | 10m | Most MCP requests are refused authentication or authorization |
OpsKnightMcpRateLimited |
warning | 15m | Assistants are hitting MCP quotas |
OpsKnightMcpDisabledTraffic |
info | 30m | Assistants keep calling MCP while it is turned off |
OpsKnightMcpP95High |
warning | 10m | MCP tool {{ $labels.tool }} p95 latency is above 2.5 seconds |
OpsKnightMcpTokenEndpointFailing |
critical | 10m | MCP token refreshes are failing on the server |
OpsKnightMcpInvalidGrantSpike |
warning | 15m | Most MCP token requests are rejected as invalid_grant |
OpsKnightMcpMaintenanceFailing |
warning | 2m | MCP retention maintenance keeps failing |
OpsKnightMcpMaintenanceStalled |
warning | 15m | No successful MCP maintenance run in 15 minutes |
OpsKnightMcpCallLogDropping |
warning | 5m | MCP request-log rows are being dropped |
OpsKnightMcpStreamsNearLimit |
warning | 10m | MCP listen streams on {{ $labels.instance }} are close to the 32-stream cap |
Each alert's description in the rules file names the log event to search. Thresholds are starting points; tune them to your traffic.
Tracing
Each /mcp request also produces one OpenTelemetry span named mcp.request, with the attributes mcp.request_id, mcp.method, mcp.tool, mcp.outcome and mcp.duration_ms: the same bounded labels as the metrics, never a person, assistant, connection, argument or result. Server failures (error, unavailable) set the span status to error; refusals the caller caused do not. OpsKnight only calls the OpenTelemetry API, so nothing is recorded or exported until you register a tracer provider in your deployment, and there is no setting to turn on.
Dashboard
Import monitoring/grafana/opsknight-mcp.json into Grafana (choose your Prometheus data source when asked). It shows requests by outcome, the server error share, the slowest 5% of calls by tool, work slots and shedding, open listen streams against the cap, dropped request-log rows and OAuth outcomes. Every panel reads a metric listed above; none is labelled by person, assistant or connection. For per-assistant and per-person usage, use the Usage tab in the MCP settings, which is built from the request log.
Per-request log
Every /mcp request writes one mcp.request.completed line: requestId (also the X-Request-Id response header, on errors too), grantId, clientId, method, tool, outcome, durationMs and, on failure, errorCode. Arguments and results are never logged. Requests refused before the bearer resolves carry no grantId. Outcomes other than served are always written; set MCP_LOG_SAMPLE_SERVED (0 to 1, default 1) to sample served requests. Treat the field names as a stable contract for SIEM pipelines. Admin-gated sensitive reads also write an mcp.sensitive_read audit row (operation, grant, path ids, page size and result size; never query values or data). See troubleshooting for every MCP log event.
Capacity and sizing
Every MCP limit is enforced per web process. Multiply by the number of web replicas for cluster capacity. These are arithmetic caps, not measured throughput (see architecture).
| Limit (per process) | Value | Shed as | Watch |
|---|---|---|---|
| Concurrent bearer authentications | 4 | MCP_BUSY, Retry-After 1 |
opsknight_mcp_busy_total |
| Work slots | policy Concurrent requests per process (1 to 16) | waits up to 750 ms, then MCP_BUSY |
opsknight_mcp_in_flight vs opsknight_mcp_in_flight_limit |
| Work-slot queue | twice the work slots | MCP_BUSY at once |
opsknight_mcp_queued |
| Slots one person may hold | half the work slots, rounded up, across all their connections | waits, then MCP_BUSY |
mcp.request.completed with outcome busy by grantId |
| Listen streams | 32 (2 per connection, 10 minutes each) | MCP_SUBSCRIPTIONS_BUSY / MCP_SUBSCRIPTIONS_LIMIT |
opsknight_mcp_subscriptions_active |
| Operations per request | policy Operations per request, at most 8 | MCP_OPERATION_BUDGET_EXCEEDED |
tool errors by errorCode |
| MCP database pool | 2 connections (maximum 4) | waits on database connections | /api/health/deep mcp.latencyMs |
Example: three web replicas with Concurrent requests per process set to 4 admit at most 12 MCP requests at once cluster-wide, queue 24 more, and hold up to 96 listen streams; they need 6 to 12 MCP database connections on top of the web pool. /api/health/deep reports each process's inFlight, queued, authenticating, limit, queueCap, authLimit, streams and streamsLimit.
Emergency stop
Set MCP_FORCE_DISABLED=true in the environment and restart to turn MCP off for every surface (the /mcp endpoint, sign-in, token refresh, registration and discovery) whatever the stored policy says. The settings page shows Disabled by deployment configuration and refuses to turn MCP on; /api/health/deep reports forceDisabled: true. Remove the variable and restart to restore the saved policy. Connections are not revoked, so they work again afterwards.
Last updated for v2.2.0
Edit this page on GitHub