Operate7 min read

MCP monitoring and capacity

Every MCP metric, the shipped alerts, the per-request log line and how MCP capacity scales with replicas.

Tool failures reach assistants as HTTP 200 JSON-RPC results, so the generic HTTP 5xx alert never fires when MCP breaks. Scrape /api/metrics and load the opsknight.mcp groups from monitoring/prometheus/opsknight-alerts.yaml and opsknight-recording-rules.yaml; CI runs their promtool unit tests (monitoring/prometheus/mcp.test.yml).

Metrics

No metric carries a user, client, grant or incident identifier. tool is a bounded label (named tools, other for anything else, none off tools/call).

Metric Kind Labels Scope Normal When it is not
opsknight_mcp_requests_total counter outcome, tool Counter (sum across replicas) Mostly served; some tool_error and invalid_request from assistants guessing arguments. Alert on error/unavailable share (OpsKnightMcpServerErrorRateHigh). Group mcp.request.completed log lines by outcome, tool and grantId to find the cause.
opsknight_mcp_in_flight gauge — Per process (gauge) Below opsknight_mcp_in_flight_limit most of the time. Pinned at the limit with opsknight_mcp_queued above zero: raise the limit or add web replicas.
opsknight_mcp_queued gauge — Per process (gauge) Zero, with short spikes during bursts of parallel tool calls. Never empties for five minutes: OpsKnightMcpQueueSaturated.
opsknight_mcp_busy_total counter — Counter (sum across replicas) Flat. Each increment is one request shed with MCP_BUSY (work or authentication slot). Sustained growth: OpsKnightMcpShedding. Raise the limit, add replicas, or find the busy connection.
opsknight_mcp_call_log_dropped_total counter — Counter (sum across replicas) Zero. Each increment is one activity-log row (the Connections tab drawer) not written because the queue was full or a batch write failed; requests themselves are unaffected. Any growth: search the log for mcp.call_log.write_failed and check database health. Audit rows for changes are written separately and are not affected.
opsknight_mcp_subscriptions_active gauge — Per process (gauge) Open listen streams on this process, at most 32. At the cap, new streams get MCP_SUBSCRIPTIONS_BUSY (counted as outcome busy).
opsknight_mcp_in_flight_limit gauge — Per process (gauge) The policy value Concurrent requests per process, identical on every replica. Cluster capacity is this value times the web replicas.
opsknight_mcp_request_duration_seconds histogram outcome, tool Counter (sum across replicas) Only calls that reached domain work are timed, so rejections never hide slow calls. p95 above 2.5 s for a tool: OpsKnightMcpP95High. Check the MCP database pool and slow queries.
opsknight_mcp_oauth_requests_total counter endpoint, outcome Counter (sum across replicas) Mostly issued; invalid_grant when assistants retry stale refresh tokens. Token server_error/unavailable: OpsKnightMcpTokenEndpointFailing. A high invalid_grant share: OpsKnightMcpInvalidGrantSpike.
opsknight_mcp_maintenance_runs_total counter task, result Counter (sum across replicas) One success per task per scheduler maintenance pass. Failures: OpsKnightMcpMaintenanceFailing. No success in 15 minutes: OpsKnightMcpMaintenanceStalled.

outcome on opsknight_mcp_requests_total: served, task_created, tool_error, notification, protocol_rejected, subscription_opened, subscription_rejected, unauthenticated, forbidden, rate_limited, busy, result_too_large, disabled, unavailable, invalid_request, error. Every HTTP 429 counts as rate_limited. The duration histogram observes only served, tool_error, task_created, result_too_large.

opsknight_mcp_oauth_requests_total: endpoint is authorize, token, register, revoke, device_authorization; outcome is issued, invalid_request, invalid_grant, invalid_client, invalid_scope, rate_limited, pending, denied, expired, unavailable, server_error, read from the status and error code actually sent (issued covers any accepted request, including a redirect to sign-in or consent).

Alerts

Alert Severity For Summary
OpsKnightMcpServerErrorRateHigh critical 10m More than 5% of MCP requests fail on the server
OpsKnightMcpToolErrorRateHigh warning 15m More than a quarter of MCP calls return tool errors
OpsKnightMcpShedding warning 10m MCP is shedding requests with MCP_BUSY
OpsKnightMcpQueueSaturated warning immediately MCP requests have waited for a work slot for five minutes
OpsKnightMcpAuthFailuresSpike warning 10m Most MCP requests are refused authentication or authorization
OpsKnightMcpRateLimited warning 15m Assistants are hitting MCP quotas
OpsKnightMcpDisabledTraffic info 30m Assistants keep calling MCP while it is turned off
OpsKnightMcpP95High warning 10m MCP tool {{ $labels.tool }} p95 latency is above 2.5 seconds
OpsKnightMcpTokenEndpointFailing critical 10m MCP token refreshes are failing on the server
OpsKnightMcpInvalidGrantSpike warning 15m Most MCP token requests are rejected as invalid_grant
OpsKnightMcpMaintenanceFailing warning 2m MCP retention maintenance keeps failing
OpsKnightMcpMaintenanceStalled warning 15m No successful MCP maintenance run in 15 minutes
OpsKnightMcpCallLogDropping warning 5m MCP request-log rows are being dropped
OpsKnightMcpStreamsNearLimit warning 10m MCP listen streams on {{ $labels.instance }} are close to the 32-stream cap

Each alert's description in the rules file names the log event to search. Thresholds are starting points; tune them to your traffic.

Tracing

Each /mcp request also produces one OpenTelemetry span named mcp.request, with the attributes mcp.request_id, mcp.method, mcp.tool, mcp.outcome and mcp.duration_ms: the same bounded labels as the metrics, never a person, assistant, connection, argument or result. Server failures (error, unavailable) set the span status to error; refusals the caller caused do not. OpsKnight only calls the OpenTelemetry API, so nothing is recorded or exported until you register a tracer provider in your deployment, and there is no setting to turn on.

Dashboard

Import monitoring/grafana/opsknight-mcp.json into Grafana (choose your Prometheus data source when asked). It shows requests by outcome, the server error share, the slowest 5% of calls by tool, work slots and shedding, open listen streams against the cap, dropped request-log rows and OAuth outcomes. Every panel reads a metric listed above; none is labelled by person, assistant or connection. For per-assistant and per-person usage, use the Usage tab in the MCP settings, which is built from the request log.

Per-request log

Every /mcp request writes one mcp.request.completed line: requestId (also the X-Request-Id response header, on errors too), grantId, clientId, method, tool, outcome, durationMs and, on failure, errorCode. Arguments and results are never logged. Requests refused before the bearer resolves carry no grantId. Outcomes other than served are always written; set MCP_LOG_SAMPLE_SERVED (0 to 1, default 1) to sample served requests. Treat the field names as a stable contract for SIEM pipelines. Admin-gated sensitive reads also write an mcp.sensitive_read audit row (operation, grant, path ids, page size and result size; never query values or data). See troubleshooting for every MCP log event.

Capacity and sizing

Every MCP limit is enforced per web process. Multiply by the number of web replicas for cluster capacity. These are arithmetic caps, not measured throughput (see architecture).

Limit (per process) Value Shed as Watch
Concurrent bearer authentications 4 MCP_BUSY, Retry-After 1 opsknight_mcp_busy_total
Work slots policy Concurrent requests per process (1 to 16) waits up to 750 ms, then MCP_BUSY opsknight_mcp_in_flight vs opsknight_mcp_in_flight_limit
Work-slot queue twice the work slots MCP_BUSY at once opsknight_mcp_queued
Slots one person may hold half the work slots, rounded up, across all their connections waits, then MCP_BUSY mcp.request.completed with outcome busy by grantId
Listen streams 32 (2 per connection, 10 minutes each) MCP_SUBSCRIPTIONS_BUSY / MCP_SUBSCRIPTIONS_LIMIT opsknight_mcp_subscriptions_active
Operations per request policy Operations per request, at most 8 MCP_OPERATION_BUDGET_EXCEEDED tool errors by errorCode
MCP database pool 2 connections (maximum 4) waits on database connections /api/health/deep mcp.latencyMs

Example: three web replicas with Concurrent requests per process set to 4 admit at most 12 MCP requests at once cluster-wide, queue 24 more, and hold up to 96 listen streams; they need 6 to 12 MCP database connections on top of the web pool. /api/health/deep reports each process's inFlight, queued, authenticating, limit, queueCap, authLimit, streams and streamsLimit.

Emergency stop

Set MCP_FORCE_DISABLED=true in the environment and restart to turn MCP off for every surface (the /mcp endpoint, sign-in, token refresh, registration and discovery) whatever the stored policy says. The settings page shows Disabled by deployment configuration and refuses to turn MCP on; /api/health/deep reports forceDisabled: true. Remove the variable and restart to restore the saved policy. Connections are not revoked, so they work again afterwards.

Last updated for v2.2.0

Edit this page on GitHub