Scalability and capacity planning

OpsKnight v1.4 does not publish a universal user, incident, notification, or streaming-connection capacity. Throughput depends on application and PostgreSQL resources, replica count, incident fan-out, query history, external-provider latency, and the traffic shape. Treat any capacity number that was not measured on your deployment as an assumption, not a product guarantee.

This guide separates code-defined protective limits from measured capacity and provides a repeatable way to establish safe operating limits for your environment.

Model the workload

Build a representative workload before choosing infrastructure. Record at least:

Average traffic is insufficient for incident-management capacity planning. Test the alert storm, notification fan-out, and recovery burst you need to survive.

Distinguish limits from capacity

The following v1.4 values are implementation guardrails. They prevent one path from consuming unlimited resources; they are not benchmark results or service-level objectives.

See API rate limiting for the complete client-visible contract. Do not add the values above together to infer total notification or incident throughput. A slow provider, expensive query, or wide escalation fan-out can lower observed capacity significantly.

Understand the scaling topology

One Next.js application serves the UI, API routes, provider webhooks, real-time streams, and the internal scheduler. PostgreSQL stores product state, rate-limit counters, scheduler coordination, and durable background jobs. Redis and a separately deployed worker service are not part of the v1.4 runtime.

Multiple application replicas can use the same PostgreSQL database. For every replica:

1. Deploy the same application version and runtime configuration.
2. Use the same NEXTAUTH_SECRET, ENCRYPTION_KEY, public URL, and provider settings.
3. Connect to the same migrated PostgreSQL database.
4. Keep the scheduler enabled across instances. PostgreSQL coordinates scheduler ownership via CronSchedulerState distributed locks with 30-second heartbeats and 5-minute timeout. Standby replicas poll with randomized jitter (15–30s) to ensure automated leader failover if the active leader replica stops.
5. Allow long-lived SSE responses at the load balancer or reverse proxy and disable response buffering for them.
6. Drain traffic and in-flight work before stopping the process.

Horizontal application scaling does not turn process-local state into shared state. Each replica has its own immediate notification queue, real-time cache, and circuit-breaker state. A process exit can lose items that have not reached durable storage. Review Technical architecture before choosing a high-availability topology.

Budget PostgreSQL connections

Do not copy a fixed connection_limit, max_connections, or memory setting from another deployment. Budget connections across the entire database environment:

(application replicas × maximum application pool per replica)
+ migration and administrative reserve
+ monitoring, backup, and other client connections
≤ PostgreSQL connection capacity

Choose the application pool and database limit together. Leave reserve for migrations, health checks, recovery access, and expected failover behavior. Increasing a connection limit without sufficient PostgreSQL CPU, memory, I/O, and query capacity can increase contention instead of throughput.

During a load test, monitor active/waiting connections, pool wait time, transaction/query latency, locks, deadlocks, CPU, memory, storage latency, and WAL/replication lag where applicable. Change one variable at a time and retain the before/after evidence.

Establish a capacity envelope

1. Define measurable objectives

Set objectives for the workflows users depend on, for example:

• accepted-event rate and HTTP p50/p95/p99 latency;
• time from accepted trigger to committed incident;
• time from trigger to first attempted and first successful notification;
• acknowledge/resolve success and latency during an alert storm;
• maximum age and count of due background jobs;
• interactive/API error rate and latency with concurrent SSE users;
• PostgreSQL utilization, connection headroom, lock time, and storage growth; and
• recovery time after a provider, database, or application fault.

Define an error budget and a minimum headroom target before testing. A test does not pass merely because requests eventually complete.

2. Build a production-like test environment

Use the intended application image, replica topology, PostgreSQL major version, connection path, proxy timeouts, and realistic data volume. Use synthetic or properly sanitized data. Never direct a capacity test at real responders, customer webhooks, or paid messaging channels without explicit safeguards.

Replace external providers with controlled test endpoints that can reproduce normal latency, throttling, timeouts, and failures. Preserve realistic notification fan-out and payload sizes.

3. Run progressive and failure tests

Run each test long enough to expose queue growth, cleanup, connection pressure, and storage effects:

1. Establish an idle and normal-traffic baseline.
2. Increase one workload dimension in steps and hold each step at steady state.
3. Test bursts at and beyond the expected peak without bypassing route rate limits accidentally.
4. Combine inbound load, responder activity, SSE connections, and scheduled work.
5. Add slow/throttled providers and confirm backlogs and retries remain bounded.
6. Restart and drain application replicas; verify the process-local queue boundary is understood.
7. Introduce PostgreSQL latency or a controlled disconnect; verify readiness, recovery, and producer behavior.
8. Exercise scheduler-owner loss and confirm another eligible process advances scheduled work after ownership becomes available.
9. Stop new load and measure backlog drain time and final delivery/error state.

Use stable dedup_key values when measuring correlation and unique keys when measuring incident creation. Mixing the two produces a misleading result.

4. Record the safe operating limit

The capacity envelope is the highest tested workload that meets every objective with the required headroom and without an unbounded queue, connection, latency, error, memory, or storage trend. Record:

• application/database/proxy versions and exact resources;
• replica and connection-pool settings;
• dataset size and retention configuration;
• workload generator, payload mix, fan-out, duration, and provider behavior;
• results, bottleneck, headroom, and failure/recovery observations; and
• the date, owner, and conditions that require a retest.

Retest after material changes to the application, schema/indexes, PostgreSQL, proxy, replicas, notification providers, retention, or expected traffic.

Monitor saturation in production

Alert on trends before users experience an incident-delivery failure:

The application includes health/readiness output, system logs, notification history, integration health, and an admin SLA-query performance page. These are useful evidence, but v1.4 does not publish a complete Prometheus-compatible capacity metrics endpoint. Collect platform and PostgreSQL telemetry independently.

Respond to saturation

1. Identify whether ingress, application CPU/memory, PostgreSQL, scheduled work, or a provider is the first constrained layer.
2. Protect incident writes and acknowledgement/resolve paths before analytics, exports, or other expensive non-critical work.
3. Pace producers and respect Retry-After; use stable deduplication keys for idempotent event retries.
4. Reduce accidental fan-out or noisy-source traffic at its owner while preserving required alerts.
5. Scale or tune the constrained layer using a tested change. Adding application replicas cannot fix a saturated database or provider quota.
6. Confirm queues drain, latency returns to baseline, and synthetic delivery succeeds.
7. Preserve evidence, update the capacity envelope, and correct the monitoring threshold or architecture that allowed the surprise.

Do not disable rate limits or raise queue/database limits as a first response without understanding the next bottleneck and failure mode.

Implementation map

• src/lib/cron-scheduler.ts — scheduler cadence, ownership, and job-cycle limits.
• src/lib/jobs/queue.ts — durable job claims, concurrency, retry, and statistics.
• src/lib/notification-queue.ts — process-local queue limits and channel guards.
• src/lib/realtime-cache.ts — process-local cache TTLs and bound.
• src/app/api/realtime/stream/route.ts — authenticated dashboard SSE polling.
• src/lib/rate-limit.ts and src/lib/integrations/rate-limiter.ts — database-backed request limits.
• src/lib/admin-health.ts — administrator health checks and SLA-query observations.

Related topics

• Technical architecture
• Architecture diagrams
• Monitoring
• Maintenance
• API rate limiting
• Kubernetes deployment
• Troubleshooting