Configure split runtime with Helm

Prerequisites

Read Runtime roles, calculate resources and connections for every replica, and use an image supporting split roles. Configure stable secrets and a direct database URL.

Set runtime.mode: split, enable the migration Job, and define Web, Scheduler, General Worker, Critical Worker, Bulk Worker, Runbook Worker, and Status Projector replicas, resources, connection pool sizes, probes, termination grace periods, and PDBs. Direct ingress only to Web. Set database.maxApplicationConnections to the reviewed ceiling.

Create values.split.yaml with the complete role configuration:

yaml

runtime:
  mode: split

# Enforce High Availability by spreading pods across distinct Kubernetes nodes/zones
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule

database:
  port: 5432
  maxApplicationConnections: 200

migrations:
  job:
    enabled: true

# Web Tier (handles HTTP/UI/API traffic)
web:
  replicaCount: 2
  resources:
    requests: { cpu: 250m, memory: 512Mi }
    limits: { cpu: '1', memory: 1Gi }
  database:
    poolSize: 10
  podDisruptionBudget:
    enabled: true
    minAvailable: 1
  autoscaling:
    enabled: false
    minReplicas: 2
    maxReplicas: 10

# Scheduler Tier (runs scheduled incident sweeps & alert evaluations)
scheduler:
  replicaCount: 2
  profile: maintenance
  resources:
    requests: { cpu: 150m, memory: 256Mi }
    limits: { cpu: 500m, memory: 512Mi }
  database:
    poolSize: 3
  podDisruptionBudget:
    enabled: true
    minAvailable: 1

# General Worker (background async job execution)
generalWorker:
  replicaCount: 2
  resources:
    requests: { cpu: 200m, memory: 512Mi }
    limits: { cpu: 500m, memory: 1Gi }
  database:
    poolSize: 15
  podDisruptionBudget:
    enabled: true
    minAvailable: 1

# Critical Worker (p1/p2 pages, voice call dispatch, webhook deliveries)
criticalWorker:
  replicaCount: 2
  resources:
    requests: { cpu: 250m, memory: 512Mi }
    limits: { cpu: '1', memory: 1Gi }
  database:
    poolSize: 15
  podDisruptionBudget:
    enabled: true
    minAvailable: 1

# Bulk Worker (reporting rollups, data retention purges, telemetry processing)
bulkWorker:
  replicaCount: 2
  resources:
    requests: { cpu: 200m, memory: 512Mi }
    limits: { cpu: 500m, memory: 1Gi }
  database:
    poolSize: 10
  podDisruptionBudget:
    enabled: true
    minAvailable: 1

# Runbook Worker (isolated remediation orchestration and reconciliation)
runbookWorker:
  replicaCount: 2
  resources:
    requests: { cpu: 200m, memory: 512Mi }
    limits: { cpu: 500m, memory: 1Gi }
  database:
    poolSize: 3
  podDisruptionBudget:
    enabled: true
    minAvailable: 1

# Status Projector (real-time status page projections & incident state updates)
statusProjector:
  replicaCount: 2
  resources:
    requests: { cpu: 150m, memory: 256Mi }
    limits: { cpu: 500m, memory: 512Mi }
  database:
    poolSize: 5
  podDisruptionBudget:
    enabled: true
    minAvailable: 1

Keep migration and non-Web roles on direct PostgreSQL. Add PgBouncer only for Web.

Deploy split runtime

Lint, render, and inspect the chart. The integrated Deployment must be absent.

helm upgrade --install opsknight deploy/kubernetes/helm/opsknight \
  --namespace opsknight \
  -f values.production.yaml \
  -f values.split.yaml

Require the migration hook to complete before all role rollouts become ready:

kubectl -n opsknight get job,deployment,pod -w

Verify every role

Confirm migration success, Web readiness, scheduler heartbeat, progress in all worker lanes, status projection, and no integrated ownership. Run a synthetic incident through notification, acknowledgement, resolution, and projection while observing database connections.

Operate and scale in production

Alert on each role and lane. Scale only after checking oldest age, throughput, resource saturation, provider limits, and connection headroom. Keep critical work isolated and preserve graceful termination.

Troubleshooting

A role is missing: inspect rendered enablement/replica values and chart schema errors.

A role CrashLoops: inspect its Secret mapping, runtime role, database route, resources, and prior logs.

Queues grow with Ready Pods: investigate heartbeat, lane routing, provider/database bottlenecks, and throughput; readiness alone is insufficient.

Change or undo the configuration

Upgrade one reviewed values revision at a time. To return to integrated, stop split ownership before starting integrated and run full acceptance.