Upgrade OpsKnight

An upgrade changes an immutable application image and may change the PostgreSQL schema. Treat those as separate operations with one migration owner, an explicit compatibility decision, a verified backup, and rollback criteria agreed before the window.

For enterprise SSO 2.1, use the existing SSO upgrade cutover with downtime. Old/new application replicas must not overlap during migration. The Helm pre-upgrade gate verifies stopped application runtimes before the migration owner runs; other topologies must enforce the same stop-all sequence explicitly.

For the Service Automation observation provenance migration (20261021040000), workers running predecessor code target the predecessor 4-column unique index. Stop or drain background workers (opsknight-general-worker and integrated replicas) before applying schema migrations, or cut over worker replicas sequentially (spec.replicas: 0 before migration, followed by new replica launch).

Prerequisites

Prepare:

Release notes and supported source-to-target upgrade path.
The target image pinned by digest, with provenance verified.
Current image digest, configuration revision, topology, runtime roles, and schema/migration state.
A recent backup restored successfully in an isolated environment using the matching encryption keys and secrets.
Direct PostgreSQL connectivity for the migration owner; do not migrate through transaction-mode PgBouncer.
A maintenance/communication plan, named decision owner, and monitored soak period.
Baseline readiness, deep health, queue ages, notification failures, scheduler state, database connections, and key journey results.

Do not start if the backup is untested, the target image is mutable, migration ownership is ambiguous, database headroom is insufficient for rollout overlap, or the old version's compatibility with the post-migration schema is unknown.

Configure compatibility and rollback before deployment

Classify the release:

Application-only or backward-compatible schema change: the previous runtime can operate safely on the migrated schema for the rollback window.
Expand/contract transition: old and new runtimes can overlap only during a documented phase; destructive contract work happens later.
Schema-breaking change: the previous runtime cannot operate on the new schema. Rollback requires stopping writes and restoring the pre-upgrade backup, with data loss bounded by the approved RPO.

Write down which class applies and the evidence supporting it. Prisma migration success alone does not prove backward compatibility.

Define rollback triggers such as migration failure, sustained readiness failure, critical queue-age regression, notification failure, login failure, data-integrity error, or unacceptable latency. Define who can call rollback and the last point where image-only rollback remains safe.

Rehearse

Restore a recent backup to an isolated database, deploy the target digest with production-equivalent configuration, and run:

bash

npm run prisma:validate
npm run prisma:health

Apply migrations using the same mechanism planned for production. Verify sign-in, services, schedules, incident create/acknowledge/resolve, notification delivery, integrations, status publication, mobile routes, scheduler/worker progress, reports, and encrypted provider configuration. Measure migration and rollout time against the maintenance window.

Pre-upgrade checks

Announce the change window and freeze unrelated configuration/schema work.
Confirm current alerts are understood and active incidents have an owner.
Record all running image digests and replica counts.
Capture and verify the final pre-upgrade backup.
Run migration validation and health against the production database without applying changes.
Confirm no unfinished migration exists in _prisma_migrations.
Validate runtime/database connection capacity for rollout overlap.
Confirm old images, old configuration, backup, keys, and restore commands are immediately available.
For a 1.x to 2.0 upgrade, preserve the existing NEXTAUTH_SECRET and configure a separate API_KEY_SECRET of at least 32 characters before starting 2.0. Production startup rejects a missing, placeholder, or reused value.
Preserve the existing ENCRYPTION_KEY, or the complete ENCRYPTION_KEYS keyring with unchanged key IDs, and deliver it to every web, worker, scheduler, and migration role.

Existing 1.x API keys and status-page API tokens keep working without operator action. 1.x stored scrypt hashes salted with API_KEY_SECRET, otherwise NEXTAUTH_SECRET, otherwise a secret derived from ENCRYPTION_KEY. 2.0 checks each retained value, plus the shipped 1.x NEXTAUTH_SECRET placeholders, during legacy lookup, then rewrites a matched record to the current HMAC hash under API_KEY_SECRET. Keep NEXTAUTH_SECRET stable through this migration window; rotating it at the same time can strand credentials that have not yet been used or recreated.

Do not combine the upgrade with an encryption-key rotation. After the upgrade is accepted, move from ENCRYPTION_KEY to ENCRYPTION_KEYS=k2:<new>,k1:<existing> as described in Migrate from OpsKnight 1.x. If you roll back the image without restoring the database, API keys already migrated to the 2.0 hash and v3: ciphertext written by 2.0 are not readable by 1.x; see Roll back an upgrade.

Apply schema changes once

Use exactly one supported migration owner. Long-running split roles should set OPSKNIGHT_SKIP_MIGRATIONS=true; they must not race the migration Job/service.

Follow Operate database migrations for the exact Compose, Helm, and Swarm commands, safe recovery behavior, and optional SLA scheduler index boundary. Require migration exit code zero and a clean npm run prisma:health result before replacing application roles.

Do not edit applied migration SQL, delete migration records, use prisma db push in production, or mark a failure resolved merely to continue rollout.

Roll out by topology

Docker Compose

Render the exact file set and record its images:

bash

docker compose <files> config --quiet
docker compose <files> config --images
docker compose <files> pull

For split mode, run the one-shot migration service and confirm successful exit before updating Web, Scheduler, General Worker, Critical Worker, Bulk Worker, Runbook Worker, and Status Projector. Then:

bash

docker compose <files> up -d --wait
docker compose <files> ps -a

For integrated mode, only one integrated application process should own background responsibilities. Complete the backup, allow the controlled startup migration owner, and replace the application. Compose is single-host; keep host-level recovery available.

Helm

Render and review the candidate first:

bash

helm lint deploy/kubernetes/helm/opsknight --values values.production.yaml
helm template opsknight deploy/kubernetes/helm/opsknight \
  --namespace opsknight --values values.production.yaml > rendered.yaml

Pin image.digest, keep migrations.job.enabled=true, and run the upgrade with the approved values:

bash

helm upgrade --install opsknight deploy/kubernetes/helm/opsknight \
  --namespace opsknight --values values.production.yaml --wait

The pre-upgrade migration hook must complete before workloads roll. Inspect the migration Job and rollout status; do not bypass a failed hook. Confirm PodDisruptionBudget, topology spread, NetworkPolicy, Secret references, direct migration URL, and readiness probes in the rendered manifests.

Kustomize or raw Kubernetes

Render the selected overlay, review immutable image and Secret references, and apply the migration Job as the single schema owner before rolling Deployments. Wait for each role to become available and retain enough old capacity only when schema compatibility explicitly permits overlap.

Docker Swarm

Set the target digest and use the repository orchestrator:

bash

OPSKNIGHT_IMAGE='ghcr.io/opsknight-labs/opsknight@sha256:<digest>' \
  ./deploy/swarm/scripts/deploy.sh

It validates capacity, manages versioned secrets, runs the one-shot migration service, deploys with --prune, waits for convergence, and runs readiness. Do not replace this routine with an ad hoc docker stack deploy.

Verify the upgrade

Complete every check before accepting the release:

All expected replicas run the target deploymentId/image digest; no unintended mixed version remains.
Liveness and readiness are healthy per role; authenticated deep health shows current scheduler, worker, realtime, queue, notification, and rollup state.
npm run prisma:health reports no unfinished or unknown migration.
Database connection use, locks, CPU, and latency remain within the planned envelope.
Queue oldest-age trends decline or remain normal across critical, general, bulk, and projection work.
Administrator/OIDC login, session refresh, and expected denied access work.
A synthetic incident is created/ingested, notified, acknowledged, and resolved.
At least one configured inbound provider and outbound notification provider succeeds.
Status-page publication, Jira/ChatOps projections, reports, and mobile routes used by your organization work.
Logs contain no sustained migration, serialization, provider, encryption, or authorization failures.

Keep the release in a monitored soak period that covers scheduled work and representative provider traffic. Record the image digest, schema result, test evidence, observed metrics, reviewer, and acceptance time.

Operate the accepted release

Retain source and target digests, rendered configuration, migration output, acceptance evidence, and rollback criteria with the change record. Continue heightened monitoring through a representative paging and scheduled-maintenance window before retiring the previous image and backup.

Failure handling

Migration fails: stop rollout, preserve logs and database state, and follow Migration fails.
New runtime fails before migration: return to the old image/configuration.
New runtime fails after a compatible migration: use the image-only rollback procedure.
New runtime fails after an incompatible migration or writes incompatible data: stop writes and use the approved restore-based recovery path.
Only one projection/provider fails: preserve canonical incident state, repair that subsystem, and decide whether the release-wide rollback trigger is met.

See Roll back an upgrade before executing any reversal.