On-Prem Upgrade Operator Guide

Overview

How the upgrade works

This guide covers deployments on your own servers. Upgrading an AWS deployment is a different procedure — a second environment is built alongside the running one and traffic moves across by DNS. See the AWS upgrade operator guide.

The upgrade runs blue/green. Blue is the system you run today; green is a fresh install of the new version. Your live data is copied out of blue, rebuilt on green in a private staging area, and promoted to live only at the very end — so nothing your users depend on changes until you're ready.

What moves between the two systems

Blue — your current systemread only, never modified--backup signed bundle

Green stagingisolated rebuild of your data--restore green_* databases

Green livethe only write to the running system--run-transforms promoted & live

The bundle is GPG-signed and carries every datastore — databases, search indices, certificates, secrets and Kafka topics — plus a full Velero safety copy of the cluster.

You run three commands, in order. Back up on the blue machine, then restore and promote on the green machine. Each command checkpoints its progress, so a re-run continues from where it stopped rather than starting over.

Before you begin

What you need

Free disk space — roughly 3× your data size on the working path (for example /home/blinq/upgrades).
A bundle destination both machines can reach — a remote host over SSH (blinq@host:/backups/) or a mounted path (/mnt/external/).
Passwordless SSH to that host, if remote — set up once with sudo ssh-copy-id blinq@<host>.
The new NetLiNQ version installed on the green machine before you restore (Phase 2).
Root access — every command runs under sudo.

No internet access is required. The whole process is air-gap friendly — the bundle carries everything green needs.

Phase 1 · on the blue machine

Back up the current system

Capture every datastore into one signed bundle and push it to your destination. Your live system keeps running throughout — this step only reads it.

back up

$ sudo ./scripts/upgrade-onprem/upgrade.sh --backup \
    -b /home/blinq/upgrades --dest blinq@192.168.5.125:/backups/ -y

Variations

local destination

# Push to a mounted local path instead of a remote host
$ sudo ./scripts/upgrade-onprem/upgrade.sh --backup -b /home/blinq/upgrades --dest /mnt/external/ -y

dry run

# Check prerequisites only — capture nothing
$ sudo ./scripts/upgrade-onprem/upgrade.sh --backup -b /home/blinq/upgrades --dry-run -y

What you'll see. Each step logs its progress and time, ending with a summary and three files that travel together — the .tar.gz bundle, its .sig signature, and the public key. The Velero safety copy is the longest part — tens of minutes on a large system.

Phase 2 · on the green machine

Restore into staging

Install the new NetLiNQ version on green first, then run this. The bundle's signature and checksums are verified before anything is restored.

Fetch the bundle and rebuild every dataset into an isolated staging copy. Green's live databases are not modified in this step.

restore

$ sudo ./scripts/upgrade-onprem/upgrade.sh --restore \
    -b /home/blinq/upgrades --dest blinq@192.168.5.125:/backups/ --skip-certs -y

bundle already on disk

# Restore from a bundle already on this machine — no fetch
$ sudo ./scripts/upgrade-onprem/upgrade.sh --restore \
    -b /home/blinq/upgrades --source /home/blinq/upgrades/netlinq-upgrade-backup-*.tar.gz --skip-certs -y

Choose your certificates. Restore includes a certificate step that asks a yes/no question, so a hands-off run must answer it up front: --skip-certs keeps green's own certificates (the safe default), or --restore-certs adopts the blue-side certificates. Pick one.

What you'll see. The signature is verified, checksums are checked, then each datastore is rebuilt into staging — green_* databases, a temporary search index, and so on. The run ends with a summary of what was staged.

Phase 3 · on the green machine

Promote staging to live

Promote your staged data into the live databases with a single standalone command — no bundle or destination needed, since the staged data is already on the green cluster from Phase 2. This is the one and only step that writes to the live system.

promote

$ sudo ./scripts/upgrade-onprem/upgrade.sh --run-transforms -y

Under the hood. The running services are briefly paused, the staged data is promoted into live, then the services are scaled back to their original size — even if the step fails, so your system never stays paused.

Clean up the staging index

To also remove the temporary Elasticsearch staging index once the promotion succeeds, add --cleanup-staging-es:

promote + clean up

$ sudo ./scripts/upgrade-onprem/upgrade.sh --run-transforms --cleanup-staging-es -y

Confirm

Check that it worked

Phase 3 runs an automatic health check — including a data-transfer count check that compares document/row counts live-vs-staging, flags any mismatch, and fails the run if data did not land (an index/table empty on live while staging held data) — so a partial migration cannot pass silently. To confirm by hand, every pod should be Running or Completed:

verify

$ sudo kubectl get pods -n netlinq   # all Running / Completed

Then log in to the application and spot-check your dashboards, sites, and users. The full run log is saved next to the bundle as netlinq-restore-*.log.

Recovery

If something goes wrong

Just re-run the same command. A failed run stops and keeps its progress. Fix the cause — config, network, disk — and run the exact same command again; it skips the finished steps and continues from the first incomplete one.

Your live system stays safe: Phases 1–2 never touch it, and Phase 3 always restores your services to normal on success or failure. Until Phase 3 promotes the data, nothing on live has changed — you can simply stop. The Velero backup from Phase 1 is your full-cluster safety net.

A phase failedFix the issue, then re-run the same command — it resumes automatically.

--restartStart completely fresh, ignoring saved progress.

--redo-phase <name>Re-run one specific phase, e.g. --redo-phase 4.3-elasticsearch.

--skip-veleroBackup too slow? Skip the safety copy — only if you already have another backup.

Recovery

Rolling back to the previous version

The safest rollback is to keep blue running until green is signed off. Through Phases 1–2 nothing on live changes, and even after Phase 3 blue is a separate machine still holding your previous version — if green misbehaves, keep using blue.

To revert green (or a rebuilt box) to the previous version, use the bundle you already captured. Both methods freeze writers, snapshot current live first (so the rollback is itself reversible), restore, scale back up, and verify.

Option A — Velero (full cluster; recommended for a rebuilt box)

rollback · velero

$ sudo ./scripts/upgrade-onprem/upgrade.sh --rollback --velero --source <bundle.tar.gz> -y

Restores every namespace, database and workload to the bundle's version. Requires the backup to have included Velero (not --skip-velero), loaded into the box's MinIO.

Option B — Database dumps (PostgreSQL + MongoDB + /storage + secrets)

rollback · dbs

# Reinstall the previous version first, then:
$ sudo ./scripts/upgrade-onprem/upgrade.sh --rollback --dbs --source <bundle.tar.gz> -y

Restores the raw dumps straight into the live databases. Elasticsearch is not reverted by --dbs — use --velero for a run that includes it. On a different / rebuilt box, fetch the bundle with --dest <ssh|path> instead of --source.

Rollback is destructive — it drops the live databases. It preserves current live to a rollback-preserve-* archive first, but keep the bundle and the Velero backup until you've confirmed the rollback is healthy.

Reference

Command reference

--backup / --restoreWhich phase to run.

-b <path>Local working directory for the bundle.

--dest <ssh|path>Where the bundle is pushed (backup) or fetched from (restore).

--source <file>Restore from a bundle already on disk — skip the fetch.

--run-transformsPromote staging to live (Phase 3). A standalone mode — needs no --dest, --source or -b.

--skip-certs / --restore-certsAnswer the restore's certificate prompt for a hands-off run (Phase 2).

--cleanup-staging-esRemove the temporary search index after a good promotion.

--skip-elasticsearch / --skip-kafka / --skip-veleroSkip a data phase.

--dry-runBackup: check prerequisites only, capture nothing.

-yNon-interactive — still choose a certificate flag above.

Full technical reference: scripts/upgrade-onprem/README.md. Flow diagrams: BACKUP-FLOW.md and RESTORE-FLOW.md.

Quality

Testing & validation

The upgrade tooling ships with an automated test suite that runs on every change. Each destructive path — the live database cutover, storage restore, Velero pruning, secret handling — is exercised against mocked infrastructure so failures are caught before they reach a real system.

Test files

Automated checks

269

Failures

Every check passes on the current build. The suite runs under CI on each pull request, alongside bash -n syntax checks and shellcheck static analysis on every script.

What the suite covers

Commands & argumentsFlag parsing, required-argument and mutual-exclusion guards, and loading of the settings file — test_arg_parsing, test_vars_loader.

Backup integrityBundle signature and checksum verification, file-format checks, secret and sealed-secret handling, Kafka topics, and Velero — test_bundle_validation, test_backup_helpers, test_secrets, test_kafka, test_velero.

Restore & stagingPostgreSQL restore into staging, Mongo/PG staging jobs, datastore discovery, host and credential resolution, and SpC /storage — test_pg_restore, test_staging_jobs, test_discovery, test_resolvers, test_spc_storage.

Promotion & transformsEvery promotion outcome end to end — deploy failure, verification failure, incomplete transforms, and the staging-cleanup safety gate — plus deploy/verify helpers and on-prem values rendering: test_transforms_flow, test_transforms_verify, test_values_render.

Safety & recoveryResume/checkpoint recovery, cutover scale-up guarantees, Velero pruning safety, and the full rollback-to-previous-version path (dumps-to-live + Velero restore) -- test_rollback, test_resume, test_cleanup.

Release validation

Before this guide's commands were published, the packaging behind them was validated end to end:

Command flagsEvery documented backup / restore / promote command parses without error, including the standalone --run-transforms mode.

On-prem renderingThe chart renders exactly the six transform jobs and the required RBAC / service account — and none of the cloud-only workloads.

No regressionsThe cloud (AWS) deployment renders byte-for-byte unchanged, and both platforms pass helm lint.