Shut down NetLiNQ without breaking it.

Pulling power on a running cluster leaves databases mid-write and pods wedged on nodes that never drained. The lifecycle service does that work deliberately: it records how many replicas every workload has, scales them to zero in dependency order, drains the nodes, then stops kubelet. On the next boot it reverses all of it.

Two systemd units, both backed by scripts in /opt/netlinq/installation/:

Day to day nothing calls these by hand — a normal sudo reboot or sudo shutdown -h now runs the shutdown hook, and the restore runs itself on the way back up. To drive them directly:

# scale the platform down and drain the nodes
$ sudo systemctl stop netlinq-shutdown.service

# bring it back: uncordon and restore every replica count
$ sudo systemctl restart netlinq-startup.service

# re-arm the shutdown hook for the next reboot
$ sudo systemctl start netlinq-shutdown.service

$ sudo /opt/netlinq/installation/shutdown_netlinq.sh
$ sudo /opt/netlinq/installation/startup_netlinq.sh

The shipped netlinq_onprem_install.vars sets enable_netlinq_shutdown_service=true, so a stock install enables and starts both units for you. Drop the variable entirely and the installer falls back to leaving them off.

enable_netlinq_shutdown_service="true"

$ sudo systemctl enable --now netlinq-startup.service netlinq-shutdown.service

Both paths set the two kubelet shutdown grace periods to 0s, so kubelet hands the shutdown over instead of running its own partial eviction, and write persistent IPv6 settings for physical, bond, bridge and VLAN uplinks.

Single node

$ sudo bash installation/configure_host_recovery.sh --all

It restarts kubelet only if the config actually changed. Run it before enabling the units.

Multi-node cluster

Install and enable the units on every node first. Then run the migration from a control-plane host where kubectl get nodes works for your user.

$ cd installation
$ source kubespray-venv/bin/activate
$ kubectl get nodes
$ ansible-playbook -i kubespray-repo/inventory/mycluster/hosts.yaml \
    migrate_kubelet_shutdown.yml --ask-become-pass

On a master node, in this order:

The unit allows 1800s (TimeoutStopSec) while the script budgets 1500s for itself, so it winds down on its own terms instead of being killed part-way through a drain.

The restore is allowed to take a while: up to 600s waiting on databases and another 600s on the remaining workloads, inside a 2400s TimeoutStartSec. On failure it retries up to three times in three hours, then stops. The unit is deliberately not masked to success — a failed restore stays visible in systemctl status and no longer affects the shutdown hook.

Both live in /opt/netlinq/installation/. Until the snapshot is consumed, a later shutdown merges it with live counts, so a workload still sitting at zero keeps its pending count instead of being recorded as zero.

{
  "netlinq": {
    "shutdown_order": [
      "diesel", "elastic", "keycloak",
      "kibana", "mongo-tenant", "postgresql"
    ]
  },
  "kafka": {
    "operator_deployments": [
      "strimzi-cluster-operator", "netlinq-cluster-operator"
    ]
  }
}

A new database with dependents belongs in .netlinq.shutdown_order. Leave it out and it stops in arbitrary order among the other workloads — the journal warns by name when that happens.

Commit the current state without shutting anything down:

$ sudo /opt/netlinq/installation/shutdown_netlinq.sh --sync-only

This is the old runbook — what the service now does for you. Use it when the units are not installed, or when you want the platform down without stopping the machine.

$ kubectl -n netlinq get deploy,sts \
    -o custom-columns=KIND:.kind,NAME:.metadata.name,REPLICAS:.spec.replicas \
    > replicas-before.txt

$ kubectl -n kafka scale deploy strimzi-cluster-operator \
    netlinq-cluster-operator --replicas=0

$ kubectl -n netlinq get hpa -o yaml > hpa-backup.yaml
$ kubectl -n netlinq delete hpa --all

# order: diesel, elastic, keycloak, kibana, mongo-tenant, postgresql
$ kubectl -n netlinq get deploy,sts   # check each one's kind
$ kubectl -n netlinq scale deployment/diesel --replicas=0
$ kubectl -n netlinq rollout status deployment/diesel --timeout=120s

$ kubectl -n netlinq scale deploy --all --replicas=0
$ kubectl -n netlinq scale sts --all --replicas=0

To bring it back, reverse the order: databases first, then the applications, then the operators, then re-apply hpa-backup.yaml. Restore each workload to the count in replicas-before.txt, not to 1.

Disabling the shutdown unit makes the next reboot an ordinary one: no scale-down, no drain, no snapshot, nothing to restore.

$ sudo systemctl disable --now netlinq-shutdown.service netlinq-startup.service

Set enable_netlinq_shutdown_service="false" in the vars file as well, so a later install or upgrade does not quietly arm it again.

$ systemctl status netlinq-shutdown.service netlinq-startup.service kubelet
$ cat /opt/netlinq/installation/replicas.json

Both scripts log to the systemd journal, so the record survives the reboot they were running for:

$ journalctl -u netlinq-shutdown.service -f
$ journalctl -u netlinq-startup.service -b