corvus
ADR 0004 accepted relates to ADR 0001, ADR 0002

Local storage with backups to the NAS

Context

ADR 0001 chose Longhorn as the only storage class, replicated across the five x86 nodes, with the Synology NAS as the NFS backup target. The reasons were that a NAS reboot must not stall DNS, the UniFi controller and Home Assistant, and that k3s local-path pins every stateful service to one node. ADR 0002 followed with a longhorn logical volume on every node and open-iscsi and nfs-utils in the Ansible package set.

Nothing of that has landed in the cluster yet: at the time of writing the unattended image is in review and platform/cluster/ is empty. Before the first node is flashed, the operator revisited the choice against the hardware it runs on and the state it has to hold.

The nodes are weak: the ThinkCentre M715q has 8 GB of RAM and an AMD A10 APU, and a k3s server with embedded etcd already takes about a gigabyte. Longhorn adds a manager, an instance manager and one engine and replica process per volume on every node, and with a dozen volumes that is another gigabyte or more per node. Writes go synchronously to three replicas over 1 GbE, which caps throughput well below the NVMe and adds latency to every fsync; Prometheus and Loki, the heaviest writers in the cluster, are the workloads Longhorn’s own documentation steers to local disk. The nodes run Arch, so every kernel update reboots them, and every reboot degrades the replicas on that node and triggers a rebuild.

The state to hold is small and mostly recoverable at the application level: pihole’s configuration and query database, the UniFi controller’s MongoDB, Home Assistant’s configuration and recorder, mosquitto’s persistence, and the databases of bookmarks, home and spacebot. hassio is already pinned to a node by ADR 0001 for mDNS. The monitoring stores are the largest volumes and the least valuable.

Decisions

local-path is the only storage class

The k3s built-in local-path provisioner is the default and only storage class. A volume is a directory under /var/lib/corvus/volumes on the node where the pod first ran, and the pod stays on that node for as long as the volume exists. The provisioner costs nothing in memory, CPU or network, and ships with k3s.

Longhorn was considered and is dropped: on these nodes its overhead is paid by every workload, replicated or not, and its rebuild traffic and failure modes are the noisiest part of a cluster this size, for a mobility that two or three volumes would use.

synology-csi was considered and rejected again, for the reason ADR 0001 gave: it moves every stateful pod’s failure domain to the NAS, and sqlite databases over NFS are unreliable. It remains an option for bulk, low-write data that already lives on the NAS, such as the recordings of nvr if that service ever unparks, as a secondary class declared when that day comes.

Backups to the NAS, per volume, with restic

Every persistent volume is backed up to the Synology NAS over NFS with restic, through a backup operator installed in platform/cluster/ (VolSync is the candidate; the milestone settles the tool). The backup object lives next to the workload in the service’s chart, so a service declares its own retention and interval. Databases with a write stream (the UniFi MongoDB, the Home Assistant recorder) get a short interval, in the order of an hour; everything else is daily. Retention is a week of dailies and a few weeklies. Monitoring stores are not backed up.

The NAS is only a backup target. A NAS reboot or a DSM upgrade delays a backup; it stalls no service. The restic repositories are encrypted with a key held in a Kubernetes Secret and in the operator’s password manager, so the NAS holds no plaintext.

A restore is how a volume moves

A local-path volume has fixed node affinity and cannot be rescheduled. A service moves to another node, planned or after a node loss, by restore:

  1. Take a fresh backup if the source node is alive.
  2. Scale the workload to zero and delete the claim; the volume goes with it.
  3. Let the scheduler place the workload, or pin it, and let the chart recreate the claim on the new node.
  4. Restore the latest snapshot into the new claim and scale back up.

The cost of a planned move is the restore time. The cost of a node loss is the restore time plus whatever changed since the last backup, bounded by that service’s interval. The runbook for the move is written with the milestone.

The data layer no longer encrypts

ADR 0002 declined disk encryption on the grounds that sensitive data is protected at the data layer by Longhorn’s encrypted volumes. With local-path the volumes are plain directories, so data at rest on a node is not encrypted. The threat ADR 0002 weighed, a drive stolen from a home desktop, is accepted as is: etcd encryption at rest still covers the Secrets, the backups are encrypted by restic, and the services’ own data (DNS queries, the UniFi inventory, home sensor history) is not worth a LUKS layer that an unattended reboot cannot unlock without the TPM2 work that ADR already set aside.

What changes in the nodes

The longhorn logical volume of ADR 0002 becomes data, mounted at /var/lib/corvus, with the same size and filesystem. The reason for a separate volume stands: a full data disk cannot wedge the OS. open-iscsi leaves the Ansible package set; nfs-utils stays, for the backup target.

Consequences