Repo split and cluster shape for the corvus k3s migration

Context

The s5p8/infra GitLab group holds 12 projects, each a docker-compose stack run as a systemd unit with a watchdog, glued together by shared docker networks (nginx-proxy, glances, events) and ./data bind mounts on a single host (a ThinkPad T430 running Omarchy). Those services are moving to a k3s cluster under a new identity, corvus, with the domain corvus.observer held in Cloudflare.

The question was how to split the pieces needed to build and run that cluster across git repos, while keeping the rule that each service is its own namespace with its own chart and its own deployment cycle.

Decisions

One group, one repo per concern

One GitLab group, corvus. platform is the base of everything, calqued from patagoniamove/platform and throwlines/platform: it holds what the cluster needs before any service exists and what no service may install for itself. Every namespace is its own repo, with the same name as today, transferred from s5p8/infra one at a time as each service cuts over, history kept. A service repo carries no infrastructure code: a DNS record, a tunnel hostname or a CI variable is declared in platform, which is the single place where “this service exists in the cluster” and “this service is public” are written down.

The layout of a service repo

Every service repo has the same shape, so a chart, a deploy and a CI job are found in the same place in all of them:

chart/                hand-written Helm chart; values.yaml has no secrets
deploy/
  helmrelease.yaml    Flux HelmRelease
  secrets.enc.yaml    SOPS-encrypted Secrets
  kustomization.yaml
.sops.yaml            age public key
.gitlab-ci.yml        lint + helm template on MR; build/push image on main when a Dockerfile exists
.ahoy.yml             operator helpers

The namespace is created by the service’s chart and named after the repo.

Retired and parked

revproxy and glances are retired: Traefik’s ingress replaces the first, kube-prometheus-stack replaces the second and its dashboards move to the services that own them. nvr and ollama are parked, out of scope for now: the first needs a GPU, NFS and /mnt/purple, the second has no hardware to run on.

Nodes

Three servers because etcd needs an odd quorum; two is strictly worse than one. Every node runs workloads; “server” only adds the control plane.

Parked: four Raspberry Pi 4 (arm64, would join as tainted edge agents, no Longhorn while on SD cards), MacBook Air 2013 (would join as a plain x86 agent once wired), Raspberry Pi 2B (never, armv7).

Deployment

Pull-based with FluxCD. Merging to main in a service repo is the deploy. Chosen over Argo CD because HelmRelease runs real Helm, SOPS decryption is native, the one-GitRepository-per-repo model matches the namespace-per-repo rule, and image automation closes the loop for repos that build images without a runner on the LAN.

Upstream image tags are pinned in values.yaml and bumped by Renovate, configured once in platform.

Secrets

• Tofu provider tokens: platform/.env.infra, git-ignored, local applies.
• Runtime secrets: SOPS + age, committed encrypted in each service repo, decrypted in-cluster by Flux. The age private key lives in flux-system and in the operator’s password manager; the public key in every .sops.yaml.
• GitLab CI variables: tofu remains the single writer. Only registry push credentials for repos that build images remain.

Storage

Longhorn as the only storage class, replicated across the five x86 nodes, with the Synology NAS as the NFS backup target. Chosen over synology-csi so that a NAS reboot does not stall DNS, the UniFi controller and Home Assistant. k3s local-path was considered but it pins every stateful service to one node.

Networking and exposure

• LAN-only by default. Services get name.corvus.observer, resolved by pihole (wildcard to Traefik’s MetalLB IP). Real TLS from cert-manager via Cloudflare DNS-01, so public DNS never learns internal names. The fake .s5p8 TLD is gone.
• Tailscale (the official Kubernetes operator, as a subnet router Connector) is the admin entry point to the LAN and the cluster.
• Cloudflare Tunnel (cloudflared in-cluster) is opt-in per service, declared in platform/infra, with Cloudflare Access in front.
• MetalLB in L2 mode with a ten-address reservation on the main LAN, excluded from DHCP (set by hand on day one, owned by tofu once network is in the cluster). Nodes and the range share one VLAN. Traefik, pihole (53), mosquitto (1883) and UniFi inform/STUN each get a fixed IP.
• Cluster nodes resolve DNS through the router or 1.1.1.1, never through pihole, to avoid the circular dependency.
• hassio is the only hostNetwork pod (mDNS discovery), pinned by node selector. dbus/Bluetooth passthrough is dropped.

Monitoring

platform/cluster/ installs kube-prometheus-stack and Loki + Alloy. Grafana is LAN-only. The per-service contract is a ServiceMonitor and a dashboard ConfigMap with the Grafana sidecar label, both in the service’s chart. Logs need nothing from services. Alert routing is deferred.

Migration order

1. platform: ISO, Ansible, k3s on the five boxes, Flux, Longhorn, MetalLB, cert-manager, Traefik, monitoring. Verified with a throwaway whoami ingress.
2. tunnels: lowest risk, and unlocks remote admin.
3. events: proves MetalLB on a non-HTTP port.
4. bookmarks, home, spacebot: prove Longhorn and Ingress with state.
5. network: proves the UDP LoadBalancer. Old controller stays up until the APs are healthy on the new one, then set-inform and shut it down. Then tofu import the LAN into platform/infra.
6. hassio.
7. dns, last: new pihole on its MetalLB IP in parallel, test, flip the router’s DHCP DNS, keep the old one a day.
8. The Omarchy T430 joins the cluster.

Each step transfers that project from s5p8/infra to corvus when it starts.

Consequences

• platform is the single place where “this service exists in the cluster” and “this service is public” are declared. Service repos stay self-contained and portable.
• Every chart must use multi-arch images or set a node selector if the Pis ever join.
• Losing the age key means re-encrypting every secret. It is backed up in the password manager.
• One server down costs nothing. The NAS down costs only backups.
• The design assumes a wired, single-VLAN LAN for the nodes. Announcing a service IP inside another VLAN would need a tagged interface on a node.