Running stateful applications on Kubernetes means making real decisions about storage. The cluster ends up with three distinct storage tiers, each solving a different problem.
Rook-Ceph provides distributed block storage across the three nodes. It handles the bulk of application state — databases, configuration, anything that needs a ReadWriteOnce PVC. The operational model is solid once it is running, but Ceph has a reputation for complexity that is not entirely undeserved. Initial cluster setup and the occasional OSD failure require more attention than most other components. The trade-off is genuine resilience: the cluster can lose a node and keep running without losing data.
NFS handles media. The NAS exports a single large share that Plex, Radarr, Sonarr, Jellyfin, and the rest of the media stack read from. This is the pragmatic choice — media files are large, sequential, and already living on the NAS. Pulling them into Ceph would be wasteful. The NFS PVC is mounted ReadWriteMany by whatever applications need it.
VolSync is the most interesting piece. It handles PVC backups by running a restic-based mover pod that snaps a PVC and pushes the backup to an S3-compatible destination (MinIO, in this case). Restoring from backup or rolling back to a previous point-in-time is handled by the same tooling, driven by a task command. The nice property is that this is entirely Kubernetes-native — there are no cron jobs or external backup agents involved.
The VolSync setup is expressed as a shared Flux component. Any application can opt into backups by adding a single component reference in its ks.yaml. This makes it easy to ensure new applications get backed up by default without duplicating configuration.
One subtlety worth calling out: VolSync mover pods run as a specific UID, and applications that share NFS storage need to run as the same UID for file ownership to work out correctly. Getting this wrong produces permission errors that can take a while to trace back to their root cause.