Ceph

  As much as I love Proxmox, if it is anything, it is basic. This is a double edge sword of course, it is rock solid and stable (foreshadowing), and it is also exceptionally basic with few features I want, especially as a Kubernetes user.

K10, when creating a PVC to copy data from, will only use the parent PVCs settings and as a result must create a PVC with the same access mode. This means if you backup a RWX or RWO PVC K10 will create a clone with the same access mode, and there’s the rub. Normally this would be fine but the way the Ceph CSI handles this is a little different than other CSIs (I think). When the new PVC is created based on the snapshot of another PVC, and the access mode is set to RWX or RWO, Ceph CSI will then clone the volume entirely, this causes Ceph to attempt to duplicate the entire volume. So if you have a 3TB volume and you would like to backup the volume it must be cloned in full before it will become available to a pod. As one can imagine this causes massive delays in backups and requires all the storage to be duplicated before it can be backed up.