Improving Cluster Backups On Ceph Backed Clusters
Summary
In my previous post “Backup Server Woes” I finally setup an S3 backup box, with some trials and tribulations of course. Sadly through this process I learned a couple things about my backup strategy I was currently using and they weren’t good. Previously I was using K10, K10 has a nice little web UI I can proxy through Rancher MCM, which makes management nice as the UI never gets exposed other than through the Rancher UI.
So whats wrong with K10? Technically there’s not really anything wrong with K10, but the combination of K10 + Ceph seems to be the problem. K10 creates a PVC to offload data to your destination, which isn’t abnormal, but its the way it does this that confuses and annoys me.
K10, when creating a PVC to copy data from, will only use the parent PVCs settings and as a result must create a PVC with the same access mode. This means if you backup a RWX or RWO PVC K10 will create a clone with the same access mode, and there’s the rub. Normally this would be fine but the way the Ceph CSI handles this is a little different than other CSIs (I think). When the new PVC is created based on the snapshot of another PVC, and the access mode is set to RWX or RWO, Ceph CSI will then clone the volume entirely, this causes Ceph to attempt to duplicate the entire volume. So if you have a 3TB volume and you would like to backup the volume it must be cloned in full before it will become available to a pod. As one can imagine this causes massive delays in backups and requires all the storage to be duplicated before it can be backed up.
The solution here is rather simple, create a PVC based on the snapshot but set the access mode to ROX, this will prevent Ceph from attempting to duplicate the data and you now have essentially a “linked” clone as opposed to a “full” clone. This does not appear to be possible in K10 from what I have seen, but Velero can. Sadly improving cluster backups means abandoning K10 for Velero.
Updating Ceph CSI
This can happen in any order but you will need to update you Ceph RBD CSI storage class to add ’noload’ to ‘mountOptions’ (if you have not already) before attempting to backup any applications. The updated ’noload’ option is required to mount the ROX volumes, example below.
RBD StorageClass:
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd-sc
annotations:
storageclass.kubernetes.io/is-default-class: true
provisioner: rbd.csi.ceph.com
parameters:
clusterID: <redacted>
pool: <redacted>
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: <redacted>
csi.storage.k8s.io/provisioner-secret-namespace: <redacted>
csi.storage.k8s.io/controller-expand-secret-name: <redacted>
csi.storage.k8s.io/controller-expand-secret-namespace: <redacted>
csi.storage.k8s.io/node-stage-secret-name: <redacted>
csi.storage.k8s.io/node-stage-secret-namespace: <redacted>
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- discard
- noload
The docs cover it well enough but don’t forget to annotate your ‘VolumeSnapshotClass’ for Velero, example below.
RBD VolumeSnapshotClass:
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-rbdplugin-snapclass
annotations:
velero.io/csi-volumesnapshot-class: "true"
driver: rbd.csi.ceph.com
parameters:
clusterID: <redacted>
csi.storage.k8s.io/snapshotter-secret-name: <redacted>
csi.storage.k8s.io/snapshotter-secret-namespace: <redacted>
deletionPolicy: Delete
Deploying Velero
Velero sadly tries hard to push their custom CLI, and as a result that is also how they try to push the installation path. To be clear Velero has the CLI installation path and a Helm chart so you are not forced to go one route or the other, just keep in mind the CLI and the deployed Velero versions could differ enough to cause problems. For reference: I decided to go the Helm route. The example config is a stripped down version of my config but should contain the relevant details:
deployNodeAgent: true
configuration:
uploaderType: kopia
features: EnableCSI
defaultSnapshotMoveData: true
volumeSnapshotLocation:
- name: csi
provider: csi
nodeAgent:
extraArgs:
["--node-agent-configmap=backuppvc"]
The primary fields to take note of are features, and extraArgs. The features section enables CSI based backups (as the name implies), and the “extraArgs” points the node agents to a ConfigMap named “backuppvc”. The “backuppvc” ConfigMap will define what modifications will be made to the second PVC that is created based on the snapshot. In the example below I re-use the same PVC but change some of the options upon creation. For example, by setting “readOnly” to true the new PVCs become read only and that prevents the full clone process form starting. There is also a “spcNoRelabeling” flag that is set is required due to the fact I am running an SELinux enabled cluster.
kind: ConfigMap
metadata:
name: backuppvc
namespace: velero
apiVersion: v1
data:
packuppvc.json: |
{
"backupPVC": {
"csi-rbd-sc": {
"storageClass": "csi-rbd-sc",
"readOnly": true,
"spcNoRelabeling": true
},
"csi-cephfs-sc": {
"storageClass": "csi-cephfs-sc",
"readOnly": true,
"spcNoRelabeling": true
}
}
}
With all this enabled I am finally able to deploy my new backup solution which is significantly improved.