Redeploying the Home Lab (Part 2): Enabling K10 Backups on Longhorn
-
4 mins read
Series: Redeploying the Home Lab
Summary
As discussed in the previous post I set out with redeploying my cluster and intended on enabling my cluster with backups. This time around I wanted to try out Kasten K10 (from Veeam), I fully admit at times all I want is a GUI. I just want to take a quick peak and make sure everything is good to go. This is why this time around I decided to give K10 a shot, Velero works well enough but the eye candy caught my attention.
Setup
When deploying K10 it needs to know what snapshot classes are available for use, this is set with an annotation like so:
kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
metadata:
name: longhorn
annotations:
k10.kasten.io/is-snapshot-class: "true"
driver: driver.longhorn.io
deletionPolicy: Delete
This is reflected in their documentation here, however as always things would not be that simple. I tried running a backup manually and produced the following errors below:
{"Status":"Failed","LastUpdateTime":"2024-11-08T15:49:38Z","Errors":["Failed to create Snapshot: CSI Driver failed to create snapshot for PVC (kubestr-csi-original-pvcdq7v4) in Namespace (k10): Failed to check and update snapshot content: failed to take snapshot of the volume pvc-0a15e1b9-c4d0-4efa-9a13-03025d605532: \\"rpc error: code = Internal desc = proxyServer=10.42.2.64:8501 destination=10.42.2.64:10609: failed to backup snapshot snapshot-55583a14-4b2a-4243-bea9-8d043d940b91 to backup-e3c19f1609cc48cd: rpc error: code = Internal desc = failed to create backup: failed to create backup to for volume pvc-0a15e1b9-c4d0-4efa-9a13-03025d605532: rpc error: code = Unknown desc = missing input parameter\\""]}
kind: BackupAction
apiVersion: actions.kio.kasten.io/v1alpha1
metadata:
name: manualbackup-vbvt8
namespace: homebox
uid: 8b2c9044-9e07-11ef-abf2-9af4ec3228fc
resourceVersion: "58"
creationTimestamp: 2024-11-08T19:27:50Z
labels:
k10.kasten.io/appName: homebox
k10.kasten.io/appNamespace: homebox
status:
state: Failed
startTime: 2024-11-08T19:27:50Z
endTime: 2024-11-08T19:31:40Z
restorePoint:
name: ""
result:
name: ""
error:
cause: '{"cause":{"cause":{"cause":{"cause":{"file":"github.com/kanisterio/kanister@v0.0.0-20241025011050-227a815a2c61/pkg/kube/snapshot/snapshot_beta.go","function":"github.com/kanisterio/kanister/pkg/kube/snapshot.isReadyToUseBeta","linenumber":316,"message":"Failed
to check and update snapshot content: failed to take snapshot of the
volume pvc-0fef3e68-af02-43d4-a20d-bbf8f9afb498: \"rpc error: code =
Internal desc = proxyServer=10.42.2.64:8501 destination=10.42.2.64:10170:
failed to backup snapshot snapshot-f4568f5d-3ead-4932-8dd9-54e839716caf to
backup-d42cb1c2d1534494: rpc error: code = Internal desc = failed to
create backup: failed to create backup to for volume
pvc-0fef3e68-af02-43d4-a20d-bbf8f9afb498: rpc error: code = Unknown desc =
missing input
parameter\""},"fields":[{"name":"PVCName","value":"homebox"},{"name":"Namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/phase/snapshot.go:497","function":"kasten.io/k10/kio/exec/phases/phase.CsiSnapshot","linenumber":497,"message":"CSI
Driver failed to create
snapshot"},"fields":[{"name":"pvcName","value":"homebox"},{"name":"namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:886","function":"kasten.io/k10/kio/exec/phases/backup.basicVolumeSnapshot.basicVolumeSnapshot.func1.func2","linenumber":886,"message":"Error
snapshotting
volume"},"fields":[{"name":"appName","value":"homebox"},{"name":"appType","value":"deployment"},{"name":"namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:897","function":"kasten.io/k10/kio/exec/phases/backup.basicVolumeSnapshot","linenumber":897,"message":"Failed
to snapshot
volumes"},"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:407","function":"kasten.io/k10/kio/exec/phases/backup.processVolumeArtifacts","linenumber":407,"message":"Failed
snapshots for workload"}'
message: Job failed to be executed
actionDetails:
phases:
- attempt: 3
endTime: 2024-11-08T19:31:40Z
errors:
- cause: '{"cause":{"cause":{"cause":{"cause":{"file":"github.com/kanisterio/kanister@v0.0.0-20241025011050-227a815a2c61/pkg/kube/snapshot/snapshot_beta.go","function":"github.com/kanisterio/kanister/pkg/kube/snapshot.isReadyToUseBeta","linenumber":316,"message":"Failed
to check and update snapshot content: failed to take snapshot of
the volume pvc-0fef3e68-af02-43d4-a20d-bbf8f9afb498: \"rpc error:
code = Internal desc = proxyServer=10.42.4.245:8501
destination=10.42.2.64:10170: failed to backup snapshot
snapshot-0309e35d-b0cd-4ff8-bd84-44b7176a6ae4 to
backup-3ff26f325e884e8b: rpc error: code = Internal desc = failed
to create backup: failed to create backup to for volume
pvc-0fef3e68-af02-43d4-a20d-bbf8f9afb498: rpc error: code =
Unknown desc = missing input
parameter\""},"fields":[{"name":"PVCName","value":"homebox"},{"name":"Namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/phase/snapshot.go:497","function":"kasten.io/k10/kio/exec/phases/phase.CsiSnapshot","linenumber":497,"message":"CSI
Driver failed to create
snapshot"},"fields":[{"name":"pvcName","value":"homebox"},{"name":"namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:886","function":"kasten.io/k10/kio/exec/phases/backup.basicVolumeSnapshot.basicVolumeSnapshot.func1.func2","linenumber":886,"message":"Error
snapshotting
volume"},"fields":[{"name":"appName","value":"homebox"},{"name":"appType","value":"deployment"},{"name":"namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:897","function":"kasten.io/k10/kio/exec/phases/backup.basicVolumeSnapshot","linenumber":897,"message":"Failed
to snapshot
volumes"},"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:407","function":"kasten.io/k10/kio/exec/phases/backup.processVolumeArtifacts","linenumber":407,"message":"Failed
snapshots for workload"}'
message: Job failed to be executed
- cause: '{"cause":{"cause":{"cause":{"cause":{"file":"github.com/kanisterio/kanister@v0.0.0-20241025011050-227a815a2c61/pkg/kube/snapshot/snapshot_beta.go","function":"github.com/kanisterio/kanister/pkg/kube/snapshot.isReadyToUseBeta","linenumber":316,"message":"Failed
to check and update snapshot content: failed to take snapshot of
the volume pvc-0fef3e68-af02-43d4-a20d-bbf8f9afb498: \"rpc error:
code = Internal desc = proxyServer=10.42.2.64:8501
destination=10.42.2.64:10170: failed to backup snapshot
snapshot-7b286c51-d08b-4827-9515-21ed17209e87 to
backup-a7d8b799efd44efb: rpc error: code = Internal desc = failed
to create backup: failed to create backup to for volume
pvc-0fef3e68-af02-43d4-a20d-bbf8f9afb498: rpc error: code =
Unknown desc = missing input
parameter\""},"fields":[{"name":"PVCName","value":"homebox"},{"name":"Namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/phase/snapshot.go:497","function":"kasten.io/k10/kio/exec/phases/phase.CsiSnapshot","linenumber":497,"message":"CSI
Driver failed to create
snapshot"},"fields":[{"name":"pvcName","value":"homebox"},{"name":"namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:886","function":"kasten.io/k10/kio/exec/phases/backup.basicVolumeSnapshot.basicVolumeSnapshot.func1.func2","linenumber":886,"message":"Error
snapshotting
volume"},"fields":[{"name":"appName","value":"homebox"},{"name":"appType","value":"deployment"},{"name":"namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:897","function":"kasten.io/k10/kio/exec/phases/backup.basicVolumeSnapshot","linenumber":897,"message":"Failed
to snapshot
volumes"},"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:407","function":"kasten.io/k10/kio/exec/phases/backup.processVolumeArtifacts","linenumber":407,"message":"Failed
snapshots for workload"}'
message: Job failed to be executed
- cause: '{"cause":{"cause":{"cause":{"cause":{"file":"github.com/kanisterio/kanister@v0.0.0-20241025011050-227a815a2c61/pkg/kube/snapshot/snapshot_beta.go","function":"github.com/kanisterio/kanister/pkg/kube/snapshot.isReadyToUseBeta","linenumber":316,"message":"Failed
to check and update snapshot content: failed to take snapshot of
the volume pvc-0fef3e68-af02-43d4-a20d-bbf8f9afb498: \"rpc error:
code = Internal desc = proxyServer=10.42.2.64:8501
destination=10.42.2.64:10170: failed to backup snapshot
snapshot-f4568f5d-3ead-4932-8dd9-54e839716caf to
backup-d42cb1c2d1534494: rpc error: code = Internal desc = failed
to create backup: failed to create backup to for volume
pvc-0fef3e68-af02-43d4-a20d-bbf8f9afb498: rpc error: code =
Unknown desc = missing input
parameter\""},"fields":[{"name":"PVCName","value":"homebox"},{"name":"Namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/phase/snapshot.go:497","function":"kasten.io/k10/kio/exec/phases/phase.CsiSnapshot","linenumber":497,"message":"CSI
Driver failed to create
snapshot"},"fields":[{"name":"pvcName","value":"homebox"},{"name":"namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:886","function":"kasten.io/k10/kio/exec/phases/backup.basicVolumeSnapshot.basicVolumeSnapshot.func1.func2","linenumber":886,"message":"Error
snapshotting
volume"},"fields":[{"name":"appName","value":"homebox"},{"name":"appType","value":"deployment"},{"name":"namespace","value":"homebox"}],"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:897","function":"kasten.io/k10/kio/exec/phases/backup.basicVolumeSnapshot","linenumber":897,"message":"Failed
to snapshot
volumes"},"file":"kasten.io/k10/kio/exec/phases/backup/snapshot_data_phase.go:407","function":"kasten.io/k10/kio/exec/phases/backup.processVolumeArtifacts","linenumber":407,"message":"Failed
snapshots for workload"}'
message: Job failed to be executed
name: Snapshotting Application Components
startTime: 2024-11-08T19:27:51Z
state: failed
updatedTime: 2024-11-08T19:31:40Z
- attempt: 1
endTime: 2024-11-08T19:28:23Z
name: Snapshotting Application configuration
startTime: 2024-11-08T19:27:51Z
state: succeeded
updatedTime: 2024-11-08T19:28:23Z
- attempt: 1
endTime: 2024-11-08T19:27:51Z
name: Snapshotting Workload homebox
startTime: 2024-11-08T19:27:51Z
state: succeeded
updatedTime: 2024-11-08T19:27:51Z
progress: 100
spec:
subject:
name: homebox
namespace: homebox
filters: {}
expiresAt: 2024-11-15T19:27:23Z
profile:
name: backblaze-b2
namespace: k10
The errors are perfectly vague and offer little insight into what has actually gone wrong.
The Fix
After doing a fair bit of sleuthing I found a KB article that resolved the issue, longhorn snapshot classes need an extra paramter:
kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
metadata:
name: longhorn
annotations:
k10.kasten.io/is-snapshot-class: "true"
driver: driver.longhorn.io
deletionPolicy: Delete
parameters:
type: snap
This is only needed for K10 to work, snapshots work otherwise. I now finally am back to having working backups for my cluster!