Backup & Restore: K8up

Intro

K8up (pronounced "ketchup") is a Kubernetes backup operator based on restic. It backs up your PersistentVolumeClaims (PVCs) and application dumps into an S3-compatible bucket, such as Xelon Object Storage, and restores them on request.

K8up is managed through Kubernetes resources:

• Backup: Runs a one-off backup of the PVCs in a namespace.
• Schedule: Runs backups, checks and prunes regularly.
• Check: Verifies the integrity of the backup repository.
• Prune: Removes old snapshots according to your retention policy.
• Restore: Restores a snapshot into a PVC or an S3 bucket.
• Archive: Copies the latest snapshots into a second bucket, for example for long-term storage.

Prerequisites

1. Kubernetes cluster: A running Xelon Kubernetes cluster and a kubeconfig with access to it. See How to: create Kubernetes Clusters.
2. CLI tools: kubectl and helm (v3) installed on your machine. The restic CLI is optional, but we recommend it for browsing backups manually.
3. Object Storage bucket: A bucket in Xelon Object Storage (for example k8up) and an access key / secret key pair with read and write permissions on it. We recommend a dedicated key pair that is only used for backups.
4. Repository password: A strong password for the restic repository. K8up encrypts all backups with this password.

K8up Installation

1. Add the K8up Helm repository:

helm repo add k8up-io https://k8up-io.github.io/k8up
helm repo update

2. Create a values.yaml with your settings:

k8up:
  # Cron schedules are interpreted in this timezone (default: cluster timezone, usually UTC)
  timezone: Europe/Zurich
  # Set to true if only PVCs with the annotation k8up.io/backup: "true" should be backed up
  skipWithoutAnnotation: false

3. Install the CRDs. They are bundled with the Helm chart, but Helm only installs CRDs once and never upgrades them. Therefore we recommend applying them explicitly on every install and upgrade:

kubectl apply --server-side -f https://github.com/k8up-io/k8up/releases/download/v2.16.1/k8up-crd.yaml

4. Install or upgrade the K8up operator:

helm upgrade --install k8up k8up-io/k8up \
  --namespace k8up --create-namespace \
  --version 4.10.1 \
  -f values.yaml

The --version flag pins the chart, so every installation of your cluster uses the same K8up version. To upgrade later, update the CRD URL and the chart version together, and check the release notes for breaking changes.

Verify Installation

kubectl get pods -n k8up
kubectl get crds | grep k8up.io

The operator pod should be in the Running state, and you should see the K8up CRDs (backups.k8up.io, schedules.k8up.io, restores.k8up.io, …).

Configure the Backend

Every K8up resource has a backend block that tells K8up where the restic repository lives and how to access it. The credentials are referenced from Kubernetes Secrets.

1. Create the Secret with the repository password:

kubectl create secret generic backup-repo \
  --namespace <namespace> \
  --from-literal=password='<repository-password>'

2. Create the Secret with your Object Storage credentials:

kubectl create secret generic backup-credentials \
  --namespace <namespace> \
  --from-literal=access-key='<access-key>' \
  --from-literal=secret-key='<secret-key>'

The backend block used in all following examples looks like this:

backend:
  repoPasswordSecretRef:
    name: backup-repo
    key: password
  s3:
    endpoint: https://zh1-s3.xelon.ch
    bucket: k8up/backup
    accessKeyIDSecretRef:
      name: backup-credentials
      key: access-key
    secretAccessKeySecretRef:
      name: backup-credentials
      key: secret-key

The bucket value can contain a path (<bucket>/<folder>). This lets you keep several repositories in one bucket, for example one folder per cluster or namespace.

Your first Backup

A Backup resource starts a backup immediately. It backs up all PVCs in its namespace (see Selecting Volumes).

1. Create backup.yaml:

apiVersion: k8up.io/v1
kind: Backup
metadata:
  name: backup-test
  namespace: <namespace>
spec:
  failedJobsHistoryLimit: 2
  successfulJobsHistoryLimit: 2
  backend:
    repoPasswordSecretRef:
      name: backup-repo
      key: password
    s3:
      endpoint: https://zh1-s3.xelon.ch
      bucket: k8up/backup
      accessKeyIDSecretRef:
        name: backup-credentials
        key: access-key
      secretAccessKeySecretRef:
        name: backup-credentials
        key: secret-key

2. Apply it:

kubectl apply -f backup.yaml

K8up starts a backup job that mounts your PVCs and uploads their content to the bucket. On the first run, K8up initialises the restic repository automatically.

Verify Backup

kubectl get backups -n <namespace>
kubectl get pods -n <namespace>
kubectl logs -n <namespace> -l k8up.io/type=backup
kubectl get snapshots -n <namespace>

The backup is done as soon as its status shows Completed. kubectl get snapshots lists one snapshot per backed-up PVC. You need the snapshot ID to restore.

Scheduled Backups

For production use, define a Schedule instead of single backups. A Schedule creates Backup, Check and Prune jobs automatically.

apiVersion: k8up.io/v1
kind: Schedule
metadata:
  name: schedule
  namespace: <namespace>
spec:
  backend:
    repoPasswordSecretRef:
      name: backup-repo
      key: password
    s3:
      endpoint: https://zh1-s3.xelon.ch
      bucket: k8up/backup
      accessKeyIDSecretRef:
        name: backup-credentials
        key: access-key
      secretAccessKeySecretRef:
        name: backup-credentials
        key: secret-key
  backup:
    schedule: '@daily-random'
    failedJobsHistoryLimit: 2
    successfulJobsHistoryLimit: 2
  check:
    schedule: '@weekly-random'
  prune:
    schedule: '@weekly-random'
    retention:
      keepLast: 5
      keepDaily: 14
      keepWeekly: 8
      keepMonthly: 6

• backup: How often your data is backed up. Use standard cron syntax (e.g. '0 */6 * * *', see crontab.guru), or a randomized value.
• check: Verifies the repository. Once a week is usually enough.
• prune: Removes snapshots that fall outside the retention policy. In the example above, K8up keeps the last 5 snapshots, one per day for 14 days, one per week for 8 weeks and one per month for 6 months.

Selecting Volumes

By default, K8up backs up every PVC in the namespace of the Backup or Schedule. You can change this in several ways:

1. Exclude a PVC: Add the annotation k8up.io/backup: "false", for example to PVCs that only hold caches or restore targets.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cache
  annotations:
    k8up.io/backup: "false"

2. Opt-in only: Set k8up.skipWithoutAnnotation: true in your Helm values. K8up then only backs up PVCs with k8up.io/backup: "true".

3. Label selectors: Limit a Backup or Schedule to PVCs with specific labels:

spec:
  labelSelectors:
    - matchLabels:
        app: my-app

Application-Aware Backups

A file-level backup of a running database can be inconsistent. For databases, we recommend letting K8up create a dump instead. Add the annotation k8up.io/backupcommand to the pod template. K8up executes the command in the pod and stores its output (stdout) in the repository.

Example for PostgreSQL:

template:
  metadata:
    labels:
      app: postgresql
    annotations:
      k8up.io/backupcommand: sh -c 'PGDATABASE="$POSTGRES_DB" PGUSER="$POSTGRES_USER" PGPASSWORD="$POSTGRES_PASSWORD" pg_dump --clean'
      k8up.io/file-extension: .sql

• k8up.io/backupcommand: The command that writes the dump to stdout. The tool (here pg_dump) must be available in the container image.
• k8up.io/file-extension: The file extension of the dump inside the snapshot.
• k8up.io/backupcommand-container: Optional. Names the container to run the command in, if the pod has several containers.

To avoid backing up the database files twice, annotate the database PVC with k8up.io/backup: "false".

Restore

1. List the available snapshots and pick the ID of the snapshot you want to restore:

kubectl get snapshots -n <namespace> -o custom-columns=ID:.spec.id,DATE:.spec.date,PATHS:.spec.paths

The path shows which PVC a snapshot belongs to (/data/<pvc-name>).

2. Choose a restore method below.

Restore to a PVC

The target PVC must exist before you start the restore. If it is an RWO volume, make sure no pod on another node is using it.

apiVersion: k8up.io/v1
kind: Restore
metadata:
  name: restore-to-pvc
  namespace: <namespace>
spec:
  snapshot: <snapshot-id>
  podSecurityContext:
    fsGroup: 65532
    fsGroupChangePolicy: OnRootMismatch
  restoreMethod:
    folder:
      claimName: <target-pvc>
  backend:
    # same backend block as in your Backup / Schedule

The content of the snapshot is written to the root of the target PVC.

• snapshot: The snapshot ID. Instead, you can use restoreTimeFilter: "2026-11-04" (format YYYY-MM-DD hh:mm:ss, or any prefix of it) to restore the latest snapshot matching that time. If nothing matches, K8up silently falls back to the latest snapshot.
• podSecurityContext: If the restore fails with permission denied, set the fsGroup (and if necessary runAsUser) to the user your application runs as. Avoid running the restore as root unless your application requires it.

Restore to S3

Instead of a PVC, you can restore a snapshot as files into another bucket, for example to hand the data over or to inspect it locally:

apiVersion: k8up.io/v1
kind: Restore
metadata:
  name: restore-to-s3
  namespace: <namespace>
spec:
  snapshot: <snapshot-id>
  restoreMethod:
    s3:
      endpoint: https://zh1-s3.xelon.ch
      bucket: k8up-restore
      accessKeyIDSecretRef:
        name: restore-credentials
        key: access-key
      secretAccessKeySecretRef:
        name: restore-credentials
        key: secret-key
  backend:
    # same backend block as in your Backup / Schedule

Restore manually with restic

With the restic CLI, you can browse and restore backups directly from your machine:

export RESTIC_REPOSITORY=s3:https://zh1-s3.xelon.ch/k8up/backup
export RESTIC_PASSWORD='<repository-password>'
export AWS_ACCESS_KEY_ID='<access-key>'
export AWS_SECRET_ACCESS_KEY='<secret-key>'

restic snapshots                                   # list snapshots
restic restore <snapshot-id> --target ./restore    # restore into a local folder
restic dump <snapshot-id> /<namespace>-<pod>-<container>.sql > dump.sql   # extract an application dump
restic mount ./mnt                                 # browse all snapshots as a folder

Use restic ls <snapshot-id> to see the exact file names inside a snapshot.

Example: Back up Pod 1 and restore into Pod 2

This walkthrough copies the data of one pod's volume into the volume of another pod. It is also a good way to test your restore process.

1. Create the two pods, each with its own PVC. Label the PVC of pod-1 so the backup can select it, and exclude the PVC of pod-2 from backups:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pod-1-storage
  labels:
    app: pod-1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-1
spec:
  containers:
    - name: pod-1
      image: alpine:3.20
      command: ["/bin/sh", "-c", "sleep infinity"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: pod-1-storage

pod-2 looks the same, with the name pod-2, the PVC pod-2-storage, and the annotation k8up.io/backup: "false" on its PVC.

kubectl apply -f pod1.yaml -f pod2.yaml

2. Write some test data into the volume of pod-1:

kubectl exec pod-1 -- sh -c 'echo "hello from pod-1" > /data/test.txt'

3. Back up only the volume of pod-1:

apiVersion: k8up.io/v1
kind: Backup
metadata:
  name: backup-pod-1
spec:
  labelSelectors:
    - matchLabels:
        app: pod-1
  backend:
    # same backend block as above

kubectl apply -f backup-pod1.yaml
kubectl get backup backup-pod-1 -w

Wait until the backup is Completed.

4Get the ID of the new snapshot. Its path is /data/pod-1-storage:

kubectl get snapshots -o custom-columns=ID:.spec.id,DATE:.spec.date,PATHS:.spec.paths

5. Stop pod-2. Its RWO volume could otherwise be mounted on a different node than the restore pod:

kubectl delete pod pod-2

6. Restore the snapshot into pod-2-storage:

apiVersion: k8up.io/v1
kind: Restore
metadata:
  name: restore-pod-1-to-pod-2
spec:
  snapshot: <snapshot-id>
  podSecurityContext:
    fsGroup: 65532
    fsGroupChangePolicy: OnRootMismatch
  restoreMethod:
    folder:
      claimName: pod-2-storage
  backend:
    # same backend block as above

kubectl apply -f restore-pod1-to-pod2.yaml
kubectl get restore restore-pod-1-to-pod-2 -w

7. Start pod-2 again and check the data:

kubectl apply -f pod2.yaml
kubectl exec pod-2 -- cat /data/test.txt

You should see hello from pod-1.

Archive

An Archive restores the latest snapshot of each PVC from your backup repository into a second bucket. This gives you a copy of your data at another location or for long-term storage.

apiVersion: k8up.io/v1
kind: Archive
metadata:
  name: archive
  namespace: <namespace>
spec:
  # Omit "schedule" to run the archive once
  schedule: '@monthly-random'
  # Source: your existing backup repository
  backend:
    repoPasswordSecretRef:
      name: backup-repo
      key: password
    s3:
      endpoint: https://zh1-s3.xelon.ch
      bucket: k8up/backup
      accessKeyIDSecretRef:
        name: backup-credentials
        key: access-key
      secretAccessKeySecretRef:
        name: backup-credentials
        key: secret-key
  # Destination: where the archived files are written to
  restoreMethod:
    s3:
      endpoint: <archive-endpoint>
      bucket: k8up-archive
      accessKeyIDSecretRef:
        name: archive-credentials
        key: access-key
      secretAccessKeySecretRef:
        name: archive-credentials
        key: secret-key

Monitoring

K8up exposes Prometheus metrics, for example the number of failed backups per namespace. If you run the Prometheus Operator, enable the following in your values.yaml:

metrics:
  serviceMonitor:
    enabled: true
  prometheusRule:
    enabled: true

The PrometheusRule includes default alerts for failed jobs. We recommend at least alerting on failed backups. A backup you don't monitor is a backup you can't rely on.

Best Practices

• Store the repository password outside the cluster. Without it, your backups are worthless.
• Follow the 3-2-1 rule. Keep at least one copy of your data at a different location, for example with an Archive.
• Use least-privilege keys. Use separate Object Storage keys for backups, restores and archives.
• Test your restores regularly, for example with the walkthrough above.
• Use application-aware backups for databases instead of file backups.
• Pin versions. Pin the Helm chart and the CRD version, and upgrade them together.
• Keep Secrets out of Git, or encrypt them.

Need help?

If you have questions about K8up on Xelon Kubernetes or need support, contact our support team.