k8s-snap

k8s-snap

Features

• Take a snapshot of all k8s resources which has verbs 'list', 'get', 'create' and 'delete'.
• Restore k8s resources from a snapshot on any k8s cluster.
• Select restoring resources according to 'excludeApiPathes' and 'excludeNamespaces' in preference.
• Backup data stored on S3.
• Run on a k8s with CRDs.

Restoring ditails

• Restore resources basically by 'create', not by 'update'.
• Restore apps(deployments, statefulsets, daemonsets) after other resources restored.
• Restore PV definitions and PV/PVC boundings for specified storageclasses.
• Do not restore token secrets, resources with owner references, endpoints with same name services.

TODO

• Overwriting resources for specified api pathes.
• 'Include' contexts in preference. Currently 'Exclude' only.

Options

$ /k8s-snap-controller \
--namespace=k8s-snap

Deploy

$ kubectl apply -f artifacts/crd.yaml
$ kubectl apply -f artifacts/namespace-rbac.yaml

Set access/secret key in artifacts/cloud-credential.yaml and create a secret.

apiVersion: v1
kind: Secret
metadata:
  namespace: k8s-snap
  name: k8s-snap-ap-northeast-1
data:
  accesskey: [base64 access_key]
  secretkey: [base64 secret_key]

$ kubectl apply -f artifacts/cloud-credential.yaml

Make a bucket for snapshots.
Set object store endpoint and bucket name in artifacts/objectstore-config.yaml and create a config.

apiVersion: clustersnapshot.rywt.io/v1alpha1
kind: ObjectstoreConfig
metadata:
  name: k8s-snap-ap-northeast-1
  namespace: k8s-snap
spec:
  region: ap-northeast-1
  endpoint: ap-northeast-1.amazonaws.com
  bucket: k8s-snap
  cloudCredentialSecret: k8s-snap-ap-northeast-1

$ kubectl apply -f artifacts/objectstore-config.yaml

Set image and registry key in artifacts/deploy.yaml and deploy.

$ kubectl apply -f artifacts/deploy.yaml

To take s snapshot

Create a snapshot resource

apiVersion: clustersnapshot.rywt.io/v1alpha1
kind: Snapshot
metadata:
  name: cluster01-001
  namespace: k8s-snap
spec:
  clusterName: cluster01
  kubeconfig: |
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN3akND...
        server: https://cluster01.kubernetes.rywt.io:6443
      name: cluster
    contexts:
    - context:
        cluster: cluster
        user: remote-user
      name: context
    current-context: context
    kind: Config
    preferences: {}
    users:
    - name: remote-user
      user:
        token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVz....
  ttl: 720h
  availableUntil: 2020-07-01T02:03:04Z

Snapshot status

$ kubectl get snapshots.clustersnapshot.rywt.io -n k8s-snap
NAME             CLUSTERNAME   BACKUPDATE             STATUS
cluster02-001    cluster02     2019-04-19T08:35:23Z   Completed
cluster02-002    cluster02     2019-04-22T05:04:35Z   InProgress
scluster02-001   sccluster02   2019-04-12T03:49:30Z   Completed

Completed snapshot status example

$ kubectl get snapshots.clustersnapshot.rywt.io -n k8s-snap scluster01-001 -o json | jq .status
{
  "availableUntil": "2019-06-19T03:45:08Z",     /*** Snapshot custom resource and data will be deleted on .. ***/
  "contents": [                                 /*** K8s resources backuped in snapshot ***/
    "/api/v1/namespaces/default/configmaps/kubelet-broken-pipe",
    "/api/v1/namespaces/default/endpoints/kubernetes",
    "/api/v1/namespaces/default/persistentvolumeclaims/test-pvc",
    "/api/v1/namespaces/default/secrets/default-token-gj2xd",
    :
  ],
  "numberOfContents": 429,                      /*** Number of backuped k8s resources in snapshot ***/
  "phase": "Completed",                         /*** Status of snapshot ***/
  "reason": "",
  "snapshotResourceVersion": "4521912",         /*** K8s ResourceVersion on which resources in snapshot synced ***/
  "snapshotTimestamp": "2019-05-20T03:45:08Z",  /*** Timestamp corresponding to the ResourceVersion ***/
  "storedFileSize": 138145,                     /*** File size on object store ***/
  "storedTimestamp": "2019-05-20T03:45:08Z"     /*** File timestamp on object store ***/
}

Failed snapshot status example

$ kubectl get snapshots.clustersnapshot.rywt.io -n k8s-snap scluster01-002 -o json | jq .status
{
  "availableUntil": null,
  "contents": null,
  "numberOfContents": 0,
  "phase": "Failed",
  "reason": "Unauthorized",      /*** Error message on snapshot failure including go library error message. Non predictable. ***/
  "snapshotResourceVersion": "",
  "snapshotTimestamp": null,
  "storedFileSize": 0,
  "storedTimestamp": null
}

To restore

Setup a restore preference

Edit artifacts/preference.yaml and create a preference.

• Currently only 'exclude' contexts are valid in preference.

Create a restore resource

apiVersion: clustersnapshot.rywt.io/v1alpha1
kind: Restore
metadata:
  name: cluster02-cluster01-001-001
  namespace: k8s-backup
spec:
  clusterName: cluster02
  snapshotName: cluster01-001
  restorePreferenceName: exclude-kube-system
  kubeconfig: |
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN3akND...
        server: https://cluster02.kubernetes.rywt.io:6443
      name: cluster
    contexts:
    - context:
        cluster: cluster
        user: remote-user
      name: context
    current-context: context
    kind: Config
    preferences: {}
    users:
    - name: remote-user
      user:
        token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVz....
  ttl: 168h
  availableUntil: 2020-07-01T01:02:03Z

• Set ttl with time.Duration format h/m/s. If not set, default to 168h0m0s(=7days).
• Spec.TTL will be ignored when Spec.AvailableUntil is set.

Restore status

$ kubectl get restores.clustersnapshot.rywt.io -n k8s-snap
NAME                            CLUSTER      SNAPSHOT         TIMESTAMP              RV        EXCLUDED   CREATED   UPDATED   EXIST     FAILED    STATUS
scluster02-scluster01-001-001   scluster02   scluster01-001   2019-05-20T03:46:15Z   8514809   50         46        0         13        0         Completed
scluster02-scluster01-002-001   scluster02   scluster01-002   <no value>                       0          0         0         0         0         Failed
scluster02-scluster01-001-002   scluster02   scluster01-001   <no value>                       0          0         0         0         0         InProgress

Completed restore status example

$ kubectl get restores.clustersnapshot.rywt.io -n k8s-snap scluster02-scluster01-001-001 -o json | jq .status
{
  "alreadyExisted": [             /*** K8s resources existed and not tried to update ***/
    "/api/v1/namespaces/default",
    "/api/v1/persistentvolumes/pvc-4ae96dfc-625b-11e9-9d94-005056bc5df9",
    "/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/remote-user",
    "/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/logfilter-controller",
    :
  ],
  "created": [                   /*** Created k8s resources ***/
    "/api/v1/namespaces/fluent-bit",
    "/apis/rbac.authorization.k8s.io/v1/namespaces/default/rolebindings/clusterrolebinding-dtwx4",
    "/apis/rbac.authorization.k8s.io/v1/namespaces/default/rolebindings/clusterrolebinding-vmsqk",
    "/api/v1/namespaces/fluent-bit/configmaps/cattle-agent-no-such-host",
    :
  ],
  "excluded": [                  /*** K8s resources excluded in restoring by some reason - resource-path,(reason) ***/
    "/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/canal-calico,(not-binded-to-ns)",
    "/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/canal-flannel,(not-binded-to-ns)",
    "/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cattle-admin-binding,(not-binded-to-ns)",
    "/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin,(not-binded-to-ns)",
    :
  ],
  "failed": null,                /*** K8s resources tried to create but failed - resource-path,error-message(<300chars) ***/
  "numAlreadyExisted": 13,       /*** Number of existed and not tried to update ***/
  "numCreated": 46,              /*** Number of created ***/
  "numExcluded": 50,             /*** Number of excluded in restoring by some reason ***/
  "numFailed": 0,                /*** Number of tried to create but failed ***/
  "numPreferenceExcluded": 319,  /*** Number of excluded in preference ***/
  "numSnapshotContents": 429,    /*** Number of k8s resources in the snapshot ***/
  "numUpdated": 0,               /*** Number of updated ***/
  "phase": "Completed",          /*** Status of restore ***/
  "preserveUntil": "2019-05-27T03:46:15Z",     /*** Restore custom resource will be deleted on .. ***/
  "reason": "",                  /*** Error message on restore failure including go library error message. Non predictable. ***/
  "restoreResourceVersion": "8514809",         /*** K8s ResourceVersion at restore finished ***/
  "restoreTimestamp": "2019-05-20T03:46:15Z",  /*** Timestamp corresponding to the ResourceVersion ***/
  "updated": null                /*** Updated k8s resources ***/
}

Failed restore status example

$ kubectl get restores.clustersnapshot.rywt.io -n k8s-snap scluster02-scluster01-002-001 -o json | jq .status
{
  "alreadyExisted": null,
  "created": null,
  "excluded": null,
  "failed": null,
  "numAlreadyExisted": 0,
  "numCreated": 0,
  "numExcluded": 0,
  "numFailed": 0,
  "numPreferenceExcluded": 0,
  "numSnapshotContents": 0,
  "numUpdated": 0,
  "phase": "Failed",
  "preserveUntil": null,
  "reason": "Snapshot data is not in status 'Completed'",   /*** Error message on restore failure including go library error message. Non predictable. ***/
  "restoreResourceVersion": "",
  "restoreTimestamp": null,
  "updated": null
}

To delete snapshot

Snapshot resources and files on object store automatically deleted when TTL expired.
You can delete a snapshot manually with:

$ kubectl delete snapshots.clustersnapshot.rywt.io -n k8s-snap cluster01-001

and also the corresponding file on object store automatically deleted.