generate

Backup & Restore procedure Generator

Purpose of this generator is to create procedures from templates for both Backup and Restore operations to use in automated way. Backup made using generated procedure should be possible to restore with a restore procedure in automated way.

The generator is having two output formats:

• shell script
• Kubernetes-like kind: Job and kind: Pod

Commands

backup-maker procedure backup

Generates a bash script ready for automated usage with crontab or Kubernetes kind: Cronjob.

Example usage with Kubernetes:

# for definition.yaml example see yaml file in Usage Concept
backup-maker procedure backup \
		--definition=./definition.yaml \ 
		--template postgres \
		--kubernetes \
		--gpg-key-path valid-sealed-secret.yaml \
		--output-dir=backup.yaml

Will create a backup.yaml with Kubernetes resources:

• kind: Cronjob
• kind: Secret
• kind: SealedSecret
• kind: ConfigMap

SealedSecrets

Fully GitOps support in Kubernetes-native way - generated set of Kubernetes resources can be committed to Git repository in a secure way using SealedSecrets integration.

When --gpg-key-path commandline switch points to a valid SealedSecrets yaml file instead of a plaintext GPG key then a SealedSecret will be used.

Example of valid file:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
    name: mysecret
    # namespace name must match --k8s-namespace commandline switch value
    # namespace: ... 
spec:
    encryptedData:
        gpg-key: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....

        # there you can place environment variables passed to the container, for example:
        GPG_PASSPHRASE: ....some-encoded-passphrase-with-sealed-secrets-mechanism...
        DB_PASSWORD: ...
        BR_TOKEN: ...
        # all those ENVs will be available to use in section "Params" and "Repository" in definition.yaml configuration file (--definition)

Reference

• Used Kubernetes templates: chart
• Standard available service templates: service templates

backup-maker procedure restore

Works almost the same as backup-maker procedure backup, except the produced result.

• The produced script takes additionally a parameter to specify to which version rollback to
• In case of Kubernetes a kind: Pod is outputted instead of kind: CronJob

Reference

• Used Kubernetes templates: chart
• Standard available service templates: service templates

Templates

Everytime backup-maker procedure is launched there are unpacked Helm chart templates and service templates into ~/.bm.

Rules:

• .base directories are always overwritten when CLI application is launched
• user directories allows for user customization. Files will be overwritten by app util user modifies at least one file in user directory

The directory structure looks for example like this:

#   ~/.bm
#   ├── chart
#   │         ├── .base
#   │         │         ├── configmap.yaml
#   │         │         ├── cronjob.yaml
#   │         │         └── secret.yaml
#   │         └── user
#   │             ├── configmap.yaml
#   │             ├── cronjob.yaml
#   │             └── secret.yaml
#   └── templates
#      ├── .base
#      │         ├── backup
#      │         │         ├── files.tmpl
#      │         │         └── postgres.tmpl
#      │         └── restore
#      │             ├── files.tmpl
#      │             └── postgres.tmpl
#      └── user
#          ├── backup
#          │         ├── files.tmpl
#          │         └── postgres.tmpl
#          └── restore
#              ├── files.tmpl
#              └── postgres.tmpl
#

Usage concept

1. User prepares YAML file as input definition for both backup & restore

# definition.yaml

# System-specific variables, in this case specific to PostgreSQL
# ${...} and $(...) syntax will be evaluated in target environment e.g. Kubernetes POD
Params:
    hostname: postgres.db.svc.cluster.local
    port: 5432
    db: rkc-test
    user: riotkit
    password: "${DB_PASSWORD}" # injects a shell-syntax, put your password in a `kind: Secret` and mount as environment variable. You can also use $(cat /mnt/secret) syntax, be aware of newlines!

# Generic repository access details. Everything here will land AS IS into the bash script.
# This means that any ${...} and $(...) will be executed in target environment e.g. inside Kubernetes POD
Repository:
    url: "https:#example.org"
    token: "${BR_TOKEN}"
    encryptionKeyPath: "/var/lib/backup-repository/encryption.key"
    passphrase: "${GPG_PASSPHRASE}"
    recipient: "your-gpg@email.org"
    collectionId: "111-222-333-444"

# Generic values for Helm used to generate jobs/pods. Those values will overwrite others.
# Notice: Environment variables with '${...}' and '$(...)' will be evaluated in LOCAL SHELL DURING BUILD
HelmValues:
    name: "hello-world"
    env: {}
        # if specified, then will be added to `kind: Secret` and injected into POD as environment
        # the value from ${GPG_PASSPHRASE} will be retrieved from the SHELL DURING THE BUILD
        #GPG_PASSPHRASE: "${GPG_PASSPHRASE}"

        # most secure way for Kubernetes is to not provide secrets there, but define them as environment variables
        # inside SealedSecrets - all encryptedData keys will be accessible as environment variables inside container

1. Next, User runs a generation command

In result the User gets prepared Kubernetes manifests that could be applied to the cluster manually or by ArgoCD, FluxCD, Tekton Pipelines, Argo Workflows or other CI/CD software.

Creating templates

There can be locally defined templates and templates bundled in binary, redistributed with Backup Maker. Placing your templates in ~/.bm/templates/user allows for easy testing before you make a pull request with your template.

1. Create definition file in "definition" subdirectory

A definition file consists of container image (used e.g. in Kubernetes),

{
    "defaultImage": "docker.io/mysql:8.0-debian",
    "restoreTemplate": "mysql-dump.tmpl",
    "backupTemplate": "mysql-dump.tmpl"
}

2. Create a backup template

Create a file in backup subdirectory, the filename must match the one specified in backupTemplate from definition file.

3. Create a restore template

That's the same thing as for backup, but use filename matching restoreTemplate.

Additionally, in order to restore a version there is an environment variable passed you need to handle in your script.

# RESTORE_VERSION is automatically injected in Kubernetes pod
# use it to restore a proper version
selectedVersion="${RESTORE_VERSION:-latest}"

Generic templating rules & hints

Input data supplied by the user - backup/restore configuration contains two sections - Params and Repository

• Params: Should be used for service-specific params e.g. "database name" or "aws account id"
• Repository: Use it to extract Backup Repository Server credentials and parameters

Params:
    hostname: postgres.db.svc.cluster.local
    password: "${DB_PASSWORD}"       # NOTICE: This will be evaluated in target environment e.g. in Kubernetes Pod
    # (...)
Repository:
    url: "https://example.org"
    passphrase: "${GPG_PASSPHRASE}"  # NOTICE: This will be evaluated in target environment e.g. in Kubernetes Pod
    # (...)

Use those parameters in your script:

DB_NAME="{{ .Params.hostname }}"  # -> postgres.db.svc.cluster.local

exec backup-maker restore --url "{{ .Repository.url }}" # (...)