clickhouse-operator

ClickHouse operator

A Kubernetes Operator for deploying one or more ClickHouse instances. Similar to the Altinity operator, except with the explicit goal of working out-of-the-box, making it easier to get started quickly.

The main user interface is via the ClickHouse CRD:

• Definition
• Example

Quick start

You’ll need a Kubernetes cluster to run against. You can use KIND or Minikube to get a local cluster for testing, or run against a remote cluster.

Use make kind to create a simple 3 worker node cluster for local development. This will also install all local dev dependencies in the cluster.

You can run everything from the make, but the following sections also contain equivalent kubectl commands.

Note: The controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).

Note: Run make --help for more information on all potential make targets

Build

Update the CRD yaml file and build the operator binary.

$ make manifests build
...

Install

With a working kubectl session pointing to a cluster, add the ClickHouse CRD and run the operator.

$ kubectl apply -f ./config/crd/bases/clickhouse.gitlab.com_clickhouses.yaml
customresourcedefinition.apiextensions.k8s.io/clickhouses.clickhouse.gitlab.com created
# cert-manager is an implicit requirement for the operator
$ kubectl apply -f ../scheduler/controllers/cluster/manifests/cert-manager/certmanager.crd.yaml
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created

or

$ make deploy-dev-dependencies install
...

then

$ ./bin/manager
...

or

$ make run
...

In a separate tab, deploy an example ClickHouse instance to the default namespace.

$ kubectl apply -f ./config/samples/example.yaml
secret/example-users created
clickhouse.clickhouse.gitlab.com/example created

Wait a few seconds for pods to be 1/1 Running.

$ kubectl get pods -n default
NAME          READY   STATUS    RESTARTS   AGE
example-0-0   1/1     Running   0          33s
example-1-0   1/1     Running   0          33s
example-2-0   1/1     Running   0          33s

Use

Interact with the example instance.

$ kubectl exec -it -n default statefulset/example-0-0 -- \
    clickhouse-client -f PrettyCompact -q 'SELECT name,engine,data_path FROM system.databases;'
┌─name───────────────┬─engine───┬─data_path─────────────────────────┐
│ INFORMATION_SCHEMA │ Memory   │ /var/lib/clickhouse/              │
│ default            │ Ordinary │ /var/lib/clickhouse/data/default/ │
│ information_schema │ Memory   │ /var/lib/clickhouse/              │
│ system             │ Atomic   │ /var/lib/clickhouse/store/        │
└────────────────────┴──────────┴───────────────────────────────────┘

$ kubectl exec -it -n default statefulset/example-0-0 -- \
    clickhouse-client -f PrettyCompact -q "SELECT cluster,shard_num,replica_num,host_name,host_address FROM system.clusters WHERE startsWith(host_name, 'example');"
┌─cluster────┬─shard_num─┬─replica_num─┬─host_name─┬─host_address───┐
│ replicated │         1 │           1 │ example-0 │ 192.168.57.139 │
│ replicated │         1 │           2 │ example-1 │ 192.168.35.108 │
│ replicated │         1 │           3 │ example-2 │ 192.168.60.78  │
│ sharded    │         1 │           1 │ example-0 │ 192.168.57.139 │
│ sharded    │         2 │           1 │ example-1 │ 192.168.35.108 │
│ sharded    │         3 │           1 │ example-2 │ 192.168.60.78  │
└────────────┴───────────┴─────────────┴───────────┴────────────────┘

Cleaning up

Delete the example instance and the leftover volumes. The associated pods/etc should be destroyed automatically.

$ kubectl delete clickhouse -n default example
clickhouse.clickhouse.gitlab.com "example" deleted

$ kubectl delete pvc -n default -l 'app=clickhouse,name=example'
persistentvolumeclaim "data-example-0-0" deleted
persistentvolumeclaim "data-example-1-0" deleted
persistentvolumeclaim "data-example-2-0" deleted
persistentvolumeclaim "logs-example-0-0" deleted
persistentvolumeclaim "logs-example-1-0" deleted
persistentvolumeclaim "logs-example-2-0" deleted

Tear down the operator with Ctrl+C, then delete the CRD.

$ kubectl delete crd clickhouses.clickhouse.gitlab.com
clickhouse.clickhouse.gitlab.com "example" deleted

Testing

We're using Ginkgo with Gomega to write BDD like specs for the controller and e2e tests.

make test will run the controller integration tests, which starts an envtest local control plane to check the controller is making the correct decisions. Local integration tests are in the standard Golang locations, i.e. corresponding _test.go files in the package.

E2E tests

make e2e-test creates the CRDs in an existing K8s cluster to check the correct resources are created and they are functioning. It is expected that the kubectl environment exists to be detected and the CRD is already installed via make install. E2E test code lives under ./e2e.

Prerequisite for E2E test

We also require MinIO setup working locally as well for e2e tests. You can deploy the minio service as:

make deploy-minio

^ This installs a minio cluster locally and also provisions the bucket, policy and user needed for the E2E test. See README.md for details on this setup.

Use make deploy-dev-dependencies to install everything required for dev and the controller deployment.

For e2e tests the controller should either be installed and running in the cluster via make install && make deploy or running locally via make run. The latter allows for a development cycle while running tests.

CI

In CI the E2E tests use gcloud to create a GKE cluster, install the CRD, operator and run E2E tests.

GitLab CI contains variables GOOGLE_APPLICATION_CREDENTIALS and OPSTRACE_DEV_GCP_PROJECT_ID to allow connection to the opstrace dev GCP project.

A small GCP Kubernetes cluster is created per test job, and deleted afterwards. Clusters are created in the opstrace-dev realm, with the name format ch-operator-e2e-${CI_COMMIT_SHORT_SHA}-${CI_PIPELINE_IID}.

Gotchas

GKE storage provisioner seems to provision volumes with size starting with 1Gi. Everything lower than that is rounded to 1Gi causing Cluster to never transition to Complete state as the desired storage size (e.g. 512Mi) and actual size (1Gi) of PVC differ.