model-registry-operator

model-registry-operator

Model Registry operator is a controller for deploying Openshift AI Model Registry service in a Kubernetes namespace.

Description

The controller reconciles ModelRegistry Custom Resources to create a service for the ModelRegistry API.

Getting Started

You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster. Note: Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).

ModelRegistry service needs a PostgreSQL or MySQL database. A sample Postgres configuration for testing (without TLS security) is included in postgres-db.yaml and a sample MySQL configuration for testing is included in mysql-db.yaml. To use another PostgreSQL instance, comment the line that includes the sample DB in kustomization.yaml and to use your own mysql instance comment the line that includes the sample DB in kustomization.yaml.

The operator supports creating model registries using Istio for security including Authorino for authorization. It also supports Istio Gateways for exposing service endpoints for clients outside the Istio service mesh.

Istio Configuration

Skip this section if you are not using Istio.

NOTE that this section describes how to configure a couple of files in the kustomize samples in this project. However, there are only a handful of extra configuration properties needed in an Istio model registry.

For using the Istio based samples, both Istio and Authorino MUST be installed before deploying the operator. An Authorino instance MUST also be configured as a custom authz provider in the Istio control plane.

Both Istio and Authorino along with the authz provider can be easily enabled in the Open Data Hub data science cluster instance.

If Istio or Authorino is installed in the cluster after deploying this controller, restart the controller by deleting the pod model-registry-operator-controller-manager in model-registry-operator-system namespace. If Model Registry component has been installed as an Open Data Hub operator component, the operator namespace will be opendatahub.

If Authorino provider is from a non Open Data Hub cluster, configure its selector labels in the authconfig-labels.yaml file.

To use the Istio model registry samples the following configuration data is needed in the istio.env file:

• AUTH_PROVIDER - name of the authorino external auth provider configured in the Istio control plane (defaults to opendatahub-auth-provider for Open Data Hub data science cluster with OpenShift Service Mesh enabled).
• DOMAIN - hostname domain suffix for gateway endpoints. This depends upon your cluster's external load balancer config. In OpenShift clusters, it can be obtained with the command:

oc get ingresses.config/cluster -o jsonpath='{.spec.domain}'

• ISTIO_INGRESS - name of the Istio Ingress Gateway (defaults to ingressgateway).
• REST_CREDENTIAL_NAME - Kubernetes secret in IngressGateway namespace (typically istio-system) containing TLS certificates for REST service (defaults to modelregistry-sample-rest-credential).
• GRPC_CREDENTIAL_NAME - Kubernetes secret in IngressGateway namespace containing TLS certificates for gRPC service (defaults to modelregistry-sample-grpc-credential).

Running on the cluster

1. Deploy the controller to the cluster using the latest docker image:

make deploy

1. The operator includes multiple samples that use kustomize to create a sample model registry modelregistry-sample.

• MySQL without Istio plain Kubernetes model registry services with a sample MySQL database
• PostgreSQL without Istio plain Kubernetes model registry services with a sample PostgreSQL database
• MySQL with Istio MySQL database, Istio, and plaintext Gateway
• PostgreSQL with Istio PostgreSQL database, Istio, and plaintext Gateway
• MySQL with Istio and TLS MySQL database, Istio, and TLS Gateway endpoints
• PostgreSQL with Istio and TLS PostgreSQL database, Istio, and TLS Gateway endpoints
• Secure MySQL without Istio plain Kubernetes model registry services with a sample SSL secured MySQL database
• Secure MySQL with Istio SSL secured MySQL database, Istio, and TLS Gateway

Istio Samples

WARNING: Istio samples without TLS are only meant for testing and demos to avoid having to create TLS certificates. They should only be used in local development clusters.

For all Istio samples, a Kubernetes user or serviceaccount authorization token MUST be passed in calls to model registry services using the header:

Authorization: Bearer sha256~xxx

In OpenShift clusters, the user session token can be obtained using the command:

oc whoami -t

To help authorize users and service accounts to access the registry, the model registry operator creates a Role named registry-user-<registry-name>. This role has the required permission to perform a GET on the model registry instance service, e.g. modelregistry-sample service. In addition, if running in an OpenShift cluster the operator creates an OpenShift user Group called <registry-name>-users. So, for included samples it creates a Role named registry-user-modelregistry-sample and a Group named modelregistry-sample-users.

A Kubernetes or OpenShift cluster administrator can then add users to the Group, or create RoleBinding for the Role to grant permission to specific users and serviceaccounts to access the model registry.

NOTE: The operator deletes the Group and the Role when the model registry custom resource is deleted. If you have created your own RoleBindings to this Role, the operator will not remove them automatically and hence must be removed manually.

The project Makefile includes targets to manage test TLS certificates using a self signed CA certificate. To create test certificates in the directory certs and Kubernetes secrets in the istio-system namespace and current namespace, use the command:

make certificates

The test CA certificate is generated in the file certs/domain.crt along with certificates for REST, gRPC, and Database service. See generate_certs.sh for details.

To cleanup the certificates and Kubernetes secrets, use the command:

make certificates/clean

NOTE: The sample database secret model-registry-db-credential is created with the CA cert, server key and server cert. However, in production the model registry only needs a secret with the CA cert(s). The production database server will be configured with a secret containing the private key and server cert. The sample certificates use a self-signed CA and does not do cert management like cert rotation, etc. Use your own certificate manager, e.g. https://cert-manager.io/ and create generic kubernetes secrets for REST, gRPC and database with the the keys tls.key, tls.crt, and ca.crt.

To disable Istio Gateway creation, create a kustomize overlay that removes the gateway yaml section in model registry custom resource or manually edit a sample yaml and it's corresponding replacements.yaml helper.

If using upstream Istio (i.e. not OpenShift Service Mesh), enable Istio proxy injection in your test namespace by using the command:

kubectl label namespace <namespace> istio-injection=enabled --overwrite

If using OpenShift Service Mesh, enable it by adding the namespace to the control plane (e.g. ODH Istio control plane data-science-smcp below) by using the command:

kubectl apply -f -<<EOF
apiVersion: maistra.io/v1
kind: ServiceMeshMember
metadata:
  name: default
spec:
  controlPlaneRef:
    name: data-science-smcp
    namespace: istio-system
EOF

If using OpenShift, the operator will automatically create OpenShift Routes in the ingress gateway's namespace (istio-system by default). It will create two routes <namespace>-modelregistry-sample-rest and <namespace>-modelregistry-sample-grpc for the REST and gRPC gateway endpoints respectively.

This automatic route creation can be disabled by setting the properties spec.istio.gateway.rest.gatewayRoute or spec.istio.gateway.grpc.gatewayRoute to disabled.

1. For Istio samples, first configure properties in istio.env. Install a model registry instance using ONE of the following commands:

kubectl apply -k config/samples/mysql
kubectl apply -k config/samples/postgres
kubectl apply -k config/samples/istio/mysql
kubectl apply -k config/samples/istio/postgres
kubectl apply -k config/samples/istio/mysql-tls
kubectl apply -k config/samples/istio/postgres-tls
kubectl apply -k config/samples/secure-db/mysql
kubectl apply -k config/samples/secure-db/mysql-tls

This will create the appropriate database and model registry resources, which will be reconciled in the controller to create a model registry deployment with other Kubernetes, Istio, and Authorino resources as needed.

1. Check that the sample model registry was created using the command:

kubectl describe mr modelregistry-sample

Check the Status of the model registry resource for failed Conditions.

For Istio Gateway examples, consult your Istio configuration to verify gateway endpoint creation. When OpenShift gateway route creation is enabled (by default), look for model registry routes using the command:

kubectl get route -n istio-system

Verifying the Model Registry REST Service

If using the Istio TLS Gateway sample, to verify the REST gateway service use the following command:

curl -H "Authorization: Bearer $TOKEN" --cacert certs/domain.crt https://modelregistry-sample-rest.$DOMAIN/api/model_registry/v1alpha3/registered_models

Where, $TOKEN and $DOMAIN environment variables are set to the client token and host domain. If using OpenShift, the token and domain can be set using the commands:

export TOKEN=`oc whoami -t`
export DOMAIN=`oc get ingresses.config/cluster -o jsonpath='{.spec.domain}'`

If using a non-TLS gateway sample, use the command:

curl -H "Authorization: Bearer $TOKEN" http://modelregistry-sample-rest.$DOMAIN/api/model_registry/v1alpha3/registered_models

If using a non-Istio sample and using OpenShift, enable the OpenShift Route using the command:

kubectl patch mr modelregistry-sample --type='json' -p='[{"op": "replace", "path": "/spec/rest/serviceRoute", "value": "enabled"}]'

Verify that the Route was created using the command:

kubectl get routes.route.openshift.io modelregistry-sample-http

Use the following command to get the OpenShift Route Host:

export ROUTE_HOST=`kubectl get routes.route.openshift.io modelregistry-sample-http -ojsonpath='{.status.ingress[0].host}'`

Then verify the REST service using the command:

curl http://$ROUTE_HOST/api/model_registry/v1alpha3/registered_models

The output should be a list of all registered models in the registry, e.g. for an empty registry:

{"items":[],"nextPageToken":"","pageSize":0,"size":0}

1. To delete the sample model registry, use ONE of the following commands based on the sample type deployed earlier:

kubectl delete -k config/samples/mysql
kubectl delete -k config/samples/postgres
kubectl delete -k config/samples/istio/mysql
kubectl delete -k config/samples/istio/postgres
kubectl delete -k config/samples/istio/mysql-tls
kubectl delete -k config/samples/istio/postgres-tls

Building local docker image for development

1. Build and push your image to the location specified by IMG:

make docker-build docker-push IMG=<some-registry>/model-registry-operator:tag

1. Deploy the controller to the cluster with the image specified by IMG:

make deploy IMG=<some-registry>/model-registry-operator:tag

Uninstall CRDs

To delete the CRDs from the cluster:

make uninstall

Undeploy controller

UnDeploy the controller from the cluster:

make undeploy

Contributing

// TODO(user): Add detailed information on how you would like others to contribute to this project

How it works

This project aims to follow the Kubernetes Operator pattern.

It uses Controllers, which provide a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster.

Test It Out

1. Install the CRDs into the cluster:

make install

1. Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):

make run

NOTE: You can also run this in one step by running: make install run

Modifying the API definitions

If you are editing the API definitions, generate the manifests such as CRs or CRDs using:

make manifests

NOTE: Run make --help for more information on all potential make targets

More information can be found via the Kubebuilder Documentation

License

Copyright 2023.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.