multicluster-mesh-addon

Multicluster Mesh Addon

multicluster-mesh-addon is an enhanced addon created with addon-framework, it is used to manage(discovery, deploy and federate) service meshes across multiple clusters and applied in Red Hat Advanced Cluster Management for Kubernetes.

Core Concepts

1. Mesh - a mesh resource is mapping to a physical service mesh in a managed cluster, it contains the desired state and status of the backend service mesh. For each physical service mesh in a managed cluster, a mesh resource is created in the managed cluster namespace of hub cluster. An example of mesh resource would resemble the following yaml snippet:

apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: Mesh
metadata:
  name: managedcluster1-istio-system-basic
spec:
  clusters: managedcluster1
  controlPlane:
    components: ["istio-discovery", "istio-ingress", "mesh-config", "telemetry-common", "tracing"]
    namespace: istio-system
    profiles: ["default"]
    version: v2.1
  meshMemberRoll: ["istio-apps"]
  meshProvider: Openshift Service Mesh
  trustDomain: cluster.local
status:
  readiness:
    components:
      pending: []
      ready: ["istio-discovery", "istio-ingress", "mesh-config", "telemetry-common", "tracing"]
      unready: []

1. MeshDeployment - meshdeployment resource is used to deploy physical service meshes to managed cluster(s), it supports deploying multiple physical service meshes to different managed clusters with one mesh template. An example of meshdeployment resource would resemble the following yaml snippet:

apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: MeshDeployment
metadata:
  name: mesh
spec:
  clusters: ["managedcluster1", "managedcluster2"]
  controlPlane:
    components: ["prometheus", "istio-discovery", "istio-ingress", "mesh-config", "telemetry-common", "tracing"]
    namespace: mesh-system
    profiles: ["default"]
    version: v2.1
  meshMemberRoll: ["mesh-apps"]
  meshProvider: Openshift Service Mesh
status:
  appliedMeshes: ["managedcluster1-mesh", "managedcluster2-mesh"]

1. MeshFederation - meshfederation resource is used to federate service meshes so that the physical service meshes located in one cluster or different clusters to securely share and manage traffic between meshes while maintaining strong administrative boundaries in a multi-tenant environment. An example of meshfederation resource would resemble the following yaml snippet:

apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: MeshFederation
metadata:
  name: mcsm
spec:
  meshPeers:
  - peers:
    - name: managedcluster1-mesh
      cluster: managedcluster1
    - name: managedcluster2-mesh
      cluster: managedcluster2
  trustConfig:
    trustType: Limited
status:
  federatedMeshes:
  - peer:
    - managedcluster1-mesh
    - managedcluster1-mesh

Getting Started

Prerequisites

• Ensure docker 17.03+ is installed.
• Ensure golang 1.15+ is installed.
• Prepare an environment of Red Hat Advanced Cluster Management for Kubernetes and login to the hub cluster with oc command line tool.

Build and Deploy

1. Build and push docker image:

make docker-build docker-push IMAGE=quay.io/<your_quayio_username>/multicluster-mesh-addon:latest

1. Deploy the multicluster-mesh-addon to hub cluster:

make deploy IMAGE=quay.io/<your_quayio_username>/multicluster-mesh-addon:latest

How to use

1. If you have installed Openshift Service Mesh in any managed cluster, then you should find a mesh resource created in its namespace of hub cluster:

# oc -n managedcluster1 get mesh
NAME                                CLUSTER          VERSION   PEERS    AGE
managedcluster1-istio-system-basic  managedcluster1  v2.1               80m

1. You can also deploy new service meshes to managed clusters, for example, creating the following meshdeployment resource to deploy new service meshes to managed cluster managedcluster1 and managedcluster2:

cat << EOF | oc apply -f -
apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: MeshDeployment
metadata:
  name: mesh
  namespace: open-cluster-management
spec:
  clusters: ["managedcluster1", "managedcluster2"]
  controlPlane:
    components: ["prometheus", "istio-discovery", "istio-ingress", "mesh-config", "telemetry-common", "tracing"]
    namespace: mesh-system
    profiles: ["default"]
    version: v2.1
  meshMemberRoll: ["mesh-bookinfo"]
  meshProvider: Openshift Service Mesh
EOF

1. Then verify the created service meshes:

# oc get mesh -A
NAMESPACE        NAME                  CLUSTER          VERSION  PEERS  AGE
managedcluster1  managedcluster1-mesh  managedcluster1  v2.1            19m
managedcluster2  managedcluster2-mesh  managedcluster2  v2.1            19m

1. You can also federate managedcluster1-mesh and managedcluster2-mesh by creating a meshfederation resource in hub cluster with the following command:

cat << EOF | oc apply -f -
apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: MeshFederation
metadata:
  name: mcsm
  namespace: open-cluster-management
spec:
  meshPeers:
  - peers:
    - name: managedcluster1-mesh
      cluster: managedcluster1
    - name: managedcluster2-mesh
      cluster: managedcluster2
  trustConfig:
    trustType: Limited
EOF

1. To verify the meshes are federated, you can deploy part(productpage,details,reviews-v1) of the bookinfo application in managed cluster managedcluster1:

Note: currently the verify steps have to be executed in the managed cluster, we're working on the service discovery and service federation now.

oc create ns mesh-bookinfo
oc apply -n mesh-bookinfo -f https://raw.githubusercontent.com/maistra/istio/maistra-2.1/samples/bookinfo/platform/kube/bookinfo.yaml -l 'app in (productpage,details)'
oc apply -n mesh-bookinfo -f https://raw.githubusercontent.com/maistra/istio/maistra-2.1/samples/bookinfo/platform/kube/bookinfo.yaml -l app=reviews,version=v1
oc apply -n mesh-bookinfo -f https://raw.githubusercontent.com/maistra/istio/maistra-2.1/samples/bookinfo/platform/kube/bookinfo.yaml -l service=reviews
oc apply -n mesh-bookinfo -f https://raw.githubusercontent.com/maistra/istio/maistra-2.1/samples/bookinfo/platform/kube/bookinfo.yaml -l 'account'
oc apply -n mesh-bookinfo -f https://raw.githubusercontent.com/maistra/istio/maistra-2.1/samples/bookinfo/networking/bookinfo-gateway.yaml

1. Then deploy the remaining part(reviews-v2, reviews-v3, ratings) of bookinfo application in managed cluster managedcluster2:

oc create ns mesh-bookinfo
oc apply -n mesh-bookinfo -f https://raw.githubusercontent.com/maistra/istio/maistra-2.1/samples/bookinfo/platform/kube/bookinfo.yaml -l app=reviews,version=v2
oc apply -n mesh-bookinfo -f https://raw.githubusercontent.com/maistra/istio/maistra-2.1/samples/bookinfo/platform/kube/bookinfo.yaml -l app=reviews,version=v3
oc apply -n mesh-bookinfo -f https://raw.githubusercontent.com/maistra/istio/maistra-2.1/samples/bookinfo/platform/kube/bookinfo.yaml -l service=reviews
oc apply -n mesh-bookinfo -f https://raw.githubusercontent.com/maistra/istio/maistra-2.1/samples/bookinfo/platform/kube/bookinfo.yaml -l app=ratings
oc apply -n mesh-bookinfo -f https://raw.githubusercontent.com/maistra/istio/maistra-2.1/samples/bookinfo/platform/kube/bookinfo.yaml -l 'account'

1. Create exportedserviceset resource in managed cluster managedcluster2 to export services(reviews and ratings) from managedcluster2-mesh:

cat << EOF | oc apply -f -
apiVersion: federation.maistra.io/v1
kind: ExportedServiceSet
metadata:
  name: managedcluster1-mesh
  namespace: mesh-system
spec:
  exportRules:
  - type: NameSelector
    nameSelector:
      namespace: mesh-bookinfo
      name: reviews
  - type: NameSelector
    nameSelector:
      namespace: mesh-bookinfo
      name: ratings
EOF

1. Create importedserviceset resource in managed cluster managedcluster1 to import services(reviews and ratings) from managedcluster1-mesh:

cat << EOF | oc apply -f -
apiVersion: federation.maistra.io/v1
kind: ImportedServiceSet
metadata:
  name: managedcluster2-mesh
  namespace: mesh-system
spec:
  importRules:
    - type: NameSelector
      importAsLocal: true
      nameSelector:
        namespace: mesh-bookinfo
        name: reviews
        alias:
          namespace: mesh-bookinfo
    - type: NameSelector
      importAsLocal: true
      nameSelector:
        namespace: mesh-bookinfo
        name: ratings
        alias:
          namespace: mesh-bookinfo
EOF

1. Access the bookinfo from your browser with the following address from managedcluster1 cluster:

echo http://$(oc -n mesh-system get route istio-ingressgateway -o jsonpath={.spec.host})/productpage

Note: The expected result is that by refreshing the page several times, you should see different versions of reviews shown in productpage, presented in a round robin style (red stars, black stars, no stars). Because reviews-v2, reviews-v3 and ratings service are running in another mesh, if you could see black stars and red stars reviews, then it means traffic across meshes are successfully routed.

Future Work

• Services and workloads discovery
• Federate services across meshes
• Deploy application across meshes