kontainer-engine-driver-example

Kontainer Engine Example Driver

This is the Kontainer-Engine Example Driver. The purpose of this repo is to show how to write a custom Cluster Driver for rancher. At a minimum to write a custom Cluster Driver for Rancher you will need to do the following:

1. Implement the Driver interface
2. Write a main func that will accept a port argument and start a GRPC server on the supplied port
3. Package your driver and distribute it in a way that rancher can consume

We will go over each of these steps in detail.

Implementing the driver interface

The Driver interface you will need to implement is as follows:

type Driver interface {
	GetDriverCreateOptions(ctx context.Context) (*DriverFlags, error)
	GetDriverUpdateOptions(ctx context.Context) (*DriverFlags, error)

	Create(ctx context.Context, opts *DriverOptions, clusterInfo *ClusterInfo) (*ClusterInfo, error)
	Update(ctx context.Context, clusterInfo *ClusterInfo, opts *DriverOptions) (*ClusterInfo, error)
	PostCheck(ctx context.Context, clusterInfo *ClusterInfo) (*ClusterInfo, error)
	Remove(ctx context.Context, clusterInfo *ClusterInfo) error

	GetVersion(ctx context.Context, clusterInfo *ClusterInfo) (*KubernetesVersion, error)
	SetVersion(ctx context.Context, clusterInfo *ClusterInfo, version *KubernetesVersion) error
	GetClusterSize(ctx context.Context, clusterInfo *ClusterInfo) (*NodeCount, error)
	SetClusterSize(ctx context.Context, clusterInfo *ClusterInfo, count *NodeCount) error

	GetCapabilities(ctx context.Context) (*Capabilities, error)

	RemoveLegacyServiceAccount(ctx context.Context, clusterInfo *ClusterInfo) error

	ETCDSave(ctx context.Context, clusterInfo *ClusterInfo, opts *DriverOptions, snapshotName string) error
	ETCDRestore(ctx context.Context, clusterInfo *ClusterInfo, opts *DriverOptions, snapshotName string) error
	GetK8SCapabilities(ctx context.Context, opts *DriverOptions) (*K8SCapabilities, error)
}

GetDriverCreateOptions

This func supplies the flags necessary to build the API schema in Rancher for the Cluster Driver. It also provides the arguments for the kontainer-engine CLI. Here is an example flag from the GKE driver GetDriverCreateOptions func:

	driverFlag.Options["disk-size-gb"] = &types.Flag{
		Type:  types.IntType,
		Usage: "Size of the disk attached to each node",
		Default: &types.Default{
			DefaultInt: 100,
		},
	}

After being parsed by Rancher this field will be turned into the resource field on the schema:

"diskSizeGb": {
    "create": true,
    "default": 100,
    "description": "Size of the disk attached to each node",
    "dynamicField": true,
    "type": "int",
    "update": true
},

Note that the name of the field was transformed from snake case to camel case before the schema was generated.

GetDriverUpdateOptions

This func provides the arguments when the driver is used in the kontainer- engine CLI. It is not used to populate the Rancher API schema.

Create

This func creates the cluster in the managed Kubernetes provider and populates sufficient information on the ClusterInfo return value so that PostCheck can connect to the cluster. See the PostCheck description for more info on this process.

Update

This func updates the cluster in the managed Kubernetes. Like Create it must ensure ClusterInfo is populated with sufficient information for PostCheck to generate a service account token. If no connection information has changed, then it can be left alone and it will reuse the information generated by Create.

PostCheck

This func must populate the ClusterInfo.serviceAccountToken field with a service account token with sufficient permissions for Rancher to manage the cluster. We recommend using the utilities (util.GenerateServiceAccountToken(clientset)) provided in kontainer-engine to generate this token. Note that the GenerateServiceAccountToken func requires a kubernetes.Interface be passed into it. This means that the Create and Update functions MUST populate sufficient information on the ClusterInfo metadata in order for a connection to be established. See this example from GKE driver:

	bytes, err := json.Marshal(state)
	if err != nil {
		return err
	}
	if info.Metadata == nil {
		info.Metadata = map[string]string{}
	}
	info.Metadata["state"] = string(bytes)

The state struct is marshalled to json and then stored on the Metadata["state"] field.

func (d *Driver) PostCheck(ctx context.Context, info *types.ClusterInfo) (*types.ClusterInfo, error) {
	state, err := getState(info)
	if err != nil {
		return nil, err
	}
	
	...
	
func getState(info *types.ClusterInfo) (state, error) {
	state := state{}
	// ignore error
	err := json.Unmarshal([]byte(info.Metadata["state"]), &state)
	return state, err
}

Then PostCheck calls getState, passing in the info variable, where the previously marshalled state object is unmarshalled into a new state instance. This information is later used to construct the kubernetes.Interface to communicate with the cluster and generate the service account token.

Remove

This func must remove the cluster from the cloud provider. Like PostCheck it will receive the clusterInfo populated from Create or Update and must make delete the cluster with that information.

GetVersion

This returns the current Kubernetes version in the cluster. Currently unused in Rancher. It is not necessary to implement this func (see Note below).

SetVersion

Sets the Kubernetes version in the cluster. Currently unused in Rancher. It is not necessary to implement this func (see Note below).

GetClusterSize

Gets the current number of nodes in the cluster. Currently unused in Rancher. It is not necessary to implement this func (see Note below).

SetClusterSize

Sets the current number of nodes in the cluster. Currently unused in Rancher. It is not necessary to implement this func (see Note below).

GetCapabilities

Returns the state of various optional funcs on the driver.

RemoveLegacyServiceAccount

This func does not need to be implemented by drivers it is used for internal backwards compatibility with the built in Cluster Drivers. Simply return nil for all arguments.

ETCDSave

Saves the current ETCD state. Cloud provider dependent. It is not necessary to implement this func (see Note below). If unimplemented this func should return an error message.

ETCDRestore

Restores the previous ETCD state. Cloud provider dependent. It is not necessary to implement this func (see Note below). If unimplemented this func should return an error message.

GetK8SCapabilities

Returns the capabilities of the Kubernetes cluster. See the struct definition in Rancher/Types for a full description of all fields.

Note: for some operations it is not necessary to implement them to ensure proper functioning of your driver with Rancher. Simply return nil for all return arguments if you do not wish to implement them. You can also embed the structs UnimplementedVersionAccess and UnimplementedClusterSizeAccess as appropriate.

The following funcs do not need to be implemented:

• GetVersion
• SetVersion
• GetClusterSize
• SetClusterSize
• ETCDSave*
• ETCDRestore*

*These funcs should return an error instead of nil if they are unimplemented.

Main Func

The main func in your binary needs to do the following things:

1. Accept a positional argument of an integer (the port)
2. Use this port argument to start a GRPC server on localhost at the specified port

Point 1 is as simple as accepting parsing the first positional argument as a integer (see main.go for an example). For point 2, we recommend using the GRPC servers provided in the kontainer-engine github repo. These can be invoked as follows:

go types.NewServer(&Driver{}, addr).ServeOrDie(fmt.Sprintf("127.0.0.1:%v", port))

Note that the Driver{} struct is our custom Cluster Driver and the port is the positional argument we received in our main func.

Packaging and distributing

Building

make

Will output driver binaries into the dist directory, these can be imported directly into Rancher and used as cluster drivers. They must be distributed via URLs that your Rancher instance can establish a connection to and download the driver binaries. For example, this driver is distributed via a GitHub release and can be downloaded from one of those URLs directly.

Running

Go to the Cluster Drivers management screen in Rancher and click Add Cluster Driver. Enter the URL of your driver, a UI URL (see the UI skel repo for details), and a checksum (optional), and click Create. Rancher will automatically download and install your driver. It will then become available to use on the Add Cluster screen.

License

Copyright (c) 2018 Rancher Labs, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.