kubernetes-mesos

kubernetes-mesos

When Google Kubernetes meets Apache Mesos

![GoDoc] (https://godoc.org/github.com/mesosphere/kubernetes-mesos?status.png)

Kubernetes and Mesos are a match made in heaven. Kubernetes enables the Pod, an abstraction that represents a group of co-located containers, along with Labels for service discovery, load-balancing, and replication control. Mesos provides the fine-grained resource allocations for pods across nodes in a cluster, and facilitates resource sharing among Kubernetes and other frameworks running on the same cluster.

This is still very much a work-in-progress, but stay tuned for updates as we continue development. If you have ideas or patches, feel free to contribute!

Refer to the documentation pages for a review of core concepts, system architecture, and known issues.

Tutorial

Mesosphere maintains a tutorial for running Kubernetes-Mesos on GCE, which is periodically updated to coincide with releases of this project. Please use a release from the 0.3.x series for the best experience with the tutorial.

Binaries

For a binary-only install of the Kubernetes-Mesos framework you can use the Docker-based builder. The following commands will build and copy the resulting binaries to ./bin:

# chcon needed for systems protected by SELinux
$ mkdir bin && chcon -Rt svirt_sandbox_file_t bin
$ docker run --rm -v $(pwd)/bin:/target mesosphere/kubernetes-mesos:build

Build

To build from source follow the instructions below. Before building anything please review all of the instructions, including any environment setup steps that may be required prior to the actual build.

NOTE: Building Kubernetes for Mesos requires Go 1.3+.

• See the development page for sample environment setup steps.

Kubernetes-Mesos is built using a Makefile to automate the process of patching the vanilla Kubernetes code. At this time it is highly recommended to use the Makefile instead of building components manually using go build.

$ cd $GOPATH # If you don't have one, create directory and set GOPATH accordingly.

$ mkdir -p src/github.com/mesosphere/kubernetes-mesos
$ git clone https://github.com/mesosphere/kubernetes-mesos.git k8sm
$ cd k8sm
$ make                             # compile the binaries
$ make test                        # execute unit tests
$ make install DESTDIR=$(pwd)/bin  # install to ./bin

Start the framework

NETWORKING: Kubernetes v0.5 introduced "Services v2" which follows an IP-per-Service model. A consequence of this is that you must provide the Kubernetes-Mesos framework with a CIDR subnet that will be used for the allocation of IP addresses for Kubernetes services: the -portal_net parameter. Please keep this in mind when reviewing (and attempting) the example below - the CIDR subnet may need to be adjusted for your network. See the Kubernetes release notes for additional details regarding the new services model.

The examples that follow assume that you have a running Mesos cluster. If needed you may download and install Mesos binaries through your OS package manager, or else download them directly from Mesosphere and install them manually. Once your Mesos cluster is up and running you're ready to fire up kubernetes-mesos.

To keep things simple the following guide also assumes that you intend to run the mesos-master, etcd, and the kubernetes-mesos framework processes on the same host, exposed on an IP address referred to hereafter as ${servicehost}. Alternatively, the instructions also support a multi-mesos-master cluster running with Zookeeper.

$ export servicehost=...  # IP address of the framework host, perhaps $(hostname -i)

etcd

If you are not running in a production setting then a single etcd instance will suffice. To run etcd, see github.com/coreos/etcd, or run it via docker:

$ sudo docker run -d --hostname $(hostname -f) -p 4001:4001 -p 7001:7001 coreos/etcd

kubernetes-mesos

Ensure that your Mesos cluster is started. If you're running a standalone Mesos master on ${servicehost} then set:

$ export mesos_master=${servicehost}:5050

Or if you have multiple Mesos masters registered with a Zookeeper cluster then set:

$ export mesos_master=zk://${zkserver1}:2181,${zkserver2}:2181,${zkserver3}:2181/mesos

Fire up the kubernetes-mesos framework components (yes, these are all required for a working framework):

$ ./bin/km apiserver \
  --address=${servicehost} \
  --mesos_master=${mesos_master} \
  --etcd_servers=http://${servicehost}:4001 \
  --portal_net=10.10.10.0/24 \
  --port=8888 \
  --cloud_provider=mesos

$ ./bin/km controller-manager \
  --master=$servicehost:8888 \
  --mesos_master=${mesos_master}

$ ./bin/km scheduler \
  --address=${servicehost} \
  --mesos_master=${mesos_master} \
  --etcd_servers=http://${servicehost}:4001 \
  --mesos_user=root \
  --api_servers=$servicehost:8888

For simpler execution of kubectl:

$ export KUBERNETES_MASTER=http://${servicehost}:8888

You can increase the verbosity of the logging for the API server, scheduler, and/or the controller-manager by including, for example, --v=2. This can be very helpful while debugging.

###Launch a Pod

Assuming your framework is running on ${KUBERNETES_MASTER}, then:

$ bin/kubectl create -f examples/pod-nginx.json
# -- or --
$ curl -L ${KUBERNETES_MASTER}/api/v1beta1/pods -XPOST -d @examples/pod-nginx.json

After the pod get launched, you can check it's status via kubectl, curl or your web browser:

$ bin/kubectl get pods
POD          IP           CONTAINER(S)  IMAGE(S)          HOST                       LABELS    STATUS
nginx-id-01  172.17.6.20  nginx-01      dockerfile/nginx  10.22.211.18/10.22.211.18  name=foo  Running

# -- or --
$ curl -L ${KUBERNETES_MASTER}/api/v1beta1/pods

{
  "kind": "PodList",
  "creationTimestamp": null,
  "selfLink": "/api/v1beta1/pods?namespace=",
  "resourceVersion": 2800,
  "apiVersion": "v1beta1",
  "items": [
    {
      "id": "nginx-id-01",
      "uid": "2cf6694e-bd16-11e4-90fa-42010adf71e3",
      "creationTimestamp": "2015-02-25T17:46:06Z",
      "selfLink": "/api/v1beta1/pods/nginx-id-01?namespace=default",
      "resourceVersion": 2751,
      "namespace": "default",
      "labels": {
        "name": "foo"
      },
      "desiredState": {
        "manifest": {
          "version": "v1beta2",
          "id": "",
          "volumes": null,
          "containers": [
            {
              "name": "nginx-01",
              "image": "dockerfile/nginx",
              "ports": [
                {
                  "hostPort": 31000,
                  "containerPort": 80,
                  "protocol": "TCP"
                }
              ],
              "resources": {},
              "livenessProbe": {
                "httpGet": {
                  "path": "/",
                  "port": "80"
                },
                "initialDelaySeconds": 30,
                "timeoutSeconds": 1
              },
              "terminationMessagePath": "/dev/termination-log",
              "imagePullPolicy": "PullIfNotPresent",
              "capabilities": {}
            }
          ],
          "restartPolicy": {
            "always": {}
          },
          "dnsPolicy": "ClusterFirst"
        }
      },
      "currentState": {
        "manifest": {
          "version": "",
          "id": "",
          "volumes": null,
          "containers": null,
          "restartPolicy": {}
        },
        "status": "Running",
        "Condition": [
          {
            "kind": "Ready",
            "status": "Full"
          }
        ],
        "host": "10.22.211.18",
        "hostIP": "10.22.211.18",
        "podIP": "172.17.6.20",
        "info": {
          "POD": {
            "state": {
              "running": {
                "startedAt": "2015-02-25T17:46:07Z"
              }
            },
            "ready": false,
            "restartCount": 0,
            "podIP": "172.17.6.20",
            "image": "kubernetes/pause:latest",
            "imageID": "docker://6c4579af347b649857e915521132f15a06186d73faa62145e3eeeb6be0e97c27",
            "containerID": "docker://811760e5070fd3c8e8014aff2a169831adc0c602833b540f689d76be6fadabf1"
          },
          "nginx-01": {
            "state": {
              "running": {
                "startedAt": "2015-02-25T17:46:08Z"
              }
            },
            "ready": true,
            "restartCount": 0,
            "image": "dockerfile/nginx",
            "imageID": "docker://0180c66bffa9450b438970b9350295c1b5d4345a8b4573abe8e2b46b075e63f8",
            "containerID": "docker://371ae01aa63b01875410f5bcca42ccdcc03872d64c351fbdcadcce2cba4a578f"
          }
        }
      }
    }
  ]
}

Or, you can run docker ps on the appropriate Mesos slave to verify that the example container is running:

$ docker ps
CONTAINER ID  IMAGE                     COMMAND   CREATED  STATUS  PORTS                  NAMES
371ae01aa63b  dockerfile/nginx:latest   "nginx"   11m ago  Up 11m                         k8s_nginx-01.aa49fb33_nginx-id-01.default.mesos_2cf6694e-bd16-11e4-90fa-42010adf71e3_c4ef9708
811760e5070f  kubernetes/pause:go       "/pause"  11m ago  Up 11m  0.0.0.0:31000->80/tcp  k8s_POD.5c99ea13_nginx-id-01.default.mesos_2cf6694e-bd16-11e4-90fa-42010adf71e3_3d99b13e

###Launch a Replication Controller

Assuming your framework is running on ${KUBERNETES_MASTER} and that you have multiple Mesos slaves in your cluster, then:

$ bin/kubectl create -f examples/controller-nginx.json
# -- or --
$ curl -L ${KUBERNETES_MASTER}/api/v1beta1/replicationControllers -XPOST -d@examples/controller-nginx.json

After the pod get launched, you can check it's status via kubectl, curl or your web browser:

$ bin/kubectl get replicationControllers
CONTROLLER          CONTAINER(S)        IMAGE(S)            SELECTOR            REPLICAS
nginxcontroller     nginx               dockerfile/nginx    name=nginx          2

# -- or --
$ curl -L ${KUBERNETES_MASTER}/api/v1beta1/replicationControllers

{
  "kind": "ReplicationControllerList",
  "creationTimestamp": null,
  "selfLink": "/api/v1beta1/replicationControllers?namespace=",
  "resourceVersion": 3087,
  "apiVersion": "v1beta1",
  "items": [
    {
      "id": "nginxcontroller",
      "uid": "5f091198-bd18-11e4-90fa-42010adf71e3",
      "creationTimestamp": "2015-02-25T18:01:49Z",
      "selfLink": "/api/v1beta1/replicationControllers/nginxcontroller?namespace=default",
      "resourceVersion": 3041,
      "namespace": "default",
      "desiredState": {
        "replicas": 2,
        "replicaSelector": {
          "name": "nginx"
        },
        "podTemplate": {
          "desiredState": {
            "manifest": {
              "version": "v1beta2",
              "id": "",
              "volumes": null,
              "containers": [
                {
                  "name": "nginx",
                  "image": "dockerfile/nginx",
                  "ports": [
                    {
                      "hostPort": 31001,
                      "containerPort": 80,
                      "protocol": "TCP"
                    }
                  ],
                  "resources": {},
                  "terminationMessagePath": "/dev/termination-log",
                  "imagePullPolicy": "PullIfNotPresent",
                  "capabilities": {}
                }
              ],
              "restartPolicy": {
                "always": {}
              },
              "dnsPolicy": "ClusterFirst"
            }
          },
          "labels": {
            "name": "nginx"
          }
        }
      },
      "currentState": {
        "replicas": 2,
        "podTemplate": {
          "desiredState": {
            "manifest": {
              "version": "",
              "id": "",
              "volumes": null,
              "containers": null,
              "restartPolicy": {}
            }
          }
        }
      },
      "labels": {
        "name": "nginx"
      }
    }
  ]
}

Test

Run test suite with:

$ make test.v