Saturday, August 16, 2014

Hacking Kubernetes on Azure

Kubernetes got some press recently because Microsoft, IBM, RedHat and Google agreed to work together on a cloud platform.

I was looking into a similar platform at the time (Mesos/Marathon stack from Apache) which was good, but didn't quite pass my open source litmus test. (ie. Ability to hack a working build in under 5 minutes). I had a mesos cluster running, but I had to build the zookeeper binaries from source to get the multi-machine cluster working.

Kubernetes cluster was up and running locally without any trouble in vagrant environment.

git clone
cd kubernetes
vagrant up

The one machine cluster was also up and running quite easily, but the nginx docker images didn't deploy because of port conflicts.
Then, I launched a cluster on GCE which also worked without any issues. So far, so good.

Why Azure? - I had MSDN credits that I could use.
I tried to use the azure scripts that were included, but it seemed to reference old files & didn't quite work.
It also turned out that Kubernetes was only tested on either Redhat/CentOS or debian.

Support for Ubuntu 14.04:
The installation of salt used bootstrap scripts that were not supported on 14.04 as per the website.
Instead, use the method recommended by saltstack:

apt-get install -y software-properties-common
add-apt-repository -y ppa:saltstack/salt
apt-get update
apt-get install -y salt-master salt-minion

The installation scripts for docker wasn't quite doing the job, I took the easy route on this one by disabling docker in cluster/saltbase/salt/top.sls.
Instead use the insecure one liner installation:

curl -sSL | sudo sh

Support for Azure:
Kubernetes seemed to make some assumptions about the private network, so I decided to use the configuration from vagrant.
Use the Azure console to create a network with the following configuration. (ensure the ip range and subnets are present)

Run this to create a build and upload to Azure storage:

Run this to bring up the cluster:

Once you have the cluster running, you can ssh to the master or minions:
ssh -i ~/.ssh/azure -p 22000

For reference, I posted a copy of the code that I now have running on Azure @:

Overall, the platform looks promising - It was easy to define pods to deploy all kinds of docker containers including nginx load balancers for node.js, mysql, grafana etc; the etcd service from coreos seems better than zookeeper due to better API access (REST) and can easily scale to thousands of machines. The salt framework was interesting, easy to use once you figure out how to debug the basic issues.


Anonymous said...

This is great stuff, I was planning to run this on Azure as well.. Did you script the network creation as well?

Vishnu Varadaraj said...

yeah, I used powershell for that..

Kubernetes cloud provider said...

Kubernetes cloud provider is really nice concept and I am waiting for it. Thanks for sharing