Hacking Kubernetes on Azure


Kubernetes got some press recently because Microsoft, IBM, RedHat and Google agreed to work together on a cloud platform.

I was looking into a similar platform at the time (Mesos/Marathon stack from Apache) which was good, but didn't quite pass my open source litmus test. (ie. Ability to hack a working build in under 5 minutes). I had a mesos cluster running, but I had to build the zookeeper binaries from source to get the multi-machine cluster working.

Kubernetes cluster was up and running locally without any trouble in vagrant environment.

git clone https://github.com/GoogleCloudPlatform/kubernetes
cd kubernetes
vagrant up

The one machine cluster was also up and running quite easily, but the nginx docker images didn't deploy because of port conflicts.
Then, I launched a cluster on GCE which also worked without any issues. So far, so good.

Why Azure? - I had MSDN credits that I could use.
I tried to use the azure scripts that were included, but it seemed to reference old files & didn't quite work.
It also turned out that Kubernetes was only tested on either Redhat/CentOS or debian.

Support for Ubuntu 14.04:
The installation of salt used bootstrap scripts that were not supported on 14.04 as per the website.
Instead, use the method recommended by saltstack:

apt-get install -y software-properties-common
add-apt-repository -y ppa:saltstack/salt
apt-get update
apt-get install -y salt-master salt-minion

The installation scripts for docker wasn't quite doing the job, I took the easy route on this one by disabling docker in cluster/saltbase/salt/top.sls.
Instead use the insecure one liner installation:

curl -sSL https://get.docker.io/ubuntu/ | sudo sh

Support for Azure:
Kubernetes seemed to make some assumptions about the private network, so I decided to use the configuration from vagrant.
Use the Azure console to create a network with the following configuration. (ensure the ip range and subnets are present)
AddressPrefix: 10.0.0.0/8
Subnet-1: 10.245.1.0/24
Subnet-2: 10.245.2.0/24

Run this to create a build and upload to Azure storage:
./release/build-release.sh 

Run this to bring up the cluster:
./cluster/azure/kube-up.sh

Once you have the cluster running, you can ssh to the master or minions:
ssh -i ~/.ssh/azure yourcloudserviceshere.cloudapp.net -p 22000

For reference, I posted a copy of the code that I now have running on Azure @:
https://github.com/vishnuvaradaraj/kubernetes

Overall, the platform looks promising - It was easy to define pods to deploy all kinds of docker containers including nginx load balancers for node.js, mysql, grafana etc; the etcd service from coreos seems better than zookeeper due to better API access (REST) and can easily scale to thousands of machines. The salt framework was interesting, easy to use once you figure out how to debug the basic issues.

Location: Waterloo, ON, Canada

Comments

Anonymous said…
This is great stuff, I was planning to run this on Azure as well.. Did you script the network creation as well?
3:14 PM
Vishnu Varadaraj said…
yeah, I used powershell for that..
3:15 PM
Kubernetes cloud provider said…
Kubernetes cloud provider is really nice concept and I am waiting for it. Thanks for sharing
12:33 PM
Post a Comment

Popular posts from this blog

Google Appengine

It was good while it lasted, Google AppEngine gave developers an easy way to create web applications without worrying about scaling it or being a sysadmin. There were numerous startups that built their products on Google Appengine. Even after beta, there was a fairly large free quota that made it a reasonable choice. But, the current pricing seems a bit ridiculous for those who are looking for a free or cheap solution or for those enjoyed the initial quotas. Perhaps, we were spoiled and Google’s goal is to get rid of free-riders which is not exactly evil from Google’s perspective. So, it is 2012 and welcome to reality since Google can’t maintain those quotas forever and the current free quotas are intended to only test your application before you go live. If for some strange reason, your application becomes popular and gets more than 1,000 users, you would end up scrambling for an alternate solution since you will find yourself paying through the nose. The quota system is complica...
Read more

Blackberry Playbook

RIM introduced the "Playbook" tablet computer with much fanfare yesterday.It relies on Flash & HTML5 as the key development platforms. RIM was able to build the compatibility for existing apps on top of QNX which is good. It was smart to rely on the HTML5 platform given the poor UI capabilities in the existing Java framework, it wasn't clear if the tablet supported native game development despite the support for OpenGL in QNX platform. The specs are great compared to the existing iPad, RIM did need some +ve hype at this point after the Torch fiasco. It is just 10mm thick compared to 13.4mm for the iPad and I thought iPad was thin. It has a 7" screen, but the resolution is 1024x600 whereas it is 1024x768 for the 9.7" iPad. Playbook also comes with a dual core 1GHZ processor compared to the single core CPU in iPad. Playbook also comes with front and back cameras and HD display with HDMI output compared with none for iPad. It also tethers with the Blackberr...
Read more

What is up with RIM?

RIM is one of the leading technology companies in Canada and one of the largest employers in Waterloo. There are a few strategic mistakes that seem obvious in hindsight, but it is easier to list them now than to predict in advance. RIM is losing market share in North America, but surprisingly the execs were focusing on developing markets for growth. Obviously, profit margins from developing countries are minimal when compared to the lucrative NA market. RIM’s strength is in the enterprise market. The primary growth is happening in consumer market, but RIM has had trouble gaining traction there. RIM will continue to rule the enterprise for now, because changes come at a slow pace there. Still, the consumerization of IT is the biggest threat for RIM in this area. The current instability doesn't help them advance in enterprise either. RIM is losing the high end smart phone market which is ruled by Apple and Android devices, but instead is happy with the wins in mid-low end ...
Read more