Posts

Did you ever have to analyze large log files before?

Well, I recently had to analyze a large log based dataset and decided to try out the ELK stack . Introduction: Please see this intro if you are not familiar with ELK, the rest of the blog assumes you know what they are. eg. elastic server, logstash & kibana. It is a popular server side tool to index, search & graph a large collection of logs or similar structured/unstructured data. This blog post mainly talks about my experience setting up this well known stack & the unexpected things learnt during the process. Goal : "How do I enable rich filtering & analysis on the large set of product logs beyond some simple scripting?" I thought I just had to upload a few log files to the server & then have some awesome graphs appear almost magically out of the box! Servers : "How do I get a server up and running?" There is a free trial for cloud based elastic stack. I signed up and got a 14 day free trial with 4 instance running on Google Clo...

Mobile development - to be native or not!

Whenever you start a new cross-platform project, there is always the question about whether to use native languages and tools for each platform or just build an HTML/javascript solution that works fine on all platforms. The decision to use HTML/JS is easy for most teams - you have a single code base, you are working against a tight budget, you have hardly enough time to develop for one platform, you have to develop a website anyway to handle the fringe mobile platforms or to provide ubiquitous access, the team may have expertise only on a few platforms & you can create quick prototypes to impress the investors or execs etc. The rich ecosystem around tools & open source libraries also helps with the choice. Ionic has emerged as the leading platform for building applications for mobile platforms using HTML/JS technologies. It leverages angular JS & Cordova to offer the promise of build once and deploy everywhere.  But, there is always a catch, right?  The HTML...

Hacking Kubernetes on Azure

Kubernetes got some press recently because Microsoft, IBM, RedHat and Google agreed to work together on a cloud platform. I was looking into a similar platform at the time (Mesos/Marathon stack from Apache) which was good, but didn't quite pass my open source litmus test. (ie. Ability to hack a working build in under 5 minutes). I had a mesos cluster running, but I had to build the zookeeper binaries from source to get the multi-machine cluster working. Kubernetes cluster was up and running locally without any trouble in vagrant environment. git clone https://github.com/GoogleCloudPlatform/kubernetes cd kubernetes vagrant up The one machine cluster was also up and running quite easily, but the nginx docker images didn't deploy because of port conflicts. Then, I launched a cluster on GCE which also worked without any issues. So far, so good. Why Azure? - I had MSDN credits that I could use. I tried to use the azure scripts that were included, but it seemed to ...

Google Appengine

It was good while it lasted, Google AppEngine gave developers an easy way to create web applications without worrying about scaling it or being a sysadmin. There were numerous startups that built their products on Google Appengine. Even after beta, there was a fairly large free quota that made it a reasonable choice. But, the current pricing seems a bit ridiculous for those who are looking for a free or cheap solution or for those enjoyed the initial quotas. Perhaps, we were spoiled and Google’s goal is to get rid of free-riders which is not exactly evil from Google’s perspective. So, it is 2012 and welcome to reality since Google can’t maintain those quotas forever and the current free quotas are intended to only test your application before you go live. If for some strange reason, your application becomes popular and gets more than 1,000 users, you would end up scrambling for an alternate solution since you will find yourself paying through the nose. The quota system is complica...

What is up with RIM?

RIM is one of the leading technology companies in Canada and one of the largest employers in Waterloo. There are a few strategic mistakes that seem obvious in hindsight, but it is easier to list them now than to predict in advance. RIM is losing market share in North America, but surprisingly the execs were focusing on developing markets for growth. Obviously, profit margins from developing countries are minimal when compared to the lucrative NA market. RIM’s strength is in the enterprise market. The primary growth is happening in consumer market, but RIM has had trouble gaining traction there. RIM will continue to rule the enterprise for now, because changes come at a slow pace there. Still, the consumerization of IT is the biggest threat for RIM in this area. The current instability doesn't help them advance in enterprise either. RIM is losing the high end smart phone market which is ruled by Apple and Android devices, but instead is happy with the wins in mid-low end ...

Blackberry Playbook

RIM introduced the "Playbook" tablet computer with much fanfare yesterday.It relies on Flash & HTML5 as the key development platforms. RIM was able to build the compatibility for existing apps on top of QNX which is good. It was smart to rely on the HTML5 platform given the poor UI capabilities in the existing Java framework, it wasn't clear if the tablet supported native game development despite the support for OpenGL in QNX platform. The specs are great compared to the existing iPad, RIM did need some +ve hype at this point after the Torch fiasco. It is just 10mm thick compared to 13.4mm for the iPad and I thought iPad was thin. It has a 7" screen, but the resolution is 1024x600 whereas it is 1024x768 for the 9.7" iPad. Playbook also comes with a dual core 1GHZ processor compared to the single core CPU in iPad. Playbook also comes with front and back cameras and HD display with HDMI output compared with none for iPad. It also tethers with the Blackberr...

Mobile tech landscape

Just some personal thoughts from development and business perspectives: Apple - iPhones and iOS; top of the line industrial design and beautiful user interfaces. Combine this with the maturity of the Mac OS platform, excellent native development tools and users willing to buy apps on AppStore this is a top platform to target. The requirement for Macs & to learn Objective-C are initial deterrents, but probably it just helps keep Windows developers away. The simple fact is that develepors will be where the money is and Apple Appstore is one place where open source hasn't yet taken away the revenues for indie developers. iPads have extended the success of iPhones, but has made it challenging for developers to target multiple screen sizes. Android - This is one of the hottest mobile platforms; mainly because people equate this to PC vs Mac from the good old days. (open vs closed). The Android platform is indeed one of the most developer friendly mobile platforms out there, it...