Vishnu Varadaraj's Blog

Well, I recently had to analyze a large log based dataset and decided to try out the ELK stack . Introduction: Please see this intro if you are not familiar with ELK, the rest of the blog assumes you know what they are. eg. elastic server, logstash & kibana. It is a popular server side tool to index, search & graph a large collection of logs or similar structured/unstructured data. This blog post mainly talks about my experience setting up this well known stack & the unexpected things learnt during the process. Goal : "How do I enable rich filtering & analysis on the large set of product logs beyond some simple scripting?" I thought I just had to upload a few log files to the server & then have some awesome graphs appear almost magically out of the box! Servers : "How do I get a server up and running?" There is a free trial for cloud based elastic stack. I signed up and got a 14 day free trial with 4 instance running on Google Clo...

Whenever you start a new cross-platform project, there is always the question about whether to use native languages and tools for each platform or just build an HTML/javascript solution that works fine on all platforms. The decision to use HTML/JS is easy for most teams - you have a single code base, you are working against a tight budget, you have hardly enough time to develop for one platform, you have to develop a website anyway to handle the fringe mobile platforms or to provide ubiquitous access, the team may have expertise only on a few platforms & you can create quick prototypes to impress the investors or execs etc. The rich ecosystem around tools & open source libraries also helps with the choice. Ionic has emerged as the leading platform for building applications for mobile platforms using HTML/JS technologies. It leverages angular JS & Cordova to offer the promise of build once and deploy everywhere.  But, there is always a catch, right?  The HTML...

Kubernetes got some press recently because Microsoft, IBM, RedHat and Google agreed to work together on a cloud platform. I was looking into a similar platform at the time (Mesos/Marathon stack from Apache) which was good, but didn't quite pass my open source litmus test. (ie. Ability to hack a working build in under 5 minutes). I had a mesos cluster running, but I had to build the zookeeper binaries from source to get the multi-machine cluster working. Kubernetes cluster was up and running locally without any trouble in vagrant environment. git clone https://github.com/GoogleCloudPlatform/kubernetes cd kubernetes vagrant up The one machine cluster was also up and running quite easily, but the nginx docker images didn't deploy because of port conflicts. Then, I launched a cluster on GCE which also worked without any issues. So far, so good. Why Azure? - I had MSDN credits that I could use. I tried to use the azure scripts that were included, but it seemed to ...

It was good while it lasted, Google AppEngine gave developers an easy way to create web applications without worrying about scaling it or being a sysadmin. There were numerous startups that built their products on Google Appengine. Even after beta, there was a fairly large free quota that made it a reasonable choice. But, the current pricing seems a bit ridiculous for those who are looking for a free or cheap solution or for those enjoyed the initial quotas. Perhaps, we were spoiled and Google’s goal is to get rid of free-riders which is not exactly evil from Google’s perspective. So, it is 2012 and welcome to reality since Google can’t maintain those quotas forever and the current free quotas are intended to only test your application before you go live. If for some strange reason, your application becomes popular and gets more than 1,000 users, you would end up scrambling for an alternate solution since you will find yourself paying through the nose. The quota system is complica...

RIM is one of the leading technology companies in Canada and one of the largest employers in Waterloo. There are a few strategic mistakes that seem obvious in hindsight, but it is easier to list them now than to predict in advance. RIM is losing market share in North America, but surprisingly the execs were focusing on developing markets for growth. Obviously, profit margins from developing countries are minimal when compared to the lucrative NA market. RIM’s strength is in the enterprise market. The primary growth is happening in consumer market, but RIM has had trouble gaining traction there. RIM will continue to rule the enterprise for now, because changes come at a slow pace there. Still, the consumerization of IT is the biggest threat for RIM in this area. The current instability doesn't help them advance in enterprise either. RIM is losing the high end smart phone market which is ruled by Apple and Android devices, but instead is happy with the wins in mid-low end ...

RIM introduced the "Playbook" tablet computer with much fanfare yesterday.It relies on Flash & HTML5 as the key development platforms. RIM was able to build the compatibility for existing apps on top of QNX which is good. It was smart to rely on the HTML5 platform given the poor UI capabilities in the existing Java framework, it wasn't clear if the tablet supported native game development despite the support for OpenGL in QNX platform. The specs are great compared to the existing iPad, RIM did need some +ve hype at this point after the Torch fiasco. It is just 10mm thick compared to 13.4mm for the iPad and I thought iPad was thin. It has a 7" screen, but the resolution is 1024x600 whereas it is 1024x768 for the 9.7" iPad. Playbook also comes with a dual core 1GHZ processor compared to the single core CPU in iPad. Playbook also comes with front and back cameras and HD display with HDMI output compared with none for iPad. It also tethers with the Blackberr...

Just some personal thoughts from development and business perspectives: Apple - iPhones and iOS; top of the line industrial design and beautiful user interfaces. Combine this with the maturity of the Mac OS platform, excellent native development tools and users willing to buy apps on AppStore this is a top platform to target. The requirement for Macs & to learn Objective-C are initial deterrents, but probably it just helps keep Windows developers away. The simple fact is that develepors will be where the money is and Apple Appstore is one place where open source hasn't yet taken away the revenues for indie developers. iPads have extended the success of iPhones, but has made it challenging for developers to target multiple screen sizes. Android - This is one of the hottest mobile platforms; mainly because people equate this to PC vs Mac from the good old days. (open vs closed). The Android platform is indeed one of the most developer friendly mobile platforms out there, it...

1) People don't often invest in ideas, people invest in people. eg. Heck, we are going to have fun together building 'x'. 2) Don't just create cool technology, instead produce something that solve a customer pain. hint: If they don't want to pay for it, then it is probably just a cool toy. 3) What you decide to not do is sometimes more important that what you decide to do. 4) Focus on making the pie bigger, not on cutting smaller pieces. ie. share equity if you end up with a bigger net value. 5) Hire smart, trustworthy people that share your passion, never ever hire a single poor performer or a bad team player. eg. A grader hires other A graders, but B grader hires other C graders. 6) Focus is better than hope. eg. targetting 1% of 50 million market is not a good goal. 7) Avoid raising money from friends & family; hint: few startups actually succeed. 8) Write a business plan, more to help you than to help others. Don't bother to create one if you ...

Microsoft doesn't seem to have changed it's strategy a little bit even though many have hailed this as a 180 degree turnaround. It is again copying what other startups and market creators have done (cloud computing, internet as a platform etc) and then executing on the plan so well that it is cheap(?), superior and accessible to everyone. At the same time, it also tries to win back the developers and defend its platform. So, what is new? For those who missed the announcement, Microsoft came out with a comprehensive cloud computing platform for developers and organizations which can be used to host applications developed for the platform. To its credit, the platform is so comprehensive and well integrated that Google's equivalent AppEngine seems like.. well, something that was built on 20% time. (But, note that the Sql Data services seems awfully similar to the Google Bigtable model.) It also seems to have pulled the entire company including Office, SQLServer, Developer ...

Thought Networking (from Primal Fusion) This may be unknown to anyone outside Waterloo, but yesterday a local startup (Primal Fusion) which was in stealth mode for over 3 years talked about what they are doing for the first time. It was interesting to watch the understated local hype since it was run by Yvan Couture, who had succesfully sold his startup to Agfa for a good profit and is also quite active in the community. The only reason I was interested was because I had listened to a talk by Yvan recently at UW. He seemed like a genuine and trustworthy guy which is rather rare in business nowadays. The core idea behind the company is related to thought networking. They hope to define what it is and be the leader for a new market category. The simple example given was to go from the approach of searching for something to have the required information delivered to you once you have defined your thoughts. Their goal is to capture your thoughts and represent them as semantic networks. Af...

Product quality - Dev & QA synergy. Quality is a problem which every software development team has to deal with. We have the usual bell curves etc which we use to gain confidence regarding overall stability before we ship. But, here are some observations based on the timelines of the curve & potential dev/qa synergy. There are some projects where many bugs are found after the 'code freeze'. This is a clear indication that the dev & qa team had issues. It is possible dev didn't unit test properly or many changes were made late in the cycle and either QA took time to ramp up and understand the changes or were busy with other things. There are some projects where many bugs are found immediately after 'code complete' and then taper off quickly. This may be the case with many projects and might be a good thing. It means the QA team was able to understand the changes quickly and cover new areas thoroughly. It might also mean that the Dev team didn't have e...

mobileMe - Wow! The sleeper announcement at the Apple WWDC was probably mobileMe. It is what MSFT, GOOGL etc. would have loved to build. I had earlier blog entries indicating that I liked .mac & how synchronization is the key to merge the web & devices (eg. SharpCast). Microsoft tried it with LiveMesh but failed before it even took off.(To be fair, I never tried the service) The seamless integration with iPhone, Desktop, Web & even PCs in mobileMe was amazing. No doubt it will be copied by many - it is true that even mobileMe concept is based on Exchange! And yes, I need to get a 3G iPhone this July - now, if only mobileMe was free!

Google App Engine - First impressions * It is free, but with quotas. The business model is probably to get others to build apps so that Google can get users, searchable data & then eventually offer a business edition. * You don't have to worry about hosting, load balancing & scaling which is indeed huge since writing the web application is the easiest among all of the above. * The initial release is not ready for business apps, it doesn't have an SLA nor does it have a good support for authentication & authorization. * You need to learn Python if you haven't already, but future support for ruby or php could be expected soon. * There are no issues in hosting flash or flex application except for the login to google account issue. * Python frameworks like Django are natively supported, but you can use your own framework as well. * It is possible to do GQL queries and Google full text search support. * And there is the full offline sdk for those who can't get an...

iPhone SDK - First impressions. This is probably the most comprehensive SDK released for a mobile OS, even though Windows Mobile probably is similar. It must have been easy for Apple since most of the functionality was already in in Mac OS, throw in a different UI framework & a bunch of innovations on touch, acclerometer and location sensing you have the mobile SDK. RIM's blackberry SDK is many years old but is so immature compared to this. It is not without it's own set of problems though. The reliance on old Mac tools means you need a Mac computer running Leopard to use it. The support for favorites like Java(upcoming from Sun)/C# is minimal and programming in Objective C is closest to programming in Win32. So, developer productivity takes a hit and would automatically reduce the number of developers who can produce business applications for example. It also doesn't expose some of the phone & SMS functionality to prevent misuse unlike similar offerings from compet...

Parallels vs VMWare Just a mental note to myself to keep off Parallels on Mac if I have a choice, it wasted almost my entire weekend since it had trouble after I upgraded the OS to Leopard & installed new parallels trial. XP image kept on crashing after a Windows update. Finally, I switched to VMWare and had to reinstall all OSes. Hopefully, it is just a problem with the trial copy, but I found VMWare to be trouble free. I had a great opinion about Parallels, but all that changed this weekend. My vote - VMWare!

Ruby on Rails It is hard to be excited over a new language nowadays, but recently I came across one which seemed to make web & database development fun again. I am referring to the combination of Ruby & Rails (RoR). This was a framework built for real world development problems rather than for 'building a framework' as per it's founder. It has several interesting features: 1. Lack of strong typing & support for 'duck typing' - it works even though you feel lost sometimes if you have been a C/C++/Java/C# programmer. 2. Configuration by convention - no need to configure stuff if you follow the convention, but you can configure if you need to as well. 3. Brevity of expression - introduced by C which can easily get you a cult following. eg. one line to write a program which prints itself. 4. Dynamic nature - ability to add methods or redefine methods on the fly - to any class in the system, even built-in ones. 5. script/console - a hidden gem, difficult to l...

The end of SOAP webservices. It is probably a bit early to say this, but SOAP webservices doesn't have a future. I had been a myopic fan of SOAP webservices and one of the early adopters of the technology in various implementations - .NET 1.0, axis, xfire, JAX-WS, BPEL, JSR 180 etc. It is a great example of how excitement about a new technology can blind your judgment. It failed the single most important test - it was just not _simple_ enough to succeed in the long term. 1. complexity meant few stable implementations. eg. jboss changed their webservice implementation twice or thrice already. 2. complexity meant inherent concerns about it's security after deployment. eg. how does it handle a malformed request? 3. it needed client & server support which meant changing platforms as your clients change. eg. javascript, actionscript, java, .NET clients. 4. it was supposed to be cross platform, but it required another standard (ws-i profile xx) to standardize the standard. Still,...

Future of the web. The answer is right in front of you even if you don't see it already: The web started with simple HTML. Then came the dynamic web where content is dynamically generated for the users. Then came the wikis which enabled editable webs. Then web applications got more interactive with AJAX. The next level of interactivity is where applications execute and behave like Desktop applications within the browser. Then there will be the web OS where you store all the important data in the web and access the same from everywhere. * The web OS sounds grand but the closest we have today for the client is Flash/Flex & Silverlight & XUL to a lesser extent. * This will blur the line between desktop apps & web apps further. * Making toy applications for web will no longer fly & traditional players might get another shot. (MSFT vs GOOGL) * The recent acquisition of 'Parakey' by Facebook is an indication of where the future is goi...

Sometimes you have to keep repeating a message so that everyone gets it, at the risk of sounding like a broken record. There was a quote in this month's Popular science which applies to  software world as I had been trying to say in earlier posts- "good engineers find solutions to hard problems that are complex and hard to understand, but great engineers find solutions to hard problems that are so simple that you wonder why you didn't think about it in the first place" Simplicity is not only important in software design but also to the user. I remember working on a product  which had a versioning scheme which was so complex that even the QA required several days of training to understand it. You can imagine the frustration for the customer and how long the feature existed in the product! The next level of simplicity is in the user interface, Microsoft Office 2007 UI is a great example of how to scare the user by showing all options. 37 signals has some good example...

ABCs of good design Good design and architecture has always been identified as a key requirement for a product to succeed in the long term. But how how can you measure the quality of a design or identify the recipe to come up with the best design always? These are not questions with easy answers, but help to illustrate the challenges in the area. There have been many approaches to improve the quality of design in computer software over the course of time: 1. Functions & subroutines - This helped improve the design of monolithic programs by separating commonly used functionality into functions & subroutines. This improved the maintenance and reuse of code by having less duplication etc. 2. OOP - This was another major step forward in design in which we tried to model the objects in real world using classes having methods & properties. This was more natural to humans than working with procedures. eg. Rumbaugh & Booch. 3. CASE - 'Computer aided software engineering...