Bug hunting with analytics

Finding bugs is never easy. You get customer complaints, spend tens of hours reproducing and fixing the problems, then start all over again. It's the never-ending circle of software development.

Today, we'll share the way we found a bug in a software we never used or even installed on our machines. It all comes down to data, this is where our Installer Analytics platform gave us the upper hand.

How it all started

On Monday morning, as we were reviewing our weekly report from Installer Analytics for Advanced Installer, we saw that so longed hockey stick rise. For the last 3 days we were getting x times more installations than our average daily rate.

Since Advanced Installer isn't as addictive as Flappy Bird we started looking closer at our data. First, we checked that no bugs from Installer Analytics where to blame for incorrect data display.

Once the data from the reports was checked we went back to our website, to see if Google Analytics is showing the same surge in website traffic. But everything was looking normal, all the average numbers were still there.

Validating the installs/downloads

We started digging deeper, to see if the direct download link for Advanced Installer is being accessed (this is something Google Analytics would not catch). We looked into our Apache logs from the web server and got the confirmation that indeed the number of downloads were multiplied by x times.

The same multiplication factor did not reflect in sales, so we knew somehow Advanced Installer is being installed by a huge number of people accidentally.

We went back to Installer Analytics, this time we checked the number of upgrades, to see if maybe a large number of old users were all upgrading at the same time, but that was not the case, as the number of upgrades was staying at its normal rates.

Finding the culprit

But Installer Analytics had one more ace up its sleeve, the Uninstall Surveys it collects. When we started analyzing them in detail we immediately noticed a trend among users reporting they were removing the application because they never installed it.

At that point we knew for sure some other software was automatically downloading and installing us on all those machines. Tracing the culprit was probably the easiest part. Data from the Apache logs and some messages from the uninstall surveys lead us directly to ManageEngine's Desktop Central solution and ultimately to their bug.

In the end we are sad our user base and sales did not go through the roof, but we cannot hold ourself from sharing the way Installer Analytics helped us unravel this enigma. Without it none of us would have looked in the Apache logs and the colleagues from support would have been in a total blackout when faced with users complaining about Advanced Installer appearing installed on their machines overnight.

Installer Analytics helped us save precious hours in support time for both us and our users, but it also helped us mitigate an event that could have turned against us (users reporting us as malware, etc...) if not managed correctly.