Don't let your big data end in big bang

So IDC predicts a total market for big data valued at £27bn by 2018 does it? No wonder so many people are getting heated about the potential success or failure of such an enormous investment (‘Analysts slam big data bullshit’, Decision Marketing, February 16, 2015).
The suspicion that the whole edifice is one with shaky foundations, built largely by self-interested technology vendors, is one that still lingers and is hard to shake off in some quarters of the industry. “Buy the right big data tools and your data analytics will deliver business changing results.” Quite rightly, many people are suspicious of this formulation. It just seems too good to be true.
Instead big data seems to have the potential to be a smokescreen which defines the problems faced by organisations without offering any viable solution. The theoretical physicist and cosmologist Steven Hawking believes that in our vast universe some form of life could very likely exist on another planet in another far, far distant galaxy. But how would you go about finding it out there among all the other millions of known planets – let alone those we haven’t even found yet? The pragmatic approach would be ‘one planet at a time’.
And just so with big data. As MailChimp data scientist John Foreman said in the story quoted above: “pick your big data battles”. I agree. In order to make any sense of it, and avoid being overwhelmed by the sheer scale of big data which threatens to grow at a pace no organisation can realistically keep tabs on, take each challenge as it comes.
Start by slicing off the haunch of data that looks most pertinent to the task, define the business objective and analyse what you have to see if it gives you a usable answer. If that doesn’t work take a different slice.
The probability of finding the most advanced form of life in the universe on the very first planet you visit is pretty small, as any statistician will tell you. So why do we expect analysts to come up with the perfect answer first time, especially when the universe of data they are sifting through is so very, very large and growing all the time? The term big data was coined for a very good reason.
Adopting this pragmatic approach will ensure that any investment in analysing big data is applied in the way most likely to produce actionable results.
As big data grows, so does our ability to refine and find better insights, but this takes time. So, as with looking for other forms of life, as our horizons grow, so do the places to look for answers.
New techniques and ways to hone our results will emerge – and so will the data available to analyse. By taking a measured analytical approach you can “sort the wheat from the chaff” (to quote that article again), and take manageable steps forward in a way best calculated to ensure that your big data strategy doesn’t end with an almighty big bang.