As Big Data gets hot, data geeks become cool

Recent estimates put the rate of consumer data capture at about 800 new records per person per day.

Consider the metadata describing one simple mobile call that is, the information about the call, not the information within the call. The telco stores the caller and receiver IDs, their locations, time of day and duration of the call. As an isolated datum, pretty uninspiring. But as an SK Telecom representative described at MobileWorld in Hong Kong last year, a consistent pattern of short calls between the same two numbers during daytime is a strong predictor of a marriage; long calls in the evening is a predictor of unwed lovers. And of course with that, a great marketing opportunity for the telco.

This practice of acquiring, analysing and interpreting ridiculously huge amounts of data is now known as ‘Big Data’.  And it has the technology and marketing worlds buzzing. Here we go again. Another catchphrase that simply repackages an existing idea. Big Data?Big deal.

But this time, there’s meaning in the meme. The term ‘Big Data’ means much more than companies and government agencies collecting lots of personal information about people. The data stores are so huge that our brains simply cannot comprehend the patterns and correlations contained within.

Try to visualise a billion people, or the 150 million kilometers that takes you to the Sun. Big numbers are hard enough on their own. Big Data introduces a new set of challenges resulting from amassing sets of data so large and complex that traditional data management and analytics can’t handle them. Mastering Big Data offers the promise of great rewards for those who invest. The hardest thing is knowing what question to ask and what problem you’re solving.

One of my favourite examples is American retailer Target’s ability to predict customer pregnancy based on product purchases, as published in the New York Times in February this year. Whilst it may sound relatively easy, identifying pregnant customers is harder than it sounds. Starting with transaction data from women who signed up for the Baby Register, Target identified a shift from scented to unscented body lotion purchases, and increasing quantities of mineral supplements like calcium and magnesium. The analytics team built a mathematical model based on these parameters, and found it to be highly predictive of second trimester pregnancy.

The genius in this example is not that Target managed to successfully cross-sell baby products to expectant mothers. It’s that they were addressing a much bigger problem: how to change people’s ingrained buying habits.

In the 1980s, Professor Alan Andreasen of UCLA studied purchasing patterns of FMCGs like soap, toothpaste and toilet paper, and found that most consumers made these purchases habitually with little decision making. This makes it very hard for marketers to persuade shoppers to change, no matter how attractive the offer or incentive. But becoming a parent is one of those times in life where habits do change.

Exhausted parents need easier, cheaper and less time consuming shopping experiences, so loyalty is up for grabs. As birth records are public, new parents are inundated with offers at the time of a baby’s birth. The key is to secure loyalty early, as Target has done by predicting pregnancy in the second trimester and commencing their targeted campaigns from this point.

Watch out for my next post when I look at how data security is changing with the emergence of Big Data.