Hurdles in Big Data Analytics for Retail

Introduction

Retail is one of the oldest platforms for a buyer to interact with a seller. Long before ECommerce and m-Commerce hit it big, the retail industry relied on great product displays, competitive pricing and effective salesmanship to get the job done.

With the growth of technology and its numerous flavors and avatars, the retail industry pulled in what it needed.

Today, with sales in the US alone over 5 trillion in 2017 and with the pressure to create a superior experience (86% of customers say that they will pay more for a superior experience), the retail industry is turning to niche technology to get a better job done.

Big Data in Retail

The niche technology here is big data. It is niche but it is no longer a technology that is in the labs.

The spending on big data was projected to reach USD 57 billion in 2017 with 6 million developers around the globe.

The retail industry is the perfect candidate for big data services as it generates data with great volume, variety and velocity.

From the products that are bought by the customers to the modes of payment used to the trends in buying in a particular festival to their buying decision factors in terms of price, quality and accessibility, there is an enormous amount of data to crunch and learn from.

To get a rough idea of that, WalMart alone generates a bewildering 2.5 petabytes of data per hour from customer transactions!

So where there is tremendous scope to use big data in the retail industry and get amazing insights into what is and what isn’t working for the customer, there are also significant hurdles that retailers have to contend with, to truly harvest the fruits of big data.

Hurdles faced by retailers when adopting or using big data analytics

Whenever a big data project is on the implementation plan for a retailer, the ‘what data to be gathered’ is an important question.

Where there is data, there is also a lot of noise which is essentially other data. Closely tied to customer modes of payment is the amount of money they transact.

Also, a customer may use multiple modes of payment for a single transaction. Is it possible to identify each mode separately and know for sure which is what?

The data about products bought is hugely significant.

But if a customer uses a relative’s credit card to purchase, you will end up building a misleading persona for the relative while missing out on data for the actual customer.

The way data is qualified in a big data scenario is extremely important and an important hurdle, since the customer is generally unpredictable who cannot be pigeonholed into fixed segments of behavior.

Collecting and collating the data from disparate systems

Even a mid-sized retailer today has a software that takes care of warehouse and inventory, a billing software that stores data on front-end transactions and a Customer Relationship Management (CRM) software (if any).

Each system stores the data in its specific format. It is quite a hurdle to gather data from these different systems that work in silos and unify that data into one single unit that is fit for analysis.

This is what the Extract and Transform processes of the famous ETL-processes of big data entails.

It is well known that it needs to be done, but it is still a challenge many a times.

Ensuring data security and compliance

A retailer in the EU who is considering big data already knows the implications of the upcoming GDPR (General Data Protection Regulation) that will soon be in effect.

Even without the compliance problems, there is the general challenge of securing the data that is painstakingly gathered.

From hardening servers to carrying out repeated penetration tests to security internal audits, it is important to secure the data today.

But with data breaches (the recent problems with Equifax, for example) being rampant, the aspect of data security is a hurdle that one must pass for a successful big data implementation.

A big data exercise can be quite daunting to execute after buying into the great promises made by your vendors.

If there is a data-gathering point (like customers viewing products while their views captured by eye-scanners or an entry into the inventory software for every new warehouse delivery) then there is a device that either needs to be run by a human or it needs some sort of routine maintenance.

Either of those increase the responsibility that your staff has and it can delay its adoption that goes back to point 1, where the veracity (widely touted as the fourth V of big data) of the data will be in question.

Drawing the right insights in a timely manner

A big data tool can help you extract, transform and load the data and even crunch it to reveal patterns and trends.

While this is the intent which looks great on paper and proposals, the actual time taken for this process can vary (due to the set-up time brought in by some of the above stated factors).

In retail, trends and patterns age out very soon.

It is a hurdle to implement big data swiftly so that there is time to draw the insights needed and take the necessary steps to utilize that information.

Even when there is time, drawing the right insights can itself take multiple board room meetings and approval cycles, all of which cost invaluable time.

Some might propose to use a ‘real-time’ big data solution but even then, the setup could take time.

Earning the customer trust to capture their data

Everyone loves to bash the so-called big corporations that are gathering data for their vested interests.

Any such news can instantly become a PR nightmare for a retailer.

While the necessary steps to ensure security are needed, it is also important to painstakingly secure the customer’s consent and to assure them that the data gathered is to be used in a safe and secure manner that will benefit no one else but the customer only.

Conclusion

While the challenges are there, the opportunity is also tremendous. According to a study published in HBR, companies creating the perfect ‘omni-channel experience’ by using data analytics increased their shareholders value to 8.5 times.

This report by Oracle says that retailers can gain a 60% increase in their operating margins by using big data.

For every problem or hurdle stated above, the solution lies in careful, thoughtful big data implementation using mature tools, guided by the right mix of developers and data scientists.

After all, this is really the next big technological bandwagon.

If you are retailing anything today, big data analytics is the best way to get an insight into the customer’s otherwise enigmatic mind.