When to Use Big Data — and When Not To

“Big data” has been on the tip of everyone’s tongue for the past several years now, and for good reason. As digital devices and touchpoints proliferate, so too does the amount of data we each create. This information can be used to help us better understand clients and customers, make more effective decisions, and improve our business operations. But only if we can make sense of it all.

By choosing the right big data sources and applications, we can put our organizations at a competitive advantage. But to do that, we need to understand big data’s definition, capabilities, and implications.

Big data already has widespread applications. From Netflix recommendations to health care monitoring, it drives all types of predictive models that improve our daily lives. But the more we depend on it, the more we need to question how it shapes our lives and whether we should be relying on it so much. While progress is inevitable and something to embrace, big data’s contribution should not be measured by how many companies apply it, but by how much better off it makes society as a whole.

Defining Big Data and Its Relationship to Artificial Intelligence (AI)

Big data is more than just large datasets. It is defined by the three Vs of data management:

• Volume: Big data is often measured in terabytes.
• Variety: It can contain structurally different datasets, such as text, images, audio, etc.
• Velocity: Big data must be processed quickly because of the increasing speed at which data is generated.

As the volume, variety, and velocity of data expands, it morphs into big data and becomes too much for humans to handle without assistance. So we leverage artificial intelligence (AI) and machine learning to help parse it. While the terms big data and AI are often used interchangeably and the two go hand-in-hand, they are, in fact, distinct.

“In many cases, it is simply no longer feasible to resolve every issue via human interaction or intervention due to the speed, scale or complexity of the data that needs to be observed, analyzed, and acted upon. Driven by AI-powered automation, machines can be imbued with the ‘intelligence’ to understand the situation at hand, assess a range of options based on available information, and then select the best action or response based on the probability of the best outcome.” — Ilan Sade

Simply put, big data powers AI with the fuel it needs to drive automation. But there are risks.

“However the tendency to add too much data in AI can cause the quality of the AI decision to suffer. So it is important to take the benefits from big data and analytics to prepare your data for AI and to ensure and measure the quality, but don’t get carried away by adding data or complexity to your AI projects. Most AI projects, which are mainly narrow artificial intelligence projects, do not require big data to provide its value. They just need a good quality of data and a big quantity of records.” — Christian Ehl

Realizing Big Data’s Business Potential

Properly applied, big data helps companies make more informed — and therefore better — business decisions.

“A few examples include the hyper-personalization of a retail experience, location sensors that help companies route shipments for greater efficiencies, more accurate and effective fraud detection, and even wearable technologies that provide detailed information about how workers are moving, lifting or their location to reduce injuries and increase safety.” — Melvin Greer

But this crucial competitive advantage is underused because so many companies struggle to sift through all the data and distinguish the signal from the noise.

Five principal challenges keep companies from realizing big data’s full potential, according to Greer:

1. Resources: Not only are data scientists in short supply, the current pool also lacks diversity.
2. Data aggregation: Data is constantly being created and it is a challenge to collect and sort it from all the disparate channels.
3. Erroneous or missing data: Not all data is good or complete. Data scientists need to know how to separate the misleading from the accurate.
4. Unfinished data: Cleaning data is time-consuming and can slow down processing. AI can help manage this.
5. Truth seekers: We should not assume data analysis will yield a definitive answer. “Data science leads to the probability that something is correct,” Greer writes. “It’s a subtle but importance nuance.”

Addressing the first challenge is of paramount importance. The only way to solve the other issues is to first create the necessary human capital and provide them with the necessary tools.

The True Promise of Big Data

Data is a wonderful instrument, but it is not a cure-all. Indeed, “too much of a good thing” is a real phenomenon.

“In my years working with many businesses, I have indeed seen some companies that fell into the situation of not using data enough. However, these occurrences paled in comparison to the number of times I have seen the reverse issue: companies with an over-reliance on data to the point that it was detrimental. The idea that data is needed to make a good decision is a destructive one.” — Jacqueline Nolis

To illustrate her point, Nolis describes Coca-Cola’s introduction of Cherry Sprite. What motivated the decision? Data. People were adding cherry-flavored “shots” to Sprite at self-service soda dispensers. So score one for big data.

But as Nolis points out, the very similar-tasting Cherry 7UP already existed — and had since the 1980s. So the data team might have come up with the new flavor more efficiently simply by perusing the soft drink aisle at the local grocery store. The lesson: Too heavy a reliance on data can be a barrier to commonsense decision making.

Big Data Applications: When and How

So how do we know when to put big data to work for our business? That decision needs to be made on a case-by-case basis according to the demands of each individual project. The following guidelines can help determine whether it is the right course:

• Consider the desired outcome. If it’s to catch up with a competitor, investing in something the competitor has already done may not be a good use of resources. It might be better to let their example serve as guidance or inspiration and reserve big data analysis for more complicated projects.

• If disruption is the goal, big data can be applied to test new ideas and hypotheses and maybe reveal other possibilities. But we need to beware of the downsides: Data can kill creativity.

• If a business decision is urgent, the “data is still being analyzed” is not an excuse to delay it. Amid a PR crisis, for example, we won’t have the time to mine the available data for insights or guidance. We have to rely on our existing knowledge of the crisis and our customers and take immediate action.

Of course, sometimes big data is not just useful but essential. Some scenarios call for big data applications:

• To determine if a strategy is working as planned, only the data will tell the story. But before we measure whether success has been achieved, we first have to establish our metrics and define the business rules that determine what success looks like.

• Big data can help process and create models out of vast amounts of information. So as a general rule, the larger and more data-intense the project, the greater the likelihood big data could be helpful.

Big data might be the trendy topic in technology today, but it is more than a buzzword. Its potential to improve our businesses and our lives over the long term is real.

But that potential needs to be leveraged purposefully and in a targeted fashion. Big data is not the business equivalent of a wonder drug. We need to be mindful of where its applications can help and where they are superfluous or harmful.

Indeed, the full promise of big data can only be realized when it is guided by thoughtful human expertise.

If you liked this post, don’t forget to subscribe to the Enterprising Investor.

---

All posts are the opinion of the author. As such, they should not be construed as investment advice, nor do the opinions expressed necessarily reflect the views of CFA Institute or the author’s employer.

Image credit: ©Getty Images / Who_I_am