Skip to main content Skip to Footer

LATEST THINKING


Does intuition matter in a big data world?

In this new digital era of “big data” and quantitative analytics, what’s intuition got to do with it?

Overview

The hype around data and analytics may lead many executives to assume that the answers to difficult questions—how much to raise prices, where to site a new retail store, whether one product will cannibalize another—can be found through aggressive number crunching. But does that mean that expert judgment and managerial intuition are obsolete? Hardly.

In fact, the full potential of quantitative analytics can be unlocked only when combined with business intuition. Mike Flowers, chief analytics officer for New York City under Mayor Bloomberg, explains the fallacy behind either-or thinking: “Intuition versus analytics is not a binary choice. I think expert intuition is the major missing component of all the chatter out there about analytics and being data driven.”

More successful companies seem to get that. In a survey of 600 companies in the United States and United Kingdom, Accenture Analytics found that data and intuition, when used in concert, correlate with higher returns on investment. Among those businesses that reported more than a 75 percent return on their analytics investments made over the last two years, slightly more than half said that their best decisions incorporated human expertise with rigorous data. For those companies that had a lower ROI, only 37 percent said they combined executive judgment with quantitative analytics.

Intuition VS Data Driven

The trick, though, is in knowing when and how to incorporate expert intuition into data-driven decision making. Intuition plays a pivotal role in shaping analytics efforts from their inception. Managerial instincts, for example, are critical to identifying where to focus analytical efforts for the greatest impact. Jack Welch, the former CEO of General Electric, is said to have had an uncanny sense for knowing where to do a “deep dive” into the data.

Like the best executives, top analysts aren’t just numbers-focused quants. They instead bridge the world of data and intuition. For example, they may help decision makers reframe a question to ensure that they get at the outcome they really need, as opposed to mechanically answering the question was posed. They also help executives examine and test their underlying assumptions about their business, markets and customers. “If during your initial project scoping all you do is talk to management and then have the IT guy get the data transfer, you’re setting yourself up for colossal failure,” says Flowers. To ask the right questions, analysts and decision makers alike need a deep intuitive understanding of the organization, its strategy, and its objectives.

Many top analysts also spend time “in the field,” gleaning insights they would otherwise miss if they spent all their time staring at numbers. For example, Flowers and his team were tasked with predicting which buildings were most likely to be fire hazards. They began by looking at lots of data that might suggest a building in bad condition and thus at risk for fire: number of complaints about rodents, ambulance visits, delinquent property taxes.

But when the analyst joined building inspectors on their rounds, they learned how the inspector’s intuition guided them to focus on the most likely perpetrators: “Look at the bricks! The bricks are fine. This guy [cares], so that’s a good tell.” The team then added data from city permits, which are required for doing any exterior brickwork, to its analysis. “Most of the time we use data in ways that it was never intended to be used,” says Flowers.

Reconciling Analytics and Intuition

Companies don’t want to have to rely on luck—finding a needle-in-a-haystack analyst—when it comes to ensuring that they get the right mix of analytics and intuition in their decision making. Further, in too many enterprises, a line separates the quants from IT and line executives, with each side speaking a different language.

In our research, we’ve seen five approaches that can bridge this divide:

Invite executives to the analytics party. CEOs should set clear expectations that decisions will be based on data and analysis, but they should also make sure experienced IT and business executives are an integral part of the analytical process from the beginning. Otherwise, the output from a black-box analytics exercise runs the risk of being feared and ignored.

Ensure two-way communication. Your executive corps needs to bring a broad understanding of and appreciation for the power of analytics to the table. They don’t have to be experts, but they should respect the power of data-driven decisions and be comfortable with the basic concepts of analytics. And traffic must run in the other direction, too: Your analytics teams need to be able to speak and understand the language of business.

Build an analytical ecosystem. The quants do not have all the answers, nor do the line executives who have to make the tough decisions. By creating an analytical ecosystem that includes line employees, IT, external analytical partners, suppliers, and business partners, companies can avoid the binary fight of numbers versus experience.

Routinely seek out the whitespace. Intuition can point to creative solutions. An analyst’s intuition can lead to experimenting with new or unconventional analytical techniques and data sources. Some of the greatest breakthroughs come when people ask the unasked questions, examine the absence of data, or test unquestioned assumptions.

Accept the limits of analytics. Leaders know when to trust their intuition and how to blend the art and science of business decision-making. They take full advantage of data and analytics wherever they can, without losing sight of the big picture.


Even as big data gets bigger and quantitative analytic techniques become ever more powerful, they will never supplant the need for human intuition and creativity. As Mike Flowers puts it, “Intuition and data analytics are like two wings on a bird. You can’t really use one effectively without the other.”

To download a copy of "Does Intuition Matter in a Big Data World?" please login to WSJ.com.

This copy is for your personal, non-commercial use only. Distribution and use of this material are governed by our Subscriber Agreement and by copyright law. For non-personal use or to order multiple copies, please contact Dow Jones Reprints at 1-800-843-0008 or visit www.djreprints.com


FIND OUT MORE ABOUT THE ACCENTURE INSTITUTE FOR HIGH PERFORMANCE.


Author

Jeanne Harris

Jeanne G. Harris

Jeanne G. Harris is the former global managing director of information technology research at the Accenture Institute for High Performance.