The path from data to knowledge
June 19, 2019
June 19, 2019
Imagine a large energy utility with an ambitious and talented team of sales and marketing professionals. The team is focused on growing the customer base and improving retention. Multiple data systems hold the utility’s most important information. Across these systems there is a wide variety of data types, access patterns, quality levels and use cases.
Due to the volume and diversity of data the company wants to use, it struggles to establish a satisfactory, enterprise-wide master data repository. Each line of business relies on manual extractions and data transformations in a spreadsheet format to generate static customer records. The results are often incomplete, out of date or inconsistent with other records. Enhancing insights by adding external or real-time data is practically impossible. The mis-match between the ambition of the utility and its supporting data systems will, most likely, lead to missed opportunities and potentially the loss of top talent.
This example reflects a reality many companies are facing—across almost all industries. Traditional MDM has served organizations well for many years. MDM processes and technologies have been used to create and maintain an authoritative, reliable, accurate, timely and secure records of the organization’s most important data—a single, high-quality version of the truth, also known as “golden records.” MDM products and services are designed to remove duplicates, standardize data and apply rules that maintain data integrity.
Traditional MDM, however, is starting to look out of date. It is not designed to cope with today’s data challenges. Data volumes, for example, are growing at a breakneck pace. The research firm IDC estimates that the “global datasphere”—defined as all the data created, replicated or captured in all the world’s data centers, infrastructure and devices—will grow more than fivefold between 2018 and 2025, from 33 Zettabytes to 175 Zettabytes by 2025.1
Traditional MDM, however, is starting to look out of date. It is not designed to cope with today’s data challenges.
Data requirements are not just growing in volume, but also in the diversity of types and sources of data used in models. Data is no longer limited to structured, transactional, internally generated types. Today, when looking to enable greater use of artificial intelligence, organizations should incorporate machine-generated, unstructured, semi-structured, streaming, external and probabilistic data.
As data proliferates and becomes more heterogeneous, business requirements for it are changing. For example:
When it comes to integrating these data types, “point-to-point” or “match and merge” approaches of traditional MDM would be too complex and slow; new techniques are required to adequately capture, link and curate these newer types of data.
Instead, a flexible “Digital Master” model is emerging that can be tailormade to deliver new business value from vast amounts of diverse data.
The tasks performed by traditional MDM, focused around identifying missing or erroneous entries in highly structured data sets and eliminating duplicate records, remain vital to modern business requirements. But MDM should be expanded to meet the business requirements described above. The solution is the “Digital Master”: an evolution of traditional MDM, designed to bring data to the heart of the business and organize data at scale, combining data from a wide variety of sources.
The Digital Master helps companies to move beyond having a limited and static "golden record" (of a customer, for example) to having a "golden profile" that not only contains more types of data and insights but can be updated and expanded in real time. (More on Golden Profiles in the full report on page 12.)
Compared with the use of traditional MDM, the Digital Master offers several advantages:
For example, we helped a large hospitality company build a digital data platform that allowed it to capture, curate, process and store emerging data types. The company’s objectives included a radical improvement in customer experience and personalization, to help retain loyal customers and increase market share. Now, with its Digital Master in place, the company has a 360-degree view of existing and potential customers—they are able to recognize individuals at any point of interaction across its channels—and it can therefore enrich offers with intelligent insight and get ever closer to its key objectives.
Organizations tend to transition to the new Digital Master in one of two ways:
In both cases, aggregating the organization’s existing data into a data lake is the starting point. This involves bringing together data from past transactions, customer records, activity on the corporate website (including from cookies) and social media channels. To make sense of this basic data, we need to add more meaning to it in the form of contextual insights. For this we can use a knowledge graph. This is flexible enough to work like a bridge between various data types and formats, while at the same time adding descriptions and associations that help machines to discern what the data is all about.
This process helps us to create rich, “golden” profiles of dimensions such as customers, businesses and products. It can also provide us with powerful insights into customer attitudes and behaviors (for example, by identifying correlations between different buyers of the same product).
Within a business, a knowledge graph that is constantly expanded with contextual and other data can identify and expand an increasing number of use cases for products, services or any number of business missions.
Knowledge graphs have been around for a couple of decades. Their ability to generate insights have been transformed in recent years, however, thanks to the proliferation of (mostly unstructured) data and enormous increase in computing power at organizations’ disposal. Knowledge graphs are what enable Apple’s, Amazon’s and Google’s virtual assistants to provide extremely relevant answers to users’ questions. Within a business, a knowledge graph that is constantly expanded with contextual and other data (such graphs can scale relatively easily) can identify and expand an increasing number of use cases for products, services or any number of business missions. And applications or dashboards can be created for some use cases, making it easy for users to quickly glean insights from the data.
With all the elements of the Digital Master in place, enterprises can harness artificial intelligence and machine learning across their operations, supporting powerful breakthroughs in insights, data-driven decision-making and predictive capabilities.
The knowledge graph bridges the gap between traditional data stores and AI capabilities. This is because AI models can directly leverage the graph, so they can learn from massive amounts of meaningful data, bringing predictive power to new domains and increasing accuracy. Unsupervised learning algorithms can explore the knowledge graph and discover patterns that humans could never see.
eBay offers an example of how AI can leverage knowledge graphs to expand its capabilities. The company’s popular “ShopBot”—an AI-driven shopping assistant—is powered by knowledge graphs, according to its creators. eBay’s knowledge graph incorporates data from roughly 16 million active buyers. The bot has powerful algorithms, but when a user voices a query through the app about a certain product, its desired features and intended uses, it is the data provided by the knowledge graph that enables the bot to provide intelligent and relevant responses to the user.
Not many companies today can leverage such capabilities at scale. In addition to eBay, online giants such as Amazon, Google, Microsoft, Facebook, Alibaba, Baidu and others are using AI in conjunction with knowledge graphs to drive their core operations and services. There is no reason, however, that other types of organizations cannot leverage these capabilities to drive positive outcomes across their business.
In this article we have explained why organizations should be making the shift toward the Digital Master model and the advantages that it confers in the digital age. To learn more about how to make the transition, our whitepaper will help executives understand the evolving landscape and assess the architectural options.
As indicated earlier, moving from MDM to a Digital Master can be done incrementally, without much disruption. It is the ideal path towards harnessing AI at scale across your operations. If you are at the beginning of this journey, the first steps you should take include the following:
1 IDC White Paper, sponsored by Seagate, Data Age 2025: The Digitization of the World from Edge to Core, November 2018