We live in the middle of a data revolution that’s showing no signs of slowing down. By 2025, we will have 200 billion connected devices1, including more than 6 billion smartphones2, generating more than 163 ZB of data3. With numerous technological advancements in data processing and management, we’ve shifted from Traditional Data to Big Data—and now, we welcome the age of New Data.

This new era urges enterprises to become data-driven or be left behind. To keep up and stay ahead of the market, companies need to build on New Data as their foundation. But they must first understand what it is.

To help explain New Data, we take inspiration from TV host and comedian David Letterman’s popular Top Ten List segment of his late-night talk show. Allow me to present 10 ways to describe New Data, Letterman style.

#10 New Data is vast, fast and complex. New Data comes from many sources—apps, websites, sensors, connected devices and more—and can be either structured or unstructured. Numerous relationships, associations and correlations are embedded in datasets to make them more complex and valuable. Despite its complexity and diversity, New Data should be extracted and processed quickly for faster consumption.

#9 New Data serves many purposes. New Data follows a model of capture and curate once, but consume many times for numerous purposes. This includes creating analytics reports or training artificial intelligence or machine learning models to hone predictive capabilities or for operational use by applications. Data’s many uses also enable finding value in multiple ways.

#8 New Data delivers value at various stages. New Data allows users to access it for consumption at various stages during curation and refinement to enable agility and speed in deriving value. With a multi-step data supply chain, data is collected from sources in raw, and then cleansed, refined and integrated through many stages. Users can then gain access to valuable data from any stage and on demand.

#7 New Data comes in flexible form to fit specific needs. To enable agile data processing, the data supply chain converts New Data into various “shapes"—star schemas, 3rd normal form, text or json files - for ease of consumption. The fabric of New Data is made up of file stores, relational databases, NoSQL stores, MPP engines and graph stores—whatever shape of data you need, New Data has it.

#6 New Data makes data storage and processing cheaper. Technological innovations cut down the cost of managing data significantly. From spending US$15,000–US$80,000 to store 1 TB of data, companies now have the option of paying only US$200–US$2,000 for Hadoop or even a few dollars per hour using the cloud—driving the increased adoption of more flexible and cost-effective cloud platforms for data storage and processing.

#5 New Data carries unique features for faster insight gathering. Each data asset contains self-identifying characteristics that define the context, meaning and usage for data consumers to understand it better. These unique physical and operational features—including location, access and activity history details—allow New Data to be discovered, understood and used easily, paving the way for data democratization and self-service.

#4 New Data enables better visibility. New Data’s embedded self-describing attributes allow users to easily index, catalog and search the data they need. It brings dark data, which is 80–90 percent of enterprise data, to light4. Better data visibility also speeds up the process for enterprises to derive valuable insights.

#3 New Data prioritizes security. New Data protects companies from millions in financial loss and reputational damage due to data breaches. It provides multiple layers of proactive security measures—including encryption, masking, tokenization, access control, and usage tracking and auditing—to ensure full protection against malicious misuse or unintentional access. Securing New Data also enables data sharing without risk of violating security or compliance regulations.

#2 New Data values trust and veracity. New Data addresses poor quality and bias issues that skew 97 percent of business decisions due to inaccurate, unverified or manipulated data5. It measures data quality, risk and relevance based on specific metrics to come up with a Data Veracity score—allowing users to determine the best use of data that they can fully trust. By eliminating uncertainties, New Data allows for more robust decision making.

#1 New Data must be treated as a strategic enterprise asset. New Data encourages data sharing across the organization for broader consumption and even possible discovery of new insights. It also avoids duplication and enables reuse. With such a sharing ecosystem, the New Data era leads to better products and services, faster speed to market and higher profitability.

1A Guide to the Internet of Things [How billions of online objects are making the web wiser]. Retrieved from Internet of Things.
2Marr, B. (2015, November 19). Big Data: 20 Mind-Boggling Facts Everyone Must Read. Retrieved from Big Data: 20 Mind-Boggling Facts Everyone Must Read.
3Reinsel, D., Gantz, J., & Rydning, J. (2018, November). The Digitization of the World from Edge to Core. Retrieved from The Digitization of the World from Edge to Core.
4Kay, C. (2016, September 26). Cognition and the future of marketing. Retrieved from Cognition and the future of marketing.
5Ernst and Young. (2015). Becoming an analytics-driven organization to create value. Retrieved from Becoming an analytics driven organization to create value.

Sharad Kumar

CTO – Cloud First, Data & AI

Subscription Center
Subscribe to Software Engineering Blog Subscribe to Software Engineering Blog