How to develop products from a data-driven perspective?

There has been a massive shift in the way we do product management in the past decade. It started with writing user stories that were backed by extensive customer research, however there was very little control over the outcomes. Product success involved a considerable dose of wishful thinking and/or hoping for positive business outcomes.

By Kirthi Vani and Steve Roberts, Accenture

What began as hoping for positive outcomes in the past has now evolved into driving better results through data and experimentation.

As many product managers have found, however, designing with the customer in mind, while necessary, is not enough to ensure a positive outcome. To increase the odds of a successful product, product managers need data.

A recent survey by Splunk showed that organizations that place a strategic emphasis on data and have an advanced strategy to extract business value have added 83 percent more revenue to their topline and 66 percent more profit to their bottom line in the past 12 months. In addition, 93 percent of these organizations feel they tend to make better, faster decisions than competitors. And 91 percent believe that their organization is in a strong position to compete and succeed in its markets over the next few years.

We believe that companies can leverage data-driven product strategies that lead to differentiation and competitive advantage. To do this organizations must be able to pivot quickly based on their data experiments:

1. Embrace the experimental culture

If data-driven product management had a mantra, it would be “test early and pivot quickly.“ Failure is an essential part of learning and growth. Product managers should not have to come up with a perfect idea on the first try. Instead, they should be encouraged to treat everything as an experiment—iterating quickly in short build/measure/learn cycles.

Amazon CEO Jeff Bezos considers Amazon‘s experimental culture to be a significant strategic advantage and a major reason why the company reached $100 billion in sales faster than any other company.

An experimental culture calls for product managers to:

• Adopt a hypothesis-driven approach to define the product roadmap, minimum lovable product (MLP), and user stories. Product managers define these elements based on an underlying business case. Still, the data that drives the underlying assumptions and expected outcomes for the business case is not always specified. Managers should set up an "experimental framework" that defines what data needs to be captured, how they are instrumented, and which observations will show success.
• Avoid bias in the experimental framework. Biases can creep into experiments. For example, if the research is too theoretical or too leading, one can start with the wrong hypothesis. To avoid this, all hypotheses should be framed as solutions to customer problems. This means the product managers should outline the circumstances of the customer‘s problem, the problem itself, and the idea for solving it. Another potential source of bias may come from building code early in the process. This can reinforce bias as people seek to protect investments in such resources. To avoid this, wireframes or design mockups can be tested as prototypes with customers.
• Measure actual outcomes through systems and automation to avoid bias. Data should be instrumented wherever possible, with tools adopted to automate the measurement of actual results.

2. Democratize data

An experimental culture relies upon data democratization, meaning that everybody has access to the data they need to make decisions. Everyone in the organization should understand and value data.

Data governance is essential to data democratization. Leadership oversees how data is collected, annotated, and accessed. Leading companies often do this by setting up a center of excellence for data management or by appointing a Chief Data Officer, Chief Digital Officer, or Chief Data Product Leader.

Similarly, the organization’s data strategy should go beyond product development. Successful data-led product managers measure and instrument product usage to determine best offerings, licensing, and product strategies. This data can also help customer support, supply chain and manufacturing improve their performance and metrics.

Data quality is another vital consideration in democratization. Many product managers are somewhat focused on data quality, but often, this isn’t enough. Data quality needs to evolve continuously, through constant, iterative improvement. Data quality metrics should be shared across the organization to ensure transparency and a high level of confidence.

Data democratization provides access to multiple sources of data. Most organizations have a lot of dark data generated by system logs—usually unstructured, untagged, and untapped.

This should be leveraged and used in combination with product data to determine product strategy. Tools like Splunk can help mine, model, and analyze unstructured log data.

In data democratization, self-service analytics is critical. The right set of tools can connect siloed data and make it accessible. These tools should be customized with filters and analytics based on the people accessing the data.

Finally, in a truly data-driven organization, everyone should be trained in the basic concepts of data, analytics, and the tools required to access this information.

3. Practice responsible data collection

The critical element in data collection is value in exchange for privacy. For example, collecting location information for advertising purposes only is not ideal. However, if that information is used to help the user geo-tag a tweet, get the most relevant tweets for the customer in that location, or use that location information to auto-tag photos taken from a camera, then there is value created for the end-user.

Product managers can practice responsible data collection by following these guidelines:

• Obtain clarity on the organization’s commitment to customer data privacy and security.
• Anonymize the data as much as possible and understand the trade-offs.
• Have a clear rationale as to why any data is collected.
• Provide clear justification on why the returns are worth the risk.
• Understand security and privacy measures in terms of policy and regulations in place to protect customer data.
• Embed adequate controls so that the user can manage how much data they want to share.
• Provide information about the source and validity of the data collected, and about how the organization plans to use it.
• Leverage tools that can help anonymize data at earlier stages of analysis itself; this is critical in today’s regulated environment with rules protecting data privacy and confidentiality.

4. Don’t confuse correlation with causation

With data democratization, the product manager can apply data to every question and decision for product strategy and development. However, product managers should be very careful in separating correlation from causation. In statistics, correlation tells us how strongly a pair of variables are related and how they change together. Causation takes this further and states that any change in the value of one variable will cause a change in the value of another variable.

For example, a correlation might be the increase in usage of new functionality “x” with an increase in user retention. The product manager can predict a causation and create a primary hypothesis—“Improving user engagement with feature ‘x‘ will directly impact conversion”—but she or he must then test it.

To minimize the confusion, managers should verify that the null hypothesis—in this case, the statement that “there is no relationship between feature ‘x‘ and conversion“—can be disproved with statistical significance before testing the primary hypothesis.

5. Create an AI flywheel

The product manager of the future will rely on artificial intelligence (AI) as well as data. The AI-led product manager knows what data is being collected and plans a foundation layer along with a core infrastructure to create an AI “flywheel.” The flywheel runs on the momentum generated by structured and unstructured data from internal and external sources, combined with multiple machine learning algorithms. The whole is exponentially greater than the sum of its parts. At most companies, AI initiatives are not integrated, but sharing AI and machine learning models across teams can help create the flywheel effect. Of course, this depends not only on product managers’ efforts but also on those of data engineers, data scientists, and DevOps engineers.

An efficient operating model and governance are also critical because there is a cost to maintaining artificial intelligence. So, managing the AI flywheel prudently means combining use cases wherever feasible and making smart choices about data; for example, re-using existing cases as often as possible and thinking creatively about data from “unexpected sources." In short: Always build models that are fit for purpose and the problem at hand.

6. Cultivate “Growth” or “Outbound” Product Managers

Product managers are typically responsible for driving business results from a specific product or portfolio of products over time. However, data democratization and AI flywheel efforts also require driving and measuring short-term growth metrics across the organization.

This has led to the emergence of the Growth Product Manager (sometimes also called “Outbound Product Manager“). These experts typically focus on growing traffic, users, engagement and other elements to drive short-term results. They often own the growth strategy and plans to optimize revenues to support multiple products in the organization.

The Growth Product Manager defines the key growth metrics for the product—such as user acquisition, renewals, conversion, or reduction in churn—and regularly reviews growth trends through insights published on tools such as Adobe Analytics or Google Analytics. Other product managers then use these insights to improve the product user experience.

The Growth Product Manager should support the product manager to ensure that short term metrics are trending in the right direction. The focus should be on understanding the customer and on solving the right problems within the product area.

Acknowledgements

The Authors of the article thank the following for providing valuable insights:

• Jeff Chancey , Managing Director, Ecosystem & Client Services, Accenture
• Skip Bacon, VP Product Management, Splunk
• Mohan Rajagopalan, Senior Director Product Management, Splunk
• Sendur Sellakumar, CPO, Splunk
• Lakshmi Shankar, Senior Director, Strategy & Operations, Twitter Product