The Scientific Method of User Data Analysis
I’m proud to say that at Periscope, we’re a data-driven company. We need to be. That means that data does more than just retroactively measure projects that have been completed. We use data to do more than just answer questions, we also use it to tell us which questions to ask. We even use data to tell us if the ultimate value is worth committing the resources necessary to answer the questions we’ve identified.
We’re a data company, our case isn’t universal, but every company has different high-level KPIs they’re tracking, and the path to moving those metrics is unique, but has the same starting point: asking questions of data. Knowing which questions you want to ask data, knowing how to ask them and knowing how to react to the findings are all separate things.
The data-centric way to optimize all of those steps looks a lot like the scientific method. It starts with a hypothesis, collects information and analyzes visualizations to come to an informed conclusion before restarting.
For data-driven product teams with concerns about how to ask their data questions and get valid answers, I’d like to explain how we do it at Periscope Data. I’ll also be presenting a few examples from a user growth project I’m leading now to give a more complete picture of how our process works.
The first part of any good data-based investigation is to condense your initial information into a hypothesis. That should help your team narrow the scope of the data that needs to be analyzed and give your investigation a more pointed approach.
At Pericope Data, we’re always looking for data-driven ways to increase our revenue. For the particular project that I’m leading now, our hypothesis was that more improved user-level engagement would drive business account retention and expansion. We don’t charge on a per-user basis, so there isn’t an easy, linear relationship between the two, but it still seems very likely that increasing the number of people using our product will be a good way to increase our revenue growth.
Ask data questions to gather information
After the hypothesis is determined, it’s time to start digging into the data you have available. You’re not looking for a direct answer here, you’re just trying to find as much relevant information as possible about the question you’re investigating.
It’s very important at this step that you realize the types of questions data can answer. It’s far too broad to simply ask your data if more users equals more revenue. You might find a way to see that both numbers are moving in the same direction, but that’s correlation instead of causation. While it's often impractical to determine causation on historical data, it's possible to find multiple consistent correlations that lead to a strong hypothesis for investigation. It’s a good practice to learn as much as you can about multiple subjects (both on their own and together) so you can pinpoint these consistent correlations.
In the case of the user growth project that I’m working on, we asked around 100 questions of our data for this stage of the research. Most of them were simple queries that returned results we anticipated. That’s ok, it's critical to see how strongly the data agrees with our hypothesis. By asking 100 questions, you can find the strongest signal to act on.
At this stage, you should be looking for opportunities to clarify your hypothesis and dig deeper. If a single question reveals a phenomenon that is driving broad movement, it’s probably important to dig in and find ask more related questions. Often the most valuable research can come from lines of questioning that you never expected to investigate. Information that is contradictory to your hypothesis is critical to your research because it helps shine a light on previously dark areas of your overall knowledge.
More analysis is better here because every piece of information contributes to the overall picture we’re trying to uncover. Results that one person expects may be contradictory to another person’s expectations, so proactively build a wide base of knowledge.
Visualize and study the results
Once you’ve asked the data all the necessary questions, it’s time to build charts and assemble a dashboard to see the big picture.If your investigation started with an unorganized sprawl of questions, that sprawl will only make sense to the original author. Organizing and curating your data is critical to conveying the signals that were discovered in a way others can understand.
Ideally, you’re updating your hypothesis after each of the 100 questions you started with. This visualization is a chance to show how your thinking has changed and how explain why your hypothesis has moved.
In our user growth project, our hypothesis was confirmed when we proved that accounts with lower user level engagement were at an increased risk of churn. Customers with higher user engagement levels were likely to remain customers. In particular, accounts with the most engaged users had 100% account retention for the year.
When we looked at user activity across those accounts, we found that users who were viewing only one dashboard were likely to stop using Periscope data and those who were logging in and checking information in more than one place were likely to increase their usage of the platform.
When we put all that information together, we can update our hypothesis to state that accounts with more users can create more data and find more ways to use that broader set of information. It’s a snowball effect — when more users are adding and analyzing information in Periscope Data, those people can find more value in their analysis. When the data ecosystem provides increased value, more people want access, which creates ... more users.
Sharpen and Repeat
Once you’ve discovered trends or outliers, it’s time to start asking new questions about those phenomena. The best approach is to alternate between asking questions and visualizing your findings for analysis, tightening your hypothesis along the way to reflect what you’ve learned.
Remember, the goal of this process is to uncover the truth in your analytics. It might take several iterations of this analysis to get to a point where you can take action on the data. As you learn more and more about your business, if might require more rounds of this analytical process to find information you didn’t already know. That’s normal. Keep asking questions, keep analyzing your findings and keep updating your hypothesis. This isn’t just poking around for a number to put in a report, it’s a scientific inquiry into the way your business works.
The more your company studies data, the more you’ll start engaging more people and more teams in your analysis. It’s important to be exhaustive in your research reporting, because insights that you consider to be intuitive might be new information to another team or another member of your project.
There are an infinite number of questions you could ask the data, so stay focused on adding to the overall story the data is telling. The goal of any dive into data should be increased information, which will lead to a more complete understanding of your business landscape and more valuable decisions.
To learn more about answering product questions with data, download our Data-Driven Product Management guide.