In a relatively short time, we’ve seen the world of big data evolve through major challenges. Before 2006, we didn't have the technology to make sense of big data. It was too expensive to store all the data a company generated, and there wasn't enough compute to make sense of it even if they did. Then Amazon launched Amazon Web Services (AWS), and over the next few years it led the way in making limitless storage and compute affordable for everyone.
On the building blocks of AWS, whole new categories of data processing technology were developed. Hadoop and MapReduce took the early spotlight and were quickly followed by new, cloud-scale distributed warehouses and data stream processors. Today, the pace of data technology innovation has never been faster.
It's an exciting time. The compute has caught up to the ambitions of the early pioneers of the data industry. That’s an exciting place to be, but it's not enough. The tools have caught up rapidly, but the strategy hasn’t evolved at the same pace. We have the tools to manage any query, but we're still not addressing the hardest problem in data: asking the right questions.
Problems with technology and talent are resource issues, but not asking the right questions is a human problem. With highly-trained people using powerful technology, the only limit is their imagination, which is a new problem. To best explore the powers of data, we have to start empowering data professionals to identify questions they didn’t even know to ask.
Traditional analysis tools help us with the second half of a question-then-answer research method. We ask questions like “How many widgets did we sell last quarter?” or “How did the last release affect user retention?” and analyze the data to arrive at a conclusion that can be used to make decisions.
These pieces of information are necessary to operate a business, but they're uninspired. They are about protecting the downside of past decisions, not about exploring what's possible. Your results in this standard line of questioning are limited by the human element — the answers can only provide value at the depth of the person or people asking the questions.
The most disruptive companies in the next decade will have data teams showing their value by exploring large sets of raw data, looking for new connections. Instead of digging into data searching for specific metrics, they'll identify significant correlations and new connections, while refining their analysis until they arrive at insights.
This approach reverses the question-then-answer process. It starts with strong, data-first answers and then works backward to find the questions that should have been asked. It’s a wide-net approach, but it’s the only way to truly let data drive your decisions. Data can still be used to find answers when you have specific needs, but when it comes to finding the most impactful ways to move the business forward, teams that let the data provide the questions will get the best results.
A lot of employees, teams and companies claim to be “data-driven,” but they're limited by only using data as a way to continue their standard line of inquiry. It’s only by realizing that data can ask questions that you can truly take the human hands off the wheel and let the data drive.
Beyond identifying superior questions to ask, this approach allows teams to adapt to questions that can’t be anticipated. Since they aren’t focused on narrow analysis, they can be more flexible in diagnosing issues that have to be handled in real time. Instead of trying to force unanticipated questions into defined answers, you’re free to examine every possible angle and tell a more complete story with your data.
No matter where your company is regarding data infrastructure and staffing, it’s never too soon to start giving your data team the freedom to look for new connections in your data. You never know exactly what the results will be, but trusting your data to reveal the right questions is the best way to move forward.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
Harry Glaser is a Co-Founder & CEO of Periscope Data, an end-to-end analytics platform for data teams, with over 1000 customers globally.