How to Build a Modern Data Stack
With all of the recent acquisitions in the BI industry (ours included), there has never been a more critical time to think about the future of the choices that you are making about your data stack. At Periscope Data by Sisense, our core mission is to offer choice, flexibility, and speed. Each part of your data stack should be tailor-made for your needs, whether it’s allowing you to choose the right methods for gathering your data through any ETL method, picking infrastructure that aligns with your needs, or keeping yourself platform agnostic by building BI in native data languages such as SQL, Python, or R.
There are some top partners that have a clear path to being best-in-class and who remain independently focused on the core value that their tool delivers. Snowflake stands out amongst the data warehousing pack as the leading way to build a managed warehouse, with the unique benefit of separating the storage and compute. This choice allows companies to store massive amounts of data and harness the power of using and scaling compute only when they need it.
A modern data stack
To help cut through the jargon in the industry, here is a synopsis of the different types of backend data sources that our customers use today to fuel their dashboards.
Databases: These are the basic workhorses. Sometimes these are the appropriate choice, especially when queries are simple, budget is a concern, or you have highly efficient queries and require massive parallelization.
Data warehouses: This data storage method was built to bring together many disparate sources, including data from multiple databases, business applications such as Salesforce, and large data volumes (for example, logging for product analytics). These are the most common sources we see modern data teams using to power their workflows because they can handle the scale and complexity of modern data problems.
Data lakes: When data volumes hit a massive level and organization is less of a problem than taking in the deluge of data, a data lake offers a simple way to catalog and store information. We most often see this type of system when logs are the main source of data for analysis, such as tracking every button click in an application.
Fast, flexible BI needs a strong engine
Data does not come in a perfect form with all the metrics needed to make business critical decisions. It is up to each company to find ways to connect, analyze, visualize, and share data so decision makers reach the right insights quickly. As a result, Business Intelligence problems often require some level of complexity in the logic necessary to align data to the business need. Because of this need for centralizing data and performing complex calculations, you’ll want a data warehouse that can house a lot of information for quick access. This is another reason we love our partnership with Snowflake; they offer both advantages for the most effective scaling cost.
At Periscope Data by Sisense, the primary way most analysts access data is by adjusting queries through the use of filters. To make this style of dynamic access faster, we can prepare the data through Materialized Views. By precalculating the metrics and storing the results back into your warehouse, you can make the data more readily available. For some of our critical dashboards, loading times used to take upward of ten minutes when a filter was changed. With Data Engine, those same queries happen in less than a second. That’s some serious power to allow your dashboards to scale to tens, hundreds, or even thousands of concurrent users!
Putting it all together (quickly)
A lot of customers come to us when they are looking to upgrade to the next level in their use of data. The most common concern is how long they think it will take to get multiple vendors up and running, connect them all, and then get value out of that data by visualizing it in a way that informs decisions. What most people don’t realize is just how fast everything can be set up.
We recommend getting a managed data source up and running. Snowflake makes this as easy as clicking a few buttons. From there, you’ll be able to pipe data in using an ETL tool that simplifies the process of extracting data from all your sources and loading it into the warehouse. While that data is loading, you can spin up a Periscope Data by Sisense site through the partner connect, or create a user in your Snowflake Warehouse and connect it to your existing site! Here are some exact details depending on where you’re starting:
Existing Customers: If you are interested in trialing a Snowflake warehouse, feel free to reach out to your customer success manager, who can share best practices of using Snowflake with our data platform.
New Customers: For more information about getting started with Periscope Data by Sisense and Snowflake, visit our partners page for more details. See Snowflake walk through the simple setup process for tools in Partner Connect in the video below.