No items found.
Advanced SQL

R & Python 101: Natural Language Processing

Most data analysis is performed on quantitative data, but companies that limit their research to just numbers could be missing out on valuable information. To maximize their value to an organization, data teams should start looking into ways to extract value from an additional type of information – text. To perform this analysis, teams use Natural Language Processing (NLP) to decode human language.

While SQL has some light NLP capabilities, Python offers huge libraries that can analyze this type of information at incredible depth. More advanced languages, such as Python, can adapt and grow as new discoveries are made in NLP, so as human speech processing evolves, companies will be able to gain the right insights, including meaning and context, from their text database.

Using Natural Language Processing to generate new insights

The problem with qualitative data in general is a lack of structure. Numerical data is inherently more sequential and can be organized and analyzed easily. Text-based data is unstructured and often disorganized. Libraries like Natural Language Toolkit (NLTK) in Python have been built to help create order out of human speech patterns and perform intensive analysis on what is being communicated. NLTK does more than just convert words into searchable sets of characters, it actually works to dissect human language construction and derive the intended meaning from a sequence of words.

NLP is still in its infancy, but companies are already using it to generate some meaningful insights. A basic example is sentiment analysis, where data teams can identify what is being said about a product or a brand and analyze those statements for meaning. It’s much simpler than putting together a formal focus group and provides a more concrete understanding of the conversation.

Any company that uses text as a major unit of data can generate value from NLP technology, they just need to think of a way to provide structure around the outcome they intend to get. NLTK is an advanced tool that is capable of sorting through volumes of text-based data, but just like your quantitative data sets, the best insights only come from a well-structured, carefully designed analysis. The possibilities for NLP-based insights are limitless and tools like NLTK will only continue to improve. With the right vision and analytical execution, data teams that utilize the tools correctly will give their companies a major advantage.

Natural Language Processing in Periscope Data

One Periscope Data customer that is making the most of NLP is Crisis Text Line, a free, anonymous 24/7 text-based crisis intervention system that aims to mitigate crises by connecting people to counselors who are trained to cool down hot moments. They use natural language processing and machine learning to pull insights from their rich data set and identify keywords in texts to help steer a counselor toward a safe resolution. Later, a second phase of this process utilizes a large community of professional counselors to analyze conversations based on common keywords and tags to help assess trends and train counselors to have high-quality conversations with texters.

This innovative approach to predictive modeling allows Crisis Text Line to detect keywords that identify and predict trends in real time. The Crisis Text Line data team uses Periscope Data to conduct this complex analysis and quickly visualize the results. In the near future, the team plans to set up a self-service data environment that will empower counselors to access information without help from the data team. This setup would give counselors quicker access to data and ultimately lead to better-informed conversations with texters. Often, the end users have difficulty predicting the needs of texters ahead of time, so a data tool that relies on upfront modeling is ineffective. An agile data environment like Periscope’s allows the team of counselors to find answers on their own.

To learn more about how you can use Periscope Data to incorporate Natural Language Processing into your data analysis, download our guide.

Want to discuss this article? Join the Periscope Data Community!

Neha Kumar
Neha is passionate about creating powerful data visualizations, spreading awareness of analytics best practices and learning about the latest and greatest advancements in data science. She strives to leverage multiple programming languages for analyses that are more efficient and effective than current methodologies.