Qualitative or Quantitative Customer Feedback? How to Get the Best of Both Worlds

Qualitative or quantitative? When soliciting feedback from customers, it’s a big question that doesn’t feel like it has one right answer.

For Mule Design co-founder Erika Hall, UX problems are best solved with qualitative feedback from customers. At the recent UX London conference she stated, “Quantitative approaches feel better, but are often actually worse.” Since human beings are largely irrational, she argues, “Human problems require illogical approaches.”

So they developed a system that turned qualitative data into quantitative data.

That’s exactly why Groove used Qualaroo to get feedback from customers. They received over 4,000 responses, all full of “Juicy, open-ended answers that gave [them] insight into the minds of [their] prospects.” While they were incredibly useful, it was really time-consuming to go through them all. As much as Groove agrees with Erika that those 4,000 answers held the solution to their problems, they needed a better solution for analyzing them.

So they developed a system that turned qualitative data into quantitative data.

Groove isn’t the only one doing this. There are a lot of people turning qualitative data into numeric data sets. For one, they’re more scalable to mine for insight than quantitative data. Qualitative data, like reams and reams of written customer feedback, can be analyzed like quantitative data through text mining and data processing. (These methods might even hold the answer to who authored Shakespeare’s plays).

Ultimately, quantitative and qualitative data are incredibly important for amplifying the voice of your customer and reaching data-informed conclusions. The key is to not rely on these shortcuts to fabricate the conclusions that you want to see.

Here’s how to get the best of both worlds when processing large amounts of text from your customers—without distorting it.

How to Turn Qualitative Feedback into Quantitative Data

If you’ve got a couple dozen responses coming in, it’s not that time-consuming to read them all. But if you have hundreds or thousands, it takes forever to make sense of the data you’re receiving.

The real problem with qualitative data is you can’t group it for easy synthesis. Unless, of course, you create a formula to take a deep dive into the data that the responses are giving you. The only way to process large amounts of data is to create a function that allows you to organize text like you would numbers.

Developing a program that converts qualitative responses into quantitative ones allows computers to do the heavy lifting—but only if carefully crafted. With a tailored program that accounts for nuances in open-ended responses, you can tackle the fundamentally “human problems” involved in customers support.

1. Create an open-ended survey

The first step is to develop a survey question and put it somewhere on your site where users will actually fill it out. Whether you’re using NPS or some other kind of open-ended survey, you’ll be able to ask a question that yields a wide range of responses. As Erika pointed out at UX London, you’ll probably get some that you might not expect that can help you improve in ways you didn’t know possible. Human problems will yield interesting, human answers.

Here’s how you can develop your own surveys in Qualaroo to get customer verbatims.

Screen Shot 2016-06-28 at 1.55.58 PM

A multiple choice question might offer a couple different answers your team thinks people are using your site for. An open-ended one allows users to say whatever they want, which can allow for all kinds of insights. This question, for example, might yield use-cases for your product that your team hasn’t even discovered yet.

Screen Shot 2016-06-28 at 1.56.04 PM

If you’re using customer feedback best practices, you’ll yield a lot of results. Now it’s time to synthesize them, without reading every single one with a fine-toothed comb.

2. Flag keywords in each verbatim

Rather than read each response individually, you can group responses into meaningful segments. That way, your team is analyzing a much smaller subset of data. The first step is to find a way to take all this information, all these reviews, and pare it down into approachable segments that you can do something with, rather than just a bunch of data.

Meaningful segments might include commonly mentioned categories that people bring up. For example, the word “tables” is important to flag, but “powerful” isn’t, since it’s too vague to mean anything on its own.

Here’s how Atlassian does just that. They use a tagging method to assign points for each keyword a verbatim mentions.

 

Screen Shot 2016-06-28 at 2.00.19 PM

It divides qualitative data into categories—usability, formatting, tables, etc.—that are commonly mentioned problems.

In order to parse qualitative data, you need to know what you’re looking for. By computing this electronically, you can do this at scale.

3. Quantify keywords

There are a couple different ways to approach analyzing this newly formatted data, depending on what you want to get out of it. Here are two popular ones we’ve seen.

  • Frequency of response: Atlassian assigns one point for every category a verbatim touches on. In the final analysis, they see how often each one is mentioned, which allows them to see what customers are bringing up the most often.
  • Sentiment of response: This tells you roughly how happy the customer giving the verbatim is. Atlassian does this through NPS scores. Another way is to use a sentiment analysis API. If a sentiment is negative, it will assign a poor score to the verbatim. If it’s positive, you’ll have a high score.

Once you have these quantified outcomes, you can check them against one another. Here’s what Atlassian’s looks like, which shows the correlation of categories users mention and the NPS score. It’s clear that a lot of people really liked the performance, but not the tables.

Screen Shot 2016-06-28 at 1.56.18 PM

This tells you which ones to read, which ones are the most important, and what’s coming up the most often, allowing you to get through a lot of data really quickly.

How to Visualize Customer Feedback

The other benefit of quantitative feedback, of course, is that it’s really easy to parse through visualization. Groove talks about why they love visualizing data from NPS scores into something easy-to-read like this:

Screen Shot 2016-06-28 at 1.56.24 PM
The problem, of course, is that while this tells you what customers love, it doesn’t tell you why they love it. You get a pat on the back, but without more specific details, you have no real answers about how to keep doing what you’re doing. That’s where qualitative data comes in. Of course, it’s much harder to visualize qualitative data and get the same easy-to-read, shareable results like the Groove example.

But it’s not impossible—you just have to roll up your sleeves and get your hands dirty.

When airline KLM (Royal Dutch Airlines) launched their new app design in May 2015, they immediately received negative reviews. A simple data visualization like this one, which shows their mobile app review ratings, tells a clear story:

Screen Shot 2016-06-28 at 1.56.29 PM
But it doesn’t tell you why it happened. What were customers seeing that made them change their minds? To answer that question, KLM asked for open-ended feedback from users to find out what they did and didn’t like about the redesign.

They flagged keywords and asked users what they thought of the app, and made it into a matrix. so that they could easily see what the flaws and strengths were, and weighted them all based on the frequency with which it was mentioned in customer verbatims.

KLM mapped two variables: how frequently something came up, and the sentiment users expressed in that result. The final product was a matrix that described exactly what the flaws and strengths of the product were, and how many customers brought them up.

Screen Shot 2016-06-28 at 1.56.42 PM

While check-in and flight info are both strengths, for example, they’re not created equal. Visualizing qualitative data into this matrix allowed the team at KLM to systematically parse the open-ended feedback they received, and identify commonalities within customer reviews.

What the team at KLM understood was that they couldn’t just use this method to tag and slot away customer feedback—they actually had to dive into it in order to improve.

The value of a matrix like this is it allows you to use qualitative feedback to drive a course of action:

  • Key Strengths: These are what drive users back to your application again and again—they’re your stickiest features. You should regularly update and monitor these parts of your application, but they’re not an immediate priority.
  • Other Strengths: These are features that users like, but use infrequently. They’re worth giving a second look and seeing if there’s something that will turn it into a Key Strength.
  • Other Flaws: These are features that users don’t like, but don’t bother to use very often. Consider cutting them from your application.
  • Key Flaws: These are features that your customers want to work, but they fall short of the mark. This is the key area where you need to concentrate your efforts to improve.

Taken as a whole, open-ended feedback can be overwhelming. The value of scoring open-ended feedback or on a matrix is that it allows you to prioritize what qualitative feedback you need to tackle first.

The Future of Qualitative Feedback Analysis

Groove’s 4,000 responses to their open-ended survey didn’t go to waste. They were carefully analyzed by employees who were able to draw up a report. But they’re in the process of further optimizing customer feedback. As they write, using NPS has been a great exercise in “the way we listen to our customers.” The value of open-ended feedback, no matter how you process it, isn’t as a score-card. It’s in being able to get real, human insights.

These techniques help us process qualitative feedback more efficiently. At the same time, the trick is to put our own biases and preconceptions aside and approach open-ended feedback with a clean slate.

Things like text mining, correlating open-ended response with NPS scores, or sentiment analysis are only going to get more powerful in the future. They won’t replace Customer Success Reps necessarily, but will allow product teams to quickly and accurately get the data they need to improve the product in ways that help the customer.

That’s exactly why Groove used Qualaroo to get feedback from customers. They received over 4,000 responses, all full of “Juicy, open-ended answers that gave [them] insight into the minds of [their] prospects.” While they were incredibly useful, it was really time-consuming to go through them all. As much as Groove agrees with Erika that those 4,000 answers held the solution to their problems, they needed a better solution for analyzing them.

  • Sofia Quintero

    This is a great a post, thanks for putting it together. Quantifying qualitative feedback and making user insights available to the entire organisation is something we’ve been working on for some time now at http://www.nomnom.it We have talked to a large number of product teams and we’ve seen very interesting attempts to automate the analysis of multiple sources of feedback. Some of the biggest challenges we see is around classification, for example, customer support data is very different to NPS data, not only on length but context, extracting the important bits from a long customer conversation is a very manual job. We are trying to solve that in different ways by aggregating sources of feedback, then allowing users to automate the classification process based on rules. Regarding sentiment, using APIs can help to certain extend but it can also become very time consuming if you are trying to analyse different data sets at once with high levels of accuracy, this is why you find sentiment libraries for twitter or domain specific. This is an exciting area and we see more and more product teams trying to balance quantitative and qualitative data insights, specially a focus on making those insights available to other teams in the business. A Qualaroo integration is in the pipeline for sure! Great post guys! 👍🏻