The effectiveness of an e-learning tool can only be judged by how well it delivers a course content to educate and train its intended set of learners. But grading their performance at the end or asking them to fill out multiple surveys are not the only ways to judge its effectiveness.
Why Product Managers Need NPSMay 31, 2016
Lots of product managers shy away from customer feedback. They think it takes too much time and effort to gather, and that if they did everything people said they want, the result would be a bloated product that fails because it’s trying to be everything for everyone.
Their wariness isn’t totally misplaced. Feature creep is a real problem PMs have to avoid. But at the end of the day, the product determines what kind of experience the customer has. If that experience isn’t up to snuff, the PM has failed, and the product will fail too.
That means PMs are walking a fine line. They need to understand the voice of the customer (VOC), but they also need to separate the wheat from the chaff and pinpoint the feedback that will actually improve the product.
Net promoter score (NPS) is the weapon of choice here. It gets right to the heart of whether or not customers (you know, the folks who write you the checks) are successful with your product. While some criticize NPS data as overly simplistic, that simplicity actually makes it easy to analyze feedback from huge samples of users and figure out what product changes will have the biggest impact on customer experience. Let’s dive more into why NPS is so valuable to PMs and how they can get the most out of it.
NPS Gets Product Managers Aligned with Customers
PMs have a million different metrics to look at. Daily active users, free trial conversion rate, churn—the list goes on and on. When they hear NPS, they probably think it’s something for marketing or customer success to worry about.
On the face of it, the question NPS asks—“What’s the likelihood you would recommend this product to a friend or colleague?”—does sound more tied to customer satisfaction or word of mouth marketing than it does to product.
But think about the deeper meaning behind that question. It’s really asking customers, “Does this product solve a problem for you, and do you think it could for others?” That’s the entire reason customers buy software products in the first place. If they’re not likely to recommend you, they’re saying that they don’t think your product creates enough value to justify buying it. Getting the “why” behind each customer’s NPS rating lets PMs pinpoint what changes to the product could create more of that value.
How Intercom’s Product Team Studies the Customer
Intercom has built an entire business around enabling companies to communicate directly with customers. So, it’s no surprise that customer feedback is a core internal value in addition to a key part of their product offering.
Naturally, that applies to their product team as well. Every week, Intercom’s PMs read through hundreds of conversations between users and customer success to find ways to improve. It might strike you as a menial, annoying task, but Intercom’s PMs know that reading this feedback is the only way to understand how their product can provide more value to customers.
The entire system depends on the customer success team diligently tagging every conversation they have with the topic the customer wanted addressed: usability, feature request, bug, whatever. PMs then analyze the tags to create “hit lists” of common problems to focus on.
image source: Intercom
You can use NPS to create a similar taxonomy of user issues by collecting and analyzing each customer’s reasoning behind their NPS rating. Make sure you’re targeting customers who have had enough time to use your product and form an opinion, and then:
- Ask why. Every time someone provides an NPS rating, follow up with this simple, open-ended question: “Why?” The responses they write are called verbatims, and they’re the key to turning NPS into actionable insights. Qualaroo lets you track every response and verbatim as it comes in, and then export that data to other programs for analysis.
- Don’t let them say too much. The verbatim shouldn’t be a list of every single grievance the customer has with your product. Just the important ones. Limit customers to about 150 characters in their responses so they focus on the one or two issues affecting them the most.
- Categorize the responses. PMs then need to read through each verbatim and, like Intercom’s customer success team, tag each one based on what the user criticizes. For instance, at Spotify, a verbatim reading, “The app crashed my phone and the Southern Rap selection is lacking,” would be tagged as “performance” and “song selection.”
That process lets you tie every single NPS response back to specific parts of your product and see what each respondent needs in order to get more value.
Analyze the Responses
Should you just start trying to solve everything customers complain about indiscriminately? Of course not. That would lead to the bloated product you want to avoid in the first place, not to mention drive you completely insane.
That’s why it’s not enough to just tag each verbatim and count up which issues get the most attention. You need to dig into the data and figure out what’s worth addressing based not only on how many customer verbatims mentioned each issue, but also what kind of customers mentioned them—promoters, detractors, or passives.
Think about it. If someone complains about your UI but would readily recommend your product, then improved UI isn’t that big a deal. It’s more of a nice-to-have. Complaints from detractors, however, point to issues that are causing more damage to the customer experience.
How Atlassian Gets the Most from its NPS Data
Atlassian’s PMs are huge proponents of using NPS to guide how they build and improve products over time. Once they’ve read through all the verbatims and linked each one to specific categories, they have a simple framework for using that data to focus on where the product actually needs to improve.
The key takeaway: While the sheer number of customers who complain about an issue is important, the Atlassian PMs prioritize the issues cited disproportionately by detractors. Those problems are having a bigger effect on customer experience, so the PMs can have a bigger impact by fixing them.
That’s why they create a table of every issue that emerged in NPS responses, how many respondents complained about it, and most importantly, the average NPS scores of the users who referenced each category:
image source: Atlassian
At first glance, you might think performance is the big priority because more customers griped about it than anything else. But the NPS data shows that most of those customers are happy enough with Atlassian to recommend it. The problems with tables and usability, however, attract a lot of complaints and come from less happy customers. Fixing those issues would be the best way to improve the overall customer experience.
As the Atlassian process illustrates, NPS lets PMs separate the product changes that might be “nice to have” from the ones that would drastically increase value for currently dissatisfied customers.
Talk to Customers
Customers buy tech products because they have a specific objective they want to achieve or problem to solve. That’s the foundation of the all-important product-market fit every company needs to succeed. Yet the people in charge of those products spend most of their time cut off from customers. PMs don’t have time to be out on the front lines every day like customer success, but to get the most out of NPS data, they need to talk to respondents to understand exactly what they’re looking for.
Once you’ve sorted the data and identified the general areas where the product needs to improve, you need to find the customers with interesting responses and figure out the context of their complaints and requests. What value are they not getting from your product, and what changes could provide that value?
How Allianz Closes the Loop
Allianz isn’t a tech company, it’s an insurance provider. But its customer service agents have mastered a crucial customer success strategy known as closing the loop. Closing the loop means speaking directly with customers who have lodged complaints, learning how the organization can better meet their needs, and translating that into strategy.
image source: Bain
For example, several Allianz health insurance clients complained on NPS surveys about delayed claim reimbursements. But when customer service spoke to them, they talked more about not knowing the status of their claims. The problem was more the lack of communication than speed. So Allianz created a system to immediately call or text customers to explain any delays. The department’s NPS scores shot up. Loop closed.
Here’s how you can apply that exact strategy as a PM:
- Reach out. Contact some of the customers who referenced the issue category you’ve identified as the one to fix. Choose customers with particularly interesting, thoughtful verbatims, or the ones who rated you lowest on the NPS survey. If you integrate Qualaroo with a CRM like Salesforce, you can easily designate which customers you need to contact.
- Probe. You need to uncover the deeper meaning behind the customer’s request. If they asked for a new feature, ask them what they need it for and how it would help them get more value from the product. You might find your product can already provide that value, or that there’s another, simpler feature that can.
- Close that loop. As a PM, closing the loop for you means implementing the product change that provides the value customers are asking for.
Your users are probably the only people who spend as much time with this product as you, and whether they get value from it is the difference between success and failure. Talking to them is the best way to ensure you’re delivering what they need.
Products are Only as Good as the Experience they Provide
Entrepreneur-turned-VC Mark Suster writes here about a failed attempt to add a to-do list feature to his construction collaboration software, BuildOnline. The feature gave users a beautiful process to input tasks, assign them, and choose a due date, but since the team lead never talked to his architect and contractor customers, he overlooked a core need of theirs: They needed to input dozens of tasks at once, which would take ages under his process. The feature was useless.
There’s an important lesson in there for PMs. You can build the greatest product in the world, but it doesn’t mean anything if it doesn’t improve the reality your customers are living in.
Nothing uncovers that reality better than NPS. It cuts through the noise, lets you understand where your product falls short for customers, and gives you the data you need to fill the gap.
Measuring customer satisfaction is becoming more and more important with today’s highly aware and actively involved customers, because they have multiple platforms on which to share their views, both good and bad. This in turn can have a remarkable influence on other prospective customers about your business’s products and services.
Today we’re thrilled to introduce our newest feature, Templates! With Templates, you can now ship your Nudges faster and more confidently than ever.
You probably have a firm grasp on some of the universal metrics of SaaS success: ARR, growth rate, churn rate, CAC, LTV, etc. There is no doubt that these are critical, but in many ways these metrics do not tell the whole story of “success”. So what’s the leading indicator that can give you a fuller picture of success? Your customers’ satisfaction.
We grouped our list of 29 questions into the different topics you should consider exploring in prototype testing. Aim to choose at least one from each section to make sure all your bases are covered. We’ve also included a few pro-tips here as food for thought.
This article will cover the major Dos and Don’ts of prototype testing. We’ll walk you through the most common mistakes we see in the field and share tips on how to avoid them.
Testing prototypes is an inherent part of finalizing designs. Nobody wants to wonder why users are not utilizing an app the way it should be utilized or why they can’t seem to complete a purchase on your website. And nobody wants to rework something that’s already been shipped.
UX designers are under a lot of pressure to produce designs that add value to users’ lives. But without input from your users, it’s nearly impossible to design an experience that actually helps alleviate their pain points. If you’re pressed for time and/or don’t have the help of a researcher, getting the user input essential to design a great product can certainly be a challenge.