The Richness of Data—Guiding Principles of Voice User Research
Dr. Joan Palmiter Bajorek
Data is power. “Data is the new gold,” says entrepreneur Mark Cuban. Data can demonstrate that you’ve hit your KPIs, illuminate a system’s bugs, help you understand your users, and provide valuable insights into next steps. Today, almost anyone can build voice experiences, but since many of us come from different backgrounds, we all have room to grow our research skill set — a tool that has already stood the test of time. And user research can help us truly understand the context in which people would use a voice application, as well as the behavior of those who would use it.
Research is a systematic approach to understanding data. You can be knee-deep in data, but not understand it. If you are in voice but do not have experience with research, you may wonder where to begin. What data should you consider? How would research help your team to reach their goals? Good news: It’s easier than you think to get started. This article outlines guiding principles for harnessing insights from voice user data to support any project.
“The goal of user research should be to understand users and retain satisfied customers, not to exploit them.”
An article about data is not complete without the mention of ethical practices. No research should abuse user rights, exploit privacy, endanger users, and/or violate GDPR. Here are some fantastic checklists about data collection, storage, and analysis (Deon, 2019) (O’Reilly, 2018), collected by Salesforce’s Kathy Baxter. The goal of user research should be to understand users and retain satisfied customers, not to exploit them.
Why conduct user research?
Since some teams don’t invest a lot in research, these statements may sound familiar:
● “Our biggest priority is to hit our deadlines. We don’t have resources for research.” ● “What is the ROI for user experience (UX) research?” ● “We’ve been doing fine so far, why start now?” ● “I don’t have experience with that. Where would I even begin?”
If these statements resonate with you, you are far from alone. Yet, conducting no research at all may cause you significant grief down the line.
“Not doing user research is the quickest way to build the wrong thing,” says Hilary Hayes, Alexa Cup Champion. First-hand, I’ve seen a team build 75% of a system before finding out it did not support basic user requests. It was a huge setback. The team had to go back and rebuild significant chunks of the system. A modicum of research could have helped save significant time, effort, and money.
“I’ve seen a team build 75% of a system before finding out it did not support basic user requests. It was a huge setback.”
Beyond “All or nothing”
Research does not need to be framed as “all or nothing.” No matter how big or small, the right analyses can be impactful — a framework I call “Big R” research and “Little r” research.
“Big R” research is large-scale and might include dedicated UX researchers and data scientists, and typically is found at large tech companies and universities. While large-scale research is important, it may not be a reality for many.
Examples of “Big R” Research
● Recruit hundreds of participants to conduct remote and in-person user testing. ● Analyze the acoustics of specific participants to customize an NLU system for a target user base.
“Little r” research is small-scale research that can be effective for tackling problems in shorter time frames and with fewer resources. Even a small team can find a few hours to consider small-scale research.
Examples of “Little r” Research
● Analyze how 3 competitors approached a feature you’re building. ● Ask 5 people to read conversational flows aloud and get feedback from someone who didn’t build it. ● Look at system performance analytics at strategic times.
Smaller research projects are less resource intensive and could take just a few hours. Look at user research as a really useful tool in your toolbox. Maybe not a tool used daily, but a healthy part of a regular design and build process.
Guiding principles for voice user research
Below is a primary framework for guiding principles of where to begin conducting research with your team. Whether you’ve done research in the past or this is all brand new, a great place to start is with goal alignment.
Goal alignment on research questions
What would your team like to know about your users and product? This answer is unique to every team. Take 30 minutes and circle up the entire team. Talk about what insights you’d like know. Write down research questions. Be as specific as possible, especially with the “what,” “who,” “where,” and “why.”
Examples of voice user research questions:
● Unsupported Utterances: For user utterances, are there patterns in ones we currently do not support, the ones in the “garbage” bin? What percentage of utterances currently go to “garbage?” ● Transaction Flow: How many users complete a specific task? Do some drop off midway, if so where? ● Common User Paths: What are our highest frequency use cases? Are we supporting those well?
Common user paths expanded:
● Who: US users, Q1-Q3 ● What: Are we supporting 3 most common use cases well by percentage of users? ● Where: Within the main product ● Why: We must support our most common use cases well to retain customers. If our users are not satisfied with these common use cases, we need to change course fast. For us, “support well” means that our average customer satisfaction rating should be 4.2 out of 5 stars.
While this example is merely an example, it is clear that these could be extremely valuable insights for a team as they iterate their voice experience. Once the team is aligned about what you’d like to know and why, it is time to choose the data set and analyze it.
2. Data: What to collect and analyze
How do we know what data to collect? The answer is a function of your research questions. There are many types of data. Quantitative data is about quantifiable numbers and the “what.” Qualitative data is typically more observational and about the “why.” Some types of data have both qualitative and quantitative aspects, like surveys with multiple choice and write-in answers.
Below are some data collection techniques:
I advocate a “mixed-methods” approach where qualitative and quantitative data insights are triangulated to understand the user experience. How would that work?
Let’s continue the common user paths example to see how using both quantitative and qualitative data could provide triangulated insights:
● Analytics: Show that games are the most common use case: 40% games, 22% information, and 17% purchases. ● Surveys: Indicate that most users are satisfied with experiences in games and purchasing, averaging of 4.3 and 4.4 out of 5 stars. However, many are frustrated by the information finding experience, averaging 3.1 out of 5 stars. Yet it is not clear from the survey and analytics data where this frustration comes from. ● Interviews: When asked, users tell you they are often routed to the wrong information and get flustered. You realize an explicit confirmation might be helpful to support users on the 2nd most common use case of your product.
Use data to answer your research questions. Be open to findings and directions you weren’t expecting. It’s part of the process. Check in with the team before going down those rabbit holes.
Want someone else to do this work for you? Get support from voice research domain experts (alphabetical): Answer Labs, Applause, Bespoken, Dashbot, and Pulse Labs [Conflict of interest note, I have previously worked at Applause and have connections who work at all of these companies. No organization paid me to be mentioned here.]
3. Harness findings for optimization
You’ve done the work, now it’s time to learn from it. Organize your insights to the research questions and report them back to the team. Findings could be used for a variety of purposes including marketing materials to drive user adoption, and product optimization for user retention.
Here are some suggested uses for research data immediately and in the future:
● Today: Illuminate today’s bugs that may need attention now. ○ The 2nd most common use case frustrates users.
● Successes and room for growth: See how users are supported compared to benchmarks you have for the team or clients. ○ Users are satisfied and have quality experiences with games and purchasing.
● Steps forward: Validate a road map of features to iterate. ○ Our information experience is important and has room for iteration.
Findings from this research should be valuable today, tomorrow and beyond, serving as a benchmark when you optimize and run the numbers again. You should be able to see whether changes shifted the needle. Ideally, research would be built into your team’s workflow for regular updates and optimization potential.
Understanding the richness around you
If data is the new gold, then research helps you understand the richness all around you. Understanding your user data can be wildly valuable. Teams I’ve worked with often wish they had started a research plan sooner and begun collecting data from day one.
1. Goal alignment on research questions 2. Data: What to collect and analyze 3. Harness findings for optimization
If you’re ready to start, take this article and begin the conversation with your team about what research might best fit for your team’s goals.
Dr. Joan Palmiter Bajorek is the Head of Conversational Research and Strategy at VERSA, and the Founder and Director of Women in Voice. Her PhD research has been published by Harvard Business Review, Cambridge University Press, and UXmatters, exploring the future of voice technology, bias in AI, and multimodal user experiences.Twitter: @joanbajorek