Houndify and Mercedes Benz
Jun 17, 2020

How Mercedes Benz’s MBUX Voice Control Revolutionizes the User Experience

When Mercedes-Benz’s Research and Development team set out to revolutionize in-car experiences with a powerful new voice assistant, they had the benefit of 120 years of car making expertise and a deep understanding of their customers’ needs and desires. This experience and knowledge guided their engineers to look beyond solving for the most common use cases, including navigation, music, and making calls. Their vision for the voice-enabled MBUX infotainment system was founded on a deep understanding of the future of the auto industry and the role of the car in the internet of things.

The auto industry is in a transformation phase right now. Cars are increasingly becoming computers on wheels. This has challenges in terms of technology. We have to transition into becoming more of a software company, while still building great cars.” 

Juergen Scherder, Sr. Manager, Speech & Digital Assistants, Mercedes-Benz R&D North America

During each stage of developing their voice assistant, the Mercedes-Benz team kept their focus on two important elements, their customers and their brand. Staying true to the Mercedes-Benz value proposition meant knowing who they are as a company and making sure their voice assistant communicated those values and attributes at every level, especially when it came to adding personality, tone, and humor to the voice assistant.

In the end, the goal was simple: Get people to interact with their cars with simple voice commands and allow them to be able to ask for anything in a conversational manner, just like they’d talk to another person in the car.

Building a seamless, frictionless in-car voice experience

The Mercedes-Benz mission has always been to provide people with the most comfortable and safe driving experience possible. The development of a voice-enabled infotainment system was a natural extension of that vision. If the team was going to add an in-car voice assistant, then it had to deliver the best experience possible. 

We want to deliver an experience for our users that is seamless so they can interact with the car in multiple ways and one of them is through a voice assistant. We want the system to understand very general questions such as “I’m hungry,” or more specific questions like a very long query with a lot of requirements for example a restaurant search.” 

Francisco Rois Siso, Software Engineer, Mercedes-Benz R&D North America

To achieve this level of conversational interaction and accuracy, the team set three specific goals:

  1. Allow drivers to talk as naturally as they would to another person and be able to say things naturally like, “I’m hungry” or “I’m cold.”
  2. Support more types of queries—specifically cloud-enabled services—such as buying concert tickets or pre-ordering coffee.
  3. Integrate the voice assistant even more naturally in the overall experience in the car, so users can switch smoothly back and forth between touch controls and voice commands.

For the immediate future, the team focused on moving from a world where drivers have knobs and buttons for everything, to providing an alternate input source for complex functionality and making those capabilities easily discoverable. 

Based on the knowledge that people are getting used to using voice assistants in their homes and on their phones, they set out to make the experience similar, but also relatable to the in-car environment. They knew that their customers want to have an easy interaction with the car in a way that helps them on a daily basis—by making routines easier, drives safer, and everything faster.

Basically it becomes your personal digital device that starts understanding you and supports you in all of your lifestyles. All of a sudden you’ve unlocked an ecosystem of opportunities as a customer that you weren’t able to do before.

Ben Boeser, Director of Open Innovation, Mercedes-Benz R&D North America

Overcoming the mindset of rigid voice commands

Developing a voice assistant for any product or application is not an easy task. With the help of our Houndify engineers, the team at Mercedes-Benz was able to extend the voice assistant to the cloud while helping their users change ingrained behaviors around addressing and interacting with voice assistants in a very rigid way.

Changing how we talk to voice assistants—shifting from barking a specific set of commands to having a natural conversation—was a challenge. And the change needed to start within the engineering team.

To this day when some of our own engineers or testers get into an A-Class or a CLA to test the voice assistant in the car, sometimes they say things like, “Find POI.” Nobody actually talks like that. What you want to say is, “Hey Mercedes, I want coffee.” 

Juergen Scherder

Making sure that every driver has the same positive experience requires the assistant to understand accents, dialects, and common languages used by drivers in the countries where their cars are sold. Even within the U.S., the multitude of accents and pronunciations posed a unique challenge. 

To really tweak the voice assistant in a way to understand different accents from different nationalities was something we didn’t anticipate in the beginning, but became quite core to a seamless customer experience.

Ben Boeser

Overcoming initial hurdles and creating a voice user interface that closely resembles that of human-to-human interaction is only the first step for the team at Mercedes-Benz. They are constantly looking to the future and for ways to improve the driving experience with voice technology. 

We’ve barely scratched the surface. There are many more things to come—you want to ask for the news, you want to ask for sports scores, you want to buy tickets, you want to pre-order and pre-pay for your coffee or a meal on your next break on a long road trip. There’s a lot of things we can do around mobility specifically that benefit a lot from having a voice assistant.” 

Juergen Scherder

The Mercedes Voice Assistant – Powered by Houndify

The partnership between Mercedes-Benz and SoundHound Inc. opened the door to a unique, powerful, and exceptionally-helpful cloud-based user experience for drivers.

For us, Houndify is really the door opener to the magic world of voice. And now we have that implemented, we are really refining the use cases. What are the capabilities that we now have? What else can we do? For us, Houndify was the absolute foundation to open up the world of possibilities that we didn’t have before.

Ben Boeser

Houndify’s extensive library of hundreds of content domains helped the MBUX voice team reach their goals faster, eliminating the need for them to start from scratch to build out all the responses to queries themselves, while still giving them the flexibility to customize any response.

The coolest thing is that the platform already provides a lot of domains that we can just plug and play into our cars and have them available right away. On the other hand, it also provides the opportunities to build your own experience, to build your own use cases. It’s a big platform that provides basically all the things you need, and this is why we were able to integrate it so fast into our products” 

Cassidy Schwarze, Senior UI Software Engineer, Mercedes-Benz R&D North America

Houndify’s hybrid engine and domain libraries enabled the team to create an intuitive and truly conversational voice assistant and has extended the “Hey Mercedes” experience from only controlling in-car functionality (like windows and air-conditioning) to delivering results for external things like sports scores, general knowledge questions, weather, and so much more.

That’s what’s really useful for our customers, to not only be able to control your car, but ask for anything. Mercedes and Houndify will change the way people talk to their cars, forever!” 

Juergen Scherder

From the beginning, the teams were focused on communicating the unique Mercedes-Benz brand with a voice assistant that would eventually come to be like a friend. Houndify’s proprietary Speech-to-Meaning ® technology combines the traditional two-step ASR and NLU models into one integrated action that processes speech and delivers results in real time, giving the illusion of talking to a real person.

Just having a system where you can shout commands at your car might have been great 10 years ago, it’s certainly not going to be great 10 years from now. These voice assistants over time will feel more and more like a real person. It’s never going to be a real person but in your perception it’s almost like you’re talking to a friend.

Juergen Scherder

Building the MBUX was a multi-year project with many components. Once the team started working with SoundHound Inc., the cloud-based component of the voice user interface took two years to implement—a fairly short sprint in an otherwise long journey. The MBUX infotainment system as drivers experience it now is only the foundation for their vision of the future.

 “There was a pretty big innovation when touch screens hit the market and people saw it as the next big thing. I really believe that voice is that next big thing. Every device that you’re interacting with that has a bit of technology you will probably expect it to have some form of voice enablement so you can talk to it, control it, and it will talk back to you. Of course, a car is a pretty big part of your daily life.

Mihai Antonescu, Senior Software Engineer, Mercedes-Benz R&D North America

Collaboration and teamwork make all the difference

The mantra of voice user interface (VUI) design is to build, test, iterate, test, iterate, and then iterate some more. The keys to making that process work is agility, good user data, and engineering expertise to execute the vision. The partnership between Mercedes-Benz and SoundHound Inc. had all three.

It was really an iterative process of working together with SoundHound and their engineering teams. We had to be very agile in taking this as soon as possible to actual customers, to actual users, and then learn what works and what doesn’t work and then in collaboration with SoundHound’s engineering team, refine it as we go.

Ben Boeser

The Mercedes-Benz team began with a vision of a luxury driving experience that exceeded customer expectations. The addition of a voice-enabled infotainment system was a natural extension of those efforts and a logical space for innovation. The MBUX infotainment system is improving the driving experience for Mercedes-Benz customers by allowing them to control the in-car environment, and connect with the outside world without ever taking their eyes off the road.

With an eye to the more distant future, the team is continuing to iterate the voice integration to become a true personal assistant that helps their customers navigate every aspect of their lives.

Developers interested in exploring the Houndify independent voice AI platform can visit Houndify.com to register for a free account. Want to know more? Talk to us about how we can bring your voice strategy to life.

Karen Scates is a storyteller with a passion for helping others through content. Argentine tango, good books, and great wine round out Karen’s interests.

Interested in Learning More?

Subscribe today to stay informed and get regular updates from SoundHound Inc.

Subscription Form Horizontal