Xperi Cockpit Combines Safety With Entertainment, Can Detect Bored Passengers

When you’re driving along software company Xperi wants you and your passengers to be both safe and entertained, somehow using the same technology. It’s a combination of detecting who’s in the vehicle, where they’re sitting, what they’re doing and if they’re happy.

“While there’s a safety component, if you can take some of that information and tie in what you’re doing from an entertainment perspective, like who’s in the car, where are they sitting, what are they doing you can create this idea of a more immersive entertainment experience,” explained Jeff Jury, Xperi senior vice president and general manager, connected car. “We’re taking what would normally be safety technology, combining it with traditional entertainment to create that more intimate, personal ability to understand what are the individuals in the car doing, what are they enjoying, not enjoying.”

Two key technologies create this kind of compound experience—AutoStage and AutoSense, which Jury and a couple of colleagues demonstrated for Forbes.com at the company’s Troy, Mich. location.

Driving first in a Mercedes-Benz S-Class to better understand image-based AutoStage, Jury explained the display provides much more than simply a radio station’s frequency, call letters and name of the song or program playing and is destined to go further in the future.

“It’s almost like a media platform. It can start doing recommendations,” said Jury. “It can potentially start putting lyrics out there. Listening to a song it could list events. Let’s say you’re listening to a song you could have information that says when is this individual playing a concert in town.”

The current version of AutoStage is currently available in all S-Class vehicles. Additional features will be coming in “a few months” according to Jury.

The safety half of the equation was demonstrated in a Honda Odyssey using Xperi’s AutoSense technology.

A single camera captures all of the vehicle’s occupants then the software takes over detecting whether or not the driver is distracted and what anyone else on-board is doing.

Using a special display that would not be in a production vehicle, Joseph Ayo-Vaughan, Xperi staff engineer, integration showed how the system could detect if he put on or removed his eyeglasses, picked up his phone or was eating a snack.

Using one camera rather than several is an obvious cost savings says Jury who points out his San Jose, Calif.-based company is building off technology already long in use in smartphones.

“Facial recognition, computer vision capabilities for phones, we took that and we built off that to build OEM car implementations,” Jury said. “We build the algorithms, not the camera, to look at eye gaze, facial expressions, attributes to determine if people are distracted or leave a baby in the back seat. We take the same logic and say how do you use that same technology to then tie to what you’re offering to what you’re (automakers) offering in an entertainment system.”

To further carry the intersection of technologies, Jury says Xperi’s algorithms could even be used to help solve a sticky situation—what to do about passengers who don’t love the music or program pumping out of the speakers.

“Are people distracted, are they interested? If you’re listening to this maybe you’ll like that. As part of that you can gauge people’s emotional reaction to the content and based on that and make other recommendations ,” said Jury.

That’s something likely to take on added importance one autonomous vehicles are more prevalent leaving occupants with more idle time while traveling.

Xperi is working with around eight automakers right now building a full array of vehicles from what Jury calls “everyday sedans” to EVs, declaring “if you’re getting a more immersive entertainment experience you have to pay attention to safety.”