Smart devices are listening to more than our words

This episode was originally published May 17, 2021.

Picture this: You’re not feeling so hot and you say to your smart speaker, “Robot, I’m hungry,” and you cough. And the device says, “Would you like a recipe for chicken soup?” And then, “By the way, would you like to order cough drops with one-hour delivery?”

This is the scenario laid out in one of Amazon’s patents. And it shows how voice recognition technology could be used to learn things about us, beyond the words we say to our devices, like whether we’re sick or depressed.

I spoke with Joseph Turow, a communications professor at the University of Pennsylvania, who writes about all of this in his new book “The Voice Catchers: How Marketers Listen In to Exploit Your Feelings, Your Privacy, and Your Wallet.” I asked him if the technology is there yet. The following is an edited transcript of our conversation.

Joseph Turow: Well, companies believe it’s very sophisticated. They think that by using technologies that look at the sound that you have, and [your] syntax, and sometimes in combination with the actual words you use — they think it helps them sell and make people feel better. How do they know? Where do they get the data from? Amazon’s pretty close lipped about it.

Amy Scott: You argue in the book that voice profiling is a gateway to a whole new era of biometric profiling by marketers, even more than facial recognition. Why is that?

Turow: Yeah, well, facial is, of course, important. The thing about facial, though, is people are worried about it. And it’s not as integrated into the home as voice is becoming. I mean, some people have three, four or five of these devices in their home. They talk to their phones every day. So these are going to become part of our everyday experiences — “Turn on the light. Turn down the shades.” And as a result, the notion of giving data to companies and then having them evaluate them is going to become more plausible.

Scott: You say we are doing this all the time. But I think a lot of folks are pretty resistant to this technology. What about you? Do you have any of these devices in your house?

Turow: Yeah, well, to be honest, I did it in the beginning because I felt I had to learn about it. You can’t write about this stuff without having it. I have a Google device in my study here and I have a couple of Amazon devices around the house. I’m not too happy with the idea, however. And if I got the sense that they were beginning to go more into profiling me, I would probably shut them off.

Scott: For people who are really creeped out about marketers using our supposed mood to sell us things, are there any positive societal benefits to this technology? For example, someone living alone and a device kind of paying attention to how they’re sounding that day.

Turow: There are companies that are looking into trying to see whether voice can be a predictor of someone’s health. Looking, for example, at Alzheimer’s disease — can a person’s voice predict whether a person is becoming more and more likely to have Alzheimer’s? And these things are really important. My suggestion in the book: This stuff should be banned from marketing. Think of the dangers of a society where immigrants, for example, are listened to not just by what they say, but for their sounds of supposedly where they come from. There are a lot of really scary things that one can spin off in all of this, and we have to be scared of that.