Smart devices are listening to more than our words

Picture this: You’re not feeling so hot and you say to your smart speaker, “Robot, I’m hungry,” and you cough. And the device says, “Would you like a recipe for chicken soup?” And then, “By the way, would you like to order cough drops with one-hour delivery?”

This is the scenario laid out in one of Amazon’s patents. And it shows how voice recognition technology could be used to learn things about us, beyond the words we say to our devices, like whether we’re sick or depressed.

I spoke with Joseph Turow, a communications professor at the University of Pennsylvania, who writes about all of this in his new book “The Voice Catchers: How Marketers Listen In to Exploit Your Feelings, Your Privacy, and Your Wallet.” I asked him if the technology is there yet. The following is an edited transcript of our conversation.

Joseph Turow: Well, companies believe it’s very sophisticated. They think that by using technologies that look at the sound that you have, and [your] syntax, and sometimes in combination with the actual words you use — they think it helps them sell and make people feel better. How do they know? Where do they get the data from? Amazon’s pretty close lipped about it.

Amy Scott: You argue in the book that voice profiling is a gateway to a whole new era of biometric profiling by marketers, even more than facial recognition. Why is that?

Turow: Yeah, well, facial is, of course, important. The thing about facial, though, is people are worried about it. And it’s not as integrated into the home as voice is becoming. I mean, some people have three, four or five of these devices in their home. They talk to their phones every day. I spoke to new home builders who are integrating them directly into the walls of the home. So these are going to become part of our everyday experiences — “Turn on the light. Turn down the shades.” And as a result, the notion of giving data to companies and then having them evaluate them is going to become more plausible.

Scott: You say we are doing this all the time. But I think a lot of folks are pretty resistant to this technology. What about you? Do you have any of these devices in your house?

Turow: Yeah, well, to be honest, I did it in the beginning because I felt I had to learn about it. You can’t write about this stuff without having it. I have a Google device in my study here and I have a couple of Amazon devices around the house. I’m not too happy with the idea, however. And if I got the sense that they were beginning to go more into profiling me, I would probably shut them off.

Scott: For people who are really creeped out about marketers using our supposed mood to sell us things, are there any positive societal benefits to this technology? For example, someone living alone and a device kind of paying attention to how they’re sounding that day.

Turow: There are companies that are looking into trying to see whether voice can be a predictor of someone’s health. Looking, for example, at Alzheimer’s disease — can a person’s voice predict whether a person is becoming more and more likely to have Alzheimer’s? And these things are really important. My suggestion in the book: This stuff should be banned from marketing. To me, that’s a red line. Think of the dangers of a society where immigrants, for example, are listened to not just by what they say, but for their sounds of supposedly where they come from. There are a lot of really scary things that one can spin off in all of this, and we have to be scared of that.

Scott: What do you want both consumers and, I should say, regulators to take away from your book?

Turow: For consumers, I want us to be wary, to realize that this is a kind of seductive surveillance. There are so many historical precedents to the idea that we can get used to almost anything. I mean, we can get used to things because they seem like they’re terrific. And down the line, we see negative aspects. So I say we have to be wary. When it comes to regulators, my point is that regulators have not been so great at actually regulating a lot of our marketing technologies. And I think when it comes to something like voice and other biometrics, sometimes it’s OK to say, “No, we’re not going to allow it.”

Scott: Can consumers opt out of this? Is there a way to have the convenience of these devices without the voice analysis?

Turow: Reading the privacy policies of a lot of these companies, it’s extremely difficult to figure out exactly what they’re doing.

Related links: More insight from Amy Scott

Devices aren’t just listening to us in our homes. Ford just announced it will soon offer a three-year complimentary subscription to Amazon’s Alexa in hundreds of thousands of its cars and trucks. After those three years, drivers have to pay for it. Alexa was already available in many models via smartphone app, but now it will be embedded in the vehicle operating system. It’s not just Amazon; Google and Apple have also been forging partnerships with carmakers.

Our producer Stephanie Hughes tried to see if Alexa would actually try to sell her cough drops, like Amazon’s patent imagined. She asked a series of questions, including one that was kind of leading: “I could use some cough drops.” Alexa’s first response was to recite the Wikipedia definition of a throat lozenge. Stephanie had to specifically ask for cough drops before Alexa served up some links on Amazon. So, you know, work in progress.

Finally, Joe Turow talked about how people are worried about facial recognition. Well, The Washington Post had a good piece on one reason to worry. Upload a photo to the face-search site PimEyes, and it’ll scan through more than 900 million images in less than a second to find matches. What makes PimEyes stand out isn’t necessarily its technology, but that anyone can use it — not just law enforcement. Including stalkers. You can block photos from showing up in a search, but it’ll cost you $80 a month.