On Spotify, music listens to you: Streaming platform wins patent to surveil users’ emotions to recommend music

Streaming music platform Spotify has won a patent enabling it to snoop on users’ speech and even background noise in order to gauge emotional states and location types to serve up an appropriate soundtrack. Not creepy at all!

Spotify has received a patent that will allow it to use speech recognition and sound analysis to assess a user’s demographic attributes, determine their emotional state, and even glean insight into their location. The information will be used – hypothetically, at least – to pick the perfect song to play without requiring any conscious data input from the listener.

Arguing that expecting users to input the details of their own tastes and preferences was asking too much of the platform’s average user (and consumes valuable time that could be spent streaming music), Spotify applied for a patent to automatically perform these functions in February 2018. It was granted earlier this month, though went unreported until it was picked up by music press on Wednesday. 

Using speech recognition, the app will not only be able to pinpoint the user’s age, gender, and other demographic basics – it will analyze their voice for “intonation, stress, rhythm and the likes of units of speech” in order to determine emotional state. As if that wasn’t intrusive enough, the app will pick up “environmental metadata,” parsing background noise to find out whether the user is out for a walk, riding the bus, at a party, and so on.

The newly patented system cascades depending on how the user reacts to the selected song, seeking “positive metadata/emotional state” in order to fine-tune the algorithm. Listening and rating history, as well as friends’ histories, are also factored in. Which emotions Spotify will encourage and which it will seek to shift the user out of through music suggestions are not explained in the patent.

The dubious new ‘feature’ is far from Spotify’s first step over the creepy line. In October, it secured a patent for “methods and systems for personalizing user experience based on [user] personality traits,” seemingly laying the groundwork for the ‘feature’ it unveiled earlier this month. The patent would allow the streaming service to “humanize the user interface… in accordance with the user’s personality,” perhaps speaking in a more bubbly tone to listeners identified as extroverts while using a more subdued voice for introverts.

In a paper published in July, Spotify cited a three-month study it had done analyzing 17.6 million songs listened to by 5,808 users, explaining AI analysis showed “moderate to high accuracy” in personality traits predicting musical tastes. The platform was quite open about its desire to do more research “to link streaming behavior with brain scanning, genetic, and physiological data.” 

Also in October, Spotify won a patent to deliver geo-targeted advertising using 3D audio, which would give the impression that streamed sound is coming directly from a geolocated source – say, a coffee shop literally beckoning to the listener as they walk down the street. It’s not clear if Spotify users will be given a chance to opt out of these features or whether they’ll be silently slipped into an app upgrade, ready to freak the user out the next time a shopfront starts talking to them.

Spotify seems aware of how powerful the complex emotional profiles built up by the app will be, noting in the July paper that “a user’s digital history is extraordinarily personal and sensitive and should be treated with proper consideration of the conceivable misuses and unintended externalities.” And should such a bevy of behavioral data fall into the wrong hands? It’s not like Spotify hasn’t been hacked before.

If you like this story, share it with a friend!