Machines on music — AI helps figure out how music tickles your brain and your body

Originally published Nov. 22, 2019

Music can take us on an emotional journey. Now with the help of artificial intelligence, a team of scientists has broken down the different features in musical compositions that tickle our brains, gives us goose bumps and move us emotionally. 

"I think oftentimes when I'm looking at art or listening to a song, sometimes I'll get shivers and not really know why or not really understand what the reason is behind that," said Timothy Greer in an interview on CBC Radio's Quirks & Quarks.  Greer, a Ph.D candidate in computer science at the University of Southern California, led a new study exploring these questions.

His team studied the effect different musical features had on the brain using advanced brain imaging, and on the body measuring heart rate and sweat response. They also had listeners report how the music made them feel to get a better understanding of the holistic picture of how we respond to music. AI then crunched the numbers to help give them insight into how different features affected the body, brain, and emotion.

The results were counterintuitive. Greer said he expected there to be more overlap between how our brains, bodies and emotions are affected by music. That wasn't the case. "The finding was actually quite surprising." 

Complex music comes together in such a way that affects our different response modes in different ways.- Timothy Greer, University of Southern California

Instead, he discovered the brain, body and reported emotional response all respond to different features in music, which may give insight into why different musical compositions can affect us in complex ways. 

The team explored several different features in music, including:

• Rhythm.
• Dynamics, or the varying volume of the music.
• Spectrum, or the frequencies of the musical notes.
• Harmony, or the relationship between notes.
• Timbre, or the tone quality or texture of the music.

Tracking down emotional music

To study how music can trigger emotional responses, the first thing they did was track down really happy and sad music. 

"We only really wanted to look at happiness and sadness in response to music," explained Greer. "These are two obviously very simple emotions, but we wanted to have a clear difference between the happy song that we chose and the two sad songs that we chose."

To track down these happy and sad songs, they went to musical streaming and social media sites to find instrumental songs tagged as "happy" or "sad" with the fewest play counts to minimize any previous — positive or negative — associations people may have with the songs or lyrics. 

"We wanted songs that would be enjoyable enough for people to listen to all the way through, but we also wanted them to have an emotional response to the music. And we didn't want them to be too familiar with the songs before we played the songs to them," Greer said. 

They then narrowed down their more than two dozen choices with an online survey.

Test subjects for their study either listened to the three pieces of music in an fMRI brain scanner, or while researchers measured their heart and sweat rate. All of the study participants also had to rate in real time how happy or sad the song made them feel. 

"We use techniques in artificial intelligence to synthesize the musical information that was contained in the stimuli to predict and study how these stimuli really affect how we respond to music," said Greer. 

The 'Happy Song' the scientists used in the study: Racing Against The Sunset by Lullatone

Interacting musical features generate emotional response

They discovered that features related to the dynamics of the song — variations in how loud or soft it was — and the rhythm, best predicted how the participants' brains responded to the music. 

On the other hand, features related to the harmonies and spectrum of the musical notes played in the song best predicted the bodily responses to the music. 

When it came to how the participants rated their emotional responses — whether happy or sad — these were best predicted by the timbre, harmony and spectrum musical features of the song.

"This very complex music comes together in such a way that affects our different response modes in different ways," said Greer. 

So, for example, the overall intensity of emotional responses come from features like dynamics and rhythm, reflected in brain activity, "but we're finding that these features that are related to harmony or spectrum are actually more indicative of how a song is happy or sad."

Simply switching a major key with a minor key — or vice versa — can flip the emotional flavour of the music from happy to sad or the other way around. 

Knowing how different musical features affect our emotional response could come in handy for applications in musical therapy.  It might also be used to generate more emotionally cohesive musical playlists.

Greer said this could also help out songwriters like himself understand how to provoke specific emotional responses in the music they create. 

He added how this study has given him a greater appreciation for composers and songwriters who come up with songs that can leave an emotional imprint on us, "because they seem to be able to already synthesize the information that's in this study to create music that really speaks to me, and really affects the people that are listening to it."

One of the two 'Sad Songs' the scientists used in the study: Fyrsta by Ólafur Arnalds