How can you read her emotions without looking at her face?

“If he really knew me. If he really, truly knew me…”

I sang that song to myself in my teens whenever I felt doubt that my budding romantic relationship with someone who loved music was amounting to anything worth both our time. I sang it through my very young love feelings – for the boy playing wonderful piano, or for the boy and his soulful guitar, or the boy who breathed through soothing flute music. As it is with music, singing that song saw me through some emotional in-betweens that are now woven into my own adult emotional repertoire. That’s the context of “If He Really Knew Me” for me.

But it is also a song in the 1979 musical They’re Playing Our Song. It’s about a male composer and a female lyricist who are trying to work out a romantic relationship, and they wonder to themselves how much of their relationship is about the music they make together and how much of it can survive, without their music. That’s “If He Really Knew Me” in that context. 

See how the lyrics of “If He Really Knew Me” change in meaning depending on the context? Context is essential for words to have meaning. Words by themselves will simply float if context does not anchor them. 

Now, imagine the different moving settings for each. First, a teenage me trying to sing the song while in the background is a young boy playing the guitar. Remove all the audio. By reading my facial expression and body language, you would most likely be able to read what the story of that scene is. 

With audio still removed, imagine the same song being sung by a female lyricist who, with her pen and paper, is beside a male composer on the piano. You also would most likely be able to tell what the story is about as you watch her and her expressions. 

But in both moving scenes above, if you blot out the face and the body of the one singing “If He Really Knew Me,” do you think you would be able to track the left to right (negative to positive feelings) and the ups and downs (high and low arousal) of what she is feeling?   

Surprisingly, a very creatively designed study that came out this year showed you most likely would. 

The researchers wanted to find out if you could track someone’s emotions without seeing her face or her gestures but by simply reading the dynamic (moving) visual context (other people in the scene and other elements in the background) in which she is seen.

By “tracking” emotions, the study referred to whether you can plot changes in someone’s emotions along a cross path – horizontally (if they are expressing negative to positive feelings) or vertically (if they are expressing high and low arousals) – as the scene unfolds. This kind of tracking is considered in the field as some kind of “emotional dashboard” where emotions could be plotted in relation to each other. This did not involve identifying “distinct” feelings of anger, joy, or fear. 

Indeed, the study found that the dynamic visual context is not just essential in detecting the wave of emotions but that it can be used by itself (without observing the face or body language) to track the run of a person’s emotions. In tracking this “emotional dashboard,” the study also found that when we try to figure out what someone is feeling throughout a scene, we study not just the other people in the scene but also the other nonemotional elements in the scene like objects.

So imagine the 3-minute-34-second musical scene where the lyricist’s face and body are blotted out as she expresses her emotions in song. But you can see the male composer’s face occupied with the piano, and him putting the notes on the sheet and not really looking at her direction. Then you can also notice the pen of the lyricist that’s lying flat on the sheet and not moving. The study’s findings showed you would most likely be able to track her emotions with the same accuracy as if you’re observing her face and body language in the same scene. This was observed in a range of video materials, including those with or without interpersonal interactions, with posed or spontaneous facial expressions, and with staged or natural scenes. 

The scientists who did the study think that we could do this because certain scenes or elements of scenes are familiar to us and we have, in various ways, been “placed” in the context of those visual elements or similar ones at some point in our lives. They also think that visual context corresponds to certain “mental states” which we can then attribute to the person whose emotions we are trying to read. 

This is a big reveal because facial recognition technology, for better or worse, is now being widely used and continually being refined to detect what we are really feeling. If background visuals are essential to track the emotions of a person, then facial recognition technology will have to smarten up and include context to really get the meaning of any scene. This means that “static” snapshots of mere facial expressions are probably the worst way you can read through someone’s emotions, as emotions do not stay still. They move in multilateral paths and at times, they even run. 

The composer of “If He Really Knew Me” is Marvin Hamlisch, and he once sang this onstage in front of a big audience. He slightly changed the lyrics to “If You Really Knew Me” as he sang to the audience. At one part, he goes: “Does the man make the music? Or does the music make this man? And am I everything I try so hard to be?” Without hearing him sing the song or seeing him as he sings and plays on the piano, you will only see the piano keys, the faces of the other orchestra members looking at him throughout the song, and the audience reacting to his music. I wonder if with music rendered like this – so personal and so singularly dependent that it is Hamlisch singing it – if context is still so crucial that you would still be able to track the pluses, the minuses, and the highs and lows of Hamlisch as he sings it. Without the music as he embodies it, will we able to, as the song says, “find” him, the part he left behind him? – Rappler.com

Maria Isabel Garcia is a science writer. She has written two books, “Science Solitaire” and “Twenty One Grams of Spirit and Seven Ounces of Desire.” You can reach her at sciencesolitaire@gmail.com.