Music is more than sound waves: What you see and how you move, affects what you hear
When you hear a melody, your perception is formed by the shapes and movements you associate with it.
When you hear Beyoncé sing, how long does it take before you visualize her dancing across the stage? Or when Jimi Hendrix's guitar solos are pounding out of the speakers—can you see Hendrix posing with his guitar?
Whether you're miming in front of the mirror and using your hairbrush as a microphone or listening with one ear while cleaning the house, the movements you associate with the music play a role in what you actually hear.
That's because music is more than a good lyric or melody. Music is the interplay between everything you sense.
"Just think about how you feel when someone is singing really high notes," says music researcher Tejaswinee Kelkar, and continues:
"What we actually notice is the effort being made by the singer. We recognize it because it's physical. We don't even need to see the singer, because we are so fine-tuned to interpreting nuances in the voice—which, for example, tells us about the singer's feelings."
Kelkar studies the shapes we associate with music and has investigated this by studying what gestures people make when listening to music. Facial expressions and the position of one's legs are just a few of the things that she believes affect one's listening experience.
When singing, you also use your arms
Tejaswinee Kelkar is herself a performing singer, and her interest in gestures developed when she became aware that there is a difference between how we use our hands when singing Western and Indian music.
"As a child, I learned to sing North Indian music. It is common there to use hand gestures in order to help children when they are learning to sing. When you're on stage, you should sing in the same way as you do when you rehearse. You focus on the song as being sung between you and the room, instead of thinking about the audience."
When she was later trained in Western classical singing, she began using similar hand gestures.
"But I was told quite firmly that that was not how it was done."
She became curious: What did the hand gestures really mean?
"You might think that these movements provide you some kind of anatomical assistance, or that they shape how you use your voice. In that case some hand gestures may be suitable for Indian music and not for Western music."
To Kelkar, the different rules that applied to gestures also served as proof of something else: in order to understand music, you need to think about how it fills the space in a room.
Melodies can have different shapes
Music plays on all our senses because it is multimodal—it takes place in different modes. For Kelkar, the main mode is spatiality. She refers to the mathematician René Thom who says: "In order to understand something, we need to understand the geometry of it."
"I believe he's right: everything has a spatiality. Time, which is important in music, is a good example. We relate the past and future to our bodies—that something lies in front of or behind us."
In order to understand more about how music is perceived spatially, Kelkar has conducted several experiments. In one of these, she asked the participants to listen to the same piece of music several times and draw or explain what they were visualizing in their minds.
"People often perceive specific shapes or use movement metaphors. Several of the participants described or drew the music as a wave that was passing by them—like sound waves or ECG (electrocardiography)," she explains.
"While others visualized a circle, especially if they noticed that one motif in the melody kept recurring."
In another experiment, she asked participants to move their arms in a way that they felt matched the music. She documented everything by using motion capture tracking technology in order to search for patterns.
"Several tried to draw the contours of the music going up or down. This reflected the pitch, but also other characteristics of the music, such as timbre, motifs and patterns."
Facial expressions affect the sounds we hear
The fact that our brains make a connection between what we see and what we hear is something that has already been studied by language researchers.
"This has been observed as linguistic phenomena, including one called the McGurk effect," says Kelkar.
The McGurk effect describes how we can listen to a sound while observing a face that expresses a different sound and hear something which is sort of in between. A classic example of this is when we hear a B pronounced while seeing a face expressing a G, we will end up hearing a D.
Kelkar recently conducted a study with her colleagues Bruno Laeng and Sarjo Kuyateh which was designed to see if the same thing happens when we sing:
"We actually found signs to indicate that what the singer does with his or her face affects how people perceive a melody. The interval between two tones may sound different if the singer's mimicry varies."
Shazam for movement
Just as melody and rhythm allows us to recognize music, the shapes that we associate with a melody can help us with the same thing. In her research, Tejaswinee Kelkar has fed artificial intelligence with documentation of the different movements derived from her experiments, thus allowing the technology to use the movements to recognize music. This type of technology can be used for developing new tools.
"Imagine a technology similar to the music-recognition app 'Shazam' but imagine it scanning movements instead of sounds. If you were to make gestures that would be match 'Happy Birthday,' such an app might be able to find the song for you."
Although this technology is relatively new, searching for melodies based on contours is an old idea. In 1975, Denys Parsons published his "The Directory of Tunes and Musical Themes" in which he cataloged about 15,000 classical pieces based on their melodic contours—how the pitch moves up and down. The identification of music based on its contours is thus called the "Parsons Code."
Kelkar's method is also similar to other tools that are available online.
"For example, today we have musipedia.org which allows you to search for music based on its contours," she says.
Listeners change the music
Sound, text and space are some of the modes included in the music. The same applies to actions, or simply thinking about them.
"If you visualize dancing, the music you hear will probably sound different to how it would if you didn't."
The music researcher also highlights how we experience concerts, something which has changed a lot over the years.
"We have musical genres where people sit still and listen respectfully, but that's something new, because many of our classical composers created music for dancing. We now play mazurkas and minuets in concert halls while the audience sits there watching respectfully. The same applies to jazz, which was a club music genre that was intended for people to dance to."
She points out that this says something fundamental about the multimodality of music.
"Your perception of music is shaped by what you see and do, but the opposite also applies; what you do will affect the music."People listen to streamed music during specific blocks of time
More information: Tejaswinee Kelkar, Computational Analysis of Melodic Contour and Body Movement. www.duo.uio.no/bitstream/handl … quence=4&isAllowed=y
Bruno Laeng et al, Substituting facial movements in singers changes the sounds of musical intervals, Scientific Reports (2021). DOI: 10.1038/s41598-021-01797-z
Journal information: Scientific Reports
Provided by University of Oslo
Perceptual links between sound and shape may unlock origins of spoken words
Most people around the world agree that the made-up word 'bouba' sounds round in shape, and the made-up word 'kiki' sounds pointy—a discovery that may help to explain how spoken languages develop, according to a new study.
Language scientists have discovered that this effect exists independently of the language that a person speaks or the writing system that they use, and it could be a clue to the origins of spoken words.
The research breakthrough came from exploring the 'bouba/kiki effect,' where the majority of people, mostly Westerners in previous studies, intuitively match the shape on the left to the neologism 'bouba' and the form on the right to 'kiki.'
An international research team has conducted the largest cross-cultural test of the effect, surveying 917 speakers of 25 different languages representing nine language families and ten writing systems—discovering that the effect occurs in societies around the world.
Publishing their findings today in Philosophical Transactions of the Royal Society B, the team, led by experts from the University of Birmingham and the Leibniz-Centre General Linguistics (ZAS), Berlin, says that such iconic vocalizations may form a global basis for the creation of new words.
Co-author Dr. Marcus Perlman, Lecturer in English Language and Linguistics at the University of Birmingham, commented: "Our findings suggest that most people around the world exhibit the bouba/kiki effect, including people who speak various languages, and regardless of the writing system they use."
"Our ancestors could have used links between speech sounds and visual properties to create some of the first spoken words—and today, many thousands of years later, the perceived roundness of the English word 'balloon' may not be just a coincidence, after all."
The 'bouba/kiki effect' is thought to derive from phonetic and articulatory features of the words, for example, the rounded lips of the 'b' and the stressed vowel in 'bouba,' and the intermittent stopping and starting of air in pronouncing 'kiki.'
To find out how widespread the bouba/kiki effect is across human populations, the researchers conducted an online test with participants who spoke a wide range of languages, including, for example, Hungarian, Japanese, Farsi, Georgian, and Zulu.
The results showed that the majority of participants, independent of their language and writing system, showed the effect, matching 'bouba' with the rounded shape and 'kiki' with the spiky one.
Co-author Dr. Bodo Winter, Senior Lecturer in Cognitive Linguistics at the University of Birmingham, commented: "New words that are perceived to resemble the object or concept they refer to are more likely to be understood and adopted by a wider community of speakers. Sound-symbolic mappings such as in bouba/kiki may play an important ongoing role in the development of spoken language vocabularies."
Iconicity—the resemblance between form and meaning—had been thought to be largely confined to onomatopoeic words such as 'bang' and 'peep,' which imitate the sounds they denote. However, the team's research suggests that iconicity can shape the vocabularies of spoken languages far beyond the example of onomatopoeias.
The researchers note that the potential for bouba/kiki to play a role in language evolution is confirmed by the evidence they collected. It shows that the effect stems from a deeply rooted human capacity to connect speech sound to visual properties, and is not just a quirk of speaking English.
Study finds hidden emotions in the sound of words
More information: Aleksandra Ćwiek et al, The bouba/kiki effect is robust across cultures and writing systems, Philosophical Transactions of the Royal Society B: Biological Sciences (2021). DOI: 10.1098/rstb.2020.0390
Journal information: Philosophical Transactions of the Royal Society B
Provided by University of Birmingham
No comments:
Post a Comment