Perception Of Emotion In Music Research Articles

Theperception of emotion in music by people with hearing loss and people withcochlear implants.

Music is an important part of life for many people. It can evoke a wide range of emotions, including sadness, happiness, anger, tension, relief and excitement. People with hearing loss and people with cochlear implants have reduced abilities to discriminate some of the features of musical sounds that may be involved in evoking emotions. This paper reviews these changes in perceptual abilities and describes how they affect the perception of emotion in music. For people with acquired partial hearing loss, it appears that the perception of emotion in music is almost normal, whereas congenital partial hearing loss is associated with impaired perception of music emotion. For people with cochlear implants, the ability to discriminate changes in fundamental frequency (associated with perceived pitch) is much worse than normal and musical harmony is hardly perceived. As a result, people with cochlear implants appear to judge emotion in music primarily using tempo and rhythm cues, and this limits the range of emotions that can be judged. This article is part of the theme issue 'Sensing and feeling: an integrative approach to sensory processing and emotional experience'.

Exploring the Role of Traditional Music Elements in Music Art Appreciation

Abstract Music appreciation class is to appreciate and evaluate musical works through listening, experiencing, exploring and evaluating, so as to cultivate students’ aesthetic perception ability and music cultural literacy. This paper examines the relationship between music emotion and music elements, selects traditional music elements that have a significant impact on people’s perception of music emotion for labeling, and creates a data set. Then, based on the emotional characteristics of music elements, the role mechanism of traditional music elements in music art appreciation was explored. Based on the coupling model, we also explored the law of time evolution of the coupling system of “traditional elements-music empathy level.” Traditional music elements have a significant positive correlation (P=0.0021) with students’ empathy in music appreciation, as demonstrated in the results. The results show that students’ music appreciation ability can be significantly enhanced through the synergy of excellent traditional Chinese music elements in music and art appreciation, which can help them to establish a correct worldview, outlook on life, and values.

Evaluation of musical emotion perception and language development in children with cochlear implants

ObjectivesWhile the primary purpose of cochlear implant (CI) fitting is to improve individuals' receptive and expressive skills, musical emotion perception (MEP) is generally ignored. This study assesses the MEP and language skills (LS) of children using CI. Methods26 CI users and 26 matched healthy controls between the ages of 6 and 9 were included in the study. The Test of Language Development (TOLD) was applied to evaluate the LS of the participants, and the Montreal Emotion Identification Test (MEI) was applied to evaluate the MEP. ResultsMEI test scores and all subtests of TOLD were statistically significantly lower in the CI group. Also, there was a statistically significant and moderate correlation between the listening subtest of TOLD and the MEI test. ConclusionsMEP and language skills are poor in children with CI. Although language skills are primarily targeted in CI performance, improving MEP should also be included in rehabilitation programs. The relationship between music and the TOLD's listening subtest may provide evidence that listening skills can be improved by paying attention to the MEP, which is frequently ignored in rehabilitation programs.

Automatic music emotion classification model for movie soundtrack subtitling based on neuroscientific premises

The ability of music to induce emotions has been arousing a lot of interest in recent years, especially due to the boom in music streaming platforms and the use of automatic music recommenders. Music Emotion Recognition approaches are based on combining multiple audio features extracted from digital audio samples and different machine learning techniques. In these approaches, neuroscience results on musical emotion perception are not considered. The main goal of this research is to facilitate the automatic subtitling of music. The authors approached the problem of automatic musical emotion detection in movie soundtracks considering these characteristics and using scientific musical databases, which have become a reference in neuroscience research. In the experiments, the Constant-Q-Transform spectrograms, the ones that best represent the relationships between musical tones from the point of view of human perception, are combined with Convolutional Neural Networks. Results show an efficient emotion classification model for 2-second musical audio fragments representative of intense basic feelings of happiness, sadness, and fear. Those emotions are the most interesting to be identified in the case of movie music captioning. The quality metrics have demonstrated that the results of the different models differ significantly and show no homogeneity. Finally, these results pave the way for an accessible and automatic captioning of music, which could automatically identify the emotional intent of the different segments of the movie soundtrack.

Development of Novel Musical Stimuli to Investigate the Perception of Musical Emotions in Individuals With Hearing Loss.

Many studies have examined the perception of musical emotion using excerpts from familiar music that includes highly expressed emotions to classify emotional choices. However, using familiar music to study musical emotions in people with acquired hearing loss could produce ambiguous results as to whether the emotional perception is due to previous experiences or listening to the current musical stimuli. To overcome this limitation, we developed new musical stimuli to study emotional perception without the effects of episodic memory. A musician was instructed to compose five melodies with evenly distributed pitches around 1 kHz. The melodies were created to express the emotions of happy, sad, angry, tender, and neutral. To evaluate whether these melodies expressed the intended emotions, two methods were applied. First, we classified the expressed emotions of melodies with selected musical features from 60 features using genetic algorithm-based k-nearest neighbors. Second, forty-four people with normal hearing participated in an online survey regarding the emotional perception of music based on dimensional and discrete approaches to evaluate the musical stimuli set. Twenty-four selected musical features produced classification for intended emotions with an accuracy of 76%. The results of the online survey in the normal hearing (NH) group showed that the intended emotions were selected significantly more often than the others. K-means clustering analysis revealed that melodies with arousal and valence ratings corresponded to representative quadrants of interest. Additionally, the applicability of the stimuli was tested in 4 individuals with high-frequency hearing loss. By applying the individuals with NH, the musical stimuli were shown to classify emotions with high accuracy, as expressed. These results confirm that the set of musical stimuli can be used to study the perceived emotion in music, demonstrating the validity of the musical stimuli, independent of innate musical bias such as due to episodic memory. Furthermore, musical stimuli could be helpful for further studying perceived musical emotion in people with hearing loss because of the controlled pitch for each emotion.

Music and neuro-cognitive deficits in depression.

BackgroundCognitive deficits are one of the core features of major depressive disorder (MDD) that play crucial role in functional recovery. Studies have explored cognitive deficits in MDD, however, given inconsistent results, especially in mild-moderate MDD. Recently, studies have explored music as cognitive ability in various clinical conditions. In MDD, large focus has been on evaluating emotion deficits and just a handful on music cognition. With growing evidence on use of music based intervention to target cognitive deficits, it is imperative to explore nature of music cognitive ability in MDD.AimTo examine musical and neuro-cognitive deficits in patients with mild-moderate MDD.MethodsPatients diagnosed with mild or moderate MDD (n = 19) and matched healthy controls (HC) (n = 18) were evaluated on selected tests from NIMHANS Neuropsychological test battery and Montreal battery for evaluation of amusia (MBEA).ResultsMDD group performed significantly lower than HC on working memory (p = 0.007), verbal learning (p = 0.02) and retention (p = 0.03). Three indices were computed for a comprehensive evaluation. Groups did not differ significantly in any of the indices- focused attention, executive function, learning and memory as well as on music cognition. Focused attention and memory index predicted music cognition in HC and the combined group (MDD + HC) (p < 0.01). Attention alone contributed to 62.1% of variance in music cognition. Similarly, music cognition significantly predicted focused attention (p < 0.01).ConclusionIndividuals with mild-moderate MDD show significant deficits in working memory, verbal learning and memory, however, not in music cognition. There exists a significant relationship between music cognition and attention, which could be implicated in use of music interventions to ameliorate cognitive deficits. Limitations of study include small sample size and heterogeneity. Future studies on larger cohort examining musical emotion perception and neurocognition is imperative to have deeper understanding of this debilitating condition.

Multisensory integration of musical emotion perception in singing

We investigated how visual and auditory information contributes to emotion communication during singing. Classically trained singers applied two different facial expressions (expressive/suppressed) to pieces from their song and opera repertoire. Recordings of the singers were evaluated by laypersons or experts, presented to them in three different modes: auditory, visual, and audio–visual. A manipulation check confirmed that the singers succeeded in manipulating the face while keeping the sound highly expressive. Analyses focused on whether the visual difference or the auditory concordance between the two versions determined perception of the audio–visual stimuli. When evaluating expressive intensity or emotional content a clear effect of visual dominance showed. Experts made more use of the visual cues than laypersons. Consistency measures between uni-modal and multimodal presentations did not explain the visual dominance. The evaluation of seriousness was applied as a control. The uni-modal stimuli were rated as expected, but multisensory evaluations converged without visual dominance. Our study demonstrates that long-term knowledge and task context affect multisensory integration. Even though singers’ orofacial movements are dominated by sound production, their facial expressions can communicate emotions composed into the music, and observes do not rely on audio information instead. Studies such as ours are important to understand multisensory integration in applied settings.

Physiological State Can Help Predict the Perceived Emotion of Music: Evidence from ECG and EDA Signals

As the soul of music, emotion information is widely used in music retrieval and recommendation systems because the pursuit of emotional experience is the main motivation for music listening. In the field of music emotion recognition, computer scientists investigated computation models to automatically detect the perceived emotion of music, but this method ignores the differences between listeners. To provide users with the most accurate music emotion information, this study investigated the effects of physiological features on personalized music emotion recognition (PMER) models, which can automatically identify an individual’s perceived emotion of music. Applying machine learning methods, we formed relations among audio features, physiological features, and music emotions. First, computational modeling analysis shows that physiological features extracted from electrocardiogram and electro-dermal activity signals can predict the perception of music emotion for some individuals. Second, we compared the performance of physiological feature-based perception and feeling models and observed substantial individual differences. In addition, we found that the performance of the perception model and the feeling model is related in predicting happy, relaxed, and sad emotions. Finally, by adding physiological features to the audio-based PMER model, the prediction effect of some individuals was improved. Our work investigated the relationship between physiological state and perceived emotion of music, constructed models with practical value, and provided a reference for the optimization of PMER systems.

Recognition of musical emotions and their perceived intensity after unilateral brain damage

For the hemispheric laterality of emotion processing in the brain, two competing hypotheses are currently still debated. The first hypothesis suggests a greater involvement of the right hemisphere in emotion perception whereas the second hypothesis suggests different involvements of each hemisphere as a function of the valence of the emotion. These hypotheses are based on findings for facial and prosodic emotion perception. Investigating emotion perception for other stimuli, such as music, should provide further insight and potentially help to disentangle between these two hypotheses. The present study investigated musical emotion perception in patients with unilateral right brain damage (RBD, n = 16) or left brain damage (LBD, n = 16), as well as in matched healthy comparison participants (n = 28). The experimental task required explicit recognition of musical emotions as well as ratings on the perceived intensity of the emotion. Compared to matched comparison participants, musical emotion recognition was impaired only in LBD participants, suggesting a potential specificity of the left hemisphere for explicit emotion recognition in musical material. In contrast, intensity ratings of musical emotions revealed that RBD patients underestimated the intensity of negative emotions compared to positive emotions, while LBD patients and comparisons did not show this pattern. To control for a potential generalized emotion deficit for other types of stimuli, we also tested facial emotion recognition in the same patients and their matched healthy comparisons. This revealed that emotion recognition after brain damage might depend on the stimulus category or modality used. These results are in line with the hypothesis of a deficit of emotion perception depending on lesion laterality and valence in brain-damaged participants. The present findings provide critical information to disentangle the currently debated competing hypotheses and thus allow for a better characterization of the involvement of each hemisphere for explicit emotion recognition and their perceived intensity.

Emotion Classification System for Digital Music with a Cascaded Technique

Music selection is difficult without efficient organization based on metadata or tags, and one effective tag scheme is based on the emotion expressed by the music. However, manual annotation is labor intensive and unstable because the perception of music emotion varies from person to person. This paper presents an emotion classification system for digital music with a resolution of eight emotional classes. Russell’s emotion model was adopted as common ground for emotional annotation. The music information retrieval (MIR) toolbox was employed to extract acoustic features from audio files. The classification system utilized a supervised machine learning technique to recognize acoustic features and create predictive models. Four predictive models were proposed and compared. The models were composed by crossmatching two types of neural networks, i.e., Levenberg-Marquardt (LM) and resilient backpropagation (Rprop), with two types of structures: a traditional multiclass model and the cascaded structure of a binary-class model. The performance of each model was evaluated via the MediaEval Database for Emotional Analysis (DEAM) benchmark. The best result was achieved by the model trained with the cascaded Rprop neural network (accuracy of 89.5%). In addition, correlation coefficient analysis showed that timbre features were the most impactful for prediction. Our work offers an opportunity for a competitive advantage in music classification because only a few music providers currently tag music with emotional terms.

Musical Emotion Perception in Bimodal Patients: Relative Weighting of Musical Mode and Tempo Cues.

Several cues are used to convey musical emotion, the two primary being musical mode and musical tempo. Specifically, major and minor modes tend to be associated with positive and negative valence, respectively, and songs at fast tempi have been associated with more positive valence compared to songs at slow tempi (Balkwill and Thompson, 1999; Webster and Weir, 2005). In Experiment I, we examined the relative weighting of musical tempo and musical mode among adult cochlear implant (CI) users combining electric and contralateral acoustic stimulation, or “bimodal” hearing. Our primary hypothesis was that bimodal listeners would utilize both tempo and mode cues in their musical emotion judgments in a manner similar to normal-hearing listeners. Our secondary hypothesis was that low-frequency (LF) spectral resolution in the non-implanted ear, as quantified via psychophysical tuning curves (PTCs) at 262 and 440 Hz, would be significantly correlated with degree of bimodal benefit for musical emotion perception. In Experiment II, we investigated across-channel spectral resolution using a spectral modulation detection (SMD) task and neural representation of temporal fine structure via the frequency following response (FFR) for a 170-ms /da/ stimulus. Results indicate that CI-alone performance was driven almost exclusively by tempo cues, whereas bimodal listening demonstrated use of both tempo and mode. Additionally, bimodal benefit for musical emotion perception may be correlated with spectral resolution in the non-implanted ear via SMD, as well as neural representation of F0 amplitude via FFR – though further study with a larger sample size is warranted. Thus, contralateral acoustic hearing can offer significant benefit for musical emotion perception, and the degree of benefit may be dependent upon spectral resolution of the non-implanted ear.

Uncertain Emotion Discrimination Differences Between Musicians and Non-musicians Is Determined by Fine Structure Association: Hilbert Transform Psychophysics.

Humans perceive musical sound as a complex phenomenon, which is known to induce an emotional response. The cues used to perceive emotion in music have not been unequivocally elucidated. Here, we sought to identify the attributes of sound that confer an emotion to music and determine if professional musicians have different musical emotion perception than non-musicians. The objective was to determine which sound cues are used to resolve emotional signals. Happy or sad classical music excerpts modified in fine structure or envelope conveying different degrees of emotional certainty were presented. Certainty was determined by identification of the emotional characteristic presented during a forced-choice discrimination task. Participants were categorized as good or poor performers (n = 32, age 21.16 ± 2.59 SD) and in a separate group as musicians in the first or last year of music education at a conservatory (n = 32, age 21.97 ± 2.42). We found that temporal fine structure information is essential for correct emotional identification. Non-musicians used less fine structure information to discriminate emotion in music compared with musicians. The present psychophysical experiments revealed what cues are used to resolve emotional signals and how they differ between non-musicians and musically educated individuals.

Musical emotion perception in bimodal patients: Relationship between bimodal benefit and neural representation of temporal fine structure using Rhodes piano stimuli

Combining electric and acoustic hearing across ears allows significant “bimodal hearing” benefit for speech recognition, sound quality, and music perception. The degree of bimodal benefit for speech recognition and musical emotion perception is significantly correlated with neural representation of F0 envelope using the frequency following response (FFR) for a 170-ms /da/ stimulus (D’Onofrio et al., in prep). The purpose of the current study is to examine the relationship between bimodal benefit for musical emotion perception and neural representation of F0 using Rhodes piano stimuli at the following fundamental frequencies: 98 Hz (G2), 262 Hz (C4), and 440 Hz (A4). Our hypotheses are (1) the correlation between bimodal benefit and neural representation of F0 and temporal fine structure will be strengthened via use of a “music” stimulus, compared to the /da/ “speech” stimulus, and (2) bimodal benefit for speech recognition will be better explained via FFR for speech stimuli. Stimuli were presented at 90 dB SPL to the non-implanted ear of bimodal listeners using magnetically shielded insert earphones. Implications regarding the clinical utility of FFR will be discussed, with particular attention given to its use as an objective measure of expected bimodal benefit for speech recognition and musical emotion perception.

Challenges and Opportunities of Predicting Musical Emotions with Perceptual and Automatized Features

Music information retrieval (MIR) is a fast-growing research area. One of its aims is to extract musical characteristics from audio. In this study, we assumed the roles of researchers without further technical MIR experience and set out to test in an exploratory way its opportunities and challenges in the specific context of musical emotion perception. Twenty sound engineers rated 60 musical excerpts from a broad range of styles with respect to 22 spectral, musical, and cross-modal features (perceptual features) and perceived emotional expression. In addition, we extracted 86 features (acoustic features) of the excerpts with the MIRtoolbox (Lartillot &amp; Toiviainen, 2007). First, we evaluated the perceptual and extracted acoustic features. Both perceptual and acoustic features posed statistical challenges (e.g., perceptual features were often bimodally distributed, and acoustic features highly correlated). Second, we tested the suitability of the acoustic features for modeling perceived emotional content. Four nearly disjunctive feature sets provided similar results, implying a certain arbitrariness of feature selection. We compared the predictive power of perceptual and acoustic features using linear mixed effects models, but the results were inconclusive. We discuss critical points and make suggestions to further evaluate MIR tools for modeling music perception and processing.

Musical emotion recognition in bimodal patients

Several cues are used to convey musical emotion, the two primary being musical mode and musical tempo. Specifically, major and minor modes are regarded as having positive and negative valence, respectively, and songs at fast tempi (i.e., quarter note = 92-196, Gosselin et al. (2005); quarter note = 80-255, Peretz et al. (1998)) are associated with more positive valence compared to songs at slow tempi (i.e., quarter note = 40-60, Gosselin et al. (2005); quarter note = 20-100, Peretz et al. (1998)). Caldwell et al. (2015) demonstrated that CI users relied on tempo (fast vs. slow) with little regard for mode (major vs. minor) when interpreting musical emotion, a finding consistent with spectral cues being poorly represented in CI users. The current study is an extension of Caldwell et al. (2015) examining how mode and tempo cues impact musical emotion recognition in bimodal listeners. Our primary hypothesis is that, unlike CI only users, bimodal listeners—with access to F0 and fine structure—will utilize both mode and tempo cues in a manner similar to normal-hearing listeners. Our secondary hypothesis is that low-frequency spectral resolution in the contralateral ear (via tuning curves and spectral modulation detection) will be significantly correlated with degree of bimodal benefit for musical emotion perception.

A comparative study of the perception of music emotion between adults with and without visual impairment

In music listening, limitations on visual experience affect a listener’s abstract information processing and conceptualization of the music. The aim of this study is to examine the differences in emotional responses to music between adults with visual impairment (VI) and adults with normal vision (NV). By using specific, emotion-inducing music reflecting happiness, sadness, anger, and fear, this study considers factors such as music emotion identification, emotional valence, arousal, intensity, and musical preference. A total of 120 participants (60 VI and 60 NV) listened to sixteen 15-second music excerpts and reported which emotion and to what extent they perceived it, using a self-reported music emotion evaluation scale. The results indicated that both of the groups showed high congruence in music emotion identification. However, the VI group showed significantly higher arousal, intensity and preference for sadness, while showing the lowest score for the intensity of fear. The main factor affecting emotion identification was preference for the VI group, and valence for the NV group.

Component Tying for Mixture Model Adaptation in Personalization of Music Emotion Recognition

Personalizing a music emotion recognition model is needed because the perception of music emotion is highly subjective, but it is a time-consuming process. In this paper, we consider how to expedite the personalization process that begins with a general model trained offline using a general user base and progressively adapts the model to a music listener using the emotion annotations of the listener. Specifically, we focus on reducing the number of user annotations needed for the personalization. We investigate and evaluate four component tying methods: single group tying, quadrantwise tying, hierarchical tying, and random tying. These methods aim to exploit the available annotations by identifying related model parameters on-the-fly and updating them jointly. In the evaluation, we use the AMG1608 dataset, which contains the clip-level valence-arousal emotion ratings of 1608 30-s music clips annotated by 665 listeners. Also, we use the acoustic emotion Gaussians model as the general model that uses a mixture of Gaussian components to learn the mapping between the acoustic feature space and the emotion space. The results show that the model adaptation with component tying requires only 10-20 personal annotations to obtain the same level of prediction accuracy as the baseline model adaptation method that uses 50 personal annotations without component tying.

The Effects of Bayan Tabla Timbre on Stimulus Discrimination and Perception of Music Emotion in a Dangdut Music Fans, Neutral, and Haters Group

Background: A theoretical issue about the control factors of the subject response on musical timbre between acoustic factors (including attack-decay) and type of music (genre) still unclear. Objective: Examine the effects of bayan tabla timbre and its attack-decay changes on stimulus discrimination and perception of music emotion in a dangdut music fans group, neutral group, and haters group. Method: A randomized experiment between-within subjects design was used in the research method. Research conducted in an Experimental Psychology laboratory at Gadjah Mada University of Indonesia. The research was supported by the participation of undergraduate and graduate students from Social-Humanities and Veterinary Medicine Sciences at Gadjah Mada University. All subjects aged 17-32 years. A dangdut music fans group/KE (N=18), neutral group/KK1 (N=18), and haters group/ KK2 (N=18). Each subject in KE, KK1, KK2 receive a random series treatment of three music: M1 (a music which has natural bayan tabla timbre [in G#1 pitch and its tone's attack was 0.403 seconds per stroke]), M2 (a music without bayan tabla timbre as the control/placebo), and M3 (a music which has synthetic bayan tabla timbre [in G#1 pitch and its tone's attack was 0.085 seconds per stroke]). Main outcome measures. The subject response was measured by StimoV1.0 software which consists of stimulus discrimination scale (similarity scaling), emotional perception scale (semantic differential scale): strong-not strong, happy-not happy, fun-not fun, erotic-not erotic, and the reaction time measurement in seconds. Results and Conclusion: In general, the results shown that the effects of musical timbre on stimulus discrimination is more controlled by acoustic factors (including attack-decay) and is not controlled by preference factors to type of music (genre). But however, the effects of musical timbre on perception of music emotion can be controlled by preference factors to type of music (genre).

The effect of adaptive music playing system on emotion regulation

We live in an industrially polluted environment, surrounded with stress and negative emotions. Since music has the marvelous power of arousing different moods and regulating emotions depending on the features of musical elements, people could be prescribed specific kinds of music to help alleviate their symptoms. Therefore, refer to entrainment, iso, and diversion principles of music therapy stated by Campbell, the investigator developed an Adaptive Music Playing System base on the two-dimensional arousal-valence emotion model. The purposes of this study were: (1) to explore the two-dimensional emotional responses of subjects when listening to specific music; (2) to survey the effects of gender, music preference on their perception of musical emotions; (3) to compare the relativities of various musical elements with musical emotions; (4) to detect the healing effect of adaptive music playing system. 56 college students were recruited as subjects in this study. Subjects imported their instant mood value, and the Adaptive Music Playing System would automatically play three music pieces. The results of this study showed that Adaptive music playing system could successfully lead the subjects in a better physical and mental condition. Gender effect wasn’t significant for subjects’ perception. Music preference affected subjects’ perception.

Polyphonic Voice Multiplicity, Numerosity, and Musical Emotion Perception

Three experimental studies suggest that music with more musical voices (higher voice multiplicity) tends to be perceived more positively. In the first experiment, participants heard brief extracts from polyphonic keyboard works representing conditions of one, two, three, or four concurrent musical voices. Two basic emotions (happiness and sadness) and two social emotions (pride and loneliness) were rated on a continuous scale. Listeners rated excerpts with higher voice multiplicity as sounding more happy, less sad, less lonely, and more proud. Results from a second experiment indicate that this effect might extend to positive and negative emotions more generally. In a third experiment, participants were asked to count (denumerate) the number of musical voices in the same stimuli. Denumeration responses corresponded closely with ratings for both positive and negative emotions, suggesting that a single musical feature or percept might play a role in both. Possible roles for both symbolic and psychoacoustic musical features are discussed.