Two-dimensional Emotion Space Research Articles

Effect of sound sequence on soundscape emotions

This study analysed the effect of sound sequence on soundscape emotions with respect to three aspects of sound sources: the number of sound source/s, changing trends in the number of sound source/s (increment/decrement) and category of sound source/s. A laboratory listening test was conducted with 31 participants and data on emotions evoked by the sound source per second, with its different characteristics, were obtained. A linear regression model was established between the three aspects of sound sources mentioned above and emotions. The results reveal the following: first, the number of sound source/s is negatively and positively correlated to the pleasing and arousing dimensions of emotion, respectively; second, changing trends in the number of sound source/s (increment/decrement) has a significant effect on variations in emotions (p < 0.05), but not on emotions itself (p > 0.05); and third, although the category of the sound source/s has a decisive effect on the coordinate range of emotions in a two-dimensional emotion space, the number and changing trends of sound source/s have a limited effect on it. Finally, the linear regression model, composed of the three aspects of sound sources, could explain the values of 33.2% and 28.8% for the pleasantness and arousal dimensions of emotion, respectively.

Recognition of Emotion According to the Physical Elements of the Video.

The increasing interest in the effects of emotion on cognitive, social, and neural processes creates a constant need for efficient and reliable techniques for emotion elicitation. Emotions are important in many areas, especially in advertising design and video production. The impact of emotions on the audience plays an important role. This paper analyzes the physical elements in a two-dimensional emotion map by extracting the physical elements of a video (color, light intensity, sound, etc.). We used k-nearest neighbors (K-NN), support vector machine (SVM), and multilayer perceptron (MLP) classifiers in the machine learning method to accurately predict the four dimensions that express emotions, as well as summarize the relationship between the two-dimensional emotion space and physical elements when designing and producing video.

An accurate emotion recognition system using ECG and GSR signals and matching pursuit method

BackgroundThe purpose of the current study was to examine the effectiveness of Matching Pursuit (MP) algorithm in emotion recognition. MethodsElectrocardiogram (ECG) and galvanic skin responses (GSR) of 11 healthy students were collected while subjects were listening to emotional music clips. Applying three dictionaries, including two wavelet packet dictionaries (Coiflet, and Daubechies) and discrete cosine transform, MP coefficients were extracted from ECG and GSR signals. Next, some statistical indices were calculated from the MP coefficients. Then, three dimensionality reduction methods, including Principal Component Analysis (PCA), Linear Discriminant Analysis, and Kernel PCA were applied. The dimensionality reduced features were fed into the Probabilistic Neural Network in subject-dependent and subject-independent modes. Emotion classes were described by a two-dimensional emotion space, including four quadrants of valence and arousal plane, valence based, and arousal based emotional states. ResultsUsing PCA, the highest recognition rate of 100% was achieved for sigma = 0.01 in all classification schemes. In addition, the classification performance of ECG features was evidently better than that of GSR features. Similar results were obtained for subject-dependent emotion classification mode. ConclusionsAn accurate emotion recognition system was proposed using MP algorithm and wavelet dictionaries.

Affective Calibration of Musical Feature Sets in an Emotionally Intelligent Music Composition System

Affectively driven algorithmic composition (AAC) is a rapidly growing field that exploits computer-aided composition in order to generate new music with particular emotional qualities or affective intentions. An AAC system was devised in order to generate a stimulus set covering nine discrete sectors of a two-dimensional emotion space by means of a 16-channel feed-forward artificial neural network. This system was used to generate a stimulus set of short pieces of music, which were rendered using a sampled piano timbre and evaluated by a group of experienced listeners who ascribed a two-dimensional valence-arousal coordinate to each stimulus. The underlying musical feature set, initially drawn from the literature, was subsequently adjusted by amplifying or attenuating the quantity of each feature in order to maximize the spread of stimuli in the valence-arousal space before a second listener evaluation was conducted. This process was repeated a third time in order to maximize the spread of valence-arousal coordinates ascribed to the generated stimulus set in comparison to a spread taken from an existing prerated database of stimuli, demonstrating that this prototype AAC system is capable of creating short sequences of music with a slight improvement on the range of emotion found in a stimulus set comprised of real-world, traditionally composed musical excerpts.

Exploring the cross-modal perception of face and voice through the lens of two-dimensional emotion space

Exploring the cross-modal perception of face and voice through the lens of two-dimensional emotion space

The impact of song-specific age and affective qualities of popular songs on music-evoked autobiographical memories (MEAMs)

Songs heard between the ages of 15 and 24 should be remembered better and have a stronger relationship to autobiographical memories when compared with music from other phases of life (“reminiscence bump effect”). Additionally, the proportion of music-evoked autobiographical memories (MEAMs) is at a maximum in these years of early adolescence and then declines up to the age of 60. In our study we tried both to replicate these important findings based on a German sample and to further investigate the influence of the affective characteristics of the songs on the frequency of participants’ autobiographical memories. In Experiment 1 a group of adults ( N = 48, Mage = 67.1 years) listened to excerpts from 80, number-one, popular music hits from 1930 to 2010 and gave written self-reports on MEAMs. In Experiment 2 the affective characteristics were rated by another group of adults ( N = 22, Mage = 66 years) and were used to predict the frequency of MEAMs. As a main result of Experiment 1, we confirmed the reminiscence bump and decline effect with a small effect size for the ratings of feelings evoked by the song and with a medium effect size for the song recognition performance of those songs released during the participants’ age range of 15 to 24 years. The total number of MEAMs was only marginally influenced by a memory bump and decline effect, and participants showed a significant proportion of MEAMs up to the fifth decade. Experiment 2 revealed that the affective ratings of the songs were unequally distributed over the two-dimensional emotion space unlike the average rate of MEAMs which was nearly equally distributed. In contrast to previous research, we therefore conclude that popular songs can be associated with autobiographical memory over five decades of life – independent of the affective character of the music.

Reliability issues regarding the beginning, middle and end of continuous emotion ratings to music

Continuous self-reported emotion expressed by four pieces of music were collected on a two-dimensional (valence and arousal) emotion space in a repeated measures (test-retest conditions) design. Initial orientation time (IOT), test-retest reliability and afterglow were examined. Median IOT was 8 seconds. Valence ratings took up to 25 (median 4), and for arousal up to 35 (median 12) seconds. Slower tempi seemed to require longer IOT. Test-retest reliability examined correlation coefficients, and compared periods of sample-by-sample good agreement in response between Test and Retest condition. About 80% of responses were reliable in both the Test and Retest conditions regardless of response dimension. Pearson correlations demonstrated better test-retest reliability for arousal responses than for valence. Retest condition ratings were within 8% of Test condition rating within participant. Average standard deviations for ratings collapsed across dimension, stimulus and conditions was 12.2% of the ratings scale range. Afterglow effects – large outliers in spread of scores just after the end of a piece – were identified. The reliability of continuous emotional response is therefore considered to be quite good, but caution must be taken as to how to deal with the opening and ending of continuous emotional response data.

Emotion Identification and Modelling on the Basis of Paired Physiological Data Features for Companion Systems

AbstractA technical companion system should be able to detect its users emotion and model the user's emotional state in order to react to it accordingly. We have developed a novel method to determine a user's most significant emotional change in the two emotion dimensions of pleasure and arousal on the basis of paired data features of physiological data when comparing two events. An experiment was set up where participants first viewed blocked IAPS picture presentations and then took part in a mental training wizard-of-oz scenario. Six meaningful features from four physiological channels of the IAPS picture presentation data - containing two electromyography channels (corrugator supercilii and zygomaticus major), skin conductance and peripheral blood volume - were extracted. Three pairs of features were found to contain valuable information about emotional changes when comparing two situations with different emotional contents. The method was then tested on a new blocked IAPS dataset and on the wizard-of-oz interaction scenario dataset to verify its performance. In 75% of the subjects, the detected emotion matched one of the two induced emotions. This information could be used by future companion technologies for modelling of the user's current emotional state in the two-dimensional emotion space of pleasure and arousal.

Music's Impact on the Visual Perception of Emotional Dyadic Interactions

Research showing that emotions can be recognized in point-light displays of human dyadic interactions was extended in the current study by investigating the impact of music on the perception of normal and exaggerated expressions of happiness, contentedness, sadness and anger in such visual stimuli. Sixteen musically untrained participants viewed short video clips of these emotion portrayals, each presented with emotionally compatible (e.g., happy music accompanies a happy interaction) and emotionally incompatible piano music (e.g., sad music accompanies a happy interaction). It was hypothesized that music will increase the accuracy of emotion judgements in displays where auditory and visual information is compatible relative to displays with incompatible audio-visual information. A two-dimensional emotion space was used to record participants' judgements of emotion in only the visual stimuli. Results indicated that music affected the accuracy of emotion judgements. Happiness and sadness were perceived more accurately in compatible than in incompatible conditions, while the opposite was the case for contentedness. Anger was perceived accurately in all conditions. Exaggerated expressions of sadness, which were evaluated more accurately than normal expressions of sadness, were also found to be resistant to the music. These findings can be interpreted in the light of previous research demonstrating that music's cross-modal impact depends on the degree of emotional ambiguity in the visual display. More generally, the results demonstrate that the perception of emotions in biological motion can be affected by music.

Music’s impact on the visual perception of emotional dyadic interactions

Research showing that emotions can be recognized in point-light displays of human dyadic interactions was extended in the current study by investigating the impact of music on the perception of normal and exaggerated expressions of happiness, contentedness, sadness and anger in such visual stimuli. Sixteen musically untrained participants viewed short video clips of these emotion portrayals, each presented with emotionally compatible (e.g., happy music accompanies a happy interaction) and emotionally incompatible piano music (e.g., sad music accompanies a happy interaction). It was hypothesized that music will increase the accuracy of emotion judgements in displays where auditory and visual information is compatible relative to displays with incompatible audio-visual information. A two-dimensional emotion space was used to record participants’ judgements of emotion in only the visual stimuli. Results indicated that music affected the accuracy of emotion judgements. Happiness and sadness were perceived more accurately in compatible than in incompatible conditions, while the opposite was the case for contentedness. Anger was perceived accurately in all conditions. Exaggerated expressions of sadness, which were evaluated more accurately than normal expressions of sadness, were also found to be resistant to the music. These findings can be interpreted in the light of previous research demonstrating that music’s cross-modal impact depends on the degree of emotional ambiguity in the visual display. More generally, the results demonstrate that the perception of emotions in biological motion can be affected by music.

Music-elicited EEG Activity and Emotional Responses are Altered in Schizophrenia

Studies of patients with schizophrenia using facial affect recognition and voice discrimination tasks have identified emotional dysfunction as a prominent clinical feature. In the present study we examine whether emotion processing in patients is also impaired in a less explicitly social context -- continuous self-report of emotions during music using a two-dimensional (pleasantness X activation) emotion space. Electroencephalographic (EEG) activity was also recorded during this task since previous studies using EEG measures have found underlying cortical processes related to emotion. Twelve patients with schizophrenia and eleven controls listened to five 25-second songs. These songs included J.S. Bach‟s Invention #13 in A minor (BWV 784) (designated the original piece), and four computer-generated pieces of which two were designed to be similar to the original, and two were designed to be different. While no significant effects were found in the activation dimension of the self-report measures, the pattern of pleasantness ratings was significantly less differentiated among songs in the patient group than in controls. EEG asymmetry indices at frontal and central electrodes provided evidence of greater hemispheric activation asymmetry (with higher activation on the left) in controls than in patients, a difference that was significant at the central electrodes (C3 and C4). These findings indicate that individuals with schizophrenia interpret emotion-eliciting music differently than do controls, even in a relatively non-social setting, possibly because of less differentiated hemispheric representations.

Quantitative and visual analysis of the impact of music on perceived emotion of film

This article presents quantitative and visual methods for the analysis of the effect of music on emotion perceived in film. We discover strong, visible, and quantifiable trends in the effect of music on perceived emotion of film. We perform studies using both selected classical and composed music segments annotated with diverse emotions, paired with ambiguous film clips. We collect and analyze viewers' ratings of stress, activity, and dominance in the silent film, and film with various music soundtracks. The results are mapped onto a three-dimensional emotion space for visual and quantitative analysis. Aggregate scatter plots and center of mass results are presented using this emotion space. We find that the center of mass of the perceived emotion of film and music combined is consistently situated on a path between the center of mass of the musicalone response and that of the film-alone response. Regression analysis based on this trajectory observation results in coefficients with high R 2 values (R 2 = 0.675 for stress, R 2 = 0.817 for activity, and R 2 = 0.813 for dominance) for the first study with classical music selections, and lower R 2 values (R 2 = 0.199 for stress, R 2 = 0.405 for activity, and R 2 = 0.660 for dominance) for the second study with composed music. We conclude that continuous treatment of the two-dimensional emotion space provides a metric for comparing and assessing emotion ratings, and that spatial interpolation in this space provides a viable method for predicting the effect of music on perceived emotion of film.

Quantitative and visual analysis of the impact of music on perceived emotion of film

This article presents quantitative and visual methods for the analysis of the effect of music on emotion perceived in film. We discover strong, visible, and quantifiable trends in the effect of music on perceived emotion of film. We perform studies using both selected classical and composed music segments annotated with diverse emotions, paired with ambiguous film clips. We collect and analyze viewers' ratings of stress, activity, and dominance in the silent film, and film with various music soundtracks. The results are mapped onto a three-dimensional emotion space for visual and quantitative analysis. Aggregate scatter plots and center of mass results are presented using this emotion space. We find that the center of mass of the perceived emotion of film and music combined is consistently situated on a path between the center of mass of the musicalone response and that of the film-alone response. Regression analysis based on this trajectory observation results in coefficients with high R2 values (R2 = 0.675 for stress, R2 = 0.817 for activity, and R2 = 0.813 for dominance) for the first study with classical music selections, and lower R2 values (R2 = 0.199 for stress, R2 = 0.405 for activity, and R2 = 0.660 for dominance) for the second study with composed music. We conclude that continuous treatment of the two-dimensional emotion space provides a metric for comparing and assessing emotion ratings, and that spatial interpolation in this space provides a viable method for predicting the effect of music on perceived emotion of film.

Emotions over time: Synchronicity and development of subjective, physiological, and facial affective reactions to music.

Most people are able to identify basic emotions expressed in music and experience affective reactions to music. But does music generally induce emotion? Does it elicit subjective feelings, physiological arousal, and motor reactions reliably in different individuals? In this interdisciplinary study, measurement of skin conductance, facial muscle activity, and self-monitoring were synchronized with musical stimuli. A group of 38 participants listened to classical, rock, and pop music and reported their feelings in a two-dimensional emotion space during listening. The first entrance of a solo voice or choir and the beginning of new sections were found to elicit interindividual changes in subjective feelings and physiological arousal. Quincy Jones' "Bossa Nova" motivated movement and laughing in more than half of the participants. Bodily reactions such as "goose bumps" and "shivers" could be stimulated by the "Tuba Mirum" from Mozart's Requiem in 7 of 38 participants. In addition, the authors repeated the experiment seven times with one participant to examine intraindividual stability of effects. This exploratory combination of approaches throws a new light on the astonishing complexity of affective music listening.

Affective video content representation and modeling

This paper looks into a new direction in video content analysis - the representation and modeling of affective video content . The affective content of a given video clip can be defined as the intensity and type of feeling or emotion (both are referred to as affect) that are expected to arise in the user while watching that clip. The availability of methodologies for automatically extracting this type of video content will extend the current scope of possibilities for video indexing and retrieval. For instance, we will be able to search for the funniest or the most thrilling parts of a movie, or the most exciting events of a sport program. Furthermore, as the user may want to select a movie not only based on its genre, cast, director and story content, but also on its prevailing mood, the affective content analysis is also likely to contribute to enhancing the quality of personalizing the video delivery to the user. We propose in this paper a computational framework for affective video content representation and modeling. This framework is based on the dimensional approach to affect that is known from the field of psychophysiology. According to this approach, the affective video content can be represented as a set of points in the two-dimensional (2-D) emotion space that is characterized by the dimensions of arousal (intensity of affect) and valence (type of affect). We map the affective video content onto the 2-D emotion space by using the models that link the arousal and valence dimensions to low-level features extracted from video data. This results in the arousal and valence time curves that, either considered separately or combined into the so-called affect curve, are introduced as reliable representations of expected transitions from one feeling to another along a video, as perceived by a viewer.

Modeling Perceived Emotion With Continuous Musical Features

The relationship between musical features and perceived emotion was investigated by using continuous response methodology and time-series analysis. Sixty-seven participants responded to four pieces of Romantic music expressing different emotions. Responses were sampled once per second on a two-dimensional emotion space (happy-sad valence and aroused-sleepy). Musical feature variables of loudness, tempo, melodic contour, texture, and spectral centroid (related to perceived timbral sharpness) were coded. Musical feature variables were differenced and used as predictors in two univariate linear regression models of valence and arousal for each of the four pieces. Further adjustments were made to the models to correct for serial correlation. The models explained from 33% to 73% of variation in univariate perceived emotion. Changes in loudness and tempo were associated positively with changes in arousal, but loudness was dominant. Melodic contour varied positively with valence, though this finding was not conclusive. Texture and spectral centroid did not produce consistent predictions. This methodology facilitates a more ecologically valid investigation of emotion in music and, importantly in the present study, enabled the approximate identification of the lag between musical features and perceived emotion. Responses were made 1 to 3 s after a change in the causal musical event, with sudden changes in loudness producing response lags from zero (nearly instantaneous) to 1 s. Other findings, interactions, and ramifications of the methodology are also discussed.

Update of the Hevner adjective checklist.

Hevner's checklist has been widely used to measure emotional responses to music. Since the revision of the checklist by Farnsworth in the 1950s and l960s, the list has not been updated. 133 musically experienced people were surveyed regarding the suitability of a list of 91 adjectives in describing music. The words consisted of the original 67 from Hevner's adjective circle, and additional words were taken from Russell's circumplex model of emotion (1980) and Whissell's dictionary of affect (1989). The words and clusters were then grouped according to their position on a two-dimensional emotion space. Some of the words used by Hevner but dropped by Farnsworth were reinstated, and 15 other words were dropped. The final list consisted of 46 words grouped into nine clusters in emotion space.

Affective synesthesia: Extracting emotion space from simple perceptual stimuli

Several experiments and reanalyses of archival data were performed in which sets of common emotions were mapped onto visual and auditory perceptual stimuli of varying degrees of complexity. Subjects showed reliability across several different experimental paradigms and diverse stimulus sets. Furthermore, the results were reducible to a two-dimensional emotion space of valence and activity, after the work of Osgood, Succi, and Tannenbaum (1957). The perceptual locus of an aesthetic stimulus in this two-dimensional space is hypothesized to provide a first approximation to how the synesthetic qualities of a percept provide a basis for emotional responses to such stimuli. More generally, this direct mapping of percept onto emotion provides an interesting conundrum for theories of emotion research.

The Position of Sex-Typical Words in Two-Dimensional Emotion Space

Data from five experiments (total N = 304) were used to examine differences in assignments of sex-typicality to personality-describing words. The several significant differences were interpreted in terms of the location of male-typical and female-typical words in two-dimensional emotion space. Words judged as typical of women were consistently more pleasant and less active than those judged as typical of men. The report includes tables of over 200 sex-typical words.