Modeling Music Emotion Judgments Using Machine Learning Methods.

Naresh N Vempala,Frank A Russo

doi:10.3389/fpsyg.2017.02239

Naresh N Vempala, Frank A Russo

Open Access

https://doi.org/10.3389/fpsyg.2017.02239

Copy DOI

Abstract

Emotion judgments and five channels of physiological data were obtained from 60 participants listening to 60 music excerpts. Various machine learning (ML) methods were used to model the emotion judgments inclusive of neural networks, linear regression, and random forests. Input for models of perceived emotion consisted of audio features extracted from the music recordings. Input for models of felt emotion consisted of physiological features extracted from the physiological recordings. Models were trained and interpreted with consideration of the classic debate in music emotion between cognitivists and emotivists. Our models supported a hybrid position wherein emotion judgments were influenced by a combination of perceived and felt emotions. In comparing the different ML approaches that were used for modeling, we conclude that neural networks were optimal, yielding models that were flexible as well as interpretable. Inspection of a committee machine, encompassing an ensemble of networks, revealed that arousal judgments were predominantly influenced by felt emotion, whereas valence judgments were predominantly influenced by perceived emotion.

Highlights

The classic philosophical debate on music emotion pits a “cognitivist” view of music emotion against an “emotivist” view
root mean-squared error (RMSE) for the perception ensemble was 0.23. These results indicate that with audio features, a linear model was sufficient to achieve prediction performance similar to a more flexible model such as a neural network; with physiology features, a flexible, nonlinear machine learning (ML) model was necessary to capture the predictive capacity of the independent variables
We revisited the classic debate on music and emotion involving the cognitivists and the emotivists

Summary

INTRODUCTION

The classic philosophical debate on music emotion pits a “cognitivist” view of music emotion against an “emotivist” view (see e.g., Kivy, 1989). In contrast with the discrete view of emotions that argues independent processes for distinct emotions (e.g., Ekman, 1992, 1999), the dimensional approach proposes that all affective states may be characterized on the basis of underlying dimensions of emotion This approach is in widespread use in music cognition research (e.g., Schubert, 1999; Gomez and Danuser, 2004; Witvliet and Vrana, 2007), and has been found to be effective in characterizing emotionally ambiguous stimuli (Eerola and Vuoskoski, 2011). We assumed that if the emotivist positions were true, we should be able to model emotion judgments on the basis of physiological responses. Another possibility that we considered is that emotion judgments are the result of a meta-level cognitive decisionmaking process that combines output from a perception module and a feeling module (Figure 1). We compared the success of our committee machine with two other ML approaches with the intent of highlighting the relative merits of the different approaches

Participants

DISCUSSION

Findings

ETHICS STATEMENT