Emotion in Speech, Singing, and Sound Effects

Abstract

This chapter explores the theoretical context of emotion studies in terms of speech and sound effects, and in particular the concept of affective potential. Voice actors in game soundtracking can have a particularly powerful impact on the emotional presentation of a narrative; and this affective control can go beyond that of the actor alone if combined with emotionally-targeted signal processing (for example, sound design and audio processing techniques). The prospect of synchronousing emotionally congruent sound effects remains a fertile area for further work, but an initial study which will be presented later in this chapter suggests that timbral features from speech and sound effects can exert an influence on the perceived emotional response of a listener in the context of dynamic soundtracking for video games. This chapter extends upon material originally presented at the Audio Engineering Society conference on video game soundtracking in London, UK, 2015 (Williams et al. 2015), and subsequently on the specific design of affect in vocal production at the Audio Engineering society convention in New York, 2015 (Williams 2015a). Prosodic (nonverbal) speech features have been the subject of a considerable amount of research (Gobl 2003; Pell 2006). The role of such features as a communicative tool in emotion studies suggests that acoustic manipulation of prosody could be a useful way to explore emotional communication (Frick 1985; Baum and Nowicki 1998). For example, in studies which use dimensional approaches to emotion, known acoustic correlations found in prosody include emotional arousal with pitch height, range, rate of speech, and loudness. Some emotional cues can be derived acoustically from prosody (Bach et al. 2008) by time series analysis in a manner which is analogous to the temporal characteristics used to determine such cues in musical sequences (Gobl 2003; Juslin and Laukka 2006; Kotlyar and Morozov 1976; Deng and Leung 2013, for example pitch height and range, loudness, and density are suggested to correlate strongly with affective arousal by some research).