Bidirectional Cross-Modal Influence on Emotion Ratings of Auditory and Visual Stimuli

Abstract

BIDIRECTIONAL CROSS-MODAL INFLUENCE ON Bidirectional Cross-Modal EMOTION RATINGS Influence on Emotion Ratings OF AUDITORY AND VISUAL STIMULI of Auditory and Visual Stimuli Abstract: Previous research concerning cross-modal influences on emotional perception has focused primarily on how auditory stimuli affect emotional responses to visual stimuli. The present study examines whether such effects are bidirectional. Different participants were tested in one of these two directions of influence, using a slider-bar rating task to judge the emotionality (sadness/happiness) of stimuli in an attended modality (auditory or visual). Stimuli were presented in auditory and visual pairs, with instructions to ignore stim- uli from the irrelevant modality. All stimuli (auditory and visual) had been previously categorized as sad, ambiguous, or happy. Results showed that ratings depended primarily on the emotional categories of stimuli in the attended modality (auditory or visual). In addition, participants were subject to smaller cross-modal influences from the unattended modality in both the auditory and visual attentional conditions. Bidirectional influences were thus observed, showing that perceptual influence is not limited to a single cross-modal direction. BY HARRISON JAMES RAMSAY Introduction Auditory and visual information are frequently com- bined to produce unique effects in many types of entertain- ment and performance. The simultaneous presentation of mu- sic and film, in particular, is a very powerful combination that is utilized as a way to elicit emotion in cinema. The ability of music to influence emotion has been well documented in the literature. 1 Similar evidence documents the effectiveness of film segments to 2 evoke emotions within participants and across participants. While these studies individually high- light the capabilities of music and film to manipulate emo- tion, it is important to consider specific effects that may result when auditory and visual stimuli are experienced together. curring activity. Therefore, it naturally appeals to questions about whether and why the addition of music matters in these examples. Multiple studies have combined auditory and visual information in order to assess the resulting effects of such cross-modality 3-9 pairings on various parameters of emotion and cognition. For present purposes, it will be necessary to recognize the distinction between experienced emotion and the perception of stimulus emotionality. With regards to the perception of emotionality in stimuli, fewer studies 11, 12 have illustrated the possibility of a cross-modal influence. While research on this topic is less plentiful, it does begin to elu- cidate the phenomenon of interest. Until recently, research on the topic of cross-modal influence has focused primarily on the auditory-to-visual direction. This is not surprising, as combinations of auditory and visual information are frequent- ly found in television, movies, and theatric performances as mentioned earlier, where the primary focus is on visually oc- In summary, while much of the research in the au- ditory-to-visual direction appears to focus mainly on the experience of affect, some studies have instead chosen to explore the perception of stimulus emotionality. When comparing the two directions of influence, it is apparent that the visual-to-auditory direction as a whole, has been less explored. From reviewing available research in this di- rection, the lack of emphasis on the topic of affect in gener- al is clear. There are few studies of experienced affect, and even fewer studies suggesting cross-modal effects on the perception of stimulus emotionality. While some of these studies do explore participant ratings of auditory stimuli, few ask whether the emotionality of auditory stimuli can be altered by the presence of emotional visual informa- tion. Berkeley Scientific Journal | FALL 2016 The present study addresses whether these sorts of phenomena can also occur in the opposite direction. The handful of studies that have investigated the visual-to-au- ditory direction of influence seem to focus less on the top- ics of experienced affect and stimuli emotionality that have been studied in the auditory-to-visual direction, showing instead, a greater emphasis on the perception of other stimuli characteristics. 13-17