Effects of denotative congruency on audio-visual impressions

Abstract

Abstract: We examined how the denotative congruency in audio-visual stimuliaffects the pleasant and restful impressions of the stimuli. In Experiment 1, in whichwe combined a single sound and a single motion picture, we found that the impres-sions of the audio-visual stimuli would be determined in terms of averaging theimpressions obtained from each of the audio and visual stimuli. In Experiment 2, inwhich we combined two sounds and a single motion picture, we found that thepleasant and restful impressions positively shifted when one of the sounds wascombined with the denotatively congruent motion picture, even if the unpleasant andunrestful sound was combined with an unpleasant and unrestful motion picture.These results suggested that the denotative congruency between the audio andvisual stimuli would improve the pleasant and restful impressions. We are proposingthat the stimulus complexity, which would be reduced by the denotative congruencybetween the audio and visual stimuli, underlies the positive shift of the pleasant andrestful impressions in observing the stimuli.Key words: impression formation, integration process, relaxing music, noise, motionpicture.Some previous studies concerned with audio-visual interaction in perception and cognitionhave demonstrated that audio processing isaffected by visual stimuli. For instance, theMcGurk effect shows that audio processing ismodulated by visual stimulus in speech per-ception (McGurk & MacDonald, 1976). Theeffects, known as “visual capture” (Hay, Pick,& Ikeda, 1965; Jackson, 1953), or the “ventrilo-quism effect” (Bertelson & Radeau, 1981;Howard & Templeton, 1966; Jack & Thurlow,1973) imply that the perceived location of thesound source and events tends to be deter-mined by the location of the visual stimulus.Other previous studies have demonstratedthat visual processing is affected by audiostimuli. For instance, the studies pertaining tothe streaming-bouncing phenomena show thatvisual perception is influenced by a transientaudio stimulus (Sekuler, Sekuler, & Lau, 1997;Shimojo & Shams, 2001). The perceivedfrequency of the visual flash would varywith the frequency of the accompaniedaudio stimuli (Shams, Kamitani, & Shimojo,2000; Shimojo & Shams, 2001). The apparentdistance (Ichikawa & Masakura, 2006) andacceleration (Wada, Kitagawa, & Noguchi,2002) of visual motion are modulated by audiostimuli. These studies indicate that the interac-tion between audio processing and visual pro-cessing is very strong in perception andcognition.