Audiovisual Perception Research Articles

Overall environmental assessment in urban parks: Modelling audio-visual interaction with a structural equation model based on soundscape and landscape indices

This study developed guidelines for the design of park environments for urban residents by conducting an overall environmental assessment from the perspective of audio-visual interaction. We collected information about audio-visual stimuli at 18 locations in three urban parks and created an assessment environment within a laboratory using virtual reality. A subjective assessment questionnaire was administered to 30 participants regarding audio-visual elements, perceived affective quality, and overall environmental assessment. Further, soundscape and landscape indices were defined using the audio-visual perception ratio between natural-related elements and traffic/human-related elements. Based on the subjective assessment responses, an environmental perception model using a structural equation model was developed. The results demonstrated that pleasantness should be prioritized over eventfulness. Water-related elements are efficient for increasing pleasantness, while providing greenery-related elements efficiently controls eventfulness. We found that audio-visual satisfaction improves when there is a higher share of water- and greenery-related elements than traffic-related elements. However, regarding the visual perception ratio of the human presence to natural elements, water features were found to be effective as a design factor for visual satisfaction and greenery for sound satisfaction. Considering the importance of human-related elements, we confirmed that designs are needed that will induce an influx of people into parks at an appropriate level to enhance their overall environmental satisfaction. This study's findings are expected to serve as design guidelines and supporting data to create pleasant urban park environments.

A comfortable soundscape perspective in acoustic environmental planning and management: a case study based on local resident audio-visual perceptions

Soundscapes are an important factor related to audio-visual perception and human health; however, research on how local residents perceive the audio-visual environment remains insufficient. This study, therefore, was mainly conducted to examine the effects of sound sources, the sociodemographic factors of the local residents, visual aesthetic quality and quiet landscape experiences on rural soundscapes through on-site and in-home questionnaire surveys focusing on three components of a soundscape. The results indicated that although there were significant differences in the audio-visual perceptions among typical locations, road traffic sounds were the dominant sound category affecting acoustic comfort on site and in memory. The residents’ age affected the acoustic comfort of background sounds and sound marks in certain ways, while positive landscape experiences made sound marks predominantly perceived and the acoustic comfort of each sound category remarkably improved. This study also developed an agile practical soundscape resource optimization process through an audio-visual perceptual investigation.

Effects of an Audiovisual Emotion Perception Training for Schizophrenia: A Preliminary Study

Individuals with schizophrenia show a reduced ability to integrate facial and vocal information in emotion perception. Although emotion perception has been a target for treatment, no study has yet examined the effect of multimodal training on emotion perception in schizophrenia. In the present study, we developed an audiovisual emotion perception training and test in which a voice and a face were simultaneously presented, and subjects were asked to judge whether the emotions of the voice and the face matched. The voices were either angry or happy, and the faces were morphed on a continuum ranging from angry to happy. Sixteen patients with schizophrenia participated in six training sessions and three test sessions (i.e., pre-training, post-training, and generalization). Eighteen healthy controls participated only in pre-training test session. Prior to training, the patients with schizophrenia performed significantly worse than did the controls in the recognition of anger; however, following the training, the patients showed a significant improvement in recognizing anger, which was maintained and generalized to a new set of stimuli. The patients also improved the recognition of happiness following the training, but this effect was not maintained or generalized. These results provide preliminary evidence that a multimodal, audiovisual training may yield improvements in anger perception for patients with schizophrenia.

Audiovisual integration and the P2 component in adult Asperger’s syndrome: An ERP-study

BackgroundA widened temporal window of integration (TWI) in audiovisual processing has been detected for children with autism spectrum disorder. However, research indicates a narrowing of this TWI and an associated change in audiovisual integration in the course of development. MethodsTo further elucidate audiovisual integration processes in adulthood, we compared adult participants with Asperger’s Syndrome (AS) to healthy controls (HC) by using the sound-induced double flash illusion. For a better understanding of underlying neural mechanisms, event-related potentials were measured via electroencephalography (EEG). ResultsThe number of reported sound-induced flash illusions indicated audiovisual integration. A similar TWI size for both, participants with AS and HC, was found. Additionally, enhanced P2 amplitudes were detected for participants with AS compared to HC. ConclusionResults indicate an involvement of attentional processes in audiovisual perception in participants with AS.

Mirror Neuron Activity During Audiovisual Appreciation of Opera Performance.

Opera is a performing art in which music plays the leading role, and the acting of singers has a synergistic effect with the music. The mirror neuron system represents the neurophysiological mechanism underlying the coupling of perception and action. Mirror neuron activity is modulated by the appropriateness of actions and clarity of intentions, as well as emotional expression and aesthetic values. Therefore, it would be reasonable to assume that an opera performance induces mirror neuron activity in the audience so that the performer effectively shares an embodied performance with the audience. However, it is uncertain which aspect of opera performance induces mirror neuron activity. It is hypothesized that although auditory stimuli could induce mirror neuron activity, audiovisual perception of stage performance is the primary inducer of mirror neuron activity. To test this hypothesis, this study sought to correlate opera performance with brain activity as measured by electroencephalography (EEG) in singers while watching an opera performance with sounds or while listening to an aria without visual stimulus. We detected mirror neuron activity by observing that the EEG power in the alpha frequency band (8–13 Hz) was selectively decreased in the frontal-central-parietal area when watching an opera performance. In the auditory condition, however, the alpha-band power did not change relative to the resting condition. This study illustrates that the audiovisual perception of an opera performance engages the mirror neuron system in its audience.

Considerations in Audio-Visual Interaction Models: An ERP Study of Music Perception by Musicians and Non-musicians.

Previous research with speech and non-speech stimuli suggested that in audiovisual perception, visual information starting prior to the onset of corresponding sound can provide visual cues, and form a prediction about the upcoming auditory sound. This prediction leads to audiovisual (AV) interaction. Auditory and visual perception interact and induce suppression and speeding up of the early auditory event-related potentials (ERPs) such as N1 and P2. To investigate AV interaction, previous research examined N1 and P2 amplitudes and latencies in response to audio only (AO), video only (VO), audiovisual, and control (CO) stimuli, and compared AV with auditory perception based on four AV interaction models (AV vs. AO+VO, AV-VO vs. AO, AV-VO vs. AO-CO, AV vs. AO). The current study addresses how different models of AV interaction express N1 and P2 suppression in music perception. Furthermore, the current study took one step further and examined whether previous musical experience, which can potentially lead to higher N1 and P2 amplitudes in auditory perception, influenced AV interaction in different models. Musicians and non-musicians were presented the recordings (AO, AV, VO) of a keyboard /C4/ key being played, as well as CO stimuli. Results showed that AV interaction models differ in their expression of N1 and P2 amplitude and latency suppression. The calculation of model (AV-VO vs. AO) and (AV-VO vs. AO-CO) has consequences for the resulting N1 and P2 difference waves. Furthermore, while musicians, compared to non-musicians, showed higher N1 amplitude in auditory perception, suppression of amplitudes and latencies for N1 and P2 was similar for the two groups across the AV models. Collectively, these results suggest that when visual cues from finger and hand movements predict the upcoming sound in AV music perception, suppression of early ERPs is similar for musicians and non-musicians. Notably, the calculation differences across models do not lead to the same pattern of results for N1 and P2, demonstrating that the four models are not interchangeable and are not directly comparable.

Perceived Loudness Sensitivity Influenced by Brightness in Urban Forests: A Comparison When Eyes Were Opened and Closed

Soundscape plays a positive, health-related role in urban forests, and there is a competitive allocation of cognitive resources between soundscapes and lightscapes. This study aimed to explore the relationship between perceived loudness sensitivity and brightness in urban forests through eye opening and closure. Questionnaires and measuring equipment were used to gather soundscape and lightscape information at 44 observation sites in urban forested areas. Diurnal variations, Pearson’s correlations, and formula derivations were then used to analyze the relationship between perception sensitivity and how perceived loudness sensitivity was influenced by lightscape. Our results suggested that soundscape variation plays a role in audio–visual perception in urban forests. Our findings also showed a gap in perception sensitivity between loudness and brightness, which conducted two opposite conditions bounded by 1.24 dBA. Furthermore, we found that the effect of brightness on perceived loudness sensitivity was limited if variations of brightness were sequential and weak. This can facilitate the understanding of individual perception to soundscape and lightscape in urban forests when proposing suitable design plans.

Rapid Temporal Recalibration to Audiovisual Asynchrony Occurs Across the Difference in Neural Processing Speed Based on Spatial Frequency.

Audiovisual integration relies on temporal synchrony between visual and auditory stimuli. The brain rapidly adapts to audiovisual asynchronous events by shifting the timing of subjective synchrony in the direction of the leading modality of the most recent event, a process called rapid temporal recalibration. This phenomenon is the flexible function of audiovisual synchrony perception. Previous studies found that neural processing speed based on spatial frequency (SF) affects the timing of subjective synchrony. This study examined the effects of SF on the rapid temporal recalibration process by discriminating whether the presentation of the visual and auditory stimuli was simultaneous. I compared the magnitudes of the recalibration effect between low and high SF visual stimuli using two techniques. First, I randomly presented each SF accompanied by a tone during one session, then in a second experiment, only a single SF was paired with the tone throughout the one session. The results indicated that rapid recalibration occurred regardless of difference in presented SF between preceding and test trials. The recalibration magnitude did not significantly differ between the SF conditions. These findings confirm that intersensory temporal process is important to produce rapid recalibration and suggest that rapid recalibration can be induced by the simultaneity judgment criterion changes attributed to the low-level temporal information of audiovisual events.

Differential Activation and Functional Plasticity of Multimodal Areas Associated with Acquired Musical Skill

Training of a musical skill is known to produce a distributed neural representation of the ability to perceive music and perform musical tasks. In the present study we tested the hypothesis that the audiovisual perception of music involves a wider activation of multimodal sensory and sensorimotor structures in the brain, including those containing mirror neurons. We mapped the activation of brain areas during passive listening and viewing of the first 40 s of “Ode to Joy” being played on the piano by an expert pianist. To do this we performed brain functional magnetic resonance imaging during the presentation of 6 different stimulus contrasts pertaining to that musical melody in a pseudo-randomized order. Group data analysis in musically trained and untrained adults showed robust activation in broadly distributed occipitotemporal, parietal and frontal areas in trained subjects and much restricted activation in untrained subjects. A visual stimulus contrast focusing on the visual motion percept of moving fingers on piano keys revealed selective bilateral activation of a locus corresponding to the V5/MT area, which was significantly more pronounced in trained subjects and showed partial linear dependence on the duration of training on the left side. Quantitative analysis of individual brain volumes confirmed a significantly greater and wider spread of activation in trained compared to untrained subjects. These findings support the view that audiovisual perception of music and musical gestures in trained musicians involves an expanded and widely distributed neural representation formed due to experience-dependent plasticity.

Time as the ink that music is written with: A review of internal clock models and their explanatory power in audiovisual perception

The current review addresses two internal clock models that have dominated discussions in timing research for the last decades. More specifically, it discusses whether the central or the intrinsic clock model better describes the fluctuations in subjective time. Identifying the timing mechanism is critical to explain and predict timing behaviours in various audiovisual contexts. Music stands out for its prominence in real life scenarios along with its great potential to alter subjective time. An emphasis on how music as a complex dynamic auditory signal affects timing accuracy led us to examine the behavioural and neuropsychological evidence that supports either clock model. In addition to the timing mechanisms, an overview of internal and external variables, such as attention and emotions as well as the classic experimental paradigms is provided, in order to examine how the mechanisms function in response to changes occurring particularly during music experiences. Neither model can explain the effects of music on subjective timing entirely: The intrinsic model applies primarily to subsecond timing, whereas the central model applies to the suprasecond range. In order to explain time experiences in music, one has to consider the target intervals as well as the contextual factors mentioned above. Further research is needed to reconcile the gap between theories, and suggestions for future empirical studies are outlined.

Audiovisual emotion perception develops differently from audiovisual phoneme perception during childhood.

This study investigated the developmental paths in the use of audiovisual information for the perception of emotions and phonemes by Japanese speakers. Children aged 5 to 12 years and adults aged 30 to 39 years engaged in an emotion perception task in which speakers expressed their emotions through their faces and voices, and a phoneme perception task using phonemic information in speakers’ lip movements and speech sounds. Results indicated that Japanese children’s judgement of emotions by using auditory information increased with increasing age, whereas the use of audiovisual information for judging phonemes remained constant with increasing age. Moreover, adults were affected by visual information more than children. We discuss whether these differences in developmental patterns are due to differential integration processes for information indicative of emotions and phonemes, as well as possible cultural / linguistic reasons for these differences.

Individual differences in multiple object tracking, attentional cueing, and age account for variability in the capacity of audiovisual integration.

There has been a recent increase in individual differences research within the field of audiovisual perception (Spence & Squire, 2003, Current Biology, 13(13), R519-R521), and furthering the understanding of audiovisual integration capacity with an individual differences approach is an important facet within this line of research. Across four experiments, participants were asked to complete an audiovisual integration capacity task (cf. Van der Burg, Awh, & Olivers, 2013, Psychological Science, 24(3), 345-351; Wilbiks & Dyson, 2016, PLOS ONE 11(12), e0168304; 2018, Journal of Experimental Psychology: Human Perception and Performance, 44(6), 871-884), along with differing combinations of additional perceptual tasks. Experiment 1 employed a multiple object tracking task and a visual working memory task. Experiment 2 compared performance on the capacity task with that of the Attention Network Test. Experiment 3 examined participants' focus in space through a Navon task and vigilance through time. Having completed this exploratory work, in Experiment 4 we collected data again from the tasks that were found to correlate significantly across the first three experiments and entered them into a regression model to predict capacity. The current research provides a preliminary explanation of the vast individual differences seen in audiovisual integration capacity in previous research, showing that by considering an individual's multiple object tracking span, focus in space, and attentional factors, we can account for up to 34.3% of the observed variation in capacity. Future research should seek to examine higher-level differences between individuals that may contribute to audiovisual integration capacity, including neurodevelopmental and mental health differences.

Audiovisual Modulation in Music Perception for Musicians and Non-musicians.

In audiovisual music perception, visual information from a musical instrument being played is available prior to the onset of the corresponding musical sound and consequently allows a perceiver to form a prediction about the upcoming audio music. This prediction in audiovisual music perception, compared to auditory music perception, leads to lower N1 and P2 amplitudes and latencies. Although previous research suggests that audiovisual experience, such as previous musical experience may enhance this prediction, a remaining question is to what extent musical experience modifies N1 and P2 amplitudes and latencies. Furthermore, corresponding event-related phase modulations quantified as inter-trial phase coherence (ITPC) have not previously been reported for audiovisual music perception. In the current study, audio video recordings of a keyboard key being played were presented to musicians and non-musicians in audio only (AO), video only (VO), and audiovisual (AV) conditions. With predictive movements from playing the keyboard isolated from AV music perception (AV-VO), the current findings demonstrated that, compared to the AO condition, both groups had a similar decrease in N1 amplitude and latency, and P2 amplitude, along with correspondingly lower ITPC values in the delta, theta, and alpha frequency bands. However, while musicians showed lower ITPC values in the beta-band in AV-VO compared to the AO, non-musicians did not show this pattern. Findings indicate that AV perception may be broadly correlated with auditory perception, and differences between musicians and non-musicians further indicate musical experience to be a specific factor influencing AV perception. Predicting an upcoming sound in AV music perception may involve visual predictory processes, as well as beta-band oscillations, which may be influenced by years of musical training. This study highlights possible interconnectivity in AV perception as well as potential modulation with experience.

Multi-modal cross-linguistic perception of fricatives in clear speech.

Research shows that acoustic modifications in clearly enunciated fricative consonants (relative to the plain, conversational productions) facilitate auditory fricative perception, particularly for auditorily salient sibilant fricatives and for native perception. However, clear-speech effects on visual fricative perception have received less attention. A comparison of auditory and visual (facial) clear-fricative perception is particularly interesting since sibilant fricatives in English are more auditorily salient while non-sibilants are more visually salient. This study thus examines clear-speech effects on multi-modal perception of English sibilant and non-sibilant fricatives. Native English perceivers and non-native (Mandarin, Korean) perceivers with different fricative inventories in their native languages (L1s) identified clear and conversational fricative-vowel syllables in audio-only, visual-only, and audio-visual (AV) modes. The results reveal an overall positive clear-speech effect when visual information is involved. Considering the factor of AV saliency, clear speech benefits sibilants more in the auditory domain and non-sibilants more in the visual domain. With respect to language background, non-native (Mandarin and Korean) perceivers benefit from visual as well as auditory information, even for fricatives non-existent in their respective L1s, but the patterns of clear-speech gains are affected by the relative AV weighting and "nativeness" of the fricatives. These findings are discussed in terms of how saliency-enhancing and category-distinctive cues of speech sounds are adopted in AV perception to improve intelligibility.

Multisensory Integration Develops Prior to Crossmodal Recalibration.

It has been hypothesized that crossmodal recalibration plays a crucial role for the development of multisensory integration capabilities [1]. To test the developmental trajectory of multisensory integration and crossmodal recalibration, we used a combined ventriloquist/ventriloquist aftereffect paradigm [2] in children aged 5-9 years. The ventriloquist effect (indicating multisensory integration), that is, the shift of auditory localization toward simultaneously presented but spatially discrepant visual stimuli, was larger in children than in adults, which was attributed to a lower auditory localization precision in the children. In fact, the size of the ventriloquist effect depended on the visual stimulus reliability in both children and adults. In all groups, the ventriloquist effect was best explained by a causal inference model. In contrast to their multisensory integration capabilities, 5-year-old children did not recalibrate. The immediate ventriloquist aftereffect (indicating recalibration after a single exposure to a spatially discrepant audio-visual stimulus) emerged in 6- to 7-year-old children, whereas the cumulative ventriloquist aftereffect (reflecting recalibration to the audio-visual spatial discrepancies over the complete experiment) was not observed before the age of 8 years. First, in contrast to common beliefs, the present results provide evidence that multisensory integration precedes rather than follows crossmodal recalibration during development. Second, we report developmental evidence for a dissociation of the processes involved in multisensory integration and immediate as well as cumulative recalibration. We speculate that multisensory integration is a prerequisite for crossmodal recalibration, because the multisensory percept, rather than unimodal cues, might comprise a crucial signal for the calibration of the sensory systems.

Сучасний погляд на викладання кінорежисури як дисципліни

Going back to history of film editing we can see quite a number of systematic attempts to classify the editing types. Béla Balázs, Vsevolod Pudovkin, Semen Timoshenko, Sergei Eisenshtein, Raymond Spottiswoode were doing their best at the time of 1920s-1930s to construct the classification of film editing types. Being at that time at the edge of the “film practice”, nowadays these attempts become aware of the lack of conjunction in “film practice” and “film science” that have been developed up to now. Moreover, the mentioned attempts were accented mainly on emotional component of audience perception. The real functionality of the cut, at this very moment, just at this very frame happened to be out of the look. In addition, all classifications were built on linear principle – i.e. the simple list of types. It is clear now that the simple listing cannot cover all the variety of existing editing types, and that it is not possible to mix the editing types and editing techniques in one simple listing.The scope of the article is within the multi-level hierarchical classification that is based on the motivation principle of the cut. That means the functional causes for film director or film editor to cut at a frame to achieve the desired impact on audiovisual perception. This makes it easy to construct the conjunction of editing types and editing techniques, the latter being the consequently subordinated to functions of editing.There are three main functions of editing in terms of primitive audiovisual perception of shot sequences that apply on content perception. These are:continuity (or continuity editing), meaning just to tell the story without the redundant details (i.e. shots);intellectual (or intellectual comparison of the shots), meaning the effect of any shot differs sharply from that of its precursor and successor, resulting in an impact of the sensation and concept deriving from contiguous shots; and from this impact may arise a third concept different from that of either of the components which produced it;disjunctive (or provocative editing), meaning the effect of combining the shots together that are against the editing rules, with the deliberate aim to attract the audience attention, sometimes even to shock (the best samples could be found with the almost every music video, however, it happened to be the distinctive practice for many feature films, as well).

Complex sensory environments alter mate choice outcomes.

Noise is a common problem in animal communication. We know little, however, about how animals communicate in the presence of noise using multimodal signals. Multimodal signals are hypothesised to be favoured by evolution because they increase the efficacy of detection and discrimination in noisy environments. We tested the hypothesis that female túngara frogs' responses to attractive male advertisement calls are improved in noise when a visual signal component is added to the available choices. We tested this at two levels of decision complexity (two and three choices). In a two-choice test, the presence of noise did not reduce female preferences for attractive calls. The visual component of a calling male, associated with an unattractive call, also did not reduce preference for attractive calls in the absence of noise. In the presence of noise, however, females were more likely to choose an unattractive call coupled with the visual component. In three-choice tests, the presence of noise alone reduced female responses to attractive calls and this was not strongly affected by the presence or absence of visual components. The responses in these experiments fail to support the multimodal signal efficacy hypothesis. Instead, the data suggest that audio-visual perception and cognitive processing, related to mate choice decisions, are dependent on the complexity of the sensory scene.

Supplemental Material for Combining the Senses: The Role of Experience- and Task-Dependent Mechanisms in the Development of Audiovisual Simultaneity Perception

Supplemental Material for Combining the Senses: The Role of Experience- and Task-Dependent Mechanisms in the Development of Audiovisual Simultaneity Perception

A systematic approach for English education model based on the neural network algorithm

This paper algorithms based on neural network model designed for English education, to develop a model education system with artificial intelligence, summarized the dimensions were can be used for data analysis related indicators. These indicators include not only the contents of the learning behavior, test behavior, cooperation behavior and resource search behavior and other human-computer interaction behavior data, also includes demographic background information, learning ability, learning attitude, and other characteristic data that affect the learning effect. We tried to collect relevant indicators to the maximum extent. An audiovisual fusion method based on Convolutional Neural Network (CNN) is proposed. The independent CNN structure is used to realize independent modeling of audiovisual perception and asynchronous information transmission and obtain the description of audiovisual parallel data in the high-dimensional feature space. Following the shared fully connected structure, it is possible to model the long-term dependence of audiovisual parallel data in a higher dimension. Experiments show that the AVSR system built using a CNN-based audiovisual fusion method can achieve a significant performance improvement, and its recognition error rate is relatively reduced by about 15%. The speech recognition system trained with the cross-domain adaptive method can obtain a significant performance improvement, and its recognition error rate is more than 10% lower than that of the baseline system..

Perception of audiovisual infant directed speech.

Infant perception often deals with audiovisual speech input and afirst step in processing this input is to perceive both visual and auditory information. The speech directed to infants has special characteristics and may enhance visual aspects of speech. The current study was designed to explore the impact of visual enhancement in infant-directed speech (IDS) on audiovisual mismatch detection in a naturalistic setting. Twenty infants participated in an experiment with a visual fixation task conducted in participants' homes. Stimuli consisted of IDS and adult-directed speech (ADS) syllables with a plosive and the vowel /a:/, /i:/ or /u:/. These were either audiovisually congruent or incongruent. Infants looked longer at incongruent than congruent syllables and longer at IDS than ADS syllables, indicating that IDS and incongruent stimuli contain cues that can make audiovisual perception challenging and thereby attract infants' gaze.