Audiovisual Environment Research Articles

Guest Editorial Special Section on the 2011 IEEE International Symposium on Haptic Audio–Visual Environments and Games

Guest Editorial Special Section on the 2011 IEEE International Symposium on Haptic Audio–Visual Environments and Games

Synchronized Audio-Visual Transients Drive Efficient Visual Search for Motion-in-Depth

In natural audio-visual environments, a change in depth is usually correlated with a change in loudness. In the present study, we investigated whether correlating changes in disparity and loudness would provide a functional advantage in binding disparity and sound amplitude in a visual search paradigm. To test this hypothesis, we used a method similar to that used by van der Burg et al. to show that non-spatial transient (square-wave) modulations of loudness can drastically improve spatial visual search for a correlated luminance modulation. We used dynamic random-dot stereogram displays to produce pure disparity modulations. Target and distractors were small disparity-defined squares (either 6 or 10 in total). Each square moved back and forth in depth in front of the background plane at different phases. The target’s depth modulation was synchronized with an amplitude-modulated auditory tone. Visual and auditory modulations were always congruent (both sine-wave or square-wave). In a speeded search task, five observers were asked to identify the target as quickly as possible. Results show a significant improvement in visual search times in the square-wave condition compared to the sine condition, suggesting that transient auditory information can efficiently drive visual search in the disparity domain. In a second experiment, participants performed the same task in the absence of sound and showed a clear set-size effect in both modulation conditions. In a third experiment, we correlated the sound with a distractor instead of the target. This produced longer search times, indicating that the correlation is not easily ignored.

Audiovisual time perception is spatially specific

Our sensory systems face a daily barrage of auditory and visual signals whose arrival times form a wide range of audiovisual asynchronies. These temporal relationships constitute an important metric for the nervous system when surmising which signals originate from common external events. Internal consistency is known to be aided by sensory adaptation: repeated exposure to consistent asynchrony brings perceived arrival times closer to simultaneity. However, given the diverse nature of our audiovisual environment, functionally useful adaptation would need to be constrained to signals that were generated together. In the current study, we investigate the role of two potential constraining factors: spatial and contextual correspondence. By employing an experimental design that allows independent control of both factors, we show that observers are able to simultaneously adapt to two opposing temporal relationships, provided they are segregated in space. No such recalibration was observed when spatial segregation was replaced by contextual stimulus features (in this case, pitch and spatial frequency). These effects provide support for dedicated asynchrony mechanisms that interact with spatially selective mechanisms early in visual and auditory sensory pathways.

A new hybrid and dynamic fusion of multiple experts for intelligent porch system

Intelligent porch research is an important issue in smart home development; however, such a field was rarely investigated in the literature. This investigation proposes a new hybrid and dynamic fusion of multiple experts for the intelligent porch system. First, a new hybrid priority tree with decision fusion (HPTD-fusion) is proposed to eliminate the problems of tag-based authentication outdoors. The HPTD-fusion first verifies the vocal entrance code (VEC), and subsequently the remaining experts are performed in the cases of AND, OR or majority voting for decision fusion. Second, the post-mapping dynamic weighted fusion (PMDW-fusion) scheme is presented to adapt the indoor porch audio-visual environment. The PMDW-fusion dynamically assigns the higher weight to experts with higher performance, and then sums all participating experts for score fusion. The experimental results demonstrate that FRR and FAR can reach up to 0.18 and 0.19, respectively, when the system is tested in the outdoor environment. Furthermore, the indoor recognition accuracy can be increased to 86.1% using the proposed fusion scheme. The experiments have verified the effectiveness and feasibility of the proposed system. Restated, the contribution of this work is to develop a novel intelligent porch system incorporating a natural and unobtrusive method for identity recognition. The proposed system has been installed and tested in a real-world environment in the Technologies for Ubiquitous Computing and Humanity (TOUCH) Center at National Cheng Kung University.

Headphone-based Spatial Sound with a GPU Accelerator

Multichannel acoustic signal processing has undergone major development in recent years. The incorporation of spatial information into an immersive audiovisual virtual environment or into video games provides better sense of “presence” to applications. Spatial sound consists in reproducing audio signals with spatial cues (spatial information embedded in the sound) through headphones. This spatial information allows the listener to identify the virtual positions of the sources corresponding to different sounds. Headphone-based spatial sound is obtained by filtering different sound sources through special filters called Head Related Transfer Functions (HRTFs) prior to render them through headphones. Efficient computation plays an important role when the number of sources to be managed is high. This situation increases the number of filtering operations, requiring high computing capacity specially when the virtual sources are moving. Graphics Processing Units (GPUs) are high parallel programmable co-processors that provide massive computation when the needed operations are properly parallelized. This paper discusses the design, the implementation and the performance of a headphone-based spatial audio application whose processing is totally carried out on a GPU. This application is able to interact with the listener who can select and change the location of the sound sources in real-time. This work analyzes also specific computational aspects inside the CUDA environment in order to successfully exploit GPU resources. Results show that the proposed application is able to move up to 2500 sources simultaneously, while leaving free CPU resources for other tasks. This work emphasizes the importance of analyzing all CUDA aspects, since they can influence drastically the performance.

Audiovisual synchrony perception of walkers as a function of distance and depth cues

Audiovisual perception is still an intriguing phenomenon, especially when we think about the physical and neuronal differences underlying the perception of sound and light. Physically, there is a delay of ∼3 ms/m between the emission of a sound and its arrival to the observer but, on the other hand, we know that acoustic transduction is a very fast process (∼1 ms). Conversely, light speed makes negligible the physical delay while phototransduction is quite slow (∼50 ms). Audio and visual stimuli that are temporally mismatched can be perceived as a coherent audiovisual stimulus, but a sound delay is often required to achieve a better synchrony perception. In this study, we analyze the Point of Subjective Synchrony (PSS) as a function of stimulus distance to understand if individuals take into account sound velocity or if they compensate for differences in transduction time when judging synchrony. Using an audiovisual virtual-reality environment (CAVE-Like) with Point Light Walkers (PLW) as visual stimulus and sound of steps as audio stimulus, audiovisual sequences were presented from −285 to +300 ms of audio asynchrony, at different distances from the observer (10, 15, 20, 25, 30 and 35 m), and in three different conditions which differ only in the number of visual and auditory depth cues. The results show a relation between PSS and stimulus distance congruent with the differences in propagation velocity between sound and light. Depending on the number of depth cues presented, this relation appears to be increasingly closer to a model based on compensation for these physical differences.

Comparing perceived auditory width to the visual image of a performing ensemble in contrasting bi-modal environments

Despite many studies investigating auditory spatial impressions in rooms, few have addressed the impact of simultaneous visual cues on localization and the perception of spaciousness. The current research presents an immersive audiovisual environment in which participants were instructed to make auditory width judgments in dynamic bi-modal settings. The results of these psychophysical tests suggest the importance of congruent audio visual presentation to the ecological interpretation of an auditory scene. Supporting data were accumulated in five rooms of ascending volumes and varying reverberation times. Participants were given an audiovisual matching test in which they were instructed to pan the auditory width of a performing ensemble to a varying set of audio and visual cues in rooms. Results show that both auditory and visual factors affect the collected responses and that the two sensory modalities coincide in distinct interactions. The greatest differences between the panned audio stimuli given a fixed visual width were found in the physical space with the largest volume and the greatest source distance. These results suggest, in this specific instance, a predominance of auditory cues in the spatial analysis of the bi-modal scene.

Los parámetros que identifican el subtitulado para sordos: análisis y clasificación

Subtitling for the Deaf and the Hard-of-Hearing (SDH) was long considered a “simple” variant of standard subtitling. Only recently, uprising social demands together with a growing presence of SDH materials in different international audiovisual environments, have promoted the proliferation of research initiatives on SDH practices. As a result, the systematic application to SDH of some of the parameters originally adopted for standard subtitling has proven to be controversial. This paper presents a proposal for the specific analysis of SDH parameters. Based on a taxonomy developed by Bartoll (2008), the new taxonomy describes the restricted application of standard parameters to this accessibility modality. The new proposal focuses on the parameters that are specific to SDH –the representation of extralinguistic information– and sheds light into the tight connection established among all the agents involved. The new taxonomy tries to provide researchers and SDH professionals with a tool to evaluate SDH practices and analyze the implications of potential modifications on parameters.

Interactive video projections as augmented environments

The aim of the work presented in this paper is to investigate and demonstrate the potential for sensitive, physical and spatial interaction, bringing together the electronic and physical environments in an ecological approach. Through the research and design processes in the author’s studio such electronic ecologies as spatial audiovisual environments are developed, in which the audience co-creates the final result through their body movements. An open design approach is applied, the audience participates in the development process at any stage. The paper presents a media framework to place interactivity and time-based material in a context. Several recent pieces are described, interactive video environments such as Trainflow and Facets, including a presentation of technical factors. The paper has a section on mapping and other design considerations, and audience feedback and experiences. The final section of this paper concludes with a discussion and reflection on the process and outcomes, and it presents future directions.

A qualitative analysis of sensory phenomena induced by perceptual deprivation

Previous studies have shown that misperceptions and illusory experiences can occur if sensory stimulation is withdrawn or becomes invariant even for short periods of time. Using a perceptual deprivation paradigm, we created a monotonous audiovisual environment and asked participants to verbally report any auditory, visual or body-related phenomena they experienced. The data (analysed using a variant of interpretative phenomenological analysis) revealed two main themes: (1) reported sensory phenomena have different spatial characteristics ranging from simple percepts to the feeling of immersion in a complex multisensory environment and (2) the active contribution of the perceiver where participants report engaging in exploratory processes even when there is nothing to find. Detailed analysis of the qualitative data further showed that participants who reported more perceptual phenomena were more likely to report internal bodily sensations, move more during the experiment and score higher on the Revised Hallucination Scale than those reporting fewer percepts explicitly linking perceptual deprivation to somatic phenomena. The results demonstrate how the variety of sensory experiences induced by perceptual deprivation can give further insight into the factors mediating conscious awareness and may suggest ways in which the brain imposes meaning on the environment under invariant sensory conditions.

Guest Editorial for Special Section on IEEE Haptic Audio Visual Environments and Games 2010

The nine papers in this special section are extended versions of some accepted papers from the IEEE International Symposium on Haptic Audio Visual Environments and Games (HAVE) 2010.

Synchronized audio-visual transients drive efficient visual search for motion-in-depth

In natural audio-visual environments, a change in depth is usually correlated with a change in loudness. In the present study, we investigated whether correlating disparity and loudness provides a functional advantage in binding disparity and sound amplitude in a visual search paradigm. To test this assumption, we used a method similar to that used by Van der Burg et al. (2008) to show that non-spatial modulations of loudness can drastically improve spatial visual search for a correlated luminance modulation. Subsequently (2010), they varied the shape of temporal modulation and demonstrated that transient events (square modulations) are required for this search efficiency, and that sinusoidal audiovisual modulations do not support efficient search. We used dynamic random-dot stereogram displays to produce pure disparity modulations. Target and distractors were 0.35 × 0.35 degrees disparity-defined squares (either 6 or 10 in total) presented on a ring at 2.5 deg eccentricity. Each square moved back and forth in depth from zero to +12 arcmin (crossed) disparity at different phases. The target's depth modulation was synchronized with an amplitude-modulated 500 Hz tone. Visual and auditory modulations were always congruent (both sinewave or squarewave). Four observers were asked to give speeded responses in a discrimination task on the target. Because binocular matching processes are known to favor smooth over abrupt changes of disparity across space and time, we expected the sine modulation condition to be at least as efficient as the square modulation in supporting efficient search. However, results show a significant improvement in visual search in the square condition compared to the sine condition, suggesting that transient auditory information can efficiently drive visual search in the disparity domain. In a second experiment, correlating sound with a distractor led to longer search times, indicating that the correlation is not easily ignored.

Influences of audio-visual environments on feelings of deliciousness during having sweet foods: An electroencephalogram frequency analysis study

Feelings of deliciousness during having foods are mainly produced by perceptions of sensory information extracted from foods themselves, such as taste and olfaction. However, environmental factors might modify the feeling of deliciousness. In the present study, we investigated how the condition of audio-visual environments affects the feeling of deliciousness during having sweet foods. Electroencephalograms (EEGs) were recorded from the frontal region of the scalp of healthy participants under virtual scenes of tearoom and construction work, respectively. The participants were asked to rate deliciousness after the recordings. Frequency analyses were performed from the EEGs. During having the foods, occupancy rates of beta frequency band between tearoom scenes and construction work scenes were markedly different, but not in other frequency bands. During having no food, in contrast, there was no difference of occupancy rates in respective frequency bands between the two different scenes. With regard to deliciousness during having sweet foods, all participants rated high scores under the scenes of tearoom than those under the scenes of construction work. Interestingly, there is a positive correlation between occupancy rates of beta frequency band and scores of deliciousness. These findings suggest that comfortable audio-visual environments play an important role in increasing the feeling of deliciousness during having sweet foods, in which beta frequency rhythms may be concerned with producing comprehensive feelings of deliciousness.

Emotional control and visual representation using advanced audiovisual interaction

Modern interactive means combined with new digital media processing and representation technologies can provide a robust framework for enhancing user experience in multimedia entertainment systems and audiovisual artistic installations with non-traditional interaction/feedback paths based on user affective state. In this work, the ‘Elevator’ interactive audiovisual platform prototype is presented, which aims to provide a framework for signalling and expressing human behaviour related to emotions (such as anger) and finally produce a visual outcome of this behaviour, defined here as the emotional ‘thumbnail’ of the user. Optimised, real-time audio signal processing techniques are employed for monitoring the achieved anger-like behaviour, while the emotional elevation is attempted using appropriately selected combined audio/visual content reproduced using state-of-the-art audiovisual playback technologies that allow the creation of a realistic immersive audiovisual environment. The demonstration of the proposed prototype has shown that affective interaction is possible, allowing the further development of relative artistic and technological applications.

Looking Back—A Lesson Learned: From Videotape to Digital Media

This paper chronicles the development of Drehort Neubrandenburg Online, an interactive, content-rich audiovisual language learning environment based on documentary film material shot on location in Neubrandenburg, Germany, in 1991 and 2002 and aimed at making language learning more interactive and more real. The paper starts with the description of the first film project and the struggle to make the linguistic material accessible to students learning German. It continues with a description of further development stages using emerging technologies--laserdisc and digitized media files, HyperCard, Libra, and internet applications such as Flash--describing which new tools are effective and for what reasons and what the use of technology may mean for language learning from a theoretical and from an empirical perspective.

A system to simulate and reproduce audio–visual environments for spatial hearing research

The article reports the experience gained from two implementations of the “Simulated Open-Field Environment” (SOFE), a setup that allows sounds to be played at calibrated levels over a wide frequency range from multiple loudspeakers in an anechoic chamber. Playing sounds from loudspeakers in the free-field has the advantage that each participant listens with their own ears, and individual characteristics of the ears are captured in the sound they hear. This makes an easy and accurate comparison between various listeners with and without hearing devices possible. The SOFE uses custom calibration software to assure individual equalization of each loudspeaker. Room simulation software creates the spatio-temporal reflection pattern of sound sources in rooms which is played via the SOFE loudspeakers. The sound playback system is complemented by a video projection facility which can be used to collect or give feedback or to study auditory–visual interaction. The article discusses acoustical and technical requirements for accurate sound playback against the specific needs in hearing research. An introduction to software concepts is given which allow easy, high-level control of the setup and thus fast experimental development, turning the SOFE into a “Swiss army knife” tool for auditory, spatial hearing and audio–visual research.

Adapting to altered image statistics using processed video

Perceptual systems can be altered by immersing observers in environments with statistical properties that differ from those naturally encountered. Here we present a novel method for placing observers in naturalistic audio visual environments whose statistics can be manipulated in very targeted ways. We present the results of a case study that used this method. Observers were exposed to an environment where there was a novel statistical relationship between two simple, visual patterns in otherwise natural scenes. Exposure to this altered environment strengthened perceptual interactions between the two patterns.

Creative engagement in interactive immersive environments

This paper presents two independent case studies selected from n = 11, an 11-year-old male with Down's Syndrome and another 12-year-old male with Autistic Spectrum Disorder (ASD). An audiovisual immersive interactive environment was developed to encourage creative interaction and expression from the participants. The feedback was via a microphone attached to a sound processor to capture any vocal utterances or speech, additionally the child's gestures were captured through a digital camera and mapped to computer vision algorithms to create a ‘moving painting’, and then further to a MIDI sound source that responded to the movements of the child. Results indicated increased engagement spans whilst in the environment, atypical communicative responses in the case of the ASD child; highly expressive gestural movements that vividly illustrated aspects of the theory of movement as postulated by Laban (1963) and Laban and Lawrence (1974); and increased vocal utterances.

A Study on the Audiovisual Environment and Theatrical Culture in Kyeong-Sung(京城) City in 1930's

A Study on the Audiovisual Environment and Theatrical Culture in Kyeong-Sung(京城) City in 1930's

The Development of an Instrument for a Technology-integrated Science Learning Environment

This study developed, validated, and utilized the Technology Integrated Classroom Inventory (TICI) to examine technology-integrated science learning environments as perceived by secondary school students and teachers. Using technology-oriented classroom climate instruments and considering the science classroom’s characteristics, TICI was developed. More than 1,100 seventh through ninth grade science students validated the instrument, revealing eight scales: technological enrichment, inquiry learning, equity and friendliness, student cohesiveness, understanding and encouragement, competition and efficacy, audiovisual environment, and order, with alpha reliabilities ranging between 0.69 and 0.91 (0.93 for the entire questionnaire). In measuring actual and preferred learning environments, TICI results indicated that both students and teachers ranked equity and friendliness highest. The largest actual–preferred discrepancy was order (students) and inquiry learning (teachers). TICI offers additional utilities for technology-enriched science leaning environments.