Stereophonic Research Articles

A Quality of Experience and Visual Attention Evaluation for 360° Videos with Non-spatial and Spatial Audio

This article presents the results of an empirical study that aimed to investigate the influence of various types of audio (spatial and non-spatial) on the user quality of experience (QoE) of and visual attention in 360° videos. The study compared the head pose, eye gaze, pupil dilations, heart rate, and subjective responses of 73 users who watched ten 360° videos with different sound configurations. The configurations evaluated were no sound; non-spatial (stereo) audio; and two spatial sound conditions (first- and third-order ambisonics). The videos covered various categories and presented both indoor and outdoor scenarios. The subjective responses were analyzed using an ANOVA (Analysis of Variance) to assess mean differences between sound conditions. Data visualization was also employed to enhance the interpretability of the results. The findings reveal diverse viewing patterns, physiological responses, and subjective experiences among users watching 360° videos with different sound conditions. Spatial audio, in particular third-order ambisonics, garnered heightened attention. This is evident in increased pupil dilation and heart rate. Furthermore, the presence of spatial audio led to more diverse head poses when sound sources were distributed across the scene. These findings have important implications for the development of effective techniques for optimizing processing, encoding, distributing, and rendering content in virtual reality (VR) and 360° videos with spatialized audio. These insights are also relevant in the creative realms of content design and enhancement. They provide valuable guidance on how spatial audio influences user attention, physiological responses, and overall subjective experiences. Understanding these dynamics can assist content creators and designers in crafting immersive experiences that leverage spatialized audio to captivate users, enhance engagement, and optimize the overall quality of VR and 360° video content. The dataset, scripts used for data collection, ffmpeg commands used for processing the videos, and the subjective questionnaire and its statistical analysis are publicly available.

EchoSee: An Assistive Mobile Application for Real-Time 3D Environment Reconstruction and Sonification Supporting Enhanced Navigation for People with Vision Impairments.

Improving the quality of life for people with vision impairments has been an important goal in the research and design of assistive devices for several decades. This paper seeks to further that goal by introducing a novel assistive technology platform that leverages real-time 3D spatial audio to promote safe and efficient navigation for people who are blind or visually impaired (PVI). The presented platform, EchoSee, uses modern 3D scanning technology on a mobile device to construct a live, digital 3D map of a user's environment as they move about their surroundings. Spatialized, virtual audio sources (i.e., virtual speakers) are dynamically placed within the digital 3D scan of the world, providing the navigator with a real-time 3D stereo audio "soundscape." The digital 3D map, and its resultant soundscape, are continuously updated as the user moves about their environment. The generated soundscape is played back through headphones connected to the navigator's device. This paper details (1) the underlying technical components and how they were integrated to produce the mobile application that generates a dynamic soundscape on a consumer mobile device, (2) a methodology for analyzing navigation performance with the application, (3) the design and execution of a user study investigating the effectiveness of the presented system, and (4) a discussion of the results of that study along with a proposed future study and possible improvements. Altogether, this paper presents a novel software platform aimed at assisting individuals with vision impairments to navigate and understand spaces safely, efficiently, and independently and the results of a feasibility study analyzing the viability of the approach.

Language in interaction: turn-taking patterns in conversations involving individuals with schizophrenia

Individuals with schizophrenia generally show difficulties in interpersonal communication. Linguistic analyses shed new light on speech atypicalities in schizophrenia. However, very little is known about conversational interaction management by these individuals. Moreover, the relationship between linguistic features, psychopathology, and patients’ subjectivity has received limited attention to date. We used a novel methodology to explore dyadic conversations involving 58 participants (29 individuals with schizophrenia and 29 control persons) and medical doctors. High-quality stereo recordings were obtained and used to quantify turn-taking patterns. We investigated psychopathological dimensions and subjective experiences using the Positive and Negative Syndrome Scale for Schizophrenia (PANSS), the Examination of Anomalous Self Experience scale (EASE), the Autism Rating Scale (ARS) and the Abnormal Bodily Phenomena questionnaire (ABPq). Different turn-taking patterns of both patients and interviewers characterised conversations involving individuals with schizophrenia. We observed higher levels of overlap and mutual silence in dialogues with the patients compared to dialogues with control persons. Mutual silence was associated with negative symptom severity; no dialogical feature was correlated with anomalous subjective experiences. Our findings suggest that individuals with schizophrenia display peculiar turn-taking behaviour, thereby enhancing our understanding of interactional coordination in schizophrenia.

Cross-modal generative model for visual-guided binaural stereo generation

Binaural stereo audio is recorded by imitating the way the human ear receives sound, which provides people with an immersive listening experience. Existing approaches leverage autoencoders and directly exploit visual spatial information to synthesize binaural stereo, resulting in a limited representation of visual guidance. For the first time, we propose a visually guided generative adversarial approach for generating binaural stereo audio from mono audio. Specifically, we develop a Stereo Audio Generation Model (SAGM), which utilizes shared spatio-temporal visual information to guide the generator and the discriminator to work separately. The shared visual information is updated alternately in the generative adversarial stage, allowing the generator and discriminator to deliver their respective guided knowledge while visually sharing. The proposed method learns bidirectional complementary visual information, which facilitates the expression of visual guidance in generation. In addition, spatial perception is a crucial attribute of binaural stereo audio, and thus the evaluation of stereo spatial perception is essential. However, previous metrics failed to measure the spatial perception of audio. To this end, a metric to measure the spatial perception of audio is proposed for the first time. The proposed metric is capable of measuring the magnitude and direction of spatial perception in the temporal dimension. Further, considering its function, it is feasible to utilize it instead of demanding user studies to some extent. The proposed method achieves state-of-the-art performance on 2 datasets and 5 evaluation metrics. Qualitative experiments and user studies demonstrate that the method generates space-realistic stereo audio.

Using deep learning for recreating binaural audio

In recent years, there has been an increase in research around generating spatialized audio using a mono audio signal. Methods like using neural networks which combine image segmentation with object location for adding back the spatial qualities are often developed. Instead, our project focuses on taking arbitrary mono input sound and an input angle, and outputting spatial stereo audio of the input sound with the directionality of the angle. This is different from current implementations as it is a simpler approach to what spatial audio generation is, and it allows for the use in the model in new areas. Using a binaural microphone and a custom-made anechoic chamber, 120 hours of labelled binaural audio was recorded for use in our model. The audio consists of frequency sweeps, pink noise, and phonetically balanced speech. Our method is to predict a complex short-time Fourier transform mask which will contain the phase and amplitude within it. The model is an autoencoder based on the U-NET model, which is applied to the mono input before being compared to the labelled data for training. With this simpler approach, we hope to make spatial audio more accessible for a variety of applications.

Quantifying active brain areas at spatial hearing process using Electroencephalography (EEG) source localization approach

The human ability for spatial hearing encourages the development of spatial audio technology to support many human activities. Spatial audio mimics the real-life sound and provides a more immersive hearing experience as if the listener were present in the environment where the sound source was recorded. Recently, research on how spatial hearing processes are encoded in the brain has begun to be developed. As a modality for brain activity measurement that is non-invasive and has a high temporal resolution, Electroencephalography (EEG) is suitable for studying brain responses to moving sound stimuli. This study compared brain activity in hearing stereo and spatial audio stimulus. Through measurements using 21 EEG electrodes on the scalp, the results showed an increase in the mean PSD for all Theta, Alpha, and Beta waves in the spatial audio stimulus compared to stereo audio. The most significant increase in the mean PSD occurred in the Beta wave of 93.8008 µV2/Hz. Furthermore, the source localization approach with Independent Component Analysis (ICA) and Low-resolution Brain Electromagnetic Tomography (LORETA) method was used to estimate and quantify the active brain area of this process. The results showed an activation of the non-auditory cortex when the subject was hearing spatial audio stimulus. By using Paired T-Test of current density for both hearing processes, the results showed there was no significant difference (p > 0.05) in Brodmann area (BA) 41 (Primary Auditory Cortex) and BA 42 (Secondary Auditory Cortex). Meanwhile, for the spatial audio hearing process, there was a significant difference (p < 0.05) in BA 6 (Premotor Cortex), which is related to spatial orientation, and BA 9 (Dorsolateral Prefrontal Cortex), which is associated with executive functions, including working memory and selective attention. This study offers potential insights into spatial hearing research and immersive audio production.

Assistive mobile application for real-time 3D spatial audio soundscapes toward improving safe and independent navigation

Assistive technologies are used in a variety of contexts to improve the quality of life for individuals that may have one or more vision impairments. This paper describes a novel assistive technology platform that utilizes real-time 3D spatial audio to aid its users in safe and efficient navigation. This platform leverages modern 3D scanning technology on a mobile device to digitally construct a live 3D map of a user's surroundings as they move about their space. Within the digital 3D scan of the world, spatialized, virtual audio sources (i.e., speakers) provide the navigator with a real-time 3D stereo audio "soundscape." As the user moves about the world, the digital 3D map, and its resultant soundscape, are continuously updated and played back through headphones connected to the navigator's device. This paper details (1) the underlying technical components and how they were integrated to produce the mobile application that generates a dynamic soundscape on a consumer mobile device and (2) a methodology for analyzing the usage of the application. It is the aim of this application to assist individuals with vision impairments to navigate and understand spaces safely, efficiently, and independently.

Wrapped into sound: Development of the Immersive Music Experience Inventory (IMEI).

Although virtual reality, video entertainment, and computer games are dependent on the three-dimensional reproduction of sound (including front, rear, and height channels), it remains unclear whether 3D-audio formats actually intensify the emotional listening experience. There is currently no valid inventory for the objective measurement of immersive listening experiences resulting from audio playback formats with increasing degrees of immersion (from mono to stereo, 5.1, and 3D). The development of the Immersive Music Experience Inventory (IMEI) could close this gap. An initial item list (N = 25) was derived from studies in virtual reality and spatial audio, supplemented by researcher-developed items and items extracted from historical descriptions. Psychometric evaluation was conducted by an online study (N = 222 valid cases). The N = 222 Participants (female = 112, mean age = 38.6) were recruited via mailing lists (n = 34) and via a panel provider (n = 188). Based on controlled headphone playback, participants listened to four songs/pieces, each in the three formats of mono, stereo, and binaural 3D audio. The latent construct “immersive listening experience” was determined by probabilistic test theory (item response theory, IRT) and by means of the many-facet Rasch measurement (MFRM). As a result, the specified MFRM model showed good model fit (62.69% of explained variance). The final one-dimensional inventory consists of 10 items and will be made available in English and German.

Fast non-uniform searching strategy for ambient phase estimation in stereo recordings with sparse primary components

A stereo recording can be considered to consist of primary and ambient components in both left and right channels. Decomposing a stereo recording into primary and ambient components is a crucial step in upmixing to retain its spatial information. The state-of-the-art algorithm to carry out the primary ambient extraction (PAE), namely the ambient phase estimation with a sparsity constraint (APES), utilized the sparsity of the primary components in the time–frequency domain to formulate the ambient phase estimation as a non-convex optimization problem. Hence, the discrete searching method was adopted, resulting in a computationally complicated solving process. In this paper, a fast non-uniform searching strategy is proposed to improve the efficiency of the APES, resulting in the ambient phase estimation with a sparsity constraint and non-uniform searching (APEN). Objective and subjective results validate that the extraction error of the APEN is almost the same as that of the APES, while the algorithm complexity of the APEN is reduced to one third that of the APES.

The development of sex differences in song in a tropical duetting wren.

The study of song development has focused on temperate zone birds in which typically only males sing. In the bay wren, Cantorchilus nigricapillus, both sexes sing, performing precisely timed, female-initiated duets in which birds alternate sex-specific song phrases. We investigated the development of these sex differences by collecting bay wren eggs and nestlings, and hand-raising them in individual acoustic isolation chambers. Each bird was tutored with either monophonic or stereophonic recordings of bay wren duets or heard no song. As adults, each tutored bird sang repertoires of complete duets, singing both male and female phrases. In addition, some birds sang only the male or female part of some duets to which they were exposed. Mono-tutored birds showed no sex-specificity in these solo songs, whereas stereo-tutored birds only sang solos consistent with their sex. In addition, stereo-tutored birds acquired songs over a longer period than did mono-tutored birds, and stereo-tutored females showed more sex-specificity than did males during early song production. Finally, we observed that tutored and acoustically isolated birds of both sexes invented male-like songs, whereas only males invent songs in the wild. These results reveal the relative roles of environmental versus innate influences in the development of sex-specific song in this species.

Piano Music Transcriber

Sheet music is a transcribed or printed type of melodic documentation that portrays the rhythms, harmonies, pitches of music with the assistance of melodic images. Printed music is the most essential instrument for artists to learn and play music. An educated musician can, from this sheet, play an entire grand composition. However, making sheet music from scratch is a very long and intricate process. A lot of musical experience is required for making sheet music from music one hears and even experts tend to make mistakes. This makes it almost impossible for beginners who are unable to distinguish between different chords to play the music that they hear. We aim to create a system that would provide the user with the needed sheet music by taking the input from the user in the form of an audio mp3 file. The proposed system automatically generates sheet music directly from recorded songs. This system proposed is mainly composed of estimation of chords then converting it into MIDI file which is a computer understandable language, and eventually into sheet music. All stereo audio recordings are converted into mono which further resampled to 16 kHz. Music generation is also applied which gives a priming sequence to the model in the types of a MIDI file and then predicts the further notes to help musicians form further melody while the composition of music is taking place. Beginners and lesser experienced musicians also struggle with procuring sheet music for melodies and are limited to only what is available from other professional transcribers. For professional musicians, it helps to lay the foundations and automates a major part of the creation process. Professionals can further improve the automatically generated sheet music without the burden of starting from scratch

Diminishing Dreams

Diminishing Dreams

Anti-forensics of fake stereo audio using generative adversarial network

Anti-forensics of fake stereo audio using generative adversarial network

Passive acoustic monitoring of sperm whales and anthropogenic noise using stereophonic recordings in the Mediterranean Sea, North West Pelagos Sanctuary

A total of 147 days spread over 4 years were recorded by a stereophonic sonobuoy set up in the Mediterranean sea, near the coast of Toulon, south of France. These recordings were analyzed in the scope of studying sperm whales (Physeter macrocephalus) and the impact anthropic noises may have on this species. With the use of a novel approach, which combines the use of a stereophonic antenna with a neural network, 226 sperm whales’ passages have been automatically detected in an effective range of 32 km. This dataset was then used to analyze the sperm whales’ abundance, the background noise, the influence of the background noise on the acoustic presence, and the animals’ size. The results show that sperm whales are present all year round in groups of 1–9 individuals, especially during the daytime. The estimated density is 1.69 whales/1000 km^2. Animals were also less frequent during periods with an increased background noise due to ferries. The animal size distribution revealed the recorded sperm whales were distributed in length from about 7 to 15.5 m, and lonely whales are larger, while groups of two are composed of juvenile and mid-sized animals.

Front Cover Picture: Stereoselective Cyclopropanation of Boron Dipyrromethene (BODIPY) Derivatives by an Organocascade Reaction (Adv. Synth. Catal. 5/2022)

The front cover picture illustrates the stereoselective synthesis of BODIPY molecules, in connection to the stereo audio player with the inscription “stereo bodipy.” The BODIPY molecule is inserted by a robot symbolizing the simplicity of preparation of such molecules. The synthesis of enantiopure chiral BODIPY molecules is of importance due to their intrinsic properties as fluorophores that could be used as probes for molecular sensing. The cover image was created by Tomáš Belloň. Details can be found in the Research Article by Veselý and co-workers (V. Dočekal, T. Koberová, J. Hrabovský, A. Vopálenská, R. Gyepes, I. Císařová, R. Rios, J. Veselý, Adv. Synth. Catal. 2022, 364, 930–937; DOI: 10.1002/adsc.202101286).

Autonomous Technology for 2.1 Channel Audio Systems

During the COVID-19 pandemic, smart home requirements have shifted toward entertainment at home. The purpose of this research project was therefore to develop a robotic audio system for home automation. High-end audio systems normally refer to multichannel home theaters. Although multichannel audio systems enable people to enjoy surround sound as they do at the cinema, stereo audio systems have been popularly used since the 1980s. The major shortcoming of a stereo audio system is its narrow listening area. If listeners are out of the area, the system has difficulty providing a stable sound field. This is because of the head-shadow effect blocking the high-frequency sound. The proposed system, by integrating computer vision and robotics, can track the head movement of a user and adjust the directions of loudspeakers, thereby helping the sound wave travel through the air. Unlike previous studies, in which only a diminutive scenario was built, in this work, the idea was applied to a commercial 2.1 audio system, and listening tests were conducted. The theory and the simulation coincide with the experimental results. The approximate rate of audio quality improvement is 31%. The experimental results are encouraging, especially for high-pitched music.

Self-Supervised Learning for Audio-Visual Relationships of Videos With Stereo Sounds

Learning cross-modal features is an essential task for many multimedia applications such as sound localization, audio-visual alignment, and image/audio retrieval. Most existing methods mainly focus on the semantic correspondence between videos and monaural sounds, and spatial information of sound sources has not been considered. However, sound locations are critical for understanding the sound environment. To this end, it is necessary to acquire cross-modal features that reflect the semantic and spatial relationship between videos and sounds. A video with stereo sound, which has become commonly used, provides the direction of arrival of each sound source in addition to the category information. This indicates its potential to acquire a desired cross-modal feature space. In this paper, we propose a novel self-supervised approach to learn a cross-modal feature representation that captures both the category and location of each sound source using stereo sound as input. For a set of unlabeled videos, the proposed method generates three kinds of audio-visual pairs: 1) perfectly matched pairs from the same video, 2) pairs from the same video but with the flipped stereo sound, and 3) pairs from a different video. The cross-modal feature encoder of the proposed method is trained on triplet loss to reflect the relationship between these three pairs (1 > 2 > 3). We apply this method to cross-modal image/audio retrieval. Compared with previous audio-visual pretext tasks, the proposed method shows significant improvement in both real and synthetic datasets.

Section Meetings

The Section Meeting was held on 9 December at the WCMH Television transmission plant in Columbus, in conjunction with the local Society of Broadcast Engineers-Chapter 52. Fifty members and guests were shown a chronological timeline of the events leading up to and including the completion of their transmitter installation by Comark field staff and WCMH engineers. Their new state-of-the-art transmitter is now on the air with ATSC 1.0 VSB signaling but can easily be upgraded to ATSC 3.0, which includes the required orthogonal frequency-division multiplexing (OFDM) signaling in less than half-day. Ken McCrimmon, WCMH’s chief engineer, began his slide presentation by showing photographs of the various phases involved in the project’s successful installation and completion. He then listed the many broadcast engineering firsts presented to the Columbus TV market over the years from their former call letters WLW-C (channels 3 then 4) and later WCMH, including: first TV station on the air in Columbus, first NBC-compatible color broadcast on 1 January 1954 with the Tournament of Roses Parade, first WLW regional color broadcasts in August 1957, first local origination color broadcasts in 1960, and first stereo audio NBC passthrough and local origination broadcasts in July 1985, with the MLB All Star baseball game coverage from NBC. He also stated that with the installation of the TV station’s new Comark broadcast transmitter, WCMH is now future-proof for many years!

Unobtrusive Pedestrian Identification by Leveraging Footstep Sounds with Replay Resistance

The ability to identify pedestrians unobtrusively is essential for smart buildings to provide customized environments, energy saving, health monitoring and security-enhanced services. In this paper, we present an unobtrusive pedestrian identification system by passively listening to people's walking sounds. The proposed acoustic system can be easily integrated with the widely deployed voice assistant devices while providing the context awareness ability. This work focuses on two major tasks. Firstly, we address the challenge of recognizing footstep sounds in complex indoor scenarios by exploiting deep learning and the advanced stereo recording technology that is available on most voice assistant devices. We develop a Convolutional Neural Network-based algorithm and the footstep sound-oriented signal processing schemes to identify users by their footstep sounds accurately. Secondly, we design a "live" footstep detection approach to defend against replay attacks. By deriving the novel inter-footstep and intra-footstep characteristics, we distinguish live footstep sounds from the machine speaker's replay sounds based on their spatial variances. The system is evaluated under normal scenarios, traditional replay attacks and the advanced replays, which are designed to forge footstep sounds both acoustically and spatially. Extensive experiments show that our system identifies people with up to 94.9% accuracy in one footstep and shields 100% traditional replay attacks and up to 99% advanced replay attacks.

Low-cost gradient amplifiers for small MRI systems

We describe low-cost gradient amplifiers for small MRI systems, such as those based on small/medium permanent magnets. The requirements for MRI gradient amplifiers are quite similar to those of modestly priced audio stereo power amplifiers. Such amplifiers for sound service can be modified to be DC coupled, extending their response down to zero frequency, as needed for MRI gradient service. We describe such modifications to one unit (Samson Servo 600) and mention a commercially available modification of another. The 600 is capable of an output greater than ±8 A and ±60 V, much greater than our needs for a greenhouse MRI. Audio amplifiers can be used this way as controlled voltage (CV) gradient amplifiers, with acceptable performance because the gradient coils typically have short L/R time constants. Superior performance can be had by using a controlled current (CC) “front end”; the circuit of our simple front end is included. The complete 3-axis CC system costs about $1200.