Virtual Sounds Research Articles

Contribution of machine learning and physics-based sound simulations for the optimization of brass instruments

Sound simulations by physical model are interesting to transcribe the physics underlying the functioning of a musical instrument. These simulations make it possible to create virtual sounds with a mode of operation representative of the interaction musician/instrument (driving a sound by the causes that create it). The work consists in studying the contribution of machine learning (ML) methods in the optimization of a musical instrument. The application concerns the brass instruments (trumpets). From a training set of virtual instruments, ML methods are trained to model the intonation (represented by the equivalent fundamental pitch) and the ease of emission (represented by the threshold pressure obtained by linear stability analysis) of sounds, according to the modal parameters of the input impedance. With the ML models, an optimization with genetic algorithms is next carried out to improve the intonation and the ease of emission. The last stage of the approach consists in the optimization of the bore of the instrument (optimization of the shape of the leadpipe). The results show that the approach gives promising results: optimal instruments are actually efficient when re-simulated with the physical model. The manufacturing of an optimal instrument is in progress to confirm the validity of the approach

KANDINSKY ON COLORS AND THE VIRTUAL OBJECTLESS VIBRATIONS

If we accept that Kandinsky developed a systematic theory of the fundamentals of painting, we must ask what is the central concept underlying this attempt. This paper argues for the thesis that objectless vibration plays a central role in the reconstruction proposed Kandinsky’s first book, On the Spiritual in Art. This kind of vibration includes as a virtual field both shapes, sounds and colors. All these “fall” in an organized way from the virtual vibrations, and the purpose of abstract painting is to lift the mind beyond the specific distinctions of the visible world of objects to the abstract level of primordial vibrations. The article examines the origin, functions, and the effect this concept has on color theory.

Proximal auditory AR rehabilitation: system integration and wellness applications-from hearing support up toward vestibular rehabilitation.

We present a new rehabilitation system based on novel principles, which consists of an auditory augmented reality (AR) headset we originated. The auditory AR headset, which does not cover both ears, allows users to hear both Real and Virtual environmental sounds at the same time. It can also be used in combination with Hearing Aids. We have studied a system to support hearing-impaired people and conducted a test evaluation. The system was able to provide convenience akin to "reading glasses for sound" to those who had mild hearing disabilities. Furthermore, by combining the system with surrounding speakers, a completely novel virtual auditory illusion was created in which the sound image jumps into the ear and runs away. We name this "proximal auditory AR (PAAR)" system. This system directly affects the unconscious level of reflexes for maintaining a standing position and can generate very subtle body motion disturbance. Using this system, we can modulate the standing posture and observe the autonomic nerve system's ability to subliminally compensate for the disturbance, using a stabilometer that measures body sways by center of pressure (COP). We observed a significant difference in the declination of COP only when using the PAAR, which is combined with array speakers and the auditory AR headphone, compared using a conventional closed-type and a bone-conduction headphone. By analyzing such big data of physical movement through machine learning, we expect to realize new systems for diagnosis, rehabilitation, function maintenance, and fall prevention.

Virtual Sound Image Reconstruction Method for Multi-objective Optimization of Folk Music Based on Evolutionary Algorithm

At present, the exchanges in various fields such as culture, economy, and politics are becoming more and more close in the world. From the perspective of the relationship between national music and cultural development, the development of national music has also received more and more attention, and it has become an inevitable trend in the development of today's era. The purpose of this paper was to perform virtual reconstruction of the sound image of folk music through multiple objective optimizations, and to model the virtual sound image of folk music as a multi-objective optimization problem. According to the research on sound image positioning, the relevant noise factors were removed, so as to achieve the playback of ethnic music that enhanced the surround effect and visual enjoyment. The evolutionary algorithm in this paper was mainly based on the multi-objective optimized sound image localization technology. For the music virtual sound image, an improved FCM algorithm and an evolutionary multi-objective optimization algorithm combining local and non-local information of the sound image were proposed, respectively. Through the analysis of the traditional sound image algorithm method, the accuracy of the sound image localization on the horizontal plane could be effectively improved. After the conversion, the audience could feel the better stereo image and surround feeling of the folk music. Through experimental analysis, it can be seen that the system can not only perform virtual conversion of audio-visual signals of different frequencies, but also provide data for different audio playback systems. The feature point registration error is low, and the reconstruction effect is good, especially the random delay processing in the range of 0∼20m, and the performance is better than the traditional method. Finally, virtual left and right surround sound image signals were obtained, which effectively improved the three-dimensional surround feeling of folk music.

BatSight: A Navigation Game to Map Environmental Information into Audio Cues

By using sound to visualize the real-world environment, audio games can offer a completely new game experience. In this paper, we study how audio cues can be visualized in computer games to enhance the navigation skills of players. To this end, we propose an audio game in which blindfolded players move through a physical maze with the help of audio cues. To realize this game, a sonar headset is designed and built based on ultrasonic sensors, which maps the external environment features into musical sounds. The game is played in a real-world environment and uses an augmented reality approach as it adds virtual sounds to the real world. The gaming environment of this game is a physical maze, where the blindfolded players enter the maze while wearing the sonar headset. A user study was conducted to evaluate the effect of using different sound mapping techniques on navigation performance and playing experience in the game proposed in this paper. The results show that producing musical sound can lead to better navigation performance, game experience, and immersion in players.

Control of Electric Wheelchair by Brain‐Computer Interface Using Mixed Reality and Virtual Sound Source

AbstractA brain‐computer interface for operating an electric wheelchair (wheelchair BCI) has been studied to support independent living for physically disabled patients. The current wheelchair BCI systems are facing the problem that the wheelchair cannot be moved or rotated to an arbitrary place or direction in a single measurement. To solve this problem, we developed a BCI that uses audiovisual stimuli based on mixed‐reality (MR) and virtual sound sources. During the online analysis, six out of seven participants were able to move or rotate the wheelchair to the target location or direction at a rate higher than the chance level. Thus, our results indicate that audiovisual stimulation with MR and virtual sound sources may be useful for intuitive wheelchair operation. © 2024 Institute of Electrical Engineer of Japan and Wiley Periodicals LLC.

Comparing the Auditory Distance and Externalization of Virtual Sound Sources Simulated Using Nonindividualized Stimuli.

When reproducing sounds over headphones, the simulated source can be externalized (i.e., perceived outside the head) or internalized (i.e., perceived within the head). Is it because it is perceived as more or less distant? To investigate this question, 18 participants evaluated distance and externalization for three types of sound (speech, piano, helicopter) in 27 conditions using nonindividualized stimuli. Distance and externalization ratings were significantly correlated across conditions and listeners, and when averaged across listeners or conditions. However, they were also decoupled in some circumstances: (1) Sound type had different effects on distance and externalization: the helicopter was evaluated as more distant, while speech was judged as less externalized. (2) Distance estimations increased with simulated distances even for stimuli judged as internalized. (3) Diotic reverberation influenced distance but not externalization. Overall, a source was not rated as externalized as soon as and only if its perceived distance exceeded a threshold (e.g., the head radius). These results suggest that distance and externalization are correlated but might not be aspects of a single perceptual continuum. In particular, a virtual source might be judged as both internalized and with a distance. Hence, it could be important to avoid using a scale related to distance when evaluating externalization.

Effects of virtual reality and nature sounds on pain and anxiety during hysterosalpingography: a randomized controlled trial.

The objective of this study was to determine the effects of listening to nature sounds alone and virtual reality plus listening to nature sounds on pain and anxiety in hysterosalpingography. This three-arm parallel randomized controlled trial included 135 (45 in each group) women who underwent hysterosalpingography in Turkey. The virtual reality+nature sounds group viewed a nature video with virtual reality glasses and listened to nature sounds during hysterosalpingography, whereas the nature sounds group only listened to nature sounds. The control group received only routine care. During hysterosalpingography, women in virtual reality+nature sounds group experienced less pain than those in control group (p=0.009). After hysterosalpingography, pain levels were lower in both virtual reality+nature sounds group and nature sounds group than in control group (p=0.000 and p=0.000, respectively), anxiety levels were lower in virtual reality+nature sounds group than in nature sounds group and control group (p=0.018 and p=0.000, respectively), and anxiety levels were lower in nature sounds group than in control group (p=0.013). Virtual reality with nature content plus listening to nature sounds and only listening to nature sounds are effective in reducing pain and anxiety related to hysterosalpingography procedures in women. Compared with only listening to nature sounds, virtual reality plus listening to nature sounds further reduced hysterosalpingography-related pain and anxiety.

Room-aware portable Auditory Augmented Reality: Real-time spatial audio generation with geometric data analysis

This study proposes an innovative and practical approach to spatial acoustic rendering for Auditory Augmented Reality (AAR) in small room usage. AAR aims to enhance user experiences by overlaying spatially appropriate sounds in the real environment, creating the illusion that virtual sounds coexist in the physical world. However, current technological constraints in hardware and AAR rendering present challenges in creating a truly immersive sensation of “being there.” To address these challenges, we present a dynamic processing framework for a typical mobile AAR setting. This framework combines a game-audio engine with 3-D visual computation both specialized for real-time processing, providing a plausible spatial impression with lightweight computation. The process involves capturing room information by the Light Detection And Ranging (LiDAR) sensor mounted on the device, followed by estimating its dominant acoustic materials. The generated data will be used to compute dynamic early reflections and late reverberation. Early reflections are calculated up to the fourth reflection using the image source method, considering the acoustic materials. Late reverberation is generated using the Feedback Delay Network (FDN), with room frequency characteristics and spatial directivity varying relative to the listener's position. All components will be represented in fifth-order Ambisonics and binauralized with head-tracking. Notably, this dynamic rendering method could be light enough to handle all processing on a single mobile device.

Proposal of a new method for learning of diesel generator sounds and detecting abnormal sounds using an unsupervised deep learning algorithm

This study is to find a method to learn engine sound after the start-up of a diesel generator installed in nuclear power plant with an unsupervised deep learning algorithm (CNN autoencoder) and a new method to predict the failure of a diesel generator using it. In order to learn the sound of a diesel generator with a deep learning algorithm, sound data recorded before and after the start-up of two diesel generators was used. The sound data of 20 min and 2 h were cut into 7 s, and the split sound was converted into a spectrogram image. 1200 and 7200 spectrogram images were created from sound data of 20 min and 2 h, respectively. Using two different deep learning algorithms (CNN autoencoder and binary classification), it was investigated whether the diesel generator post-start sounds were learned as normal. It was possible to accurately determine the post-start sounds as normal and the pre-start sounds as abnormal. It was also confirmed that the deep learning algorithm could detect the virtual abnormal sounds created by mixing the unusual sounds with the post-start sounds. This study showed that the unsupervised anomaly detection algorithm has a good accuracy increased about 3% with comparing to the binary classification algorithm.

Spatial up-sampling of HRTF sets using generative adversarial networks: A pilot study

Headphones-based spatial audio simulations rely on Head-related Transfer Functions (HRTFs) in order to reconstruct the sound field at the entrance of the listener’s ears. A HRTF is strongly dependent on the listener’s specific anatomical structures, and it has been shown that virtual sounds recreated with someone else’s HRTF result in worse localisation accuracy, as well as altering other subjective measures such as externalisation and realism. Acoustic measurements of the filtering effects generated by ears, head and torso has proven to be one of the most reliable ways to obtain a personalised HRTF. However this requires a dedicated and expensive setup, and is time-intensive. In order to simplify the measurement setup, thereby improving the scalability of the process, we are exploring strategies to reduce the number of acoustic measurements without degrading the spatial resolution of the HRTF. Traditionally, spatial up-sampling of HRTF sets is achieved through barycentric interpolation or by employing the spherical harmonics framework. However, such methods often perform poorly when the provided HRTF data is spatially very sparse. This work investigates the use of generative adversarial networks (GANs) to tackle the up-sampling problem, offering an initial insight about the suitability of this technique. Numerical evaluations based on spectral magnitude error and perceptual model outputs are presented on single spatial dimensions, therefore considering sources positioned only in one of the three main planes: Horizontal, median, and frontal. Results suggest that traditional HRTF interpolation methods perform better than the proposed GAN-based one when the distance between measurements is smaller than 90°, but for the sparsest conditions (i.e., one measurement every 120°–180°), the proposed approach outperforms the others.

Crystal sounds in contemporary film and TV

At the core of Deleuze’s theorization of the time-image is the ‘crystal image’ that appears as a direct presentation of time in an indiscernibility of the actual and the present from the virtual in various forms. Lending his ears to cinema after the Second World War, Deleuze coins the term ‘sound-crystal’, but he neither sufficiently engages with it nor thinks about it beyond music. Turning to film sound scholarship or film philosophy is futile in a similar fashion. To address these gaps in Deleuze, film philosophy and sound in cinema, I listen carefully to a number of contemporary films and TV shows in which sound crystallization splits time in an indiscernibility of the perceived actual from the virtual sounds of recollections, dreams, hallucinations, fantasies and worlds. I argue that crystallization of sounds avoids an anthropocentric approach to understanding the relation between a sound and its source which cannot be claimed to ‘voice’ it.

On the development of virtual reality tools to aid in the design interior soundscapes

Can simulations of interiors spaces (visual and acoustic) be used to improve designs of spaces within buildings? The first step in the simulation of the sounds in a space is to characterize typical sound sources and their locations within the space. The next step is to characterize the paths from the sources to the user, based on geometry and materials used. Combining the contributions from all sources, is it possible to create sounds that are similar to sounds in the real environment? How close do simulated and measured sounds need to be to evoke similar emotional responses from users? The same questions arise for the visual experience and for interactions between the visual and acoustic experiences. Are there metrics that can be used to guide improvements in the virtual environment to match the real experience in the built environments? To start to answer these questions, a sound simulation program using an image source modeling approach was developed for spaces in an existing building. The impulse responses from the image source model and the measured impulse responses are compared, as are simulated and measured sounds. Room acoustics and sound quality metrics for the virtual and real sounds are also compared.

The Influence of Binaural Room Impulse Responses on Externalization in Virtual Reality Scenarios

A headphone-based virtual sound image can not be perceived as perfectly externalized if the acoustic of the synthesized room does not match that of the real listening environment. This effect has been well explored and is known as the room divergence effect (RDE). The RDE is important for perceived externalization of virtual sounds if listeners are aware of the room-related auditory information provided by the listening environment. In the case of virtual reality (VR) applications, users get a visual impression of the virtual room, but may not be aware of the auditory information of this room. It is unknown whether the acoustic congruence between the synthesized (binaurally rendered) room and the visual-only virtual listening environment is important for externalization. VR-based psychoacoustic experiments were performed and the results reveal that perceived externalization of virtual sounds depends on listeners’ expectations of the acoustic of the visual-only virtual room. The virtual sound images can be perceived as externalized, although there is an acoustic divergence between the binaurally synthesized room and the visual-only virtual listening environment. However, the “correct” room information in binaural sounds may lead to degraded externalization if the acoustic properties of the room do not match listeners’ expectations.

The Use of a Virtual Simulator for Practicing the Skills of Auscultation of Sounds of the Lungs of Children in the Formation of Clinical Thinking

Auscultation skills are some of the most important basic methods of patient examination. The educational process of teaching auscultation is complex in itself, and in the context of the pandemic of the new coronavirus infection and the widespread epidemiological limitations, the complexity of learning is multiplied, which requires alternative teaching methods. The Virtual Lung Sound Auscultation Simulator creates a learning platform for auscultation by delivering real pre-recorded cardiopulmonary sounds on the virtual chest of a child.

A Graphical-User-Interface-Based Azimuth-Collection Method in Autonomous Auditory Localization of Real and Virtual Sound Sources.

Auditory localization of spatial sound sources is an important life skill for human beings. For the practical application-oriented measurement of auditory localization ability, the preference is a compromise among (i) data accuracy, (ii) the maneuverability of collecting directions, and (iii) the cost of hardware and software. The graphical user interface (GUI)-based sound-localization experimental platform proposed here (i) is cheap, (ii) can be operated autonomously by the listener, (iii) can store results online, and (iv) supports real or virtual sound sources. To evaluate the accuracy of this method, by using 12 loudspeakers arranged in equal azimuthal intervals of 30° in the horizontal plane, three groups of azimuthal localization experiments are conducted in the horizontal plane with subjects with normal hearing. In these experiments, the azimuths are reported using (i) an assistant, (ii) a motion tracker, or (iii) the newly designed GUI-based method. All three groups of results show that the localization errors are mostly within 5-12°, which is consistent with previous results from different localization experiments. Finally, the stimulus of virtual sound sources is integrated into the GUI-based experimental platform. The results with the virtual sources suggest that using individualized head-related transfer functions can achieve better performance in spatial sound source localization, which is consistent with previous conclusions and further validates the reliability of this experimental platform.

Virtual Sound Field of the Roman Theatre of Malaca

In Hispania (present-day Spain and Portugal), there are 25 structures documented of classical Roman open-air theatres, of which 10 are in the south, in the Roman Baetica (Andalusia). The Baetica embraced the progress of urbanisation in the time of the Roman emperor Augustus, where theatres, built in stone, were the foci of entertainment, performance, and propaganda of the empire. The Roman theatre in Malaga presents the archaeological remains of the main vestige of the Roman Malaca. It is located in the historical centre of the city, at the foot of the hill of the Muslim Alcazaba and was discovered in 1952. It is a medium-sized theatre whose design corresponds to a mixed construction that combines making use of the hillside for the terraces, in the manner of Greek theatres, with a major construction where rock is non-existent, thereby creating the necessary space for the stands. In this paper, the production process, adjustment, and validation of the 3D model of the theatre are analysed for the creation of a numerical predictive model of its sound field. Acoustic properties of the venue are examined and the effect of the Muslim Alcazaba and the hillside on the various acoustic descriptors is analysed. The results highlight the influence of this large stone surface mainly on the time decay parameters.

Towards Multisensory Perception: Modeling and Rendering Sounds of Tool-Surface Interactions.

Touch-produced sounds in tool-surface interactions convey rich information about textured surface properties and provide direct feedback about how users interact with the surface. This article presents a statistical learning-based approach for modeling and rendering touch-produced sounds in real time. We apply a data-driven modeling method, which recreates highly realistic sounds using audio signals recorded from unconstrained tool-surface interactions. The recorded sound is segmented, and each segment is labeled with the average velocity during that time. We model each segment with wavelet tree models using a moving window approach. Each window is analyzed by fast wavelet transform and is then broken down into a tree structure. During rendering, we use the user's current velocity to select tree models and synthesize new sounds by breadth-first search and inverse wavelet transform. We conducted a user study to evaluate the realism of our virtual sounds and their effect on human's perception of the texture dimensions in the presence of simultaneous real haptic cues. The results showed that in the presence of haptic cues, the virtual sound can more completely capture the texture's roughness and hardness than haptic cues alone. However, the perception on slipperiness depended mainly on touch.

Audio Legends: Investigating Sonic Interaction in an Augmented Reality Audio Game

Augmented Reality Audio Games (ARAG) enrich the physical world with virtual sounds to express their content and mechanics. Existing ARAG implementations have focused on exploring the surroundings and navigating to virtual sound sources as the main mode of interaction. This paper suggests that gestural activity with a handheld device can realize complex modes of sonic interaction in the augmented environment, resulting in an enhanced immersive game experience. The ARAG “Audio Legends” was designed and tested to evaluate the usability and immersion of a system featuring an exploration phase based on auditory navigation, as well as an action phase, in which players aim at virtual sonic targets and wave the device to hit them or hold the device to block them. The results of the experiment provide evidence that players are easily accustomed to auditory navigation and that gestural sonic interaction is perceived as difficult, yet this does not affect negatively the system’s usability and players’ immersion. Findings also include indications that elements, such as sound design, the synchronization of sound and gesture, the fidelity of audio augmentation, and environmental conditions, also affect significantly the game experience, whereas background factors, such as age, sex, and game or music experience, do not have any critical impact.

Virtual auditory aperture passability.

Two experiments investigated (1) the ability of individuals to perceive the passability of apertures that are constructed using two virtual sounds sources and (2) the nature of the perceptual information that is used when determining passability in such a way. In the first experiment, participants judged whether they could successfully walk between two sound sources, heard through headphones, without turning their shoulders. We hypothesized that judgements would be accurate and driven by the detection of a proposed informational variable that relates head rotation, forward locomotion and aperture width. To test this hypothesis, we used motion tracking and a gain manipulation to alter apparent head rotation relative to virtual sound source positions and evaluated the effect on performance. Participants were able to accurately judge aperture passability based only on acoustic information. However, the gain manipulation did not show a significant influence on perceptual reports. The unexpected significant influence of lateral head movement on perceptual accuracy, however, does suggest that an alternative informational variable, based on lateral movement, may have been used. In the second experiment, a group of participants with wide shoulders was compared to a group with narrow shoulders on a similar task. Significant differences in minimally acceptable aperture width were found between the wide and narrow groups. When these aperture widths were scaled to the participants' shoulder widths, however, the differences were no longer present. These findings are consistent with previous studies investigating perception of passability and offer promising applications of virtual reality technology in the study of auditory perceptual abilities.