Sound Synthesis Research Articles

Aeroacoustic modeling of blown-closed free reeds

We present ongoing work on the development of an aeroacoustic model of blown-closed free reeds used in a hand harmonium. Physics-based models of free reed instruments involve modeling of the oscillating free reed, the air-flow in the upstream region and around the reed, and the interaction between these two systems. The minimal model of free reeds by Millot and Baumann (2007) assumed the reed as a damped spring-mass system and approximated the air flow field by two discrete zones representing the flow near the reed and the upstream region respectively. Our recent work included the adaptation of the minimal model to match the physical setup and control parameters of the hand harmonium and the development of a distributed clamped-bar model of the free reed. These updates resulted in a closer agreement of the parameters for simulation and the observed physical variables. To address limitations of this model, we present a revised modeling of the air flow as a continuous potential flow and discuss implications for numerical stability, accuracy and real-time sound synthesis.

Physics-informed differentiable method for piano modeling

Numerical emulations of the piano have been a subject of study since the early days of sound synthesis. High-accuracy sound synthesis of acoustic instruments employs physical modeling techniques which aim to describe the system’s internal mechanism using mathematical formulations. Such physical approaches are system-specific and present significant challenges for tuning the system’s parameters. In addition, acoustic instruments such as the piano present nonlinear mechanisms that present significant computational challenges for solving associated partial differential equations required to generate synthetic sound. In a nonlinear context, the stability and efficiency of the numerical schemes when performing numerical simulations are not trivial, and models generally adopt simplifying assumptions and linearizations. Artificial neural networks can learn a complex system’s behaviors from data, and their application can be beneficial for modeling acoustic instruments. Artificial neural networks typically offer less flexibility regarding the variation of internal parameters for interactive applications, such as real-time sound synthesis. However, their integration with traditional signal processing frameworks can overcome this limitation. This article presents a method for piano sound synthesis informed by the physics of the instrument, combining deep learning with traditional digital signal processing techniques. The proposed model learns to synthesize the quasi-harmonic content of individual piano notes using physics-based formulas whose parameters are automatically estimated from real audio recordings. The model thus emulates the inharmonicity of the piano and the amplitude envelopes of the partials. It is capable of generalizing with good accuracy across different keys and velocities. Challenges persist in the high-frequency part of the spectrum, where the generation of partials is less accurate, especially at high-velocity values. The architecture of the proposed model permits low-latency implementation and has low computational complexity, paving the way for a novel approach to sound synthesis in interactive digital pianos that emulates specific acoustic instruments.

Anti-cancer effects of green synthesized gold nanoparticles using leaf extract of Annona muricata. L against squamous cell carcinoma cell line 15 through apoptotic pathway

Background: Oral cancer remains one of the most dreadful diseases in developing nations. Currently, there has been a rise in the prevalence of tongue squamous cell carcinoma (SCC), with a poor prognosis. The use of standard treatment approaches against oral cancer patients brings about several side effects. In recent years, nanomedicine has provided a versatile platform for developing new targeted therapeutic modalities. However, safety remains a concern in the synthesis of nanoparticles (NPs). Therefore, the present study aims to synthesize safer phytoconstituent-mediated gold NPs (AuNPs) utilizing leaf extracts of Annona muricata, where the biochemical components of the plant leaf act as the reducing and capping agents in the synthesis of NPs, and to evaluate its anti-cancer activity against SCC. Materials and Methods: In this in vitro experimental study, AuNPs were synthesized through an effective, simple, and ecologically sound green synthesis method. After characterization of these synthesized AuNPs, in vitro assays such as 3-(4, 5-dimethylthiazole2-yl)-2, 5-biphenyl tetrazolium bromide, wound healing, and clonogenic assays were carried out to investigate the anti-cancer potential of green synthesized AuNPs in the human tongue SCC cell line (SCC-15), and the possible mechanism of action was evaluated through gene and protein expression analysis of Bax, Bcl-2, and p53 genes. The results were expressed as mean ± standard deviation using Statistical Package for Social Sciences (SPSS) 20.0 software and Student's t-test was performed for experimental data. P ≤0.05 were considered statistically significant. Results: The in vitro assays demonstrated that the synthesized AuNPs are exhibiting anti-cancer activity by apoptosis of SCC-15 cells in a dose-dependent manner. Further, it also revealed a highly significant decrease in anti-apoptotic Bcl-2 gene expression, whereas pro-apoptotic genes p53 and Bax revealed a highly significant increase, which is statistically significant compared to the control (P < 0.05). Conclusion: Our findings demonstrated that the AuNPs synthesized from A. muricata leaf extract could act as a novel anticancer agent, particularly against SCC, after further scrutiny.

Vibrational Analysis of a Splash Cymbal by Experimental Measurements and Parametric CAD-FEM Simulations

The present study encompasses a thorough analysis of the vibrations in a splash musical cymbal. The analysis is performed using a hybrid methodology that combines experimental measurements with parametric computer-aided design and finite element method simulations. Experimental measurements, including electronic speckle pattern interferometry, and impulse response measurements are conducted. The interferometric measurements are used as a reference for the evaluation of finite element method modal analysis results. The modal damping ratio is calculated via the impulse response measurements and is adopted by the corresponding simulations. Two different approximations are employed for the computer-aided design and finite element method models: one using three-point arcs and the other using lines to describe the non-smooth curvature introduced during manufacturing finishing procedures. The numerical models employing the latter approximation exhibit better agreement with experimental results. The numerical results demonstrate that the cymbal geometrical characteristics, such as the non-smooth curvature and thickness, greatly affect the vibrational behavior of the percussion instrument. These results are of valuable importance for the development of vibroacoustic numerical models that will accurately simulate the sound synthesis of cymbals.

Retracted: Construction and Research of Guqin Sound Synthesis and Plucking Big Data Simulation Model Based on Computer Synthesis Technology

Retracted: Construction and Research of Guqin Sound Synthesis and Plucking Big Data Simulation Model Based on Computer Synthesis Technology

Neural electric bass guitar synthesis framework enabling attack-sustain-representation-based technique control

Musical instrument sound synthesis (MISS) often utilizes a text-to-speech framework because of its similarity to speech in terms of generating sounds from symbols. Moreover, a plucked string instrument, such as electric bass guitar (EBG), shares acoustical similarities with speech. We propose an attack-sustain (AS) representation of the playing technique to take advantage of this similarity. The AS representation treats the attack segment as an unvoiced consonant and the sustain segment as a voiced vowel. In addition, we propose a MISS framework for an EBG that can control its playing techniques: (1) we constructed a EBG sound database containing a rich set of playing techniques, (2) we developed a dynamic time warping and timbre conversion to align the sounds and AS labels, (3) we extend an existing MISS framework to control playing techniques using AS representation as control symbols. The experimental evaluation suggests that our AS representation effectively controls the playing techniques and improves the naturalness of the synthetic sound.

Auditory perception of the thickness of plates.

This study focuses on the auditory perception of plate thickness and investigates acoustic cues that evoke thickness in the context of sound synthesis. Three hypotheses are proposed and tested through a listening test, examining the influence of damping, nonlinear phenomena, and modal frequencies on the perceived thickness of sound sources. The stimuli are generated using the numerical resolution of the Föppl-von Kármán system. We confirm that increasing the overall damping leads to an increased perceived thickness. Additionally, the emergence of an energy cascade toward higher frequencies (characteristic of thin plates) for impacts of increasing intensity evokes a thinner object.

An Overview of Visual Sound Synthesis Generation Tasks Based on Deep Learning Networks

Visual sound synthesis (which refers to the process of recreating, as realistically as possible, the sound produced by the movements and actions of objects within a video, given specific conditions such as video content and accompanying text) is an important part of the composition of high-quality films at present. Most traditional methods of sound synthesis are based on the artificial creation of simulated props for sound effects synthesis, which is achieved by using various existing props and constructed scenes. However, traditional methods cannot meet specific conditions for sound effect synthesis and require large amounts of participant, material resources and time. It can take nearly ten hours to simulate realistic sound effects in a minute-long video. In this paper, we systematically summarize and consolidate current advances in deep learning in the field of visual sound synthesis, based on existing related papers. We focus on the exploration and development history of deep learning models for the task of visual sound synthesis, and classify detailed research methods and related dataset information based on their development characteristics. By analyzing the technical differences among various model approaches, we can summarize potential research directions in the field, thereby further promoting the rapid development and practical implementation of deep learning models in the video domain.

Probing Aesthetics Strategies for Robot Sound: Complexity and Materiality in Movement Sonification

This article presents three studies where we probe aesthetics strategies of sound produced by movement sonification of a Pepper robot by mapping its movements to sound models. We developed two sets of sound models. The first set was made by two sound models, a sawtooth-based one and another based on feedback chains, for investigating how the perception of synthesized robot sounds would depend on their design complexity. We implemented the second set of sound models for probing the “materiality” of sound made by a robot in motion. This set consisted of a sound synthesis based on an engine highlighting the robot’s internal mechanisms, a metallic sound synthesis highlighting the robot’s typical appearance, and a whoosh sound synthesis highlighting the movement. We conducted three studies. The first study explores how the first set of sound models can influence the perception of expressive gestures of a Pepper robot through an online survey. In the second study, we carried out an experiment in a museum installation with a Pepper robot presented in two scenarios: (1) while welcoming patrons into a restaurant and (2) while providing information to visitors in a shopping center. Finally, in the third study, we conducted an online survey with stimuli similar to those used in the second study. Our findings suggest that participants preferred more complex sound models for the sonification of robot movements. Concerning the materiality, participants liked better subtle sounds that blend well with the ambient sound (i.e., less distracting) and soundscapes in which sound sources can be identified. Also, sound preferences varied depending on the context in which participants experienced the robot-generated sounds (e.g., as a live museum installation vs. an online display).

RESEARCH OF THE PROCESS OF VISUAL ART TRANSMISSION IN MUSIC AND THE CREATION OF COLLECTIONS FOR PEOPLE WITH VISUAL IMPAIRMENTS

This article explores the creation of music through the automated generation of sounds from images. The developed automatic image sound generation method is based on the joint use of neural networks and light-music theory. Translating visual art into music using machine learning models can be used to make extensive museum collections accessible to the visually impaired by translating artworks from an inaccessible sensory modality (sight) to an accessible one (hearing). Studies of other audio-visual models have shown that previous research has focused on improving model performance with multimodal information, as well as improving the accessibility of visual information through audio presentation, so the work process consists of two parts. The result of the work of the first part of the algorithm for determining the tonality of a piece is a graphic annotation of the transformation of the graphic image into a musical series using all colour characteristics, which is transmitted to the input of the neural network. While researching sound synthesis methods, we considered and analysed the most popular ones: additive synthesis, FM synthesis, phase modulation, sampling, table-wave synthesis, linear-arithmetic synthesis, subtractive synthesis, and vector synthesis. Sampling was chosen to implement the system. This method gives the most realistic sound of instruments, which is an important characteristic. The second task of generating music from an image is performed by a recurrent neural network with a two-layer batch LSTM network with 512 hidden units in each LSTM cell, which assembles spectrograms from the input line of the image and converts it into an audio clip. Twenty-nine compositions of modern music were used to train the network. To test the network, we compiled a set of ten test images of different types (abstract images, landscapes, cities, and people) on which the original musical compositions were obtained and stored. In conclusion, it should be noted that the composition generated from abstract images is more pleasant to the ear than the generation from landscapes. In general, the overall impression of the generated compositions is positive. Keywords: recurrent neural network, light music theory, spectrogram, generation of compositions.

Using Spontaneous Speech Recognition as a Biomarker to Distinguish Dementia Patients

AbstractBackgroundDigital biomarkers such as individual speech recordings, are a cost‐effective tool for assessing cognitive status using linguistic and acoustic content. The purpose of this study is to investigate whether Natural Language Processing (NLP) methods and audio feature extraction can distinguish spontaneous speech outputs in dementia patients from a healthy population.Method240 audio samples and transcripts of clinician‐participant dialogues involving the Cookie‐Theft visual description were acquired from the DementiaBank Pitt corpus (Dementia group (DG) n = 169; healthy controls (CG) n = 71). Lexical variety, syntactic complexity, salience of information, disfluency patterns, and sentiment analysis were evaluated. Sound synthesis and an acoustic R analysis package were used to filter and examine acoustic properties of recorded audio such as silence and speech segmentation, pitch tracking, and frequency modulation of vibrato or jitter. DG and CG were compared using non‐parametric Kruskal‐Wallis tests, Dunn’s multiple comparison, and Mann‐Whitney U‐test in the statistical analysis. In comparison to CG, we hypothesized that DG would differ in levels of lexical diversity, grammatical complexity, and overall salience.ResultOverall, text analysis showed no variations in disfluencies between groups, but there were significant differences in lexical diversity, syntactic complexity and reference rate to reality (p<0.0001). Additionally, there were no appreciable differences in the frequency of salient information between the two groups, however the frequency of non‐salient information and disfluencies was considerably higher in the DG (p<0.05). Analysis of acoustic features revealed that DG had longer average speech duration and speech segment durations compared to CG (p<0.001).. On the other hand, the rate of change of acoustic characteristics including pitch, mean entropy, and harmonics‐to‐noise ratios did not significantly differ between DG and CG.ConclusionIn conclusion, this paradigm reveals that, in comparison to healthy controls, dementia patients exhibited increased lexical diversity yet lower linguistic complexity, and a shift towards characterizing non‐salient objects in given stimuli, as well as longer segments of speech. Further examination of the rhythmic and acoustic characteristics of speech may provide a comprehensive framework towards defining speech as a potential digital biomarker for dementia risk, and provide applications to regions experiencing increasing socioeconomic disparities.

Multilingual sound sopt synthesis systems

Localized sound zone synthesis, which can generate acoustically bright and dark zones using loudspeakers, is gaining attention as one of important acoustic communication techniques. By superposing multiple localized sound zones, multiple sound spot synthesis can simultaneously deriver different sound signals at different zones using a loudspeaker array. Additionally, it can be implemented as a user interface for real-time speech translation by combining multilingual speech synthesis technology. This paper introduces multilingual sound spot synthesis systems based on spatial Fourier transform implemented by using compact circular array of 16 loudspeakers and linear array of 64 loudspeakers combined with multilingual neural speech synthesis.

Modeling and reproduction of sound field by moving complex sound source

Previously, Wave Field Synthesis (WFS), Near-field Compensated Higher-Order Ambisonics (NFC-HOA), Boundary Surface Control (BoSC), and the Spectral Division Method (SDM) have been proposed as sound field synthesis methods. These methods are based on matching the acoustic physical quantities at the boundary of the reproduced sound field with those of the desired sound field. Sound field synthesis techniques can be classified into data-based and model-based methods. The former aim at reproducing sound fields captured by microphone arrays, while the latter aim at synthesizing sound fields based on mathematical models. Most existing sound field reproduction methods only consider simple sound source models, such as static point sources, instead of sound sources with complex radiation properties. To generate a more realistic sound image, this study investigated the formulation of a sound field generated by a sound source with complex radiation properties moving along arbitrary trajectories in a spherical harmonic domain, and reproduced the sound field using loudspeaker arrays.

Subjective evaluation of source distance in dynamic 2.5D local sound field synthesis

Many loudspeakers are required to reproduce the immersive and physically-accurate sound field up to high frequency. Especially, at high frequency, a limited number of loudspeakers degrade the reproduction accuracy because of spatial Nyquist frequency. When use the same number of loudspeakers, reducing the extent of the reproduction region can improve the reproduction accuracy. However, by narrowing the reproduction region, the extent of movement for the listener is reduced. In our previous studies, we proposed the dynamic reproduction system of local sound field by moving the reproduction region to the listener's current position. Using a limited number of loudspeakers, the proposed system achieved the sound reproduction with high accuracy allowing the freedom of listener movement. In this study, we conduct the subjective evaluation of the dynamic 2.5D local sound field synthesis for sound source distance. Especially, we investigate the distance perception by allowing listeners to move around freely.

Analysis and Synthesis of Guitar Sounds with Hammer on Strumming Technique

This research aims to analyze and synthesize periodic signals derived from guitar string plucking with hammer-on technique. This research has three stages, namely the data collection stage, the analysis stage, and the synthesis stage. The guitar string was plucked with a tension of 2.5 N and recorded using a sound sensor connected to PASCO Capstone software. The data used has two variations, namely the sound signal of a hammer-on pluck with a half tone increase and a one tone increase. Data analysis was carried out using MATLAB software to obtain deviation graphs as a function of frequency, damping coefficient values, and frequency spectra. The results showed that after hammer-on the amplitude of the tone decreased drastically as the mass per unit length of the string decreased. The initial tone before the hammer-on will appear in the tone after the hammer-on with a lower amplitude as the mass per unit length of the string increases. The synthesis of guitar sounds with this technique is done by combining the individual tones obtained and adjusting the time interval and amplitude according to the literature data

Exploring Gen in MaxMSP: Approaches to virtual analog and wavetable synthesis

The proposed poster presentation will explore using the Gen environment in MaxMSP to develop software that implements virtual analog and wavetable synthesis. Gen is a powerful tool that enables one to approach sound synthesis on an elemental level by implementing signal processing on the sample level instead of dealing with audio frames. The talk will explore both graphical patching and using the embedded text-based language GenExpr using the codebox operator. Focus will be placed on developing design patterns for developing efficient sound synthesis routines and learning to experiment in realtime with Gen.

Impact on quality and diversity from integrating a reconstruction loss into neural audio synthesis

In digital media or games, sound effects are typically recorded or synthesized. While there are a great many digital synthesis tools, the synthesized audio quality is generally not on par with sound recordings. Nonetheless, sound synthesis techniques provide a popular means to generate new sound variations. In this research, we study sound effects synthesis using generative models that are inspired by the models used for high-quality speech and music synthesis. In particular, we explore the trade-off between synthesis quality and variation. With regard to quality, we integrate a reconstruction loss into the original training objective to penalize imperfect audio reconstruction and compare it with neural vocoders and traditional spectrogram inversion methods. We use a Wasserstein GAN (WGAN) as an example model to explore the synthesis quality of generated sound effects, such as footsteps, birds, guns, rain, and engine sounds. In addition to synthesis quality, we also consider the range of sound variation that is possible with our generative model. We report on the trade-off that we obtain with our model regarding the quality and diversity of synthesized sound effects.

VSpace: A browser-based vowel synthesiser

Two-formant representation of the universal vowel space is a foundational model in phonetics (Fant & Risberg 1963; Traunmüller & Lacerda 1983), but tools for rapid synthesis of vowel sounds corresponding to points in the vowel space are limited. Stand-alone applications such as the Formant Synthesizer Demo (Beskow, 2000) provide this functionality, and rich tools for formant synthesis are available in Praat (Boersma and Weenink, 2023), Matlab (Rabiner et al., 2023) and other platforms, but these tools are not universally accessible, while web-based speech synthesis tools (e.g., Thapen, 2017) do not typically allow systematic manipulation or quantification of key acoustic parameters. VSpace is a browser-based formant synthesizer developed using the Tone.js Web Audio framework. A universal vowel space is represented as a trapezoid scaled to typical male, female or child speaker formant ranges. Clicking on any point generates a vowel sound corresponding to the first and second formant frequencies selected, synthesized with typical F3 and F4 frequencies excited by a selectable source signal. IPA symbols corresponding to typical formant frequencies for language-specific vowel phonemes may be superimposed on the vowel space. Key synthesis parameters are configurable through the web interface to allow exploration of the acoustic consequences for vowel perception.

Jet oscillation model for flute-like instruments allowing overall deflection

A new jet oscillation model for flute-like instruments is proposed. This is based on a conventional semi-empirical model for a jet in half space disturbed by an acoustic cross flow. The proposed model, however, allows a jet to deflect and to oscillate as a whole with no phase delay in response to the volume flow through the instrument's window (embouchure hole or mouth). The overall jet deflection is essential to match the volume flow swept by the lateral motion of the jet to that through the window. The complete jet oscillation is the sum of the overall oscillation and that caused by fluid dynamical sinuous instability. The latter grows exponentially with a phase delay as the jet travels. The amplitude of acoustic volume velocity triggering this oscillation is much smaller than that through the window because most of it has already contributed to the overall deflection. The proposed model reasonably reproduces the recorder's head reflection coefficient measured in [Price et al., J. Acoust. Soc. Am. 138(5), 3282–3292 (2015)]. Synthesis of recorder sounds using this model is underway.

Exploring the Impact of CPS-Based Robot-Assisted Teaching in STEM Education: Enhancing Knowledge, Skills, and Attitudes

Cyber-physical systems (CPS), along with information technology devices such as portable devices and robots, materials including knowledge contents, and Internet resources, can be integrated into curricula to guide learners in observing and experiencing STEM (science, technology, engineering, and mathematics) knowledge and skills. This integration provides opportunities for learners to cultivate their abilities to design creative works. The objective of this study was to investigate the impact of a robot-assisted teaching and learning system in STEM education on learners’ academic knowledge, skills, and attitudes. To achieve this objective, an experimental design was employed, with 30 college students assigned to learn and create a Drawdio electronic sound synthesizer using robot teachers and tablet computers. Pre- and post-tests were administered to measure students’ academic achievement in STEM, and a survey was conducted to assess their attitudes toward STEM. The findings of the study indicate that the combination of robot teachers and tablet computers served as a supplementary learning resource within the learning environment. It accelerated knowledge acquisition, expanded hands-on experiences, and improved engagement and motivation among the learners. These results suggest that robot-assisted teaching and learning systems have the potential to enhance learning outcomes and attitudes among students. However, it is important to note that human teachers still outperformed robot teachers in certain aspects, particularly in higher levels of knowledge acquisition according to Bloom’s taxonomy. Therefore, this study suggests the adoption of a collaborative model involving both robot and human teachers to effectively promote STEM education in the classroom.