Sound Synthesis Research Articles

Exploring the Impact of CPS-Based Robot-Assisted Teaching in STEM Education: Enhancing Knowledge, Skills, and Attitudes

Cyber-physical systems (CPS), along with information technology devices such as portable devices and robots, materials including knowledge contents, and Internet resources, can be integrated into curricula to guide learners in observing and experiencing STEM (science, technology, engineering, and mathematics) knowledge and skills. This integration provides opportunities for learners to cultivate their abilities to design creative works. The objective of this study was to investigate the impact of a robot-assisted teaching and learning system in STEM education on learners’ academic knowledge, skills, and attitudes. To achieve this objective, an experimental design was employed, with 30 college students assigned to learn and create a Drawdio electronic sound synthesizer using robot teachers and tablet computers. Pre- and post-tests were administered to measure students’ academic achievement in STEM, and a survey was conducted to assess their attitudes toward STEM. The findings of the study indicate that the combination of robot teachers and tablet computers served as a supplementary learning resource within the learning environment. It accelerated knowledge acquisition, expanded hands-on experiences, and improved engagement and motivation among the learners. These results suggest that robot-assisted teaching and learning systems have the potential to enhance learning outcomes and attitudes among students. However, it is important to note that human teachers still outperformed robot teachers in certain aspects, particularly in higher levels of knowledge acquisition according to Bloom’s taxonomy. Therefore, this study suggests the adoption of a collaborative model involving both robot and human teachers to effectively promote STEM education in the classroom.

Beyond sense-specific processing: decoding texture in the brain from touch and sonified movement

Texture, a fundamental object attribute, is perceived through multisensory information including touch and auditory cues. Coherent perceptions may rely on shared texture representations across different senses in the brain. To test this hypothesis, we delivered haptic textures coupled with a sound synthesizer to generate real-time textural sounds. Participants completed roughness estimation tasks with haptic, auditory, or bimodal cues in an MRI scanner. Somatosensory, auditory, and visual cortices were all activated during haptic and auditory exploration, challenging the traditional view that primary sensory cortices are sense-specific. Furthermore, audio-tactile integration was found in secondary somatosensory (S2) and primary auditory cortices. Multivariate analyses revealed shared spatial activity patterns in primary motor and somatosensory cortices, for discriminating texture across both modalities. This study indicates that primary areas and S2 have a versatile representation of multisensory textures, which has significant implications for how the brain processes multisensory cues to interact more efficiently with our environment.

DDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge

DDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge

Sound field synthesis for psychoacoustic research: In situ evaluation of auralized sound pressure levela).

The use of virtual acoustic environments has become a key element in psychoacoustic and audiologic research, as loudspeaker-based reproduction offers many advantages over headphones. However, sound field synthesis methods have mostly been evaluated numerically or perceptually in the center, yielding little insight into the achievable accuracy of the reproduced sound field over a wider reproduction area with loudspeakers in a physical, laboratory-standard reproduction setup. Deviations from the ideal free-field and point-source concepts, such as non-ideal frequency response, non-omnidirectional directivity, acoustic reflections, and diffraction on the necessary hardware, impact the generated sound field. We evaluate reproduction accuracy in a 61-loudspeaker setup, the Simulated Open Field Environment, installed in an anechoic chamber. A first measurement following the ISO 8253-2:2009 standard for free-field audiology shows that the required accuracy is reached with critical-band-wide noise. A second measurement characterizes the sound pressure reproduced with the higher-order Ambisonics basic decoder, with and without max rE weighting, vector base amplitude panning, and nearest loudspeaker mapping on a 187 cm × 187 cm reproduction area. We show that the sweet-spot size observed in measured sound fields follows the rule kr≤N/2 rather than kr≤N but is still large enough to avoid compromising psychoacoustic experiments.

Physical modeling and sound synthesis of the hi-hat

Physical modeling and sound synthesis of the hi-hat

The disconnected keyboard: Inclusive learning and musicking practice with modular synthesis

Since the 1960s, electronic sound synthesis and the keyboard interface have been so closely associated that many young musicians have come to see them as inseparable components, if not interchangeable terms. In this article, we ‘disconnect the keyboard’ and explore an alternative history of electronic sound synthesis – modular synthesis – that has remained largely overshadowed by keyboard-based synthesizers since the Minimoog. Researchers in music education signal that Eurocentric aesthetic norms, ableist performance ideals and exclusionary practices are interwoven in keyboard technologies, creating barriers that extend into popular music education. Drawing upon critical discussions in music education and science and technology studies (STS), we examine the underexplored opportunities of using modular synthesizers for music learning. We examine how modular synthesis, liberated from the keyboard-controller, serves as a basis for exploring an alternative model for sound-based inquiry and for rethinking the possibilities of instrument design and ways of musiking that are more inclusive.

Implicit and explicit schemes for energy-stable simulation of string vibrations with collisions: Refinement, analysis, and comparison

Collisions play an important role in the sound production mechanism of musical string instruments. Impactive contact modelling must therefore be incorporated in energy-stable time-stepping schemes for simulation and sound synthesis. Recently, explicit and linearly implicit schemes based on energy quadratisation have emerged as a means of numerical modelling of nonlinear string vibrations that avoids the need for iterative solvers, which significantly enhances the applicability to real-time sound synthesis. However when applied to modelling collisions, spurious oscillations and artefacts have been reported, which can be linked to the one-sided nature of the contact force. To investigate whether and how these limitations can be avoided, this paper proposes refined numerical scheme variants for both lumped and distributed contact configurations, and assesses their accuracy and efficiency. For reference, two nonlinearly implicit schemes are also included in the comparisons. Guaranteed correctness of the sign of the numerical contact force is proven for several of the scheme variants. Numerical experiments across three case studies indicate that each of the proposed schemes converges towards the exact solution, and that for sufficiently small time steps, explicit schemes generally perform well compared to implicit schemes.

Agent-Based Music Live Coding: Sonic adventures in 2D

This article describes agent-based music live coding, an approach for music performance and composition based on programming a set of agents in a 2D plane. This style of programming draws from the tradition of agent-based models and facilitates interactive algorithmic control of data-driven sound synthesis methods such as wave terrain synthesis or corpus-based concatenative synthesis. The main elements are a ‘terrain’, which may be used to access different types of data, a set of agents and their trajectories, and a set of synthesis functions associated to agents. An implementation using the SuperCollider language is demonstrated.

Enhancing Audio Signal Quality and Learning Experience with Integrated Covariance Weiner Filtering in College Music Education

In recent years, computer music technology has become increasingly prevalent in college music education, offering new possibilities for creative expression and pedagogical approaches. This paper concentrated on the music education in the colleges with the application of integrated time and frequency filtering (ITFF) with Kalman integrated covariance Weiner filtering in college music education. The ITFF technique combines time and frequency domain analysis to enhance the quality and clarity of audio signals. By integrating the Kalman integrated covariance Weiner filtering, the ITFF method provides robust noise reduction and improved signal representation. This integrated approach enables music educators to effectively analyze and manipulate audio signals in real-time, fostering a more immersive and engaging learning environment for students. The findings of this study highlight the benefits and potential applications of ITFF with Kalman-integrated covariance Weiner filtering in college music education, including audio signal enhancement, sound synthesis, and interactive performance systems. The integration of computer music technology with advanced filtering techniques presents new opportunities for exploring sound, composition, and music production within an educational context.

Continuous synthesis of artificial speech sounds from human cortical surface recordings during silent speech production

Objective. Brain–computer interfaces can restore various forms of communication in paralyzed patients who have lost their ability to articulate intelligible speech. This study aimed to demonstrate the feasibility of closed-loop synthesis of artificial speech sounds from human cortical surface recordings during silent speech production. Approach. Ten participants with intractable epilepsy were temporarily implanted with intracranial electrode arrays over cortical surfaces. A decoding model that predicted audible outputs directly from patient-specific neural feature inputs was trained during overt word reading and immediately tested with overt, mimed and imagined word reading. Predicted outputs were later assessed objectively against corresponding voice recordings and subjectively through human perceptual judgments. Main results. Artificial speech sounds were successfully synthesized during overt and mimed utterances by two participants with some coverage of the precentral gyrus. About a third of these sounds were correctly identified by naïve listeners in two-alternative forced-choice tasks. A similar outcome could not be achieved during imagined utterances by any of the participants. However, neural feature contribution analyses suggested the presence of exploitable activation patterns during imagined speech in the postcentral gyrus and the superior temporal gyrus. In future work, a more comprehensive coverage of cortical surfaces, including posterior parts of the middle frontal gyrus and the inferior frontal gyrus, could improve synthesis performance during imagined speech. Significance. As the field of speech neuroprostheses is rapidly moving toward clinical trials, this study addressed important considerations about task instructions and brain coverage when conducting research on silent speech with non-target participants.

Evaluations of Clarinet Sound Quality and Register Characteristics

The clarinet is a commonly used woodwind instrument that has a unique sound quality, making it a prominent instrument in solo performances and orchestral music. Therefore, sound quality plays an important role in the perception of clarinet sound. This study provides a review of the relevant research in the field of clarinet sound analysis and synthesis, and identifies many valuable references from previous studies. In studying the noise component of clarinet sound, this study recorded audio samples of E3, E4, and E5 and preprocessed them by slicing and performing frequency domain transformations. Using 13 representative spectral features, this paper extracted features from the spectral signals and constructed a classification model of sound quality based on the SVM algorithm of machine learning. Finally, this study analyzed the shortcomings of the SVM algorithm in high-pitch analysis and proposed future research directions of combining the SVM algorithm with different sound characteristics of clarinet registers and spectral changes throughout the entire process of playing. These results shed light on guiding further exploration of the timbre of the clarinet.

Improved Water Sound Synthesis using Coupled Bubbles

We introduce a practical framework for synthesizing bubble-based water sounds that captures the rich inter-bubble coupling effects responsible for low-frequency acoustic emissions from bubble clouds. We propose coupled-bubble oscillator models with regularized singularities, and techniques to reduce the computational cost of time stepping with dense, time-varying mass matrices. Airborne acoustic emissions are estimated using finite-difference time-domain (FDTD) methods. We propose a simple, analytical surface-acceleration model, and a sample-and-hold GPU wavesolver that is simple and faster than prior CPU wavesolvers. Sound synthesis results are demonstrated using bubbly flows from incompressible, two-phase simulations, as well as procedurally generated examples using single-phase FLIP fluid animations. Our results demonstrate sound simulations with hundreds of thousands of bubbles, and perceptually significant frequency transformations with fuller low-frequency content.

Piano Soundboard Analysis at Radiated Sound

The piano is a complex musical instrument consisting of several components influencing vibration and sound production. By understanding the sound production mechanism virtual instruments can be created (physics-based sound synthesis) and the design and manufacturing of soundboards can be supported (virtual prototyping). Based on previous results published in the literature, a piano model was built and extended by a near field sound radiation model capable for physics-based sound synthesis. In this paper a simplified piano model is presented, including hammer strike and hysteretic felt models, coupled lossy string model and a 2D FEM based stiffened plate model for soundboard. This paper contains a parametric study where the soundboard parameters, such as its material characteristics and boundary conditions, are modified and their effect on the soundboard's modal behavior and the radiated sound is analyzed. Instead of using only musical (qualitative) descriptors, e.g. brightness or coloring, the piano sounds are characterized based on standard quantitative descriptors (e.g. harmonic ratio, spectral centroid). It is shown that these descriptors are determined by soundboard admittance, string characteristics and position on the soundboard; radiated sound from wooden soundboards can be characterized as harmonic for wide range of initial material descriptors; the string position is essential, and the perceived sound can differ significantly for different listening positions, even for the same harmonic decay pattern.

Synthesis of sound field from moving complex sources with arbitrary trajectories by linear and spherical loudspeaker arrays.

The spectral division method (SDM) and near-field compensated higher order ambisonics (NFC-HOA) are sound field synthesis techniques based on the spatial Fourier representation of sound fields. Previous studies have derived the driving functions of SDM for sound field synthesis with consideration to uniformly moving point sources and moving point sources with arbitrary trajectories. However, the driving functions of NFC-HOA for synthesizing sound fields from moving sound sources have not been proposed to date. For a more realistic auditory experience, the synthesis of a sound field produced by a moving sound source with a complex radiation property is required. This study focused on deriving the driving functions for synthesizing sound fields produced by moving sound sources with arbitrary trajectories and radiation properties. Sound fields were formulated in the angular spectrum and spherical harmonic domains and applied to SDM and NFC-HOA, respectively. Numerical and measurement experiments were conducted to evaluate the proposed method. The results reveal that the proposed method can synthesize the desired sound fields.

Affective Neural Responses Sonified through Labeled Correlation Alignment

Sound synthesis refers to the creation of original acoustic signals with broad applications in artistic innovation, such as music creation for games and videos. Nonetheless, machine learning architectures face numerous challenges when learning musical structures from arbitrary corpora. This issue involves adapting patterns borrowed from other contexts to a concrete composition objective. Using Labeled Correlation Alignment (LCA), we propose an approach to sonify neural responses to affective music-listening data, identifying the brain features that are most congruent with the simultaneously extracted auditory features. For dealing with inter/intra-subject variability, a combination of Phase Locking Value and Gaussian Functional Connectivity is employed. The proposed two-step LCA approach embraces a separate coupling stage of input features to a set of emotion label sets using Centered Kernel Alignment. This step is followed by canonical correlation analysis to select multimodal representations with higher relationships. LCA enables physiological explanation by adding a backward transformation to estimate the matching contribution of each extracted brain neural feature set. Correlation estimates and partition quality represent performance measures. The evaluation uses a Vector Quantized Variational AutoEncoder to create an acoustic envelope from the tested Affective Music-Listening database. Validation results demonstrate the ability of the developed LCA approach to generate low-level music based on neural activity elicited by emotions while maintaining the ability to distinguish between the acoustic outputs.

Research on the Harmonics-Based the Optimization Algorithm for the Active Synthesis of Automobile Sound

The technology of active sound generation (ASG) for automobiles is one of the most effective methods to flexibly achieve the sound design that meets the expectations of different user groups, and the active sound synthesis algorithms are crucial for the implementation of ASG. In this paper, the Kaiser window function-based the harmonic synthesis algorithm of automobile sound is proposed to achieve the extraction of the order sounds of target automobile. And, the suitable fitting functions are utilized to construct the mathematical model between the engine speed information and the amplitude of the different order sound. Then, a random phase correction algorithm is proposed to ensure the coherence of the synthesized sounds. Finally, the analysis of simulation results verifies that the established method of the extraction and synthesis of order sound can meet the requirements of target sound quality. The algorithm proposed in this paper can be used for the development of real-time engineering of ASD technology for real vehicles.

Sound models for synthesis: a structural viewpoint

For the needs of music production and multimedia art, sound synthesis algorithms are needed which are versatile, responsive to user’s expectations, and having high audio quality. It is useful to organize our intuitive sound abstracts into models. A computational model can be used for representing and generating a whole class of sounds, depending on the internal structure of the model and on the choice of control parameters. We will review some of the most important computational models that are being used for sound synthesis in musical production from the viewpoint of the model structure. Moreover, the Centro di Sonologia Computazionale of Padova University has done research on synthesis models for a long time and the main achievements in both the scientific and musical fields will be presented.

Audio Augmented Reality in Smart Learning Environments

Image processing-based augmented reality (AR) is widely used in several fields. However, audio content is also crucial in certain cases, for example, focusing on appreciating artwork in a museum, rather than a virtually synthesized image. In this study, an attempt was made to provide ubiquitous learning (u-learning) services by using audio AR for smart cities where people do not need smartphones, tablets, or wearable devices for learning. The concept of “unmanned stores” was introduced in the proposed learning environments. A 360° camera was mounted on the ceiling of an exhibition room. An algorithm was used to detect moving objects and locate a visitor, from whose position an information-processing platform searched the showpiece nearest to the visitor and selected the corresponding audio guide. For multiple visitors, infrared (IR) sensors were used to count the visitors inside the room. A loudspeaker array was built to emit directional sounds for eliminating interference between multiple sound sources. Along with the proposed algorithm for increasing the accuracy while sensing multiple movements, the conventional sound field synthesis method was modified to improve the listening experience and learning quality. Image-based sensor technology, IR detection, and an audio signal-processing platform were integrated into a smart environment in which people can conduct u-learning without personal devices. The method can provide immersive audio tours with virtual narrators and peer-helpers in schools, museums, and art galleries. Qualitative and quantitative analyses indicated that effective hands-free and eyes-free learning was achieved.

성덕대왕신종(聖德大王神鍾)의 음향학적 에뮬레이션 연구

The purpose of this study is to analyze the possibility of using the “Sacred Bell of Great King Seongdeok” which is designated as National Treasure No. 29, as a musical resource through acoustic analysis and sound synthesis method. Based on the acoustically analyzed data of the Bell, it was tried to synthesized with frequency and fluctuation of partial sound, the beat cycle and envelop application by the additive synthesis among the method of sound synthesis The spatial sounds were modelled by an artificial neural network to estimate the exact reverberation of the source. The final result was produced with Native Instruments Reactor6, which can be easily used not only by sound designers and multimedia artists but also by many musicians, to preserve the history and identity of Sacred Bell of Great King Seongdeok, and to confirm its potential as a medium to be fused to new media.

Spatial audio for interactive hearing research

The use of sound field synthesis for hearing research has gained popularity due to the ability to auralize a wide range of sound scenes in a controlled and reproducible way. We are interested in reproducing acoustic environments for interactive hearing research, allowing participants to move freely over an extended area in the reproduced sound field. While the physically accurate sound field reproduction using sound field synthesis is limited to the sweet spot, it is unclear how different perceptual measures vary across the reproduction area and how suitable sound field synthesis is to evaluate them. To investigate the viability of listening experiments and provide a database for modelling approaches, measurements of binaural cues were carried out in the Simulated Open Field Environment loudspeaker array. Results show that the binaural cues are reproduced well close to the center, but exhibit more variance than in the corresponding free field case. Off center, lower interaural coherence is observed, which can affect binaural unmasking and speech intelligibility. In this work, we study binaural cues and speech reception thresholds over a wide area in the loudspeaker array to investigate the feasibility of psychoacoustic experiments involving speech understanding.