Sound Synthesis Research Articles

Interactive Computation of Timbre Spaces for Sound Synthesis Control

Expressive sonic interaction with sound synthesizers requires the control of a continuous and high dimensional space. Further, the relationship between synthesis variables and timbre of the generated sound is typically complex or unknown to users. In previous works, we presented an unsupervised mapping method based on machine listening and machine learning techniques, which addresses these challenges by providing a low-dimensional and perceptually related timbre control space. The mapping maximizes the breadth of the explorable sonic space covered by the sound synthesizer, and minimizes possible timbre losses due to the low­-dimensional control. The mapping is generated automatically by a system requiring little input from users. In this paper we present an improved method and an optimized implementation that drastically reduce the time for timbre analysis and mapping computation. Here we introduce the use of the extreme learning machines for the regression from control to timbre spaces, and an interactive approach for the analysis of the synthesizer sonic response, performed as users explore the parameters of the instrument. This work is implemented in a generic and open-source tool that enables the computation of ad hoc synthesis mappings through timbre spaces, facilitating and speeding up the workflow to get a customized sonic control system.

Synthesis of Normal Heart Sounds Using Generative Adversarial Networks and Empirical Wavelet Transform

Currently, there are many works in the literature focused on the analysis of heart sounds, specifically on the development of intelligent systems for the classification of normal and abnormal heart sounds. However, the available heart sound databases are not yet large enough to train generalized machine learning models. Therefore, there is interest in the development of algorithms capable of generating heart sounds that could augment current databases. In this article, we propose a model based on generative adversary networks (GANs) to generate normal synthetic heart sounds. Additionally, a denoising algorithm is implemented using the empirical wavelet transform (EWT), allowing a decrease in the number of epochs and the computational cost that the GAN model requires. A distortion metric (mel–cepstral distortion) was used to objectively assess the quality of synthetic heart sounds. The proposed method was favorably compared with a mathematical model that is based on the morphology of the phonocardiography (PCG) signal published as the state of the art. Additionally, different heart sound classification models proposed as state-of-the-art were also used to test the performance of such models when the GAN-generated synthetic signals were used as test dataset. In this experiment, good accuracy results were obtained with most of the implemented models, suggesting that the GAN-generated sounds correctly capture the characteristics of natural heart sounds.

Two methods for acoustic modeling of the saxophone mouthpiece

The mouthpiece is one of the key components in determining the sound and the playability of the saxophone. Assuming plane-wave propagation inside the mouthpiece, two different models are proposed for acoustic modeling of the saxophone mouthpiece. First, a Transfer Matrix (TM) model is derived from a finite element model and is then validated by comparing the calculated input impedance of a mouthpiece-cone structure with measurements. Then, by adapting a methodology traditionally used in modeling the human vocal tract, a Geometric Model (GM) is used to approximate the mouthpiece as a series of acoustic tubes of varying cross-sectional areas, and its accuracy is examined by comparing it with the TM mouthpiece model. We couple both models to a measured input impedance of an alto saxophone and provide a comparison of both methods and their impact on different fingering configurations. Finally, we discuss applications in sound synthesis and in mouthpiece design.

Automatic sound synthesis using the fly algorithm

Our study is demonstrated a new type of evolutionary sound synthesis method. This work based on the fly algorithm, a cooperative co-evolution algorithm; it is derived from the Parisian evolution approach. The algorithm has relatively amended the position of individuals (the Flies) represented by 3-D points. The fly algorithm has successfully investigated in different applications, starting with a real-time stereo vision for robotics. Also, the algorithm shows promising results in tomography to reconstruct 3-D images. The final application of the fly algorithm was generating artistic images, such as digital mosaics. In all these applications, the flies’representation started for simple, 3-D points, to complex one, the structure of 9-elements. Our method follows evolutionary digital art with the fly algorithm in representing the pattern of the flies. They represented in a way of having their structure. This structure includes position, colour, rotation angle, and size. Our algorithm has the benefit of graphics processing units (GPUs) to generate the sound waveform using the modern OpenGL shading language.

Perception of atomic speech

Natural speech signals are continuous and have great redundancy in time, which is learned by human beings and guarantees speech comprehension even in challenging circumstances. This work develops a sound synthesis method to study some time-related characteristics of the human auditory system. The temporal information within the individual frequency band of a speech is down-sampled using Gaussian-shaped pulses and then recombined into a new sound which may have some remaining intelligibility or feasible phonetic features. The “atomic” sound is coined to the sound with an extremely spectral-temporally sparse spectrogram generated using the method. A battery of speech perception tests was administered in normal-hearing listeners. Results show that (1) atomic sounds from clear speech can be understood as speech, although the listeners often reported a feeling of water sound textures; (2) the temporal and spectral resolution could be traded off in the atomic speech comprehension; and (3) only one maximum envelope value preserved among a 32-channel filter-bank at a rate of 400 Hz was surprisingly adequate for speech understanding, which indicates that the brain can organize the only spectral peak within each short duration no longer than 2.5-ms into sentence understanding without either explicit or implicit encoding of the first three formants.

Animating explosion with exploding sound and rigid‐body sound

AbstractVisual simulation of explosion animation can be found many applications in virtual reality. In addition to visual simulation, sound synthesis of explosion scenes is also indispensable. However, little attention has been paid to this research area. To this end, this paper proposes an automatic sound synthesis method for explosion scenes. Explosion animation consists of two major events, namely explosive event and interactive rigid‐body fracture event. Correspondingly, the sound for an explosion scene can be viewed as the combination of two components, that is, the exploding sound and the rigid‐body sound. Among them, the exploding sound often shows similarity in terms of characteristics in frequency domain, while the rigid‐body sound is rich in variations due to the unpredictable motion state and different materials of debris. As a result, for the synthesis of the rigid‐body sound, firstly, we extract the rigid‐body sounds from recording examples by the empirical mode decomposition method. On this basis, a new spectral flux based approach is developed to segment the extracted sounds into a grain dictionary. Then, a cascade sound synthesis method based on impulse response is designed to synchronize the rigid‐body sound for fracture events. For the synthesis of exploding sound, due to its similar characteristics in frequency domain, an example‐based automatic sound synthesis method is adapted in this paper. Finally, we propose a priority‐aware sound blending method to merge these two types of completely different sounds reasonably. Various experimental results demonstrated the effectiveness of our new method.

Sound field synthesis of arbitrary moving sources using spectral division method.

The spectral division method (SDM) is one of several sound field synthesis techniques of spatial sound reproduction. The synthesis accuracy of SDM is usually higher than that of wave field synthesis (WFS). However, existing SDM-based sound field synthesis methods are developed for virtual sources moving uniformly along a straight trajectory. This Letter proposes a 2.5-dimensional SDM driving function for arbitrary moving sources. Numerical simulations have validated the driving function. The results indicate that the synthesis error of the SDM with the proposed driving function is lower than that of WFS, especially when the virtual source is close to the secondary sources.

On-The-Fly Syntheziser Programming with Fuzzy Rule Learning.

This manuscript explores fuzzy rule learning for sound synthesizer programming within the performative practice known as live coding. In this practice, sound synthesis algorithms are programmed in real time by means of source code. To facilitate this, one possibility is to automatically create variations out of a few synthesizer presets. However, the need for real-time feedback makes existent synthesizer programmers unfeasible to use. In addition, sometimes presets are created mid-performance and as such no benchmarks exist. Inductive rule learning has shown to be effective for creating real-time variations in such a scenario. However, logical IF-THEN rules do not cover the whole feature space. Here, we present an algorithm that extends IF-THEN rules to hyperrectangles, which are used as the cores of membership functions to create a map of the input space. To generalize the rules, the contradictions are solved by a maximum volume heuristics. The user controls the novelty-consistency balance with respect to the input data using the algorithm parameters. The algorithm was evaluated in live performances and by cross-validation using extrinsic-benchmarks and a dataset collected during user tests. The model’s accuracy achieves state-of-the-art results. This, together with the positive criticism received from live coders that tested our methodology, suggests that this is a promising approach.

Blurred lines: Practical and theoretical implications of a DAW-based pedagogy

Digital audio workstations (DAWs) occupy a prominent space in the creative arts. Songwriters, composers, producers, and audio engineers use a combination of software and virtual instruments to record and make music. Educators increasingly find DAWs useful for teaching concepts in signal flow, acoustics and sound synthesis, and to model analogue processes. As the creative industries shift to primarily software-based platforms, the identities, roles, and responsibilities of the participants intersect and blur. Similarly, networked technologies change the space and place of creative activity. Now, the ‘studio’ exists virtually anywhere. For educators working with students, these changing paradigms present a series of challenges. This article explores the DAW’s possibilities across three areas: space and place, theory and identity, and pedagogy. The article advocates for a less technocratic model of teaching and learning with DAWs in favour of an approach that cultivates a balance of aesthetic awareness and creativity.

Implementation of sample-engineering as arrangement technology in the training of a future teacher of musical arts

The article considers the possibility of including the technology of sample-engineering in the process of training students - future teachers of Musical Arts. The purpose of the article is to highlight the concept “sample-engineering” in its historical and technical aspects and to determine the possibilities of practical application of the innovative technology for arranging musical material by future teachers of Musical Arts. The main development stages of the technology facilitating the artificial production of non-traditional sound tones are considered, it is associated with the revolutionary inventions obtained at the end of the twentieth century and a qualitative breakthrough in microprocessor technology, which allowed us to significantly increase the volume and speed of information processing as well as the role of discoveries in the field of applied music programs and provided the emergence of new ideas regarding sound synthesis. The author describes the features of three technological stages related to sound synthesis, sampling and sample-engineering, as well as to new opportunities that are opened up for musicians through their use. The distinguishing features of using these technologies and devices are presented, in particular, the features of the sampling phenomenon and the emergence of a new variety of electronic keyboard instruments – a sampler in the memory banks of which you can store sounds that are “borrowed” from the environment. Particular attention is drawn to the importance of the widespread practice of developing modern sample-engineering technology which arose due to the microprocessor technology and the opportunity to convert an analogue sound to a digital one without any noticeable loss in quality. The essence of the concept “engineering”, the significance of its use in musical practice and the main stages-tasks of using the technology of sample-engineering in the educational process are revealed. The universal role of the computer in the application of the sample-engineering technology is determined, its technical capabilities growing when connecting external sound sources (tone generators) to it. The article substantiates the feasibility and prospects for the future teachers of Musical Arts to study the sampling-engineering technology a good command of which might allow them to achieve timbre diversity and greater artistic expression in the processing of school songs arranged by them and in their own creativity. The work reveals the prospects of using the technology of sample engineering as a way to increase the capabilities of future specialists-arrangers to process musical material, the importance of mastering creative self-realisation by the future teachers in a modern innovative way. The author presents the stage-by-stage process regulating the acquisitions of skills to use sample engineering by students in educational and creative practice alongside the prospects of mastering this technology involving a wide range of teachers of institutions of higher musical and pedagogical education.

Academic music transcript modeling using electronic sound synthesis

Nowadays, computer technologies are integrated into all art forms. New artistic directions emerge which are totally based on multimedia (primarily audio-visual) instruments. The question about the role of music art in the new cultural paradigm becomes topical. Based on the example of academic music, the article considers the possibilities of using programming tools for music scores processing into a transcript. The authors consider the possible mathematical model of a piece of music which allows processing the piece of music and getting both an authentic and an experimental version. The authors analyze the set of parameters helping to form a virtual acoustic space corresponding with an artistic concept. The authors demonstrate that academic music transcript modeling using electronic sound synthesis can promote the artistic reinterpretation of the score of the piece of music. The selection of particular timber, performance and spatial parameters during transcribing helps to interpret a composition according to the artistic tasks. The use of virtual instruments also allows creating digital recordings without performing musicians. The article contains the variants of forming a virtual acoustic environment based on the example of a classical piece of music for string quartet: from a “traditional” to an “experimental, differing in musical sources positioning, a set of spatial effects and the parameters of their adjustment.   

An interdisciplinary approach to learning programming through the creation of electronic music in the sonic pi audio programming environment

With the development of computer technologies, there is a rapid development of applied music based on computer technology: music for computer games, for advertising, for voicing household and other equipment, music in movies and animations, TV music, and academic music. This indicates the relevance of such an area of use of information technologies as program (algorithmic) sound synthesis, and the need for schoolchildren to familiarize themselves with this promising IT area. The fact that music and mathematics are interconnected has been known since ancient times. Now we can observe the close connection of music and informatics. And not just the processing of music in synthesis programs, but the programmatic creation of music using code. The article shows the relationship of musical art with the art of programming, considers the concept of the Sonic Pi audio programming environment, demonstrates the main features of this environment, language tools for programming algorithms, data structures and musical compositions in Ruby.

Analyzing journeys in sound: usability of graphical interpolators for sound design

Graphical interpolation systems provide a simple mechanism for the control of sound synthesis by providing a level of abstraction above the engine parameters, allowing users to explore different sounds without awareness of the underlying details. Typically, a graphical interpolator presents the user with a two-dimensional pane where a number of synthesizer presets, each representing a collection of synthesis parameter values, can be located. Moving an interpolation cursor within the pane results in the calculation of new parameter values, based on its position, the relative locations of the presets, and the mathematical interpolation function, thus generating new sounds. These systems supply users with two sensory modalities in the form of sonic output and visual feedback from the interface. A number of graphical interpolator systems have been developed over the years, with a variety of user-interface designs, but few have been subject to formal user evaluation. Our testing studied both user interaction with, and the perceived usability of, graphical interpolation systems by comparing alternative visualizations in order to establish whether the visual feedback provided by the interface aids the locating of sounds within the space. The outcomes of our study may help to better understand design considerations for graphical interpolators and inform future designs.

Neuronal Acts

If microperformativity is seen to spatially and temporally destabilize human scales as the dominant plane of reference, as well as to ‘emphasize biological and technological micro-agencies’ that ‘relate the invisibility of the microscopic to the incomprehensibility of the macroscopic’, there may be no better performative example than the neuron. Neurons seem to have become our latest cultural-technical fetish objects that easily jump scale, context and category, from live artistic performances like CellF or Silent Barrage that utilize the electrical spikes of cultured neurons to control the action of robots or sound synthesis to the mathematical behaviour of artificial neural networks. But strangely enough there is another kind of performativity that the neuron implies – that of economic markets. In 1952, the Austrian economist Friedrich Hayek published The Sensory Order, which put forward a neuronal theory of the mind, arguing that all sense-making, learning and memory is a product of the dynamic connections that emerge from neuronal links that take place over time. According to Hayek, the sensory order is constructed from the neuronal connections classifying information – objects external to the mind have no intrinsic properties except how the nervous system classifies these properties. In other words, ‘we live in a sensory order that is created by the central nervous system’. What then does the neuron have to do markets? The concept of the ‘performativity’ of markets implies that through its mathematical models, economics shapes social reality rather than objectively describing it. Thus this article tries to understand this economic performativity through a detour - asking what work the concept of performance can do in attempting to describe ‘living technologies’ (Takashi Ikegami) or lively systems across different scales (brains, machines, economies), whose ontological and operational order is one that challenges a basic epistemological assumption: that we can actually know the world.

Microphone and Loudspeaker Array Signal Processing Steps towards a “Radiation Keyboard” for Authentic Samplers

To date electric pianos and samplers tend to concentrate on authenticity in terms of temporal and spectral aspects of sound. However, they barely recreate the original sound radiation characteristics, which contribute to the perception of width and depth, vividness and voice separation, especially for instrumentalists, who are located near the instrument. To achieve this, a number of sound field measurement and synthesis techniques need to be applied and adequately combined. In this paper we present the theoretic foundation to combine so far isolated and fragmented sound field analysis and synthesis methods to realize a radiation keyboard, an electric harpsichord that approximates the sound of a real harpsichord precisely in time, frequency, and space domain. Potential applications for such a radiation keyboard are conservation of historic musical instruments, music performance, and psychoacoustic measurements for instrument and synthesizer building and for studies of music perception, cognition, and embodiment.

Online sonification for golf putting gesture: reduced variability of motor behaviour and perceptual judgement.

This study investigates whether real-time auditory feedback has a direct behavioural or perceptual effect on novices performing a golf putting task with limited visual feedback. Due to its significant role in the success of a putt, club head speed was selected as the parameter for sonification. Different combinations of synthesisers, timbral modulations, scales, and mappings were developed to examine whether particular sound classes influenced performance. When compared to trials with static pink noise, we found that, despite their vision being limited at impact, participants were able to use different types of sonification to significantly reduce variability in their distance from the target and ball location estimation. These results suggest that concurrent sound can play an important role in reducing variability in behavioural performance and related perceptual estimations. In addition, we found that, when compared to trials with static pink noise, participants were able to use sonification to significantly lower their average impact velocity. In the discussion, we offer some trends and observations relative to the different sound synthesis parameters and their effects on behavioural and perceptual performance.

Sound model of an orchestral kettledrum considering viscoelastic effects

The modeling of the mechanical systems that describe musical instruments has long been a topic of interest for acoustic physicists. The orchestral kettledrum, distinguished by a broad spectrum of sounds, has been the subject of multiple investigations related to the goal of generating a realistic digital sound synthesis. In the present work, we apply the Green function method to estimate modal frequencies considering an air-loaded viscoelastic membrane. We propose a method that includes viscoelasticity to accurately predict the sound spectrum of the modeled kettledrum. Results are compared to real sound recordings of an orchestral kettledrum obtained in controlled conditions. The calculated modal frequencies are found to coincide well with the real values with an absolute mean error of 1.25±0.76 Hz and 1.87±1.83 Hz for the A2 and B2♭ tuned drumheads respectively. The spectral envelope of the synthesized sound spectrum coincides well with the Fourier transform of the real sound. The viscoelastic term was found to generally reduce the amplitude of the sound spectrum and in certain cases, better approximate the modal frequencies and decay times. The modal synthesis method used here is numerically lightweight and can be adapted to be used in real-time applications with low computational resources. To reproduce our experiment, the recorded kettledrum sounds and Python source code of the model are freely available.

Gaussian Process Synthesis of Artificial Sounds

In this paper, we propose Gaussian Process (GP) sound synthesis. A GP is used to sample random continuous functions, which are then used for wavetable or waveshaping synthesis. The shape of the sampled functions is controlled with the kernel function of the GP. Sampling multiple times from the same GP generates perceptually similar but non-identical sounds. Since there are many ways to choose the kernel function and its parameters, an interface aids the user in sound selection. The interface is based on a two-dimensional visualization of the sounds grouped by their similarity as judged by a t-SNE analysis of their Mel Frequency Cepstral Coefficient (MFCC) representations.

Anacoustic Modes of Sound Construction and the Semiotics of Virtuality

This article discusses technical and aesthetic aspects of sound synthesis in the context of anacoustic modes of sound construction – a neologism that underscores the unique ontology of information in the digital domain. Anacoustic modes address the computer at its most fundamental level: the syntactic level of information. This changes the nature of signification as sound is considered first as an informational construct rather than a material circumstance, rupturing the front-loaded meaning that arises from our acoustic experience. Following certain concepts encompassed by N. Katherine Hayles’s posthumanism, anacoustic modes can be viewed as an expression of the materiality of information.

Sound field synthesis based on superposition of multipoles comprising focused monopole sources

Sound field synthesis based on superposition of multipoles comprising focused monopole sources