Audio Synthesis Research Articles

Physical Modeling, Algorithms, and Sound Synthesis: The NESS Project

Abstract Synthesis using physical modeling has a long history. As computational costs for physical modeling synthesis are often much greater than for conventional synthesis methods, most techniques currently rely on simplifying assumptions. These include digital waveguides, as well as modal synthesis methods. Although such methods are efficient, it can be difficult to approach some of the more detailed behavior of musical instruments in this way, including strongly nonlinear interactions. Mainstream time-stepping simulation methods, despite being computationally costly, allow for such detailed modeling. In this article, the results of a five-year research project, Next Generation Sound Synthesis, are presented, with regard to algorithm design for a variety of sound-producing systems, including brass and bowed-string instruments, guitars, and large-scale environments for physical modeling synthesis. In addition, 3-D wave-based modeling of large acoustic spaces is discussed, as well as the embedding of percussion instruments within such spaces for full spatialization. This article concludes with a discussion of some of the basics of such time-stepping methods, as well as their application in audio synthesis.

A Comparative Study of Different Deep Learning Approaches for the Prediction of Natural Gas Demand in the United States

In this work, the impact of network architecture on the natural gas demand and price forecasts task is evaluated. Recurrent models such as GRU (Gated Recurrent Units) and LSTM (Long Short-Term Memory networks) are investigated to verify that their impressive performance on sequence modeling tasks like audio synthesis can be transferred over to this challenging domain. The effect of data decomposition using Empirical Mode Decomposition technique is explored. Finally, the effect of the encoder-decoder architecture on model performance is evaluated in this work. The results obtained by the different approaches are then compared to identify the most appropriate model for the case studies analyzed.

Pre-Alignment Guided Attention for Improving Training Efficiency and Model Stability in End-to-End Speech Synthesis

Recently, end-to-end (E2E) neural text-to-speech systems, such as Tacotron2, have begun to surpass the traditional multi-stage hand-engineered systems, with both simplified system building pipelines and high-quality speech. With a unique encoder-decoder neural structure, the Tacotron2 system no longer needs separately learned text analysis front-end, duration model, acoustic model, and audio synthesis module. The key of such a system lies in the attention mechanism, which learns an alignment between the encoder and the decoder, serving as an implicit duration model bridging the text sequence and the acoustic sequence. However, attention learning suffers from low training efficiency and model instability problems, which hinder the E2E approaches from wide deployment. In this paper, we address the problems and propose a novel pre-alignment guided attention learning approach. Specifically, we inject handy prior knowledge-accurate phoneme durations-in the neural network loss function to bias the attention learning to the desired direction more accurately. The explicit time alignment between an audio recording and its corresponding phoneme sequence can be achieved by forced-alignment from an automatic speech recognizer (ASR). The experiments show that the proposed pre-alignment guided (PAG) attention approach can significantly improve training efficiency and model stability. More specifically, the PAG updated version of Tacotron2 can quickly obtain the attention alignment using only 500 (text, audio) pairs, which is apparently not possible for the original Tacotron2. A series of subjective experiments also show that the PAG-Tacotron2 approach can synthesize more stable and natural speech.

Evaluation of near field distance perception in virtual environments

Binaural synthesis of spatial audio for virtual and augmented environments is typically performed by convolving an anechoic signal with impulse responses measured at the listener's ears using a source at a fixed distance. This method is accurate for sounds presented at the measured impulse response's distance, and for more distant sources, decreasing intensity provides a reasonable approximation. However, sources nearer than 1 m provide additional distance cues. The head-shadow effect becomes markedly stronger and exponential sound intensity falloff comes into play, exaggerating inter-aural level differences. Also, the angle to each ear diverges, changing the directionally dependent spectral filtering of the pinnae. Changes in inter-aural level differences can be approximated across listeners, but changes in pinna cues with changes in distance are highly individual, and thus less easily approximated. Reproducing some or all of these cues for each individual may be necessary to create a convincing percept of very near objects in virtual and augmented reality environments. The present work aims to determine the importance of each cue for static and dynamic sources. Listening tests have been conducted in a virtual environment using generic and individual head related impulse responses, with and without near-field compensation.

Perceptual evaluation on the influence of individualized near-field head-related transfer functions on auditory distance localization

It is desired that spatial audio for virtual reality is able to reproduce various auditory localization information so as to recreate virtual source at different directions and distances. Distance-dependent binaural cue (binaural level difference (ILD), loudness, as well as environmental reflections are considered as auditory distance localization cues. In the case of free and near-field within about 1.0 m, the binaural cue which is encoded in near-field head-related transfer function (HRTFs) is an absolute and dominant distance localization cue [D. S. Brungart, JASA, 1999]. However, HRTFs depending on individualized and non-individualized HRTFs are usually used in spatial audio synthesis. In the present work, the perceptual influence of individualized HRTFs distance localization is evaluated by a psychoacoustic experiment. The binaural signals with various bandwidths are synthesized by filtering the input stimuli with individualized and non-individualized near-field HRTFs and then reproduced by headphone. Preliminary results of virtual source localization experiment indicate that individualized HRTFs influences little on distance localization at low frequencies but have some influence at mid and high frequency. [This work was supported by the Natural Science Foundation of China, Grant No. 11574090.]

Descriptor driven concatenative synthesis tool for Python

A command-line tool and Python framework is proposed for the exploration of a new form of audio synthesis known as ‘concatenative synthesis’, a form of synthesis that uses perceptual audio analyses to arrange small segments of audio based on their characteristics. The tool is designed to synthesise representations of an input target sound using a source database of sounds. This involves the segmentation and analysis of both the input sound and database, the matching of input segments to their closest segment from the database, and the resynthesis of the closest matches to produce the final result. The project aims to provide a tool capable of generating high-quality sonic representations of an input, to present a variety of examples that demonstrated the breadth of possibilities that this style of synthesis has to offer and to provide a robust framework on which concatenative synthesis projects can be developed easily. The purpose of this project was primarily to highlight the potential for further development in the area of concatenative synthesis, and to provide a simple and intuitive tool that could be used by composers for sound design and experimentation. The breadth of possibilities for creating new sounds offered by this method of synthesis makes it ideal for digital sound design and electroacoustic composition. Results demonstrate the wide variety of sounds that can be produced using this method of synthesis. A number of technical issues are outlined that impeded the overall quality of results and efficiency of the software. However, the project clearly demonstrates the strong potential for this type of synthesis to be used for creative purposes.

Integrating Gesture Data in Computer-aided Composition: A Framework for Representation, Processing and Mapping

In this article, we discuss physical gestures from the perspective of computer-aided composition (CAC) and propose solutions for the integration of gesture signals as musical materials into these environments. After presenting background and motivations, we review related works and describe design guidelines for our developments. A particular focus is given to the idiosyncrasies of working with gestures in real-time vs differed-time paradigms. We present an implementation of our concepts as the library OM-Geste, integrated into the CAC environment OpenMusic. Two case studies are presented for the generation of a symbolic musical score and spatial audio synthesis from gesture recordings of dance and instrumental performances.

Dynamical Systems for Audio Synthesis: Embracing Nonlinearities and Delay-Free Loops

Many systems featuring nonlinearities and delay-free loops are of interest in digital audio, particularly in virtual analog and physical modeling applications. Many of these systems can be posed as systems of implicitly related ordinary differential equations. Provided each equation in the network is itself an explicit one, straightforward numerical solvers may be employed to compute the output of such systems without resorting to linearization or matrix inversions for every parameter change. This is a cheap and effective means for synthesizing delay-free, nonlinear systems without resorting to large lookup tables, iterative methods, or the insertion of fictitious delay and is therefor suitable for real-time applications. Several examples are shown to illustrate the efficacy of this approach.

ChucK: A Strongly Timed Computer Music Language

ChucK is a programming language designed for computer music. It aims to be expressive and straightforward to read and write with respect to time and concurrency, and to provide a platform for precise audio synthesis and analysis and for rapid experimentation in computer music. In particular, ChucK defines the notion of a strongly timed audio programming language, comprising a versatile time-based programming model that allows programmers to flexibly and precisely control the flow of time in code and use the keyword now as a time-aware control construct, and gives programmers the ability to use the timing mechanism to realize sample-accurate concurrent programming. Several case studies are presented that illustrate the workings, properties, and personality of the language. We also discuss applications of ChucK in laptop orchestras, computer music pedagogy, and mobile music instruments. Properties and affordances of the language and its future directions are outlined.

Touch as techne: rethinking digitality

This article introduces Pulse Project (2011–), a doctoral performance research project that engages in critical discourse between art and science through the creation of digital soundscapes that weave together artistic, medical, technological, ancient and modern methodologies. Pulse ‘reading’, case histories, notations of pulses and programming soundscape compositions are all used together as methods for exploring the cultural encounter between artist, participants and diverse medical practices. Pulse Project seeks to provide an examination of the means with which the temporal materiality of touch can be used together with audio programming to form a translation and synthesis of different ecologies and disciplines, e.g. medicine and art, Eastern and Western practices, touch and digitality, etc. Drawing upon my experience as a clinical acupuncturist (with training in biomedicine), I use Chinese pulse diagnostics together with SuperCollider (an audio synthesis programming language) to inform the composition process of each soundscape. In this way, Pulse Project interrogates the aesthetic and philosophical axioms underpinning contemporary medicine, technology and cognitive embodiment through the exploration of their corollary ‘others’—traditional Chinese medicine and music theory—in order to generate a fresh approach to embodiment and soundscape composition.

The Viability of the Web Browser as a Computer Music Platform

The computer music community has historically pushed the boundaries of technologies for music-making, using and developing cutting-edge computing, communication, and interfaces in a wide variety of creative practices to meet exacting standards of quality. Several separate systems and protocols have been developed to serve this community, such as Max/MSP and Pd for synthesis and teaching, JackTrip for networked audio, MIDI/OSC for communication, as well as Max/MSP and TouchOSC for interface design, to name a few. With the still-nascent Web Audio API standard and related technologies, we are now, more than ever, seeing an increase in these capabilities and their integration in a single ubiquitous platform: the Web browser. In this article, we examine the suitability of the Web browser as a computer music platform in critical aspects of audio synthesis, timing, I/O, and communication. We focus on the new Web Audio API and situate it in the context of associated technologies to understand how well they together can be expected to meet the musical, computational, and development needs of the computer music community. We identify timing and extensibility as two key areas that still need work in order to meet those needs.

An interactive audio synthesizer for investigating formants and timbre

Computer-based investigations of the nature of formants and their role in characterizing vowel sounds and the timbre of musical instruments will be described. One version of this activity requires students to obtain detailed spectra of different vowel sounds, measure relative amplitudes of harmonics in the recorded sounds, and perform additive synthesis using Mathematica to recreate the different vowels. Students can investigate the minimum number of harmonics and formants needed to distinguish vowels. A new version of this activity relies on a recently developed interactive audio synthesis program that reduces the emphasis on tedious measurements and the need to edit Mathematica notebooks, allowing students to devote intellectual effort on investigation. The new program and the associated lab activity will be tested on students during Fall quarter of 2013 as part of the “Physics of Musical Sound” course at Central Washington University.

The Systems Concepts Digital Synthesizer: An Architectural Retrospective

In the mid 1970s, specialized hardware for synthesizing digital audio helped computer music research move beyond its early reliance on software synthesis running on slow mainframe computers. This hardware allowed for synthesis of complex musical scores in real time and for dynamic, interactive control of synthesis. Peter Samson developed one such device, the Systems Concepts Digital Synthesizer, for Stanford University's Center for Computer Research in Music and Acoustics. The “Samson Box” addressed the classical problems of digital audio synthesis with an elegance that still rewards study. This article thoroughly examines the principles underlying the Box's design—while considering how it was actually employed by its users—and describes the architecture's advantages and disadvantages. An interview with Samson is included.

Dynamic Convolution Modeling, a Hybrid Synthesis Strategy

This article outlines a hybrid approach to the synthesis of percussion sounds. The synthesis method described here combines techniques and concepts from physical modeling and convolution to produce audio synthesis of percussive instruments. This synthesis method not only achieves a high degree of realism in comparison with audio samples but also retains some of the flexibility associated with waveguide physical models. When the results are analyzed, the method exhibits some interesting detailed spectral features that have some aspects in common with the behavior of acoustic percussion instruments. In addition to outlining the synthesis process, the article discusses some of the more creative possibilities inherent in this approach, e.g., the use and free combination of excitation and resonance sources from beyond the realms of the purely percussive examples given.

An Interactive 3-D Audio System With Loudspeakers

Traditional 3-D audio systems using two loudspeakers often have a limited sweet spot and may suffer from poor performance in reverberant environments. This paper presents a novel binaural 3-D audio system that actively combines head tracking and room modeling into 3-D audio synthesis. The user's head position and orientation are first tracked by a webcam-based 3-D head tracker. The system then improves its robustness to head movement and strong early reflections by incorporating the tracking information and an explicit room model into the binaural synthesis and crosstalk cancellation process. Sensitivity analysis on the room model shows that the method is reasonably robust to modeling errors. Subjective listening tests confirm that the proposed 3-D audio system significantly improves the users' perception and ability for localization.

El sintesista: contexto, evolución y nuevas perspectivas de desarrollo

Luego de una revisión histórica de la evolución del sintetizador y del estudio de las aproximaciones cognitivas de la enseñanza musical, se enuncian y definen factores que son importantes tener en cuenta en el diseño y la construcción de nuevos instrumentos e interfaces para la síntesis de audio. Multidimensionalidad, espacios tímbricos, retroalimentación multisensorial y expresividad se muestran como aspectos decisivos en la construcción de sintetizadores, formando el paradigma a seguir por la nueva generación de diseñadores de instrumentos digitales. Su reto principal es desarrollar herramientas cada vez más intuitivas que estimulen y faciliten el proceso de acercamiento del músico a la síntesis de audio.

Virtual Conference Audio Reconstruction Based on Spatial Object

This paper proposed a virtual conference audio reconstruction model based on spatial audio object. The aim of the model is to enhance the realistic experience of virtual conference. Firstly, the conference audio synthesis method is given according the principle of the virtual conference. Then the spatial audio parameters interaural level difference (ILD) are used to reconstruct the spatial sound field for each listener based on the theory of spatial audio object coding.

A Virtual Model of Spring Reverberation

The digital emulation of analog audio effects and synthesis components, through the simulation of lumped circuit components has seen a large amount of activity in recent years; electromechanical ef...

Simulation of Directional Microphones in Digital Waveguide Mesh-Based Models of Room Acoustics

Digital waveguide mesh (DWM) models are time-domain numerical methods providing computationally simple solutions for wave propagation problems. They have been used in various acoustical modeling and audio synthesis applications including synthesis of musical instrument sounds and speech, and modeling of room acoustics. A successful model of room acoustics should be able to account for source and receiver directivity. Methods for the simulation of directional sources in DWM models were previously proposed. This paper presents a method for the simulation of directional microphones in DWM-based models of room acoustics. The method is based on the directional weighting of the microphone response according to the instantaneous direction of incidence at a given point. The direction of incidence is obtained from instantaneous intensity that is calculated from local pressure values in the DWM model. The calculation of instantaneous intensity in DWM meshes and the directional accuracies of different mesh topologies are discussed. An intensity-based formulation for the response of a directional microphone is given. Simulation results for an actual microphone with frequency-dependent, non-ideal directivity function are presented.

Analysis and synthesis of humpback whale songs.

The humpback whale song is one of the most complex, non‐human, acoustic displays in the animal kingdom. In this paper, we analyze and synthesize the humpback song using additive synthesis which is one of the most powerful techniques for the analysis, modification, and synthesis of complex audio or speech signals. This method creates complex sounds by adding together individual sinusoidal signals called “partials.” A partial’s frequency and amplitude are each time‐varying functions, so it is a more flexible version of the harmonic associated with a Fourier series decomposition of a periodic waveform. Applying the fast Fourier transform to all overlapping frames (small time segments), we can detect peaks from the frequency spectrum of each frame and calculate the frequency, magnitude, and phase for all the peaks. These extracted parameters are then used as components of the sinusoidal signals (partials) for the synthesis process. This process finally reconstructs and concatenates each frame, generating the synthesized humpback song. We observed that this synthesized song resembles much more closely the original humpback song by the time/frequency domain representation and listening. In the future, we will analyze and synthesis other baleen whale songs with this powerful synthesis method. [Work supported by IITA (IITA‐2008‐(C1090‐0801‐0039)).]