Sound Matching Research Articles

Towards multisensory control of physical modeling synthesis

Physical models of musical instruments offer an interesting tradeoff between computational efficiency and perceptual fidelity. Yet, they depend on a multidimensional space of user-defined parameters whose exploration by trial and error is impractical. Our article addresses this issue by combining two ideas: query by example and gestural control. Prior publications have presented these ideas separately but never in conjunction. On one hand, we train a deep neural network to identify the resonator parameters of a percussion synthesizer from a single audio example via an original method named perceptual-neural-physical sound matching (PNP). On the other hand, we map these parameters to knobs in a digital controller and configure a musical touchpad with MIDI polyphonic expression. Hence, we propose a multisensory interface between human and machine: it integrates haptic and sonic information and produces new sounds in real time as well as visual feedback on the percussive touchpad. We demonstrate the interest of this new kind of multisensory control via a musical game in which participants collaborate with the machine in order to imitate the sound of an unknown percussive instrument as quickly as possible. Our findings show the challenge and promise of future research in musical "Human-AI parternships".

Neural correlates of phonetic categorization under auditory (phoneme) and visual (grapheme) modalities.

We tested whether the neural mechanisms of phonetic categorization are specific to speech sounds or generalize to graphemes (i.e., visual letters) of the same phonetic label. Given that linguistic experience shapes categorical processing, and letter-speech sound matching plays a crucial role during early reading acquisition, we hypothesized sound phoneme and visual grapheme tokens representing the same linguistic identity might recruit common neural substrates, despite originating from different sensory modalities. Behavioral and neuroelectric brain responses (ERPs) were acquired as participants categorized stimuli from sound (phoneme) and homologous letter (grapheme) continua each spanning a /da/ - /ga/ gradient. Behaviorally, listeners were faster and showed stronger categorization of phoneme compared to graphemes. At the neural level, multidimensional scaling of the EEG revealed responses self-organized in a categorial fashion such that tokens clustered within their respective modality beginning ∼150-250 ms after stimulus onset. Source-resolved ERPs further revealed modality-specific and overlapping brain regions supporting phonetic categorization. Left inferior frontal gyrus and auditory cortex showed stronger responses for sound category members compared to phonetically ambiguous tokens, whereas early visual cortices paralleled this categorical organization for graphemes. Auditory and visual categorization also recruited common visual association areas in extrastriate cortex but in opposite hemispheres (auditory = left; visual=right). Our findings reveal both auditory and visual sensory cortex supports categorical organization for phonetic labels within their respective modalities. However, a partial overlap in phoneme and grapheme processing among occipital brain areas implies the presence of an isomorphic, domain-general mapping for phonetic categories in dorsal visual system.

The role of handwriting in English word acquisition among elementary students

In order to investigate whether handwriting has an advantage in learning word form, sound, and meaning, this study randomly selected 40 elementary school student participants (20 males, 20 females, aged 11.4 ± 1.34 years). Using an experimental approach, we compared the learning outcomes of word sound matching, word meaning matching, and word form judgment tasks under two conditions: handwriting and visual learning. After three consecutive days of learning and testing, we found that handwriting generally outperformed visual learning in terms of accuracy and response time in word form, sound, and meaning learning. Additionally, we observed differences in the timing of significant discrepancies in learning outcomes between the two methods across the three tasks. Specifically, in terms of accuracy, discrepancies first appeared in the word sound matching task on the first day, followed by the word form judgment task, and lastly the word meaning matching task. Regarding response time, significant differences between learning methods first emerged in the word form judgment task, followed by the word sound and word meaning tasks. Thus, combining accuracy and response time data, we conclude that handwriting is more advantageous than visual learning for word acquisition, with a differential impact on word form, sound, and meaning, where word form and sound are prioritized over meaning.

Blind extraction of guitar effects through blind system inversion and neural guitar effect modeling

Audio effects are an ubiquitous tool in music production due to the interesting ways in which they can shape the sound of music. Guitar effects, the subset of all audio effects focusing on guitar signals, are commonly used in popular music to shape the guitar sound to fit specific genres or to create more variety within musical compositions. Automatic extraction of guitar effects and their parameter settings, with the aim to copy a target guitar sound, has been previously investigated, where artificial neural networks first determine the effect class of a reference signal and subsequently the parameter settings. These approaches require a corresponding guitar effect implementation to be available. In general, for very close sound matching, additional research regarding effect implementations is necessary. In this work, we present a different approach to circumvent these issues. We propose blind extraction of guitar effects through a combination of blind system inversion and neural guitar effect modeling. That way, an immediately usable, blind copy of the target guitar effect is obtained. The proposed method is tested with the phaser, softclipping and slapback delay effect. Listening tests with eight subjects indicate excellent quality of the blind copies, i.e., little to no difference to the reference guitar effect.

A review of differentiable digital signal processing for music and speech synthesis

The term “differentiable digital signal processing” describes a family of techniques in which loss function gradients are backpropagated through digital signal processors, facilitating their integration into neural networks. This article surveys the literature on differentiable audio signal processing, focusing on its use in music and speech synthesis. We catalogue applications to tasks including music performance rendering, sound matching, and voice transformation, discussing the motivations for and implications of the use of this methodology. This is accompanied by an overview of digital signal processing operations that have been implemented differentiably, which is further supported by a web book containing practical advice on differentiable synthesiser programming (https://intro2ddsp.github.io/). Finally, we highlight open challenges, including optimisation pathologies, robustness to real-world conditions, and design trade-offs, and discuss directions for future research.

Hybrid NVH modeling approach: High quality of NVH results enables psychoacoustic analysis of numerical computations

In the field of acoustics, the human perception of sound is often more important than the overall sound pressure level. The human perception is highly individual and is linked to how the user's expectations of the sound and the actual perceived sound match. Thus, generalization is difficult. However, psychoacoustic parameters that are developed and validated by means of jury tests are an established tool in the field of experimental measurements to quantify the human perception for specific acoustic phenomena like tonality, roughness, and loudness. Since the precision of numerical simulations results is continuously improving, these psychoacoustics parameters can be applied as well to the results of numerical simulation. In this Paper, the advantages of this approach are shown using the example of a numerical model of an e-bike. E-bikes have become increasingly popular in recent years, used for commuting, transporting loads or, in the case of e-mountain bikes, just for fun. As already mentioned, the acceptance of a sound is strongly linked to the expectations of the user. Since the electric drive unit of an e-bike is added to an acoustically known system, the user expects the sound to be quiet rather than dominant. Different e-bike analyses are presented using numerical and experimental methods. The numerical model used in this project predicts the sound generated by the electrical drive unit and radiated by the frame. For the calculation of airborne noise the distribution of velocities on the surface of the frame is used as an input, which is taken from a previous simulation of structure-borne noise. The model to simulate the structure-borne noise consists of a detailed representation of the carbon frame and simplified representations of different components such as battery, fork etc. With the help of the numerical model, different combinations of frame and electrical drive can be analyzed and compared by psychoacoustic parameters. The validity of the approach is shown by means of data from experimental measurements.

Learning to Solve Inverse Problems for Perceptual Sound Matching

Learning to Solve Inverse Problems for Perceptual Sound Matching

Spanish Phonological Awareness in Kindergarten Uniquely Supports Second-Grade English Morphological Awareness in Spanish-English Dual Language Learners.

This study investigated the longitudinal, cross-linguistic developmental relationships of phonological awareness (PA), letter identification (letter ID), and morphological awareness (MA) in 71 heritage Spanish-English dual language learners (DLLs) in kindergarten and second grade. Multiple linear regression was used to test if kindergarten Spanish and English PA (sound elision and sound matching) and letter ID significantly predicted later English MA (oral derived word stress judgment, oral derivational morpheme blending, written derived word decomposition, and morphologically complex word spelling) performance in second grade. Cross-linguistically, the PA skill of sound matching in kindergarten was the most reliable predictor of MA in second grade for Spanish-English DLLs. Spanish PA explained the majority of variation in oral MA skills in English. English PA was only uniquely predictive of written MA skills in English. Both the cognitive operation of sound sequence manipulation in PA (elision or matching) and the modality of morpheme representation in MA (oral or written) appear to mediate the transfer of metalinguistic knowledge in Spanish-English DLL development in early elementary school. Results are discussed within the context of classroom practices.

TOWARDS A PAN-EU BUILDING FOOTPRINT MAP BASED ON THE HIERARCHICAL CONFLATION OF OPEN DATASETS: THE DIGITAL BUILDING STOCK MODEL - DBSM

Abstract. This paper presents a hierarchical conflation process applied to open datasets for the creation of a seamless pan-European map of building footprints in vector format, named Digital Building Stock Model – DBSM. The objective is the sequential addition of input components (which currently include OpenStreetMap, Microsoft GlobalML Building Footprints, European Settlement Map), taking into account their limitations, and aiming at the highest level of completeness possible, for planning and evaluating energy transition scenarios at the EU level. The results indicate how DBSM compares robustly against cadastral data from Estonia, used as reference area. The comparison of DBSM with GHS-BUILT-S, a 10 metres resolution grid with worldwide coverage that encodes the built-up surface in each pixel as derived from Sentinel-2 imagery for the year 2018, reveals a relative overestimation of the latter, factored by 0.68 at the EU scale for a sound match.

Algorithm-based mapping of products in a branded Canadian food and beverage database to their equivalents in Health Canada's Canadian Nutrient File.

There is increasing recognition of the value of linking food sales databases to national food composition tables for population nutrition research. Expanding upon automated and manual database mapping approaches in the literature, our aim was to match 1,179 food products in the Canadian data subset of Euromonitor International's Passport Nutrition to their closest respective equivalents in Health Canada's Canadian Nutrient File (CNF). Matching took place in two major steps. First, an algorithm based on thresholds of maximal nutrient difference (between Euromonitor and CNF foods) and fuzzy matching was executed to offer match options. If a nutritionally appropriate match was available among the algorithm suggestions, it was selected. When the suggested set contained no nutritionally sound matches, the Euromonitor product was instead manually matched to a CNF food or deemed unmatchable, with the unique addition of expert validation to maximize meticulousness in matching. Both steps were independently performed by at least two team members with dietetics expertise. Of 1,111 Euromonitor products run through the algorithm, an accurate CNF match was offered for 65% of them; missing or zero-calorie data precluded 68 products from being run in the algorithm. Products with 2 or more algorithm-suggested CNF matches had higher match accuracy than those with one (71 vs. 50%, respectively). Overall, inter-rater agreement (reliability) rates were robust for matches chosen among algorithm options (51%) and even higher regarding whether manual selection would be required (71%); among manually selected CNF matches, reliability was 33%. Ultimately, 1,152 (98%) Euromonitor products were matched to a CNF equivalent. Our reported matching process successfully bridged a food sales database's products to their respective CNF matches for use in future nutritional epidemiological studies of branded foods sold in Canada. Our team's novel utilization of dietetics expertise aided in match validation at both steps, ensuring rigor and quality of resulting match selections.

Quality-diversity for Synthesizer Sound Matching

Synthesizers have become commonplace in music production due to their ability to create a diverse and unique timbre. However, many users find it difficult to control a synthesizer due to the complexity of the relationship between synthesis parameters and output sound. As such, the sound matching task focuses on the estimation of synthesis parameters that recreate the target sound as closely as possible within the capabilities of the synthesizer. In this paper, we introduce a quality-diversity approach to the problem of sound matching by introducing a novelty objective to genetic algorithm-based sound matching. We show that this QD method allows for the discovery of a collection of matches that are more diverse compared to matches found by standard genetic algorithms. These matches are spread across a space of interpretable spectral features, making it useful for the intuitive discovery of unique and useful matches.

Improving Semi-Supervised Differentiable Synthesizer Sound Matching for Practical Applications

While synthesizers have become commonplace in music production, many users find it difficult to control the parameters of a synthesizer to create a sound as they intended. In order to assist the user, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sound matching</i> task aims to estimate synthesis parameters that produce a sound that is as close as possible to the query sound. Recently, neural networks have been employed for this task. These neural networks are trained on paired data of synthesis parameters and the corresponding output sound, optimizing a loss of synthesis parameters. However, query by the user usually consists of real-world sounds, different from the synthesizer output sounds used as training data. In a previous work, the authors presented a sound matching method where the synthesizer is implemented using differentiable DSP. The estimator network could then be trained by directly optimizing the spectral similarity between the original sound and the output sound. Furthermore, the network could be trained on real-world sounds whose ground-truth synthesis parameters are unavailable. This method was shown to improve the match quality in both objective and subjective measures. In this work, we experiment with different synthesizer configurations and extend this approach to a more practical synthesizer with effect modules and envelope generators. We propose a novel training strategy where the network is fully trained using both parameter loss and spectral loss. We show that models trained using this strategy is able to utilize the chorus effect effectively while models that switch completely to spectral loss underutilizes the chorus effect.

WITHDRAWN: Semi-Supervised Learning with Ensemble Deep Learning Networks for Descriptor Generation

Deep neural networks and machine learning have made many real-world tasks easier. They can analyze large data sets that humans can't. Deep neural networks can be utilized for image, video, and sound matching, making them an intriguing research area. This research focuses on semi-supervised training and proposes a "enhanced semi-supervised" learning strategy. Robots can learn from labeled and unlabeled data with semi-supervised learning. Supervised learning employs labeled data. Semi-supervised learning is used to govern the Deep neural network's output. Assigning values to input groups and accessing the output area have no interaction. This method seeks to deliver a more efficient learning approach with an equitable distribution of output throughout the output field of space, and the authors developed an ensemble strategy based on three deep learning approaches (Convolution Deep neural network, Alexnet, and MobileNetv2). The result is a more effective learning approach. The handwritten digit experiment was 90.74 percent accurate, whereas Alzheimer's detection was 99.76 percent accurate. When the proposed strategy was used on two experimental data sets, the accuracy was better than when different applications used Siamese Deep neural networks.

Two-Phase Sensor Decision: Machine-Learning for Bird Sound Recognition and Vineyard Protection

For a wireless sensor network consisting of numerous sensors, spread over a large area with no direct power supply, energy efficiency is of paramount importance. As most power is consumed by the communication module, special attention has to be paid to reduce communication needs as much as possible. The more data is sent, the larger the power requirement of the sensor module. Preprocessing can help in reducing the amount of data to send. However, it also consumes energy. This paper focuses on this tradeoff between preprocessing, pre-filtering and preselecting of sensor data on one hand, and uploading of unprocessed and unfiltered raw data on the other hand, for the special case of protecting vineyards from starlings. The paper proposes a two-phase decision mechanism based on machine learning: the less complex first phase is executed on the microcontroller of the sensor module, while the more complex, more accurate second phase is performed in the cloud. Individual noise sensors monitor the environment, and try to detect starling songs, using a simple, SVM-based classification. These sensors are grouped into clusters, through a mechanism similar to the well-known LEACH protocol, and signal to the current cluster-head the likelihood of starling presence. If several alerts are received to justify further investigation, the cluster-head asks the node with highest starling detection likelihood to upload a one second sound sample to the cloud. There, the more complex and more accurate second phase sound matching is performed, and the actuators deployed in the field are remotely triggered, if needed.

Optimization Simulation of Dance Technical Movements and Music Matching Based on Multifeature Fusion.

Music and dance videos have been popular among researchers in recent years. Music is one of the most important forms of human communication; it carries a wealth of emotional information, and it is studied using computer tools. In the feature engineering process, most present machine learning approaches suffer from information loss or insufficient extracted features despite the relevance of computer interface and multimedia technologies in sound and music matching tasks. Multifeature fusion is widely utilized in education, aerospace, intelligent transportation, biomedicine, and other fields, and it plays a critical part in how humans get information. In this research, we offer an effective simulation method for matching dance technique movements with music based on multifeature fusion. The initial step is to use music beat extraction theory to segment the synchronized dance movements and music data, then locate mutation points in the music, and dynamically update the pheromones based on the merits of the dance motions. The audio feature sequence is obtained by extracting audio features from the dancing video's accompanying music. Then, we combine the two sequences to create an entropy value sequence based on audio variations. By comparing the consistency of several approaches for optimizing dance movement simulation trials, the optimized simulation method described in this research has an average consistency of 87%, indicating a high consistency. As a result, even though the background and the subject are readily confused, the algorithm in this research can keep a consistent recognition rate for more complicated dance background music, and the approach in this study can still guarantee a certain accuracy rate.

Survival of the synthesis—GPU accelerating evolutionary sound matching

AbstractManually configuring synthesizer parameters to reproduce a particular sound is a complex and challenging task. Researchers have previously used different optimization algorithms, including evolutionary algorithms to find optimal sound matching solutions. However, a major drawback to these algorithms is that they typically require large amounts of computational resources, making them slow to execute. This article proposes an optimized design for matching sounds generated by frequency modulation (FM) audio synthesis using the graphics processing unit (GPU). A benchmarking suite is presented for profiling the performance of three implementations: serial CPU, data‐parallel CPU, and data‐parallel GPU. Results have been collected and discussed from a high‐end NVIDIA desktop and a mid‐range AMD laptop. Using the default configuration for simple FM, the GPU accelerated design had a speedup of 128 over the naive serial implementation and 8.88 over the parallel CPU version on a desktop with an Intel i7 9800X CPU and NVIDIA RTX GeForce 2080Ti GPU. Furthermore, the relative speedup over the naive serial implementation continues to increase beyond simple FM to more advanced structures. Further observations include comparisons between integrated and discrete GPUs, toggling optimizations, and scaling evolutionary strategy population size.

A Prospective Study of the Effect of Tinnitus Sound Matching Degree on the Efficacy of Customized Sound Therapy in Patients with Chronic Tinnitus

Objectives: The aim of this study was to explore and compare the customized sound therapy effect between tinnitus sound matching and nonmatching patients in tinnitus customized sound therapy and therapy-related influencing factors. Methods: This prospective study investigated a total of 100 patients with unilateral chronic tinnitus who received customized sound therapy. The participants were dichotomously divided into matching (group A) and nonmatching (group B) groups after 4 stages of tinnitus matching via the tinnitus assistant app (provided by Sound Ocean Company, SuZhou, China). Each group consists of 50 participants. Before and 6 months after the treatment, Hospital Anxiety and Depression Scale (HADS), tinnitus handicap inventory (THI), and tinnitus loudness Visual Analog Scale (VAS) were used to evaluate the customized sound therapy effect and explore other related influencing factors. Results: (1) The HADS-A, HADS-D, THI, and VAS scores of 2 groups were both significantly decreased after treatment. (2) The HADS-A and THI scores improved markedly in group A than that in group B, which could be related to the hearing loss of the tinnitus side ear before treatment; the lighter the degree of hearing loss, the better the improvement. No statistically significant differences were detected in HADS-D and VAS scores between the 2 groups, and also, these were not related to the degree of hearing loss. The differences in age, gender, and tinnitus duration did not show any statistically significant effect on the improvement of the 2 groups. Conclusions: Both tinnitus sound matching and nonmatching of the customized sound therapy brought a significant effect to tinnitus participants. Our study also suggests that THI and HADS-A scores of those with tinnitus matching participants improved markedly as compared to those of nonmatching participants, and the customized sound therapy effect is negatively correlated with the severity of hearing loss.

Difference Between Children and Adults in the Print-speech Coactivated Network

ABSTRACT Skilled reading requires the convergent brain network between spoken language and reading. Here, we examined the developmental changes in this convergent network between beginning and skilled readers. We used fMRI data from 41 adults (aged from 20 to 21 years) and 21 children (aged from 9 to 12 years) who performed a Chinese word sound matching task in speech and print. Three complementary analytic approaches, including print-speech conjunction analysis, brain-behavioral analysis, and functional connectivity analysis were performed to reveal the link between print and speech neural systems as well as the developmental changes in this link. We found that (1) adults showed broader convergence than children; (2) coactivation in anterior dorsal regions (i.e., inferior frontal gyrus and insula) predominated in children, while coactivation in the ventral posterior region (i.e., middle temporal gyrus) predominated in adults, which aligns with the dual-stream model of reading; and (3) print-speech convergence regions showed task-specific functional connectivity patterns and there was an increased visual-modal specialization in the functional connectivity patterns with reading development. Our study provides important evidence for developmentally increased coactivation in ventral posterior regions and increased task specialization in print-speech convergence regions, which is crucial for successful reading.

Auditory perceptual processing during musical imagery: An event-related potential study

The perceptual processing of a sound is facilitated when the sound matches auditory imagery. Previous studies have shown that auditory imagery and actual sound activate the auditory cortex in a similar fashion. To investigate whether auditory imagery is a modality-specific representation or an amodal representation, the current study examined how watching silent music videos affected the auditory processing of sound excerpts. Twenty university students were asked to form musical imagery of Japanese popular songs while watching the official music videos. Event-related brain potentials were recorded in response to short sound excerpts from the on-screen video or from a different video. The results showed that the amplitude of the exogenous N1 component (90–110 ms) was smaller for imagery-matched than for unmatched sound excerpts. The electrical source of the difference was estimated in the auditory cortex. After the N1, the matched excerpts elicited a larger late positive potential (400–800 ms) than the unmatched excerpts. These findings suggest that auditory imagery involves modality-specific neural processing and that imagery-matched sounds are processed efficiently at an early stage, inducing additional cognitive processing at a later stage.

Reconstruction of annual accumulation rate on firn, synchronising H&amp;lt;sub&amp;gt;2&amp;lt;/sub&amp;gt;O&amp;lt;sub&amp;gt;2&amp;lt;/sub&amp;gt; concentration data with an estimated temperature record

Abstract. This work deals with reconstructing firn layer thicknesses at the deposition time from the firn's observed thickness in ice cores, thus reconstructing the annual accumulation, yielding a timescale and an ice-core chronology. We employed a dynamic time warping algorithm to find an optimal, non-linear alignment between an H2O2 concentration data series from 98 m worth of ice cores of a borehole on the central ice divide of the Detroit Plateau, the Antarctic Peninsula, and an estimated local temperature time series. The viability and the physical reliability of the procedure are rooted in the robustness of the seasonal marker H2O2 in a high-accumulation context, which brought the entire borehole to within the operational life span of four Antarctic stations around the Antarctic Peninsula. The process was heavily based on numerical optimisation, producing a mathematically sound match between the two series to estimate the annual layering efficiently on the entire data section at once, being disposition-free. The results herein confirm a high annual accumulation rate of aN=2.8 m w.e./yr, which is of the same order of magnitude as and highly correlated with that of the Bruce Plateau and twice as large as that of the Gomez Plateau, 300 and 1200 km further south, respectively.