Monophonic Audio Research Articles

Cross-modal generative model for visual-guided binaural stereo generation

Binaural stereo audio is recorded by imitating the way the human ear receives sound, which provides people with an immersive listening experience. Existing approaches leverage autoencoders and directly exploit visual spatial information to synthesize binaural stereo, resulting in a limited representation of visual guidance. For the first time, we propose a visually guided generative adversarial approach for generating binaural stereo audio from mono audio. Specifically, we develop a Stereo Audio Generation Model (SAGM), which utilizes shared spatio-temporal visual information to guide the generator and the discriminator to work separately. The shared visual information is updated alternately in the generative adversarial stage, allowing the generator and discriminator to deliver their respective guided knowledge while visually sharing. The proposed method learns bidirectional complementary visual information, which facilitates the expression of visual guidance in generation. In addition, spatial perception is a crucial attribute of binaural stereo audio, and thus the evaluation of stereo spatial perception is essential. However, previous metrics failed to measure the spatial perception of audio. To this end, a metric to measure the spatial perception of audio is proposed for the first time. The proposed metric is capable of measuring the magnitude and direction of spatial perception in the temporal dimension. Further, considering its function, it is feasible to utilize it instead of demanding user studies to some extent. The proposed method achieves state-of-the-art performance on 2 datasets and 5 evaluation metrics. Qualitative experiments and user studies demonstrate that the method generates space-realistic stereo audio.

Using deep learning for recreating binaural audio

In recent years, there has been an increase in research around generating spatialized audio using a mono audio signal. Methods like using neural networks which combine image segmentation with object location for adding back the spatial qualities are often developed. Instead, our project focuses on taking arbitrary mono input sound and an input angle, and outputting spatial stereo audio of the input sound with the directionality of the angle. This is different from current implementations as it is a simpler approach to what spatial audio generation is, and it allows for the use in the model in new areas. Using a binaural microphone and a custom-made anechoic chamber, 120 hours of labelled binaural audio was recorded for use in our model. The audio consists of frequency sweeps, pink noise, and phonetically balanced speech. Our method is to predict a complex short-time Fourier transform mask which will contain the phase and amplitude within it. The model is an autoencoder based on the U-NET model, which is applied to the mono input before being compared to the labelled data for training. With this simpler approach, we hope to make spatial audio more accessible for a variety of applications.

Multi-space channel representation learning for mono-to-binaural conversion based audio deepfake detection

Audio deepfake detection (ADD) aims to detect the fake audio generated by text-to-speech (TTS), and voice conversion (VC), etc., which is an emerging topic. Traditionally we read the mono signal and analyze the artifacts directly. Recently, the mono-to-binaural conversion based ADD approach has attracted increasing attention since the binaural audio signals provide a unique and comprehensive perspective on speech perception. Such method attempts tried to first convert the mono audio into binaural, then process the left and right channels respectively to discover authenticity cues. However, the acoustic information from the two channels exhibits both differences and similarities, which have not been thoroughly explored in previous research. To address this issue, we propose a new mono-to-binaural conversion based ADD framework that considers multi-space channel representation learning, termed “MSCR-ADD”. Specifically, (1) the feature representations of the respective channels are learned by the channel-specific encoder and stored in the channel-specific space; (2) the feature representations capturing the difference between the two channels are learned by the channel-differential encoder and stored in the channel-differential space; (3) after which the channel-invariant encoder learn the channel commonality representations in the channel-invariant space. Note that we propose orthogonal and mutual information maximization losses to constrain the channel-specific and invariant encoders. At last, three representations from various spaces are mixed together to finalize the deepfake detection. It is worth noting that the feature representations in the channel-differential and invariant spaces unveil the differences and similarities between the two channels in binaural audio, enabling us to effectively detect artifacts in fake audio. The experimental results on four benchmark datasets demonstrate that our MSCR-ADD is superior to existing state-of-the-art approaches.

Teaching Integration of Piano and Traditional Music Elements in Colleges and Universities Based on Network Flow Optimization

Abstract In this paper, from the basis of piano music theory, pitch and twelve equal temperaments are studied, and due to the characteristics of the piano’s construction, there are different timbres and harmonic structures. In order to realize the digitization of music signals for piano teaching, the collected analog signals are converted into digital signals by Fourier transform, the samples are inputted into the neural network model to get the output results, and the loss function based on Markov’s music language model in piano teaching is used to find the partial derivatives of the parameters to realize the maximization of the efficiency of the piano pitch and note recognition. According to the multi-task learning in neural networks and the envelope curve of notes, we construct the automatic music transcription model in piano teaching based on CNN-HMM and use the simulation experiment analysis to empirically analyze the piano teaching based on CNNHMM. The results show that the dual-channel audio results are improved by 2.68% in F1 value, 3.18% in accuracy and 2.61% in recall than the mono audio results. That is, dual-channel audio has richer information than mono audio, which enables the CNN-HNN network to learn richer features and thus improve the pitch and note recognition results. In this paper, the reliability of the teaching quality evaluation results of the CNN-HMM-based quantitative assessment model for facilitating the combination of electronic organ and piano teaching concepts ranges from 78.1% to 85.4%. This study provides guiding value for the development of digital piano teaching, which is of great significance for the innovative reform of piano teaching in colleges and universities.

Comparative Study of Musical Timbral Variations: Crescendo and Vibrato Using FFT-Acoustic Descriptor

A quantitative evaluation of the musical timbre and its variations is important for the analysis of audio recordings and computer-aided music composition. Using the FFT acoustic descriptors and their representation in an abstract timbral space, variations in a sample of monophonic sounds of chordophones (violin, cello) and aerophones (trumpet, transverse flute, and clarinet) sounds are analyzed. It is concluded that the FFT acoustic descriptors allow us to distinguish the timbral variations in the musical dynamics, including crescendo and vibrato. Furthermore, using the Random Forest algorithm, it is shown that the FFT-Acoustic provides a statistically significant classification to distinguish musical instruments, families of instruments, and dynamics. We observed an improvement in the FFT-Acoustic descriptors when classifying pitch compared to some timbral features of Librosa.

Exploring the relationships between teacher noticing, ambisonic audio, and variance in focus when viewing 360 video.

A growing body of research has supported the implementation of innovative and immersive video for teaching and learning across the lifespan. Immersive video, delivered through eXtended Reality (XR) tools like 360 video, provides users with new ways to see real or created environments. Unfortunately, most of the existing research has highlighted immersive video without accompanying immersive audio. This use of monophonic audio can create a disconnect for viewers as they experience close to real world video with sounds that do not match a real-world environment. The purpose of this study was to respond to this gap in the literature by exploring the use of ambisonic audio and its impact on preservice teacher noticing and variability of viewing focus when watching 360 video. Data were collected from undergraduate teacher education students who participated in a self-paced online activity that included watching 360 videos and responding to a questionnaire. A convergent mixed methods design was employed to compare participants' professional noticing and observed viewing behavior in the context of ambisonic and monophonic audio. Results showed that users in ambisonic audio conditions in 360 video environments were more likely to have higher focus. Moreover, for users who had specific professional knowledge, monophonic audio with immersive video had a negative impact on their variance in focus. The paper concludes with recommendations for future research on the use of audio in virtual and augmented reality environments.

Bit rate required for mono audio object in object-based audio program compressed with MPEG-H 3D Audio

Advantage of object-based audio (OBA) is that customization of the reproduced sound by individual listeners is allowed. To design the broadcasting system and transmission line for OBA, bit rate estimation of OBA program is essential; thus, how much bit rate is required for mono audio object is critical information. The bit rate required for mono audio object in OBA program compressed with MPEG-H 3D Audio is investigated using 12 mono test materials in terms of whether the broadcast quality is met. The obtained results are statistically analyzed and the bit rate meeting the broadcast quality and the bit rate of the detection limit for each mono test material is derived.

Similarity of Musical Timbres Using FFT-Acoustic Descriptor Analysis and Machine Learning

Musical timbre is a phenomenon of auditory perception that allows the recognition of musical sounds. The recognition of musical timbre is a challenging task because the timbre of a musical instrument or sound source is a complex and multifaceted phenomenon that is influenced by a variety of factors, including the physical properties of the instrument or sound source, the way it is played or produced, and the recording and processing techniques used. In this paper, we explore an abstract space with 7 dimensions formed by the fundamental frequency and FFT-Acoustic Descriptors in 240 monophonic sounds from the Tinysol and Good-Sounds databases, corresponding to the fourth octave of the transverse flute and clarinet. This approach allows us to unequivocally define a collection of points and, therefore, a timbral space (Category Theory) that allows different sounds of any type of musical instrument with its respective dynamics to be represented as a single characteristic vector. The geometric distance would allow studying the timbral similarity between audios of different sounds and instruments or between different musical dynamics and datasets. Additionally, a Machine-Learning algorithm that evaluates timbral similarities through Euclidean distances in the abstract space of 7 dimensions was proposed. We conclude that the study of timbral similarity through geometric distances allowed us to distinguish between audio categories of different sounds and musical instruments, between the same type of sound and an instrument with different relative dynamics, and between different datasets.

Artificial intelligence-based classification performance evaluation in monophonic and polyphonic indian classical instruments recognition with hybrid domain features amalgamation

In computer music, instrument recognition is a critical part of sound modeling. Pitch, timbre, loudness, duration, and spatialization are all components of musical sounds. All of these components play a significant part in determining the quality of the tonal sound. It is possible to alter the first four parameters, but timbre always poses a challenge [6]. It was inevitable that timbre would take center stage. Musical instruments are distinguished from one other by their distinct sound quality, independent of their pitch or volume. To distinguish between monophonic and polyphonic music recordings, this method might be used. In Musical Information Retrieval, classification plays one of the critical role. Monophonic instrument classification can be found in literature with quiet a substantial combinations of features and classifiers. Polyphonic instrument classification witnessed less references in the literature and is still an area to be explored specifically when it comes to Indian Classical domain. The present paper exactly focusses on this experimentation. Several Indian instruments were used to produce training data sets for the proposed approach’s evaluation purposes. Among the instruments utilized are the flute, harmonium, and sitar. Statistical and spectral factors are used to classify Indian musical instruments along with the Artificial Intelligence-based methods. Hybrid features from multiple domains that extract essential musical properties are extracted. Accuracy is demonstrated through an Indian Musical Instrument SVM and GMM classification. With monophonic sounds, SVM and Polyphonic produce an average accuracy of 89% and 91%. GMM outperforms SVM in monophonic recordings by a factor of 96.33 and polyphonic recordings by a factor of 93.33, according to the results of the studies. The future scope of this recognition framework can be an Artificial Intelligence System with a system linked with the Industrial Internet of Things (IIOT) framework to develop a standalone system or application which can be used for real- time classification of instruments.

Points2Sound: from mono to binaural audio using 3D point cloud scenes

For immersive applications, the generation of binaural sound that matches its visual counterpart is crucial to bring meaningful experiences to people in a virtual environment. Recent studies have shown the possibility of using neural networks for synthesizing binaural audio from mono audio by using 2D visual information as guidance. Extending this approach by guiding the audio with 3D visual information and operating in the waveform domain may allow for a more accurate auralization of a virtual audio scene. We propose Points2Sound, a multi-modal deep learning model which generates a binaural version from mono audio using 3D point cloud scenes. Specifically, Points2Sound consists of a vision network and an audio network. The vision network uses 3D sparse convolutions to extract a visual feature from the point cloud scene. Then, the visual feature conditions the audio network, which operates in the waveform domain, to synthesize the binaural version. Results show that 3D visual information can successfully guide multi-modal deep learning models for the task of binaural synthesis. We also investigate how 3D point cloud attributes, learning objectives, different reverberant conditions, and several types of mono mixture signals affect the binaural audio synthesis performance of Points2Sound for the different numbers of sound sources present in the scene.

Acoustic diversity of forested landscapes: Relationships to habitat structure and anthropogenic pressure

Understanding how human-dominated landscapes affect biodiversity and ecosystems is essential for effective conservation planning. This work aims to understand the relationship between the acoustic diversity of forested landscapes, and descriptors of habitat structure, composition, and anthropogenic pressure, as well as to identify the characteristic scale at which acoustic community diversity relates to those metrics in central Massachusetts.Ten passive acoustic recorders were placed within forest areas in central Massachusetts, during the breeding season. Mono audio recordings were collected during the dawn chorus. The relationship between acoustic indices (AI), and core habitat quality, connectivity, vegetation productivity, percent tree cover, human edge, artificial illumination, and traffic noise were assessed.Significant relationships were found between AI and variables related to habitat structure and human pressure. Sounds related to biota (biophony) and acoustic complexity were positively correlated with core habitat quality, connectivity, and vegetation while negatively correlated with human pressure variables, including nighttime lights, traffic noise, and human edge. AI can therefore act as successful indicators of habitat quality in highly modified landscapes The highest correlations were found at buffers between 1.5 and 3 Km. This response of AI to the broad spatial context and not to the local site characteristics indicate that they can act as robust landscape-scale indicators.The large characteristic scale indicates that urban planning should consider potential impacts acting at scales beyond site planning. Moreover, conservation planning can benefit from managing the context matrix to support biodiversity, particularly traffic noise and artificial illumination reduction initiatives.

Binaural Synthetic Aperture Imaging of the Field of Audition as the Head Rotates and Localisation Perception of Monophonic Sound Listened to through Headphones

A human listening to monophonic sound through headphones perceives the sound to emanate from a point inside the head at the auditory centre at effectively zero range. The extent to which this is predicted by synthetic-aperture calculation performed in response to head rotation is explored. The instantaneous angle between the auditory axis and the acoustic source, lambda, for the zero inter-aural time delay imposed by headphones is 90°. The lambda hyperbolic cone simplifies to the auditory median plane, which intersects a spherical surface centred on the auditory centre, along a prime meridian lambda circle. In a two-dimensional (2-D) synthetic-aperture computation, points of intersection of all lambda circles as the head rotates constitute solutions to the directions to acoustic sources. Geometrically, lambda circles cannot intersect at a point representing the auditory centre; nevertheless, 2-D synthetic aperture images for a pure turn of the head and for a pure lateral tilt yield solutions as pairs of points on opposite sides of the head. These can reasonably be interpreted to be perceived at the sums of the position vectors of the pairs of points on the acoustic image, i.e., at the auditory centre. But, a turn of the head on which a fixed lateral tilt of the auditory axis is concomitant (as in species of owl) yields a 2-D synthetic-aperture image without solution. However, extending a 2-D synthetic aperture calculation to a three-dimensional (3-D) calculation will generate a 3-D acoustic image of the field of audition that robustly yields the expected solution.

Binaural audio generation via multi-task learning

We present a learning-based approach for generating binaural audio from mono audio using multi-task learning. Our formulation leverages additional information from two related tasks: the binaural audio generation task and the flipped audio classification task. Our learning model extracts spatialization features from the visual and audio input, predicts the left and right audio channels, and judges whether the left and right channels are flipped. First, we extract visual features using ResNet from the video frames. Next, we perform binaural audio generation and flipped audio classification using separate subnetworks based on visual features. Our learning method optimizes the overall loss based on the weighted sum of the losses of the two tasks. We train and evaluate our model on the FAIR-Play dataset and the YouTube-ASMR dataset. We perform quantitative and qualitative evaluations to demonstrate the benefits of our approach over prior techniques.

Hybrid Feature Based Classifier Performance Evaluation of Monophonic and Polyphonic Indian Classical Instruments Recognition

Instrument recognition in computer music is an important research area that deals with sound modelling. Musical sounds comprises of five prominent constituents which are Pitch, timber, loudness, duration, and spatialization. The tonal sound is function of all these components playing critical role in deciding quality. The first four parameters can be modified, but timbre remains a challenge [6]. Then, inevitably, timbre became the focus of this piece. It is a sound quality that distinguishes one musical instrument from another, regardless of pitch or volume, and it is critical. Monophonic and polyphonic recordings of musical instruments can be identified using this method. To evaluate the proposed approach, three Indian instruments were experimented to generate training data set. Flutes, harmoniums, and sitars are among the instruments used. Indian musical instruments classify sounds using statistical and spectral parameters. The hybrid features from different domains extracting important characteristics from musical sounds are extracted. An Indian Musical Instrument SVM and GMM classifier demonstrate their ability to classify accurately. Using monophonic sounds, SVM and Polyphonic produce an average accuracy of 89.88% and 91.10%, respectively. According to the results of the experiments, GMM outperforms SVM in monophonic recordings by a factor of 96.33 and polyphonic recordings by a factor of 93.33.

Identification of Fake Stereo Audio Using SVM and CNN

The number of channels is one of the important criteria in regard to digital audio quality. Generally, stereo audio with two channels can provide better perceptual quality than mono audio. To seek illegal commercial benefit, one might convert a mono audio system to stereo with fake quality. Identifying stereo-faking audio is a lesser-investigated audio forensic issue. In this paper, a stereo faking corpus is first presented, which is created using the Haas effect technique. Two identification algorithms for fake stereo audio are proposed. One is based on Mel-frequency cepstral coefficient features and support vector machines. The other is based on a specially designed five-layer convolutional neural network. The experimental results on two datasets with five different cut-off frequencies show that the proposed algorithm can effectively detect stereo-faking audio and has good robustness.

Monophonic Musical Instrument Sound Classification Using Impulse Response Modeling

The field of music has promising commercial and social applications. Hence it has attracted the attention of researchers, engineers, sociologists and health care peoples. Therefore this particular research area has been selected.
 In this manuscript the monophonic musical classificationsystem using impulse response of the system is presented. In this research work 19 musical instruments monophonic sounds from 4 families are classified using WEKA classifier. The impulse response is of all musical instruments and families are computed in Cepstral Domain. AsImpulse response is used to model the body response of the musical instruments and helps to capture the information. It is different for different instruments. The features are extracted from impulse response and presented to WEKA Classifier.
 The Musical instrument classification for individual instruments and family is verified using impulse response modeling. It is found that the impulse response is different for different instruments. It helps to easily distinguish between instrument to instrument and family to family. For individual instruments, the average classification accuracy has been obtained is 83.23% and 85.55% for family classification.

MUSIC IN DANTE'S DIVINE COMEDY: To the 700th Anniversary of the Memory

The article examines the famous literary and artistic work of the Renaissance poet Dante Alighieri "Divine Comedy". In the Comedy a special place is occupied by the musical content, which enriches the literary images of the afterlife. Many works of Western researchers pay attention to the "philosophy of music" of Dante's work. The opinions of various scholars who single out the following musical references are considered: songs and musical instruments, liturgical sung texts and psalms, the presence of monophonic and polyphonic sounds, the reflection of the Boethian principles: musica mundana, humana, instrumentalis and assumptions about the poet's own vision of the development of music. Authors saw the relation with words used to designate the music itself. They have focused on moments of structural and literary relations in these sacred texts respect to each other, analysed the "singing" relationship between heroes of Comedy. The considered a set of key cases resonates with music and sound: infernal dissonance, cacophony emphasizes the immorality Inferno; the middle part Purgatorio, is the central part of Comedy is the musical bridge between the “anti-music” of Inferno and the divine music of Paradiso. The music running these canticas throughout psalms and hymns voiced by choirs of souls. Recent scholarship on the literary aspects and musicology enables us as one level up of the meaning of Comedy. This philosophical and metaphysical aspects of Dante`s thought maybe give us information about music life in 14th century. Interesting material is supplemented by quotations from the Ukrainian translation of Maxim Strikha, which preserves the poetics of speech and sound means by which Dante tried to portray his visions.

SPICE: Self-Supervised Pitch Estimation

We propose a model to estimate the fundamental frequency in monophonic audio, often referred to as pitch estimation. We acknowledge the fact that obtaining ground truth annotations at the required temporal and frequency resolution is a particularly daunting task. Therefore, we propose to adopt a self-supervised learning technique, which is able to estimate pitch without any form of supervision. The key observation is that pitch shift maps to a simple translation when the audio signal is analysed through the lens of the constant-Q transform (CQT). We design a self-supervised task by feeding two shifted slices of the CQT to the same convolutional encoder, and require that the difference in the outputs is proportional to the corresponding difference in pitch. In addition, we introduce a small model head on top of the encoder, which is able to determine the confidence of the pitch estimate, so as to distinguish between voiced and unvoiced audio. Our results show that the proposed method is able to estimate pitch at a level of accuracy comparable to fully supervised models, both on clean and noisy audio samples, although it does not require access to large labeled datasets.

Octave Error Reduction in Pitch Detection Algorithms Using Fourier Series Approximation Method

ABSTRACTPitch detection has been an area of research in the field on Music Information retrieval, which has always received great attention. Pitch is defined as the frequency of a pure tone signal which the brain perceives while listening to a signal. Pitch detection methods ideally identify the fundamental frequency (f0) or fundamental time period (T) of a signal and f0 is defined as the inverse of T. Several pitch detection algorithms have been proposed in the past and each one has its advantages and disadvantages. One of the prominent problems seen across all pitch detection algorithms is to identify the correct octave to which the pitch or f0 belongs to. Ideally, we are required to find the fundamental harmonic. Due to the structure of vocal tract and the acoustic nature of the instruments and due to various other factors like noise, the speech, and music signals are not perfectly periodic in nature and are termed as quasi-periodic signals. In this paper, we have proposed a method using Fourier series to approximate the quasi-periodic signals with a periodic signal, which analyses the strengths of various harmonics of the signal to arrive at a better estimate of the pitch for monophonic sounds. The brain perceives a periodic signal even while listening to a quasi-periodic signal. Hence, the idea is to approximate the quasi-periodic signal with a perfectly periodic signal and estimate the fundamental frequency of the signal which gives the best approximation. The method was tested on synthetic signals and real music signals and found to improve the pitch detection accuracy.

Attention Modulates the Auditory Cortical Processing of Spatial and Category Cues in Naturalistic Auditory Scenes.

This combined fMRI and MEG study investigated brain activations during listening and attending to natural auditory scenes. We first recorded, using in-ear microphones, vocal non-speech sounds, and environmental sounds that were mixed to construct auditory scenes containing two concurrent sound streams. During the brain measurements, subjects attended to one of the streams while spatial acoustic information of the scene was either preserved (stereophonic sounds) or removed (monophonic sounds). Compared to monophonic sounds, stereophonic sounds evoked larger blood-oxygenation-level-dependent (BOLD) fMRI responses in the bilateral posterior superior temporal areas, independent of which stimulus attribute the subject was attending to. This finding is consistent with the functional role of these regions in the (automatic) processing of auditory spatial cues. Additionally, significant differences in the cortical activation patterns depending on the target of attention were observed. Bilateral planum temporale and inferior frontal gyrus were preferentially activated when attending to stereophonic environmental sounds, whereas when subjects attended to stereophonic voice sounds, the BOLD responses were larger at the bilateral middle superior temporal gyrus and sulcus, previously reported to show voice sensitivity. In contrast, the time-resolved MEG responses were stronger for mono- than stereophonic sounds in the bilateral auditory cortices at ~360 ms after the stimulus onset when attending to the voice excerpts within the combined sounds. The observed effects suggest that during the segregation of auditory objects from the auditory background, spatial sound cues together with other relevant temporal and spectral cues are processed in an attention-dependent manner at the cortical locations generally involved in sound recognition. More synchronous neuronal activation during monophonic than stereophonic sound processing, as well as (local) neuronal inhibitory mechanisms in the auditory cortex, may explain the simultaneous increase of BOLD responses and decrease of MEG responses. These findings highlight the complimentary role of electrophysiological and hemodynamic measures in addressing brain processing of complex stimuli.