Audio Spectrogram Research Articles

Semi-supervised classification-aware cross-modal deep adversarial data augmentation

Deep neural networks are usually data-starved in real-world applications, while manually annotation can be costly—for example, the audio emotion recognition from the audio. In contrast, the continued research in image-based facial expression recognition grants us a rich source of public available labeled IFER datasets. Using images to support audio emotion recognition with limited labeled data according to their inherent correlations can be a meaningful and challenging task. This paper proposes a system that facilitates knowledge transfer from the labeled visual to the heterogeneous labeled audio domain by learning a joint distribution of examples in different modalities then the system can map an IFER example to a corresponding audio spectrogram. Next, our work reformulates the audio emotion classification into a K+1 class discriminator of GAN-based semi-supervised learning. Good semi-supervised learning requires that the generator does NOT sample from a distribution well matching the true data distribution. Therefore, we demand the generated examples are from the low-density areas of the marginal distribution in the audio spectrogram modality. Concretely, the proposed model translates image samples to audios class-wisely in the form of spectrograms. To harness the decoded samples in a sparsely distributed area and construct a tighter decision boundary, we give a solution to precisely estimate the density on feature space and incorporate low-density pieces with an annealing scheme. Our method requires the network to discriminate against the low-density data points from high-density data points throughout the classification, and we evidence that this technique effectively improves task performance.

Sound Classification Based on Multihead Attention and Support Vector Machine

Sound classification is a broad area of research that has gained much attention in recent years. The sound classification systems based on recurrent neural networks (RNNs) and convolutional neural networks (CNNs) have undergone significant enhancements in the recognition capability of models. However, their computational complexity and inadequate exploration of global dependencies for long sequences restrict improvements in their classification results. In this paper, we show that there are still opportunities to improve the performance of sound classification by substituting the recurrent architecture with the parallel processing structure in the feature extraction. In light of the small-scale and high-dimension sound datasets, we propose the use of the multihead attention and support vector machine (SVM) for sound taxonomy. The multihead attention is taken as the feature extractor to obtain salient features, and SVM is taken as the classifier to recognize all categories. Extensive experiments are conducted across three acoustically characterized public datasets, UrbanSound8K, GTZAN, and IEMOCAP, by using two commonly used audio spectrograms as inputs, respectively, and we fully evaluate the impact of parameters and feature types on classification accuracy. Our results suggest that the proposed model can reach comparable performance with existing methods and reveal its strong generalization ability of sound taxonomy.

Deep convolutional neural networks for double compressed AMR audio detection

Detection of double compressed (DC) adaptive multi-rate (AMR) audio recordings is a challenging audio forensic problem and has received great attention in recent years. Here, the authors propose to use convolutional neural networks (CNN) for DC AMR audio detection. The CNN is used as (i) an end-to-end DC AMR audio detection system and (ii) a feature extractor. The end-to-end system receives the audio spectrogram as the input and returns the decision whether the input audio is single compressed (SC) or DC. As a feature extractor in turn, it is used to extract discriminative features and then these features are modelled using support vector machines (SVM) classifier. Our extensive analysis conducted on four different datasets shows the success of the proposed system and provides new findings related to the problem. Firstly, double compression has a considerable impact on the high frequency components of the signal. Secondly, the proposed system yields great performance independent of the recording device or environment. Thirdly, when previously altered files are used in the experiments, 97.41% detection rate is obtained with the CNN system. Finally, the cross-dataset evaluation experiments show that the proposed system is very effective in case of a mismatch between training and test datasets.

Acoustic auto-encoders for biodiversity assessment

Continuous audio recordings are playing an ever more important role in conservation and biodiversity monitoring, however, listening to these recordings is often infeasible, as they can be thousands of hours long. Automating analysis using machine learning algorithms requires a feature representation. In this paper we propose a technique for learning a general feature representation from unlabelled audio using auto-encoders, which can be used for analysing environmental audio on a small timescale.We start by segmenting the audio data into non-overlapping 1-s long chunks and generating audio spectrograms. These audio spectrograms are then used to train a basic auto-encoder, with the output of the encoder network being used to generate the feature representation. We have found that at a 1-s timescale, our feature representation offers marginal improvements over “acoustic indices”, a common representation for analysing environmental audio.

Deep learning improves acoustic biodiversity monitoring and new candidate forest frog species identification (genus Platymantis) in the Philippines

One significant challenge to biodiversity assessment and conservation is persistent gaps in species diversity knowledge in Earth’s most biodiverse areas. Monitoring devices that utilize species-specific advertisement calls show promise in overcoming challenges associated with lagging frog species discovery rates. However, these devices generate data at paces faster than it can be analyzed. As such, automated platforms capable of efficient data processing and accurate species-level identification are at a premium. In addressing this gap, we used TensorFlow Inception v3 to design a robust, automated species identification system for 41 Philippine frog species (genus Platymantis), utilizing single-note audio spectrograms. With this model, we explored two concepts: (1) performance of our deep-learning model in discriminating closely-related frog species based on images representing advertisement call notes, and (2) the potential of this platform to accelerate new species discovery. TensorFlow identified species with a ~ 94% overall correct identification rate. Incorporating distributional data increased the overall identification rate to ~ 99%. In applying TensorFlow to a dataset that included undescribed species in addition to known species, our model was able to differentiate undescribed species through variation in “certainty” rate; the overall certainty rate for undescribed species was 65.5% versus 83.6% for described species. This indicates that, in addition to discriminating recognized frog species, our model has the potential to flag possible new species. As such, this work represents a proof-of-concept for automated, accelerated detection of novel species using acoustic mate-recognition signals, that can be applied to other groups characterized by vibrational cues, seismic signals, and vibrational mate-recognition.

Soundscape segregation based on visual analysis and discriminating features

The distinction of landscapes based on their sound patterns is useful for several analyses. For instance, comparisons of audio files from different periods enable the detection of changes over time in a particular habitat, signaling events of importance, such as modifications in the balance between species and presence of new ones. The handling of a large number of different sound recordings in wild environments also reduces the set of sounds to be examined. However, the current efforts towards soundscape interpretation do not provide enough elements for researchers to automatically split soundscape datasets with degrees of similarity, thus requiring users' feedback for the grouping of highly related recordings. This work introduces a strategy for the exploration and analysis of soundscapes that highlights data characteristics related to differences and similarities among distinct soundscapes. It is based on a visual and numerical evaluation of feature spaces and was applied to three feature sets, namely acoustic indices and measurements, images from audio spectrograms depicted by classic features, and the same images depicted by features automatically generated by Deep Learning techniques. The results indicate that certain combinations of acoustic indices and measurements perform well for the discrimination task, although other feature sets have not been discarded. In addition, visual techniques were able to assist this type of analysis.

Predicting outdoor ultrafine particle number concentrations, particle size, and noise using street-level images and audio data

Outdoor ultrafine particles (UFPs) (<0.1 µm) may have an important impact on public health but exposure assessment remains a challenge in epidemiological studies. We developed a novel method of estimating spatiotemporal variations in outdoor UFP number concentrations and particle diameters using street-level images and audio data in Montreal, Canada. As a secondary aim, we also developed models for noise. Convolutional neural networks were first trained to predict 10-second average UFP/noise parameters using a large database of images and audio spectrogram data paired with measurements collected between April 2019 and February 2020. Final multivariable linear regression and generalized additive models were developed to predict 5-minute average UFP/noise parameters including covariates from deep learning models based on image and audio data along with outdoor temperature and wind speed. The best performing final models had mean cross-validation R2 values of 0.677 and 0.523 for UFP number concentrations and 0.825 and 0.735 for UFP size using two different test sets. Audio predictions from deep learning models were stronger predictors of spatiotemporal variations in UFP parameters than predictions based on street-level images; this was not explained only by noise levels captured in the audio signal. All final noise models had R2 values above 0.90. Collectively, our findings suggest that street-level images and audio data can be used to estimate spatiotemporal variations in outdoor UFPs and noise. This approach may be useful in developing exposure models over broad spatial scales and such models can be regularly updated to expand generalizability as more measurements become available.

Deep attention based music genre classification

As an important component of music information retrieval, music genre classification attracts great attentions these years. Benefitting from the outstanding performance of deep neural networks in computer vision, some researchers apply CNN on music genre classification tasks with audio spectrograms as input instead, which has similarities with RGB images. These methods are based on a latent assumption that spectrums with different temporal steps have equal importance. However, it goes against the theory of processing bottleneck in psychology as well as our observation from audio spectrograms. By considering the differences of spectrums, we propose a new model incorporating with attention mechanism based on Bidirectional Recurrent Neural Network. Furthermore, two attention-based models (serial attention and parallelized attention) are implemented in this paper. Comparing with serial attention, parallelized attention is more flexible and gets better results in our experiments. Especially, the CNN-based parallelized attention models with taking STFT spectrograms as input outperform the previous work.

Using acoustic features of mothers’ infant-directed speech to predict changes in infant biobehavioral state

Infant directed speech is marked by exaggerated frequency modulation and strong high frequency power, features that may provide physiological cues for mobilization or calming (Porges and Lewis, 2010; Kolacz et al., 2018). We examined whether these features predicted changes in infant biobehavioral state during the Still Face Paradigm, a stressor in which the mother withdraws and reinstates social cues. 98 mother-infant dyads participated when infants were 4-8 months old. Infant heart rate and respiratory sinus arrhythmia (a measure of cardiac parasympathetic control) were derived from an electrocardiogram (ECG). Infant behavioral distress was measured by vocal, facial, and body movement distress. Mothers’ vocalizations were measured using spectral analysis within designated frequency bands and modulation using a 2-dimensional fast Fourier transform of the audio spectrogram. Maternal frequency modulation predicted decreases in infant heart rate (p = .030), mid-frequency acoustic power (500–5000 Hz) predicted increases in cardiac parasympathetic regulation in infants with low parasympathetic tone (p = .024), and high frequency power predicted increases in infant behavioral distress in infants who were not initially distressed (p = .011). These results suggest that mothers’ vocal frequency band power and modulation may be aspects of infant-directed speech that are relevant for regulating infant biobehavioral state.

Development and performance analysis of a low-cost MEMS microphone-based hearing aid with three different audio amplifiers

In this paper, MEMS-based capacitive microphone and low-cost amplifier are designed for low-cost power-efficient hearing aid application. The developed microphone along with the associated circuitry is mounted on a common board in the form of pocket-type (body-worn) device. The designed microphone consists of a flexible circular silicon nitrite (Si3N4) diaphragm and a polysilicon-perforated back plate with air as dielectric between them. The incident acoustic waves on the sensor cause deflection of the diaphragm to alter the air gap between the perforated back plate (fixed electrode) and the diaphragm (moving plate) which causes a change in capacitance. The acoustic pressure applied to the microphone is from 0 to 100 Pa for an operating range of 100 Hz–10 kHz which corresponds to the audible frequency range in case of human beings. The main purpose of this work is to increase the longevity of battery used in conventional hearing aids. The designed MEMS microphone with Si3N4 diaphragm is capable of identifying acoustic frequencies (100 Hz to 10 kHz) which correspond to a specific change in absolute pressure from 0 to 100 Pa for 2-micron-thick diaphragm with a sensitivity of about 0.08676 mV/Pa. The design of the sensor and the characteristics analysis are performed in FEM-based simulation software, which are later validated in real time. The prototype is designed using MEMS microphone and low-cost amplifier ICs with biasing components in the form of pocket-type (body-worn) hearing aid. In order to study the performance of proposed device, three different market-available amplifiers with controllable gain are used. Finally, the performance of the hearing aid is studied through audio spectrogram analysis to choose the best-suited amplifier among the three.

Robust Electric Network Frequency Estimation with Rank Reduction and Linear Prediction

This article deals with the problem of Electric Network Frequency (ENF) estimation where Signal to Noise Ratio (SNR) is an essential challenge. By exploiting the low-rank structure of the ENF signal from the audio spectrogram, we propose an approach based on robust principle component analysis to get rid of the interference from speech contents and some of the background noise, which in our case can be regarded as sparse in nature. Weighted linear prediction is enforced on the low-rank signal subspace to gain accurate ENF estimation. The performance of the proposed scheme is analyzed and evaluated as a function of SNR, and the Cramér-Rao Lower Bound (CRLB) is approached at an SNR level above -10 dB. Experiments on real datasets have demonstrated the advantages of the proposed method over state-of-the-art work in terms of estimation accuracy. Specifically, the proposed scheme can effectively capture the ENF fluctuations along the time axis using small numbers of signal observations while preserving sufficient frequency precision.

Efficient computation of image moments for robust cough detection using smartphones

Health Monitoring apps for smartphones have the potential to improve quality of life and decrease the cost of health services. However, they have failed to live up to expectation in the context of respiratory disease. This is in part due to poor objective measurements of symptoms such as cough. Real-time cough detection using smartphones faces two main challenges namely, the necessity of dealing with noisy input signals, and the need of the algorithms to be computationally efficient, since a high battery consumption would prevent patients from using them. This paper proposes a robust and efficient smartphone-based cough detection system able to keep the phone battery consumption below 25% (16% if only the detector is considered) during 24 h use. The proposed system efficiently calculates local image moments over audio spectrograms to feed an optimized classifier for final cough detection. Our system achieves 88.94% sensitivity and 98.64% specificity in noisy environments with a 5500× speed-up and 4× battery saving compared to the baseline implementation. Power consumption is also reduced by a minimum factor of 6 compared to existing optimized systems in the literature.

CLASSIFICATION OF EMOTIONAL RESPONSES IN VIDEO AND AUDIO RECORDINGS USING A CONVOLUTIONAL NEURAL NETWORK

Depression and depressive symptoms are associated in Alzheimer disease (AD) (Green et. al, Arch. Neurol., 2003). Depressive symptoms have been reported as occurring prior to the development of AD dementia and could be a potential indicator of impending dementia. Hence, identifying emotions that reflect underlying depressive symptoms may provide an early risk score estimate as well as serve as a quality control measure to help physicians make better decisions. Deep Learning is an application of Artificial Neural Networks with more than one hidden layer. It has been shown that with increasing data, the performance of deep learning increases significantly. Convolutional Neural Network (CNN) in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex, has proven to be highly successful in image classification. Audio features have been minimally used and integrated for video classification problems. Here we combine audio and visual features in a unique fashion to train the machine learning model and make the predictions. The dataset (Jiang et al, AAAI, 2014) labelled with 8 emotions namely ‘Anger’, ‘Anticipation’, ‘Disgust’, ‘Fear’, ‘Joy’, ‘Sadness’, ‘Surprise’ and ‘Trust’ was obtained. The model was trained using a CNN merging the video images and the corresponding audio spectrograms. The CNN was trained on pre-trained inceptionV3 model by adding a fully connected layer with 1024 neurons and a “softmax” output layer. The prototype model from the methodology described above obtained an accuracy of approximately 85% in correct classification of the emotions. We are currently in the process of verifying the result, and obtaining the Audio/Visual Emotion Challenge (AVEC) datasets – a challenge aiming to provide common benchmark test set, and to bring together research communities, to establish fusion of possible and beneficial approaches for depression and emotion analysis, to validate our model.

Speech-music discrimination using deep visual feature extractors

Speech music discrimination is a traditional task in audio analytics, useful for a wide range of applications, such as automatic speech recognition and radio broadcast monitoring, that focuses on segmenting audio streams and classifying each segment as either speech or music. In this paper we investigate the capabilities of Convolutional Neural Networks (CNNs) with regards to the speech - music discrimination task. Instead of representing the audio content using handcrafted audio features, as traditional methods do, we use deep structures to learn visual feature dependencies as they appear on the spectrogram domain (i.e. train a CNN using audio spectrograms as input images). The main contribution of our work focuses on the potentials of using pre-trained deep architectures along with transfer-learning to train robust audio classifiers for the particular task of speech music discrimination. We highlight the supremacy of the proposed methods, compared both to the typical audio-based and deep-learning methods that adopt handcrafted features, and we evaluate our system in terms of classification success and run-time execution. To our knowledge this is the first work that investigates CNNs for the task of speech music discrimination and the first that exploits transfer learning across very different domains for audio modeling using deep-learning in general. In particular, we fine-tune a deep architecture originally trained for the Imagenet classification task, using a relatively small amount of data (almost 80 min of training audio samples) along with data augmentation. We evaluate our system through extensive experimentation against three different datasets. Firstly we experiment on a real-world dataset of more than 10 h of uninterrupted radio broadcasts and secondly, for comparison purposes, we evaluate our best method on two publicly available datasets that were designed specifically for the task of speech-music discrimination. Our results indicate that CNNs can significantly outperform current state-of-the-art in terms of performance especially when transfer learning is applied, in all three test-datasets. All the discussed methods, along with the whole experimental setup and the respective datasets, are openly provided for reproduction and further experimentation.

Binary Non-Negative Matrix Deconvolution for Audio Dictionary Learning

In this study, we propose an unsupervised method for dictionary learning in audio signals. The new method, called binary nonnegative matrix deconvolution (BNMD), is developed and used to discover patterns from magnitude-scale spectrograms. The BNMD models an audio spectrogram as a sum of delayed patterns having binary gains (activations). Only small subsets of patterns can be active for a given spectrogram excerpt. The proposed method was applied to speaker identification and separation tasks. The experimental results show that dictionaries obtained by the BNMD bring much higher speaker identification accuracies averaged over a range of SNRs from $-$ 6 dB to 9 dB (91.3%) than the NMD-based dictionaries (37.8–75.4%). The BNMD also gives a benefit over dictionaries obtained using vector quantization (87.8%). For bigger dictionaries the difference between the BNMD and the vector quantization (VQ) is getting smaller. For the speech separation task the BNMD dictionary gave a slight improvement over the VQ.

Multi-Level and Multi-Scale Feature Aggregation Using Pretrained Convolutional Neural Networks for Music Auto-Tagging

Music auto-tagging is often handled in a similar manner to image classification by regarding the 2D audio spectrogram as image data. However, music auto-tagging is distinguished from image classification in that the tags are highly diverse and have different levels of abstractions. Considering this issue, we propose a convolutional neural networks (CNN)-based architecture that embraces multi-level and multi-scaled features. The architecture is trained in three steps. First, we conduct supervised feature learning to capture local audio features using a set of CNNs with different input sizes. Second, we extract audio features from each layer of the pre-trained convolutional networks separately and aggregate them altogether given a long audio clip. Finally, we put them into fully-connected networks and make final predictions of the tags. Our experiments show that using the combination of multi-level and multi-scale features is highly effective in music auto-tagging and the proposed method outperforms previous state-of-the-arts on the MagnaTagATune dataset and the Million Song Dataset. We further show that the proposed architecture is useful in transfer learning.

Bird sound spectrogram decomposition through Non-Negative Matrix Factorization for the acoustic classification of bird species.

Feature extraction for Acoustic Bird Species Classification (ABSC) tasks has traditionally been based on parametric representations that were specifically developed for speech signals, such as Mel Frequency Cepstral Coefficients (MFCC). However, the discrimination capabilities of these features for ABSC could be enhanced by accounting for the vocal production mechanisms of birds, and, in particular, the spectro-temporal structure of bird sounds. In this paper, a new front-end for ABSC is proposed that incorporates this specific information through the non-negative decomposition of bird sound spectrograms. It consists of the following two different stages: short-time feature extraction and temporal feature integration. In the first stage, which aims at providing a better spectral representation of bird sounds on a frame-by-frame basis, two methods are evaluated. In the first method, cepstral-like features (NMF_CC) are extracted by using a filter bank that is automatically learned by means of the application of Non-Negative Matrix Factorization (NMF) on bird audio spectrograms. In the second method, the features are directly derived from the activation coefficients of the spectrogram decomposition as performed through NMF (H_CC). The second stage summarizes the most relevant information contained in the short-time features by computing several statistical measures over long segments. The experiments show that the use of NMF_CC and H_CC in conjunction with temporal integration significantly improves the performance of a Support Vector Machine (SVM)-based ABSC system with respect to conventional MFCC.

Complex non-negative matrix factorization: Phase-aware sparse representation of audio spectrograms

Non-negative matrix factorization (NMF) has attracted a lot of attention after being proposed as a powerful approach for audio source separation. Factorizing the magnitude spectrogram of a mixture signal into the product of two non-negative matrices can be interpreted as approximating the observed spectra at each time frame as a linear sum of basis spectra scaled by time-varying amplitudes. One drawback of this approach is that the additivity of magnitude spectra is assumed, which holds only approximately. To address this drawback, we introduce a novel framework called the “Complex NMF,” which makes it possible to realize NMF-like signal decompositions in the complex spectrogram domain. We further develop a time-domain extension of this approach, which is noteworthy in that it allows for several extensions that were not possible with the conventional NMF framework.

Noise Reduction in the Spherical Harmonic Domain Using a Tradeoff Beamformer and Narrowband DOA Estimates

In noise reduction, a common approach is to use a microphone array with a beamformer that combines the individual microphone signals to extract a desired speech signal. The beamformer weights usually depend on the statistics of the noise and desired speech signals, which cannot be directly observed and must be estimated. Estimators based on the speech presence probability (SPP) seek to update the statistics estimates only when desired speech is known to be absent or present. However, they do not normally distinguish between desired and undesired speech sources. In this contribution, an algorithm is proposed to distinguish between these two types of sources using additional spatial information, by estimating a desired speech presence probability based on the combination of a multichannel SPP and a direction of arrival (DOA) based probability. The DOA-based probability is computed using DOA estimates for each time-frequency bin. The estimated statistics are then used to compute the weights of a spherical harmonic domain tradeoff beamformer, which achieves a balance between noise reduction and speech distortion. The performance evaluation demonstrates the effectiveness of the proposed approach at suppressing both background noise and spatially coherent noise. A number of audio examples and sample spectrograms are also provided.

Note‐based alignment using score‐driven non‐negative matrix factorisation for audio recordings

This study presents a discussion on the task of score alignment, which properly aligns an audio recording with its corresponding score. Conventional methods have difficulty performing this task because of asynchrony in the recording of simultaneous notes in the score. A note-based score alignment based on the pitch-by-time feature is proposed, called the piano-roll feature, and it presents an approach for converting the audio spectrogram to a piano-roll-like feature. Score-driven non-negative matrix factorisation is then adopted in the transformation. Furthermore, this study also proposes pitch-wise alignment considering each pitch sequence (i.e. the row of piano roll) separately. Results based on the MIDI-Aligned Piano Sounds database show that approximately 88% of notes match their onsets, deviating from the ground truth by less than 50 ms. Other results based on SCREAM Music Annotation Project database that is a manual annotation project of commercial CD recordings are presented as well.