Spectrogram Features Research Articles

Acoustic scene classification with multi-temporal complex modulation spectrogram features and a convolutional LSTM network

Acoustic scene classification with multi-temporal complex modulation spectrogram features and a convolutional LSTM network

Identifying the Acoustic Source via MFF-ResNet with Low Sample Complexity

Acoustic signal classification plays a central role in acoustic source identification. In practical applications, however, varieties of training data are typically inadequate, which leads to a low sample complexity. Applying classical deep learning methods to identify acoustic signals involves a large number of parameters in the classification model, which calls for great sample complexity. Therefore, low sample complexity modeling is one of the most important issues related to the performance of the acoustic signal classification. In this study, the authors propose a novel data fusion model named MFF-ResNet, in which manual design features and deep representation of log-Mel spectrogram features are fused with bi-level attention. The proposed approach involves an amount of prior human knowledge as implicit regularization, thus leading to an interpretable and low sample complexity model of the acoustic signal classification. The experimental results suggested that MFF-ResNet is capable of accurate acoustic signal classification with fewer training samples.

Attention Based Convolutional Neural Network with Multi-frequency Resolution Feature for Environment Sound Classification.

The environmental sound classification has great research significance in the fields of intelligent audio monitoring and other fields. A novel multi-frequency resolution (MFR) feature is proposed in this paper to solve the problem that the existing single frequency resolution time–frequency features of sound cannot effectively express the characteristics of multiple types of sound. The MFR feature is composed of three features with different frequency resolutions, which are compressed in varying degrees at the time dimension. This method not only has the effect of data augmentation but also can obtain more context information during the feature extraction. And the MFR features of Log-Mel Spectrogram, Cochleagram, and Constant Q-Transform are combined to form a multi-channel MFR feature. Also, a network named SacNet is built, which can effectively solve the problem that the time–frequency feature map of sound contains more invalid information. The basic structural unit of the SacNet consists of two parallel branches, one using depthwise separable convolution as the main feature extractor, and the other using spatial attention module to extract more effective information. Experiment results have demonstrated that the proposed method achieves the state-of-the-art accuracy of 97.5%, 93.1%, and 95.3% on three benchmark datasets of ESC10, ESC50, and UrbanSound8K respectively, which are increased by 3.3%, 0.5%, and 2.3% respectively compared with the previous advanced methods.

Self-Supervised Learning of Audio Representations From Audio-Visual Data Using Spatial Alignment

Learning from audio-visual data offers many possibilities to express correspondence between the audio and visual content, similar to the human perception that relates aural and visual information. In this work, we present a method for self-supervised representation learning based on audio-visual spatial alignment (AVSA), a more sophisticated alignment task than the audio-visual correspondence (AVC). In addition to the correspondence, AVSA also learns from the spatial location of acoustic and visual content. Based on 360$^\text{o}$ video and Ambisonics audio, we propose selection of visual objects using object detection, and beamforming of the audio signal towards the detected objects, attempting to learn the spatial alignment between objects and the sound they produce. We investigate the use of spatial audio features to represent the audio input, and different audio formats: Ambisonics, mono, and stereo. Experimental results show a 10 $\%$ improvement on AVSA for the first order ambisonics intensity vector (FOA-IV) in comparison with log-mel spectrogram features; the addition of object-oriented crops also brings significant performance increases for the human action recognition downstream task. A number of audio-only downstream tasks are devised for testing the effectiveness of the learnt audio feature representation, obtaining performance comparable to state-of-the-art methods on acoustic scene classification from ambisonic and binaural audio.

An LSTM-Autoencoder Architecture for Anomaly Detection Applied on Compressors Audio Data

The compressors used in today’s natural gas production industry have an essential role in maintaining the production line operational. Each of the compressors’ components has routine maintenance tasks to avoid sudden failures. Hence, the significant advantages and benefits of performing preventative maintenance tasks in time are decreasing equipment downtime, saving additional costs, and improving the safety and reliability of the whole system. In this paper, anomaly classification and detection methods based on a neural network hybrid model named Long Short-Term Memory (LSTM)-Autoencoder (AE) is proposed to detect anomalies in sequence pattern of audio data, collected by multiple sound sensors deployed at different components of each compressor system for predictive maintenance. In research methodology, this paper has conducted experiments that employed different RNN architectures such as GRU, LSTM, Stacked LSTM, and Stacked GRU with various functions to create a baseline for model evaluation. Each architecture used audio signals dataset received from the compressor system for experiments to consider each neural network model’s performance. According to performance results, an optimal model for anomaly detection with the best performance scores has been proposed in this research. Experiments combined one-dimensional raw audio signal features using SC and Mel spectrogram features were fed to deep learning models to evaluate performance. Hence, such hybrid methods can effectively detect normal and anomaly audio signals collected from a compressor system, increasing the compressor system’s reliability and the sustainability of the gas production line. The combination of multiple-resource features in the proposed hybrid model showed a 100% score in all four-evaluation metrics such as accuracy, precision, recall, and F1 in LSTM-based autoencoder in both test and train results.

Time-frequency analysis-based deep interference classification for frequency hopping system

It is known that interference classification plays an important role in protecting the authorized communication system and avoiding its performance degradation in the hostile environment. In this paper, the interference classification problem for the frequency hopping communication system is discussed. Considering the possibility of the presence of multiple interferences in the frequency hopping system, in order to fully extract effective features of the interferences from the received signals, the linear and bilinear transform-based composite time-frequency analysis method is adopted. Then, the time-frequency spectrograms obtained from the time-frequency analysis are constructed as matching pairs and input to the deep neural network for classification. In particular, the Siamese neural network is used as the classifier, where the paired spectrograms are input into the two sub-networks of the deep networks, and these two sub-networks extract the features of the paired spectrograms for interference-type classification. The simulation results confirm that the proposed algorithm can obtain higher classification accuracy than both traditional single time-frequency representation-based approach and the AlexNet transfer learning or convolutional neural network-based methods.

Identification of Shortwave Radio Communication Behavior Based on Autocorrelation Spectrogram Features

Cognitive communication behavior is becoming a research hotspot in the field of communication confrontation. In theory, the behavioral intention of noncooperating parties can be obtained by analyzing communication signals. Considering the complexity of the actual electromagnetic environment, even when the signal-to-noise ratio (SNR) is low, a certain accuracy still needs to be guaranteed. In this paper, according to five types of physical burst waveforms defined by the shortwave radio interoperability standard, a signal feature extraction method based on autocorrelation spectrogram features is proposed, and a two-input convolutional neural network (CNN) for classification is designed to improve the identification ability of shortwave communication behavior. The experimental results illustrate that the five kinds of shortwave radio communication behaviors can be accurately identified even when the noise is large. The research in this paper can directly analyze the communication behavior through physical layer signal without demodulation, which has the ability to grasp the communication behavior of the shortwave radio station in real time.

Automatic Classification of Bird Sounds: Using MFCC and Mel Spectrogram Features with Deep Learning

Bird species identification is a relevant and time-consuming task for ornithologists and ecologists. With growing amounts of audio-annotated data, automatic bird classification using machine learning techniques is an important trend in the scientific community. Analyzing bird behavior and population trends helps detect other organisms in the environment and is an important problem in ecology. Bird populations react quickly to environmental changes, which make their real-time counting and tracking challenging and very useful. A reliable methodology that automatically identifies bird species from audio would therefore be a valuable tool for the experts in different scientific and applicational domains. The goal of this work is to propose a methodology to identify bird sounds. In this paper, we explore deep learning techniques that are being used in this domain, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to classify the data. In deep learning, audio problems are commonly approached by converting them into images using audio feature extraction techniques such as Mel Spectrograms and Mel Frequency Cepstral Coefficients (MFCCs). We propose and test multiple deep learning and feature extraction combinations in order to find the most suitable approach to this problem.

Applying the FN-corrector to improve the quality of audio event classification

The paper deals with the problem of acoustic events classification which is actively applied to the problems of a safe city, smart home, IoT devices, and for the detection of industrial accidence. A solution to improve the accuracy of classifiers without changing their structure and collecting additional data is proposed. The main data source for the experiments was the TUT Urban Acoustic Scenes 2018, Development Dataset. The paper presents the way to increase the accuracy of audio event classification by using the FN-corrector. The FN-corrector is a linear two-stage classifier performing the transformation of the feature space into a linearly separable space and the linear separation of one class from another. If a corrector is applied, the responses of the original classifier generate four classes: positive (P), negative (N), false positive (FP), and false negative (FN). As a result, it becomes possible to train two types of correctors: the FP-corrector separating positive and false positive classifier responses, and the FN-corrector separating negative and false negative classifier responses. In the experiments, the VGGish convolutional neural network was used as the initial classifier. The audio signal is converted into a spectrogram and is fed to the input of the neural network which forms the spectrogram feature description and performs a classification. As an example, two ”confused“ classes are selected to demonstrate the increase in classification accuracy. Using the feature description of audio recordings of these classes, an FN-corrector was built, trained and connected to the original classifier. The response from the classifier, as well as the feature description, has been passed to the corrector input. Next, the corrector translated the feature space into a new basis (into a linearly separable space) and classified the classifier answer responding to the question whether the original classifier makes a mistake on such a feature vector or not. If the original classifier made a mistake, then his answer is changed by the corrector to the opposite, otherwise the answer remains the same. The results of the experiments demonstrated a decrease in the level of class confusion and, accordingly, an increase in the accuracy of the original classifier without changing its structure and without collecting an additional data set. The results obtained can be used on IoT devices that have significant limitations on the size of the models used, as well as in solving the problems of domain adaptation which is relevant in audio analytics

Classification of asphyxia infant cry using hybrid speech features and deep learning models

Single speech feature such as Mel-Frequency Cepstral Coefficient (MFCC) has been used in most of the studies to classify asphyxia cry among infants. Other speech features such as Chromagram, Mel-scaled Spectrogram, Spectral Contrast and Tonnetz have not been reported in any study related to the classification of asphyxia cry. The study investigated the use of hybrid features of MFCC, Chromagram, Mel-scaled Spectrogram, Spectral Contrast and Tonnetz and deep learning models in classifying asphyxia cry. Deep learning models such as Deep Neural Network (DNN) and Convolutional Neural Network (CNN) were used to classify infant cry between normal/non-asphyxia and asphyxia. The performance of the deep learning models was compared using concatenated hybrid features and single feature of MFCC. The Baby Chillanto Database was used in this study. CNN model performed better than DNN models when MFCC was used. DNN models performed better with hybrid features compared to that with single feature of MFCC. DNN with multiple hidden layers achieved an accuracy of 100% in classifying normal and asphyxia cry, and 99.96% for non-asphyxia and asphyxia cry when the hybrid features were used.

Hand gesture recognition based improved multi-channels CNN architecture using EMG sensors

With the continuous development of sensor and computer technology, human-computer interaction technology is also improving. Gesture recognition has become a research hotspot in human-computer interaction, sign language recognition, rehabilitation training, and sports medicine. This paper proposed a method of hand gestures recognition which extracts the time domain and frequency domain features from surface electromyography (sEMG) by using an improved multi-channels convolutional neural network (IMC-CNN). The 10 most commonly used hand gestures are recognized by using the spectral features of sEMG signals which is the input of the IMC-CNN model. Firstly, the third-order Butterworth low-pass filter and high-pass filter are used to denoise the sEMG signal. Secondly, effective sEMG signal segment from denoised signal is applied. Thirdly, the spectrogram features of different channels’ sEMG signals are merged into a comprehensive improved spectrogram feature which is used as the input of IMC-CNN to classify the hand gestures. Finally, the recognition accuracy of IMC-CNN model, three single channel CNN of IMC-CNN model, SVM, LDA, LCNN and EMGNET are compared. The experiment was carried out on the same dataset and the same computer. The experimental results showed that the recognition accuracy, sensitivity and accuracy of the proposed model reached 97.5%, 97.25% and 96.25% respectively. The proposed method not only has high average recognition accuracy on MYO collected dataset, but also has high average recognition accuracy on NinaPro DB5 dataset. Overall, the proposed model has more advantages in accuracy and efficiency than that of the comparison models.

When sub-band features meet attention mechanism while knowledge distillation for sound classification

This paper introduces a novel knowledge distillation (KD) framework that distills the knowledge from multiple deep models trained on spectrogram features obtained by dividing the spectrogram into multiple sub-band spectrograms. The deep models learned from sub-band spectrograms prevent information loss while performing knowledge distillation from a teacher model to a student model receiving the full spectrogram as an input. The student models’ performance is evaluated on three benchmark sound datasets, viz., the ESC-10, RAVDESS and Audio MNIST datasets. Impact of three state-of-the-art attention mechanisms is investigated thoroughly to enhance the final accuracy of the student model supervised by the proposed knowledge distillation framework for sound classification. Experiments and results show that the performance of the student model in the presence of state-of-the-art attention mechanisms is comparable and competitive to state-of-the-art techniques. Moreover, the student model trained on the Audio MNIST dataset attains an hitherto unpublished accuracy of 98.24%, a new benchmark for the Audio MNIST dataset. Additionally, Grad-CAM visualization of the spectrograms is provided to understand the spectrogram’s relevant regions and explain why the model classifies a signal into a specific class. The code used in this work is available at: https://github.com/achyutmani/Sub-Band-Guided-KD-for-Sound-Classification.

SSLNet: A network for cross-modal sound source localization in visual scenes

Sound source localization in visual scenes is to associate sounds and their visual producers. Although great progress has been made in this field, the mixed sounds from multiple objects make it intractable to perform efficient localization in the unconstrained scenarios. In this paper, we propose a novel cross-modal sound source localization networks (SSLNet), which is accessible to dispose audio and visual information. First, to encode spatial and temporal information in the spectrogram, we combine convolutional neural networks (CNN) and bidirectional long short-term memory networks (BiLSTM). For different time-step inputs of BiLSTMs, we propose a novel operation, called as grouped global average pooling (GGAP), to divide the 3-D spectrogram feature block into several 1-D vectors. Then, a cross-modal channel attention mechanism is introduced to alleviate the discrepancy of different modalities. Finally, for achieving the pixel-level localization, we propose a fusion approach using cosine and L2 distances to measure the similarity between audio and visual vectors. We implement extensive experiments in benchmark and FAIR-Play datasets. The qualitative results demonstrate the effectiveness of sound source localization. On benchmark dataset, the quantitative results show that SSLNet can improve the localization accuracy by 1.2% in consensus intersection over union (cIoU) and 16.4% in area under the curve (AUC). On FAIR-Play dataset, SSLNet also achieves superior performance with Pearson’s correlation coefficient (CC) of 0.779 and similarity metric (SIM) of 0.650.

Predictive evaluation of spectrogram-based vehicle sound quality via data augmentation and explainable artificial Intelligence: Image color adjustment with brightness and contrast

In this study, a novel method for selecting the optimal data augmentation method in combination with explainable artificial intelligence techniques is presented. Accordingly, a convolutional neural network-based model was designed to quantitatively evaluate the affective sound quality characteristics of vehicle driving sounds, which were classified as professional knowledge. Virtual learning data were created by adjusting the color of the image to avoid damaging the physical features of the spectrogram. By implementing the explainable artificial intelligence technique, the spectrogram features were extracted using domain knowledge. In particular, the engine noise of the vehicle, which plays a significant role in determining the characteristics of the running sound of the vehicle, was selected as a physical characteristic called the engine order line in the spectrogram. The explainable artificial intelligence technique was used to select the most influential feature among those extracted from the spectrogram. By observing the changes in the selected characteristics according to the data augmentation method, an optimal data augmentation method is proposed according to each characteristic. Furthermore, an average classification accuracy of 94.22% was obtained using the proposed data augmentation method, which is an improvement of 1.55–5.55% over the existing data augmentation methods. Moreover, according to the dataset, the standard deviation of the classification accuracy was 2.13%, which yielded an optimum result.

A study on data augmentation in voice anti-spoofing

In this paper we perform an in depth study of how data augmentation techniques improve synthetic or spoofed audio detection. Specifically, we propose methods to deal with channel variability, different audio compressions, different bandwidths and unseen spoofing attacks. These challenges, have all been shown to significantly degrade the performance of audio based systems and anti spoofing systems. Our results are based on the ASVspoof 2021 challenge, in the Logical Access (LA) and Deep Fake (DF) categories. Our study is Data-Centric, meaning that the models are fixed and we significantly improve the results by manipulating the data. We introduce two forms of data augmentation - compression augmentation for the DF part, and compression and channel augmentation for the LA part. In addition, we introduce a double sided log spectrogram feature design that improves the results significantly by centering the sub-bands of interest, where the discriminating spoofing artifacts can be localized. Furthermore, a new type of online data augmentation, SpecAverage, is introduced. This method includes masking the audio features with their average value in order to improve generalization. Our best single system and fusion schemes both achieve state of the art performance in the DF category, with an EER of 15.46% and 14.27%, respectively. Our best system for the LA task reduced the best baseline EER by 50% and the min t-DCF by 16%. Our techniques to deal with spoofed data from a wide variety of distributions can be replicated and can help anti spoofing and speech based systems enhance their results.

Automatic Grading of Student’s Presentation Skills based on PowerPoint Presentation and Audio

Creating and delivering a presentation for Project, Seminar, and Course work is an important academic activity included in the curricula of undergraduate engineering studies. The presentation should be graded based on the presentation skills, accuracy, and authenticity of the contents covered in the presentation. Educational institutes use rubrics to assess the presentation skills on different grounds, which is a cumbersome task for the teacher when the strength of students is significant. Our main objective is to automatically grade the students' presentation skills in terms of PowerPoint presentations and the student's confidence. The proposed system describes a method and dataset designed to automate grading students' presentation skills. Our research study is divided into two parts. In the first part, the PowerPoint presentation features corresponding to text appearance, tables, charts, images, footer, and hyperlinks are extracted to grade PowerPoint presentations. At the same time, Mel-frequency Cepstral Coefficients, Mel Spectrogram, and Chroma features are extracted from the students' audio to identify confidence in the second part of the study. The audio is recorded at the presentation time. Feature extraction programs are implemented in python using Python-pptx and Librosa library. The tree-based feature selection method is used to remove the irrelevant features. Random Forest Ensemble model gives 100 % accuracy while predicting the grade of PowerPoint presentations. Multilayer Perceptron model gives 88% accuracy while predicting the confidence level of the students. The output of both models is combined to grade the students' presentation skills. The quadratic Weighted Kappa (QWK) score is 0.82, which indicates a significant similarity between automated and human-rated scores.

Deep Learning-Based Artistic Inheritance and Cultural Emotion Color Dissemination of Qin Opera.

How to enable the computer to accurately analyze the emotional information and story background of characters in Qin opera is a problem that needs to be studied. To promote the artistic inheritance and cultural emotion color dissemination of Qin opera, an emotion analysis model of Qin opera based on attention residual network (ResNet) is presented. The neural network is improved and optimized from the perspective of the model, learning rate, network layers, and the network itself, and then multi-head attention is added to the ResNet to increase the recognition ability of the model. The convolutional neural network (CNN) is optimized from the internal depth, and the fitting ability and stability of the model are enhanced through the ResNet model. Combined with the attention mechanism, the expression of each weight information is strengthened. The multi-head attention mechanism is introduced in the model and a multi-head attention ResNet, namely, MHAtt_ResNet, is proposed. The network structure can effectively identify the features of the spectrogram, improve the weight information of spectrogram features, and deepen the relationship between distant information in long-time series. Through experiments, the proposed model has high emotional classification accuracy for Qin opera, and with the increase of the number of data sets, the model will train a better classification effect.

Open set recognition of underwater acoustic targets based on GRU-CAE collaborative deep learning network

Aiming at the problem of underwater acoustic target open set recognition, a new recognition method based on deep learning and template matching is proposed in this paper. The proposed method includes three stages: feature extraction based on GRU-CAE collaborative deep learning network, template establishment and template matching. Firstly, the GRU-CAE collaborative deep learning network is proposed to extract the deep collaborative features. The GRU-CAE cooperative deep learning network combines the advantages of CAE network in extracting spatial information of spectrogram and GRU network in extracting temporal structure features of spectrogram. Then, the feature template of each class of underwater acoustic target is constructed by finding an optimal vector in Euclidean space. Finally, the distance between the deep collaborative features of the test set and templates are calculated to distinguish the category of the samples. The experimental data of 5 kinds of underwater acoustic targets are used to verify the recognition performance of the proposed method. The results show that the deep collaborative features have excellent class separability, and the recognition accuracy rate of open set recognition can reach 82.21%. Compared with the other 3 open set recognition methods, the proposed method has the highest recognition accuracy rate.

Deep learning assisted continuous wavelet transform-based spectrogram for the detection of chlorophyll content in potato leaves

• A CNN assisted CWT-based spectrogram strategy is proposed for chlorophyll content detection. • The first-order derivative was performed to capture detailed spectral information. • CWT was applied to obtain effective features from spectral data. • CNN model was applied to explore deep features hidden in spectral data. Visible and near-infrared spectroscopy is a nondestructive method for the chlorophyll content detection of potato crops, in which effective feature extraction is a crucial issue for detecting accuracy in the field. This study aimed to explore comprehensive features to improve the accuracy of chlorophyll content detection in potato leaves. A feature extraction method was proposed by assisting continuous wavelet transform (CWT)-based spectrogram and convolutional neural network (CNN) of the deep learning algorithm. In the experiments, the spectral data in the range of 325–1075 nm were measured in the field. A total of 314 potato leaf samples were collected, and the chlorophyll content was determined in four growth periods. The spectral features in the spatial domain and time–frequency domain were considered in data processing. First, we compared features of first-order derivative (FOD) and Savitzky–Golay smoothing (SG) in the spatial domain to select the preprocessing method. Second, in the time–frequency domain, CWT decomposes spectral data into a series of wavelet coefficient curves at various scales and wavelengths. We supposed that combining both scales and wavelengths of the wavelet coefficients could benefit the detection of the chlorophyll content in potato crops. Thus, the spectrograms were established by transforming 1D wavelet coefficient curves into 2D wavelet coefficient spectrograms. The CNN model was applied to explore potential and comprehensive features in the spectrograms. Finally, partial least squares models were established to compare the detection capability of three features including the spatial domain, wavelet coefficients by CWT, and spectrogram features by CNN. The modeling performances showed that the FOD features obtained the coefficient of determination of 0.7856 in the prediction set ( R P 2 ) and the root mean square error in the prediction set ( RMSEP ) with 3.9357; the FOD features performed better than the SG features ( R P 2 = 0.7734, RMSEP = 4.0460); The wavelet coefficients by CWT obtained R P 2 = 0.8293 and RMSEP = 3.5117. The PLS model with spectrogram features by the CNN model performed best and achieved R P 2 = 0.8749 and RMSEP = 3.0067. It preferably captured the deeper features of spectral data compared with features in the spatial domain. This research provides an effective method for leaf chlorophyll content detection in the precision management of potato crops.

ASERS-LSTM: Arabic Speech Emotion Recognition System Based on LSTM Model

The swift progress in the study field of human-computer interaction (HCI) causes to increase in the interest in systems for Speech emotion recognition (SER). The speech Emotion Recognition System is the system that can identify the emotional states of human beings from their voice. There are well works in Speech Emotion Recognition for different language but few researches have implemented for Arabic SER systems and that because of the shortage of available Arabic speech emotion databases. The most commonly considered languages for SER is English and other European and Asian languages. Several machine learning-based classifiers that have been used by researchers to distinguish emotional classes: SVMs, RFs, and the KNN algorithm, hidden Markov models (HMMs), MLPs and deep learning. In this paper we propose ASERS-LSTM model for Arabic Speech Emotion Recognition based on LSTM model. We extracted five features from the speech: Mel-Frequency Cepstral Coefficients (MFCC) features, chromagram, Melscaled spectrogram, spectral contrast and tonal centroid features (tonnetz). We evaluated our model using Arabic speech dataset named Basic Arabic Expressive Speech corpus (BAES-DB). In addition of that we also construct a DNN for classify the Emotion and compare the accuracy between LSTM and DNN model. For DNN the accuracy is 93.34% and for LSTM is 96.81%.