Spectrogram Features Research Articles

Improving multi-talker binaural DOA estimation by combining periodicity and spatial features in convolutional neural networks

Deep neural network-based direction of arrival (DOA) estimation systems often rely on spatial features as input to learn a mapping for estimating the DOA of multiple talkers. Aiming to improve the accuracy of multi-talker DOA estimation for binaural hearing aids with a known number of active talkers, we investigate the usage of periodicity features as a footprint of speech signals in combination with spatial features as input to a convolutional neural network (CNN). In particular, we propose a multi-talker DOA estimation system employing a two-stage CNN architecture that utilizes cross-power spectrum (CPS) phase as spatial features and an auditory-inspired periodicity feature called periodicity degree (PD) as spectral features. The two-stage CNN incorporates a PD feature reduction stage prior to the joint processing of PD and CPS phase features. We investigate different design choices for the CNN architecture, including varying temporal reduction strategies and spectro-temporal filtering approaches. The performance of the proposed system is evaluated in static source scenarios with 2–3 talkers in two reverberant environments under varying signal-to-noise ratios using recorded background noises. To evaluate the benefit of combining PD features with CPS phase features, we consider baseline systems that utilize either only CPS phase features or combine CPS phase and magnitude spectrogram features. Results show that combining PD and CPS phase features in the proposed system consistently improves DOA estimation accuracy across all conditions, outperforming the two baseline systems. Additionally, the PD feature reduction stage in the proposed system improves DOA estimation accuracy while significantly reducing computational complexity compared to a baseline system without this stage, demonstrating its effectiveness for multi-talker DOA estimation.

Deciphering Human Emotions in Audio with Deep Learning Networks and MFCC Spectrogram Features

Introduction: Emotions play a fundamental role in human communication and interaction. In this paper, an effective and highly accurate sequential Deep Learning (DL) based emotion recognition model is proposed. Objectives: The main contribution of this work is to use effective Feature Extraction, Feature Selection (FS) and classification techniques so that accuracy of the model is increased while its computational complexity is reduced. Methods: The research utilizes a Surrey Audio-Visual expressed Emotion (SAVEE) dataset of audio recordings capturing emotional states. Feature extraction techniques are applied to obtain representative features from the audio signals. Specifically, Mel Spectrograms are computed and transformed into the decibel scale to capture the frequency content of the signals. These features are then fed into DL architecture for extracting relevant and important features. Moreover, Infinite Feature Selection (IFS) and Fuzzy based techniques are implemented for selecting only important and crucial features from the given feature set. For classification phase, a sequential DL based layered architecture is implemented for detecting and classifying emotions in audio signals. Results: The inclusion of convolutional layer with larger filter sizes and application of feature selection techniques improved the performance and interpretability of the approach. Conclusions: Through extensive experiments, an accuracy of 90.27% is achieved by proposed model on test dataset, outperforming traditional QPSO, WQPSO and PQPSO models by 33.027%, 31.817% and 30.897% respectively, and showcasing the potential of proposed model.

Sound recurrence analysis for acoustic scene classification

In everyday life, people experience different soundscapes in which natural sounds, animal noises, and man-made sounds blend together. Although there have been several studies on the importance of recurring sound patterns in music and language, the relevance of this phenomenon in natural soundscapes is still largely unexplored. In this article, we study the repetition patterns of harmonic and transient sound events as potential cues for acoustic scene classification (ASC). In the first part of our study, our aim is to identify acoustic scene classes that exhibit characteristic sound repetition patterns concerning harmonic and transient sounds. We propose three metrics to measure the overall prevalence of sound repetitions as well as their repetition periods and temporal stability. In the second part, we evaluate three strategies to incorporate self-similarity matrices as an additional input feature to a convolutional neural network architecture for ASC. We observe the characteristic repetition of transient sounds in recordings of “park” and “street traffic” as well as harmonic sound repetitions in acoustic scene classes related to public transportation. In the ASC experiments, hybrid network architectures, which combine spectrogram features and features from sound recurrence analysis, show increased accuracy for those classes with prominent sound repetition patterns. Our findings provide additional perspective on the distinctions among acoustic scenes previously primarily ascribed in the literature to their spectral features.

Transformative Advancements in Fetal Cardiac Health through BILSTM Networks for FPCG Classification

This article unveils a cutting-edge methodology for Fetal Phonocardiogram(FPCG) classification employing Bidirectional Long Short-Term Memory (BILSTM)networks. Acknowledging the pivotal role of fetal heart monitoring inearly anomaly detection, the research delves into the profound insights offeredby FPCG signals concerning fetal cardiac activity. The innovative approachencompasses preprocessing FPCG signals using Mel-frequency cepstral coefficients(MFCC) and spectrogram features, coupled with the strategic applicationof BI-LSTM networks, ensuring a resilient classification framework. The bidirectionalnature of the LSTM architecture elevates the model’s ability to capturetemporal dependencies in both forward and backward directions, facilitating thediscernment of intricate patterns in fetal heart sounds. Remarkably, experimentalfindings demonstrate a remarkable 98% accuracy, reaffirming the effectivenessand precision of the BILSTM approach in fetal PCG classification. This pioneeringresearch significantly advances automated methods for evaluating fetalcardiac health, promising transformative enhancements in prenatal care.

Local Pyramid Vision Transformer: Millimeter-Wave Radar Gesture Recognition Based on Transformer with Integrated Local and Global Awareness

A millimeter-wave radar is widely accepted by the public due to its low susceptibility to interference, such as changes in light, and the protection of personal privacy. With the development of the deep learning theory, the deep learning method has been dominant in the millimeter-wave radar field, which usually uses convolutional neural networks for feature extraction. In recent years, transformer networks have also been highly valued by researchers due to their parallel processing capabilities and long-distance dependency modeling capabilities. However, traditional convolutional neural networks (CNNs) and vision transformers each have their limitations: CNNs usually overlook the global features of images and vision transformers may neglect local image continuity, and both of them may impede gesture recognition performance. In addition, whether CNN or transformer, their implementation is hindered by the scarcity of public radar gesture datasets. To address these limitations, this paper proposes a new recognition method using a local pyramid visual transformer (LPVT) based on millimeter-wave radar. LPVT can capture global and local features in dynamic gesture spectrograms, ultimately improving the recognition ability of gestures. In this paper, we mainly carried out the following two tasks: building the corresponding datasets and executing gesture recognition. First, we constructed a gesture dataset for training. In this stage, we use a 77 GHz radar to collect the echo signals of gestures and preprocess them to build a dataset. Second, we propose the LPVT network specifically designed for gesture recognition tasks. By integrating local sensing into the globally focused transformer, we improve its capacity to capture both global and local features in dynamic gesture spectrograms. The experimental results using the dataset we constructed show that the proposed LPVT network achieved a gesture recognition accuracy of 92.2%, which exceeds the performance of other networks.

Abnormal Sound Detection of Wind Turbine Gearboxes Based on Improved MobileFaceNet and Feature Fusion

To solve problems such as the unstable detection performance of the sound anomaly detection of wind turbine gearboxes when only normal data are used for training, and the poor detection performance caused by the poor classification of samples with high similarity, this paper proposes a self-supervised wind turbine gearbox sound anomaly detection algorithm that fuses time-domain features and Mel spectrograms, improves the MobileFaceNet (MFN) model, and combines the Gaussian Mixture Model (GMM). This method compensates for the abnormal information lost in Mel spectrogram features through feature fusion and introduces a style attention mechanism (SRM) in MFN to enhance the expression of features, improving the accuracy and stability of the abnormal sound detection model. For the wind turbine gearbox sound dataset of a certain wind farm in Guangyuan, the average AUC of the sound data at five measuring point positions of the wind turbine gearbox using the method proposed in this paper, STgram-MFN-SRM, reached 96.16%. Compared with the traditional anomaly detection methods LogMel-MFN, STgram-MFN, STgram-Resnet50, and STgram-MFN-SRM(CE), the average AUC of sound detection at the five measuring point positions increased by 5.19%, 4.73%, 11.06%, and 2.88%, respectively. Therefore, the method proposed in this paper effectively improves the performance of the sound anomaly detection model of wind turbine gearboxes and has important engineering value for the healthy operation and maintenance of wind turbines.

Feature pyramid attention network for audio‐visual scene classification

AbstractAudio‐visual scene classification (AVSC) poses a formidable challenge owing to the intricate spatial‐temporal relationships exhibited by audio‐visual signals, coupled with the complex spatial patterns of objects and textures found in visual images. The focus of recent studies has predominantly revolved around extracting features from diverse neural network structures, inadvertently neglecting the acquisition of semantically meaningful regions and crucial components within audio‐visual data. The authors present a feature pyramid attention network (FPANet) for audio‐visual scene understanding, which extracts semantically significant characteristics from audio‐visual data. The authors’ approach builds multi‐scale hierarchical features of sound spectrograms and visual images using a feature pyramid representation and localises the semantically relevant regions with a feature pyramid attention module (FPAM). A dimension alignment (DA) strategy is employed to align feature maps from multiple layers, a pyramid spatial attention (PSA) to spatially locate essential regions, and a pyramid channel attention (PCA) to pinpoint significant temporal frames. Experiments on visual scene classification (VSC), audio scene classification (ASC), and AVSC tasks demonstrate that FPANet achieves performance on par with state‐of‐the‐art (SOTA) approaches, with a 95.9 F1‐score on the ADVANCE dataset and a relative improvement of 28.8%. Visualisation results show that FPANet can prioritise semantically meaningful areas in audio‐visual signals.

A new sensing paradigm for the vibroacoustic detection of pedicle screw loosening.

The current clinical gold standard to assess the condition and detect loosening of pedicle screw implants is radiation-emitting medical imaging. However, solely based on medical imaging, clinicians are not able to reliably identify loose implants in a substantial amount of cases. To complement medical imaging for pedicle screw loosening detection, we propose a new methodology and paradigm for the radiation-free, non-destructive, and easy-to-integrate loosening detection based on vibroacoustic sensing. For the detection of a loose implant, we excite the vertebra of interest with a sine sweep vibration at the spinous process and use a custom highly sensitive piezo vibration sensor attached directly at the screw head to capture the propagated vibration characteristics which are analyzed using a detection pipeline based on spectrogram features and a SE-ResNet-18. To validate the proposed approach, we propose a novel, biomechanically validated simulation technique for pedicle screw loosening, conduct experiments using four human cadaveric lumbar spine specimens, and evaluate our algorithm in a cross-validation experiment. The proposed method reaches a sensitivity of and a specificity of for pedicle screw loosening detection.

Recognition of Sheep Feeding Behavior in Sheepfolds Using Fusion Spectrogram Depth Features and Acoustic Features.

In precision feeding, non-contact and pressure-free monitoring of sheep feeding behavior is crucial for health monitoring and optimizing production management. The experimental conditions and real-world environments differ when using acoustic sensors to identify sheep feeding behaviors, leading to discrepancies and consequently posing challenges for achieving high-accuracy classification in complex production environments. This study enhances the classification performance by integrating the deep spectrogram features and acoustic characteristics associated with feeding behavior. We conducted the task of collecting sound data in actual production environments, considering noise and complex surroundings. The method included evaluating and filtering the optimal acoustic features, utilizing a customized convolutional neural network (SheepVGG-Lite) to extract Short-Time Fourier Transform (STFT) spectrograms and Constant Q Transform (CQT) spectrograms' deep features, employing cross-spectrogram feature fusion and assessing classification performance through a support vector machine (SVM). Results indicate that the fusion of cross-spectral features significantly improved classification performance, achieving a classification accuracy of 96.47%. These findings highlight the value of integrating acoustic features with spectrogram deep features for accurately recognizing sheep feeding behavior.

Ground penetrating radar urban road underground target classification algorithm using sequential spectral and time-domain features

Abstract Ground-penetrating radar (GPR), a highly efficient non-destructive detection method, finds extensive use in urban road underground target detection. Existing GPR data recognition algorithms often rely on singular time-domain spectrogram features, leading to potential misjudgements. To address this, we propose a novel algorithm based on sequence spectra and time-domain features. Serialized radar data, transformed through wavelets, is combined with time-domain images for input, enabling classification through a multi-scale convolutional neural network. Experiments show improved accuracy in underground target classification, offering a fresh perspective on intelligent GPR data recognition.

Comparative study of CNN, LSTM and hybrid CNN-LSTM model in amazigh speech recognition using spectrogram feature extraction and different gender and age dataset

Comparative study of CNN, LSTM and hybrid CNN-LSTM model in amazigh speech recognition using spectrogram feature extraction and different gender and age dataset

Attention Induced Dual Convolutional-Capsule Network (AIDC-CN): A deep learning framework for motor imagery classification

In recent times, Electroencephalography (EEG)-based motor imagery (MI) decoding has garnered significant attention due to its extensive applicability in healthcare, including areas such as assistive robotics and rehabilitation engineering. Nevertheless, the decoding of EEG signals presents considerable challenges owing to their inherent complexity, non-stationary characteristics, and low signal-to-noise ratio. Notably, deep learning-based classifiers have emerged as a prominent focus for addressing the EEG signal decoding process. This study introduces a novel deep learning classifier named the Attention Induced Dual Convolutional-Capsule Network (AIDC-CN) with the specific aim of accurately categorizing various motor imagination class labels. To enhance the classifier’s performance, a dual feature extraction approach leveraging spectrogram and brain connectivity networks has been employed, diversifying the feature set in the classification task. The main highlights of the proposed AIDC-CN classifier includes the introduction of a dual convolution layer to handle the brain connectivity and spectrogram features, addition of a novel self-attention module (SAM) to accentuate the relevant parts of the convolved spectrogram features, introduction of a new cross-attention module (CAM) to refine the outputs obtained from the dual convolution layers and incorporation of a Gaussian Error Linear Unit (GELU) based dynamic routing algorithm to strengthen the coupling among the primary and secondary capsule layers. Performance analysis undertaken on four public data sets depict the superior performance of the proposed model with respect to the state-of-the-art techniques. The code for this model is available at https://github.com/RiteshSurChowdhury/AIDC-CN.

Unsupervised detection and classification of fish vocalizations in coral reefs

Typical convolutional neural network detectors for acoustic signals rely on large quantities of labeled data, which can be expensive and time-intensive to generate. Unsupervised neural network architectures allow for the automatic detection and classification of acoustic signals, which may be especially useful for classifying underwater signals that a typical human labeler may not be familiar with. Here, we used a convolutional autoencoder to automatically detect and classify fish vocalizations in a coral reef off the coast of Hawaii Island. A database of more than 2 million spectrogram segments were generated with durations of one second and frequency ranges of 30–700 Hz, corresponding to the timescale and spectrum of a significant portion of fish calls. The autoencoder was successful in recreating the features of input spectrograms, and k-means clustering was applied to the latent feature space of each sample to automatically generate classes. Visual inspection of example spectrograms within each class confirms that the autoencoder was successful in classifying fish calls, including differentiating between separate call types.

Impact of Sliding Window Variation and Neuronal Time Constants on Acoustic Anomaly Detection Using Recurrent Spiking Neural Networks in Automotive Environment

Acoustic perception of the automotive environment has the potential to advance driving potentials with enhanced safety. The challenge arises when these acoustic perception systems need to perform under resource and power constraints on edge devices. Neuromorphic computing has introduced spiking neural networks in the context of ultra-low power sensory edge devices. Spiking architectures leverage biological plausibility to achieve computational capabilities, accurate performance, and great compatibility with neuromorphic hardware. In this work, we explore the depths of spiking neurons and feature components with the acoustic scene analysis task for siren sounds. This research work aims to address the qualitative analysis of sliding windows’ variation on the feature extraction front of the preprocessing pipeline. Optimization of the parameters to exploit the feature extraction stage facilitates the advancement of the performance of the acoustics anomaly detection task. We exploit the parameters for mel spectrogram features and FFT calculations, prone to be suitable for computations in hardware. We conduct experiments with different window sizes and the overlapping ratio within the windows. We present our results for performance measures like accuracy and onset latency to provide an insight on the choice of optimal window. The non-trivial motivation of this research is to understand the effect of encoding behavior of spiking neurons with different windows. We further investigate the heterogeneous nature of membrane and synaptic time constants and their impact on the accuracy of anomaly detection. On a large scale audio dataset comprising of siren sounds and road traffic noises, we obtain accurate predictions of siren sounds using a recurrent spiking neural network. The baseline dataset comprising siren and noise sequences is enriched with a bird dataset to evaluate the model with unseen samples.

Bowel Sounds Detection Method Based on ResNet-BiLSTM and Attention Mechanism

Bowel sounds can reflect the movement and health status of the gastrointestinal tract. However, the traditional manual auscultation method has subjective deviation and is time-consuming and labor-intensive. In order to better assist doctors in diagnosing bowel sounds and improve the reliability and efficiency of bowel sound detection, this study proposed a deep neural network model that combines a residual neural network (ResNet), a bidirectional long short-term memory network (BiLSTM), and an attention mechanism. Firstly, a large number of labeled clinical data was collected using the self-developed multi-channel bowel sound acquisition system, and the multi-scale wavelet decomposition and reconstruction method was used to preprocess the bowel sounds. Then, log Mel spectrogram features were extracted and sent to the network for training. Finally, the performance and effectiveness of the model were evaluated and verified by 10-fold cross-validation and an ablation experiment. The experimental results showed that the precision, recall, and F 1 score of the model reached 83%, 76%, and 79%, respectively, and it could effectively detect bowel sound segments and locate their start and end times, performing better than previous algorithms. This algorithm can not only provide auxiliary information for doctors in clinical practice but also offer technical support for further analysis and research of bowel sounds.

Damage detection and classification of carbon fiber-reinforced polymer composite materials based on acoustic emission and convolutional recurrent neural network

Carbon fiber-reinforced polymer (CFRP) composite laminates generate acoustic emission signals during the tensile process, which can be used to identify the type of acoustic emission (AE) signal at a given moment and determine the current damage condition. However, due to the large number and complexity of AE signals generated during the tensile process of carbon fiber composite laminates, it is challenging to manually identify and annotate them. To address this issue, we propose a low-complexity classification and detection network model based on the convolutional recurrent neural network (CRNN) architecture. First, we construct a dataset of composite damage signals for a single damage category and use log-mel spectrogram features to train the model for that specific category of damage signals. Then, we utilize the trained model to recognize the AE signals of CFRP composite laminates. The proposed method achieves an accuracy of 99.02% in classifying single damage modes in the test set and effectively distinguishes the types of AE signals generated during the damage process of CFRP composite laminates. Compared with the classic classification model using AE signals, the number of parameters in our proposed low-complexity CRNN model is reduced by 94.42%. It also demonstrates that 19.4% of the AE signals during the damage process are in a superimposed state, indicating the simultaneous occurrence of multiple damage modes in CFRP composite laminates.

Sugarcane disease recognition through visible and near-infrared spectroscopy using deep learning assisted continuous wavelet transform-based spectrogram

Utilizing visible and near-infrared (Vis-NIR) spectroscopy in conjunction with chemometrics methods has been widespread for identifying plant diseases. However, a key obstacle involves the extraction of relevant spectral characteristics. This study aimed to enhance sugarcane disease recognition by combining convolutional neural network (CNN) with continuous wavelet transform (CWT) spectrograms for spectral features extraction within the Vis-NIR spectra (380–1400 nm) to improve the accuracy of sugarcane diseases recognition. Using 130 sugarcane leaf samples, the obtained one-dimensional CWT coefficients from Vis-NIR spectra were transformed into two-dimensional spectrograms. Employing CNN, spectrogram features were extracted and incorporated into decision tree, K-nearest neighbour, partial least squares discriminant analysis, and random forest (RF) calibration models. The RF model, integrating spectrogram-derived features, demonstrated the best performance with an average precision of 0.9111, sensitivity of 0.9733, specificity of 0.9791, and accuracy of 0.9487. This study may offer a non-destructive, rapid, and accurate means to detect sugarcane diseases, enabling farmers to receive timely and actionable insights on the crops’ health, thus minimizing crop loss and optimizing yields.

A Semi-Supervised Lie Detection Algorithm Based on Integrating Multiple Speech Emotional Features

When people tell lies, they often exhibit tension and emotional fluctuations, reflecting a complex psychological state. However, the scarcity of labeled data in datasets and the complexity of deception information pose significant challenges in extracting effective lie features, which severely restrict the accuracy of lie detection systems. To address this, this paper proposes a semi-supervised lie detection algorithm based on integrating multiple speech emotional features. Firstly, Long Short-Term Memory (LSTM) and Auto Encoder (AE) network process log Mel spectrogram features and acoustic statistical features, respectively, to capture the contextual links between similar features. Secondly, the joint attention model is used to learn the complementary relationship among different features to obtain feature representations with richer details. Lastly, the model combines the unsupervised loss Local Maximum Mean Discrepancy (LMMD) and supervised loss Jefferys multi-loss optimization to enhance the classification performance. Experimental results show that the algorithm proposed in this paper achieves better performance.

An efficient channel recurrent Criss-cross attention network for epileptic seizure prediction

Epilepsy is a chronic disease caused by repeated abnormal discharge of neurons in the brain. Accurately predicting the onset of epilepsy can effectively improve the quality of life for patients with the condition. While there are many methods for detecting epilepsy, EEG is currently considered one of the most effective analytical tools due to the abundant information it provides about brain activity. The aim of this study is to explore potential time-frequency and channel features from multi-channel epileptic EEG signals and to develop a patient-specific seizure prediction network. In this paper, an epilepsy EEG signal classification algorithm called Channel Recurrent Criss-cross Attention Network (CRCANet) is proposed. Firstly, the spectrograms processed by the short-time fourier transform is input into a Convolutional Neural Network (CNN). Then, the spectrogram feature map obtained in the previous step is input into the channel attention module to establish correlations between channels. Subsequently, the feature diagram containing channel attention characteristics is input into the recurrent criss-cross attention module to enhance the information content of each pixel. Finally, two fully connected layers are used for classification. We validated the method on 13 patients in the public CHB-MIT scalp EEG dataset, achieving an average accuracy of 93.8 %, sensitivity of 94.3 %, and specificity of 93.5 %. The experimental results indicate that CRCANet can effectively capture the time-frequency and channel characteristics of EEG signals while improving training efficiency.

Reshaping Bioacoustics Event Detection: Leveraging Few-Shot Learning (FSL) with Transductive Inference and Data Augmentation.

Bioacoustic event detection is a demanding endeavor involving recognizing and classifying the sounds animals make in their natural habitats. Traditional supervised learning requires a large amount of labeled data, which are hard to come by in bioacoustics. This paper presents a few-shot learning (FSL) method incorporating transductive inference and data augmentation to address the issues of too few labeled events and small volumes of recordings. Here, transductive inference iteratively alters class prototypes and feature extractors to seize essential patterns, whereas data augmentation applies SpecAugment on Mel spectrogram features to augment training data. The proposed approach is evaluated by using the Detecting and Classifying Acoustic Scenes and Events (DCASE) 2022 and 2021 datasets. Extensive experimental results demonstrate that all components of the proposed method achieve significant F-score improvements of 27% and 10%, for the DCASE-2022 and DCASE-2021 datasets, respectively, compared to recent advanced approaches. Moreover, our method is helpful in FSL tasks because it effectively adapts to sounds from various animal species, recordings, and durations.