Log-mel Spectrogram Research Articles

Acoustic insights into the corn extrusion process for enhanced quality control

In industrial extrusion processes, a solid material is pressed through a die to obtain products of the desired shape and dimension. Fluctuations in the process parameters have a significant impact on the product quality. In food extrusion, the expansion noise at the die can serve as an indicator of the stability of the process. This study employs microphones to characterize the corn extrusion process, focusing on correlating acoustic emissions with predefined process parameters such as feed intake and water content. Experimental data from laboratory and industrial settings reveal distinct domain shifts, yet consistent findings confirm the distinguishability of various extrusion process parameters by analyzing acoustic emissions. Employing machine learning models, including support vector machines and convolutional neural networks, in conjunction with audio features such as log Mel spectrograms, yields promising accuracies above 90\% in discriminating between standard and non-standard process parameters. The proposed acoustic quality control approach has the potential to enhance the stability of extrusion processes and contributes to the development of automated monitoring systems in the field of food extrusion. This ensures consistent quality and reduced waste, ultimately leading to significant cost savings in production.

Environmental sound classification using raw-audio based ensemble framework

Environmental sound classification (ESC) aims to automatically recognize audio recordings from the underlying environment, such as "urban park" or "city centre". Most of the existing methods for ESC use hand-crafted time-frequency features such as log-mel spectrogram to represent audio recordings. However, the hand-crafted features rely on transformations that are defined beforehand and do not consider the variability in the environment due to differences in recording conditions or recording devices. To overcome this, we present an alternative representation framework by leveraging a pre-trained convolutional neural network, SoundNet, trained on a large-scale audio dataset to represent raw audio recordings. We observe that the representations obtained from the intermediate layers of SoundNet lie in low-dimensional subspace. However, the dimensionality of the low-dimensional subspace is not known. To address this, an automatic compact dictionary learning framework is utilized that gives the dimensionality of the underlying subspace. The low-dimensional embeddings are then aggregated in a late-fusion manner in the ensemble framework to incorporate hierarchical information learned at various intermediate layers of SoundNet. We perform experimental evaluation on publicly available DCASE 2017 and 2018 ASC datasets. The proposed ensemble framework improves performance between 1 and 4 percentage points compared to that of existing time-frequency representations.

Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models

As speech is the most natural way for humans to express emotions, studies on Speech Emotion Recognition (SER) have been conducted in various ways However, there are some areas for improvement in previous SER studies: (1) while some studies have performed multi-label classification, almost none have specifically utilized Korean speech data; (2) most studies have not utilized multiple features in combination for emotion recognition. Therefore, this study proposes deep fusion models for multi-label emotion classification using Korean speech data and follows four steps: (1) preprocessing speech data labeled with Sadness, Happiness, Neutral, Anger, and Disgust; (2) applying data augmentation to address the data imbalance and extracting speech features, including the Log-mel spectrogram, Mel-Frequency Cepstral Coefficients (MFCCs), and Voice Quality Features; (3) constructing models using deep fusion architectures; and (4) validating the performance of the constructed models. The experimental results demonstrated that the proposed model, which utilizes the Log-mel spectrogram and MFCCs with a fusion of Vision-Transformer and 1D Convolutional Neural Network–Long Short-Term Memory, achieved the highest average binary accuracy of 71.2% for multi-label classification, outperforming other baseline models. Consequently, this study anticipates that the proposed model will find application based on Korean speech, specifically mental healthcare and smart service systems.

Damage detection and classification of carbon fiber-reinforced polymer composite materials based on acoustic emission and convolutional recurrent neural network

Carbon fiber-reinforced polymer (CFRP) composite laminates generate acoustic emission signals during the tensile process, which can be used to identify the type of acoustic emission (AE) signal at a given moment and determine the current damage condition. However, due to the large number and complexity of AE signals generated during the tensile process of carbon fiber composite laminates, it is challenging to manually identify and annotate them. To address this issue, we propose a low-complexity classification and detection network model based on the convolutional recurrent neural network (CRNN) architecture. First, we construct a dataset of composite damage signals for a single damage category and use log-mel spectrogram features to train the model for that specific category of damage signals. Then, we utilize the trained model to recognize the AE signals of CFRP composite laminates. The proposed method achieves an accuracy of 99.02% in classifying single damage modes in the test set and effectively distinguishes the types of AE signals generated during the damage process of CFRP composite laminates. Compared with the classic classification model using AE signals, the number of parameters in our proposed low-complexity CRNN model is reduced by 94.42%. It also demonstrates that 19.4% of the AE signals during the damage process are in a superimposed state, indicating the simultaneous occurrence of multiple damage modes in CFRP composite laminates.

A Semi-Supervised Lie Detection Algorithm Based on Integrating Multiple Speech Emotional Features

When people tell lies, they often exhibit tension and emotional fluctuations, reflecting a complex psychological state. However, the scarcity of labeled data in datasets and the complexity of deception information pose significant challenges in extracting effective lie features, which severely restrict the accuracy of lie detection systems. To address this, this paper proposes a semi-supervised lie detection algorithm based on integrating multiple speech emotional features. Firstly, Long Short-Term Memory (LSTM) and Auto Encoder (AE) network process log Mel spectrogram features and acoustic statistical features, respectively, to capture the contextual links between similar features. Secondly, the joint attention model is used to learn the complementary relationship among different features to obtain feature representations with richer details. Lastly, the model combines the unsupervised loss Local Maximum Mean Discrepancy (LMMD) and supervised loss Jefferys multi-loss optimization to enhance the classification performance. Experimental results show that the algorithm proposed in this paper achieves better performance.

Open-Set Recognition of Pansori Rhythm Patterns Based on Audio Segmentation

Pansori, a traditional Korean form of musical storytelling, is characterized by performances involving a vocalist and a drummer. It is well-known for the singer’s expressive narrative (aniri) and delicate gesture with fan in hand. The classical Pansori repertoires mostly tell love, satire, and humor, as well as some social lessons. These performances, which can extend from three to five hours, necessitate that the vocalist adheres to precise rhythmic structures. The distinctive rhythms of Pansori are crucial for conveying both the narrative and musical expression effectively. This paper explores the challenge of open-set recognition, aiming to efficiently identify unknown Pansori rhythm patterns while applying the methodology to diverse acoustic datasets, such as sound events and genres. We propose a lightweight deep learning-based encoder–decoder segmentation model, which employs a 2-D log-Mel spectrogram as input for the encoder and produces a frame-based 1-D decision along the temporal axis. This segmentation approach, processing 2-D inputs to classify frame-wise rhythm patterns, proves effective in detecting unknown patterns within time-varying sound streams encountered in daily life. Throughout the training phase, both center and supervised contrastive losses, along with cross-entropy loss, are minimized. This strategy aimed to create a compact cluster structure within the feature space for known classes, thereby facilitating the recognition of unknown rhythm patterns by allocating ample space for their placement within the embedded feature space. Comprehensive experiments utilizing various datasets—including Pansori rhythm patterns (91.8%), synthetic datasets of instrument sounds (95.1%), music genres (76.9%), and sound datasets from DCASE challenges (73.0%)—demonstrate the efficacy of our proposed method to detect unknown events, as evidenced by the AUROC metrics.

FFA-BiGRU: Attention-Based Spatial-Temporal Feature Extraction Model for Music Emotion Classification

Music emotion recognition is becoming an important research direction due to its great significance for music information retrieval, music recommendation, and so on. In the task of music emotion recognition, the key to achieving accurate emotion recognition lies in how to extract the affect-salient features fully. In this paper, we propose an end-to-end spatial-temporal feature extraction method named FFA-BiGRU for music emotion classification. Taking the log Mel-spectrogram of music audio as the input, this method employs an attention-based convolutional residual module named FFA, which serves as a spatial feature learning module to obtain multi-scale spatial features. In the FFA module, three group architecture blocks extract multi-level spatial features, each of which consists of a stack of multiple channel-spatial attention-based residual blocks. Then, the output features from FFA are fed into the bidirectional gated recurrent units (BiGRU) module to capture the temporal features of music further. In order to make full use of the extracted spatial and temporal features, the output feature maps of FFA and those of the BiGRU are concatenated in the channel dimension. Finally, the concatenated features are passed through fully connected layers to predict the emotion classification results. The experimental results of the EMOPIA dataset show that the proposed model achieves better classification accuracy than the existing baselines. Meanwhile, the ablation experiments also demonstrate the effectiveness of each part of the proposed method.

Heart Murmur Quality Detection Using Deep Neural Networks with Attention Mechanism

Heart murmurs play a critical role in assessing the condition of the heart. Murmur quality reflects the subjective human perception of heart murmurs and is an important characteristic strongly linked to cardiovascular diseases (CVDs). This study aims to use deep neural networks to classify the patients’ murmur quality (i.e., harsh and blowing) from phonocardiogram (PCG) signals. The phonocardiogram recordings with murmurs used for this task are from the CirCor DigiScope Phonocardiogram dataset, which provides the murmur quality labels. The recordings were segmented, and a dataset of 1266 segments with average lengths of 4.1 s from 164 patients’ recordings was obtained. Each patient usually has multiple segments. A deep neural network model based on convolutional neural networks (CNNs) with channel attention and gated recurrent unit (GRU) networks was first used to extract features from the log-Mel spectrograms of segments. Then, the features of different segments from one patient were weighted by the proposed “Feature Attention” module based on the attention mechanism. The “Feature Attention” module contains a layer of global pooling and two fully connected layers. Through it, the different features can learn their weight, which can help the deep learning model distinguish the importance of different features of one patient. Finally, the detection results were produced. The cross-entropy loss function was used to train the model, and five-fold cross-validation was employed to evaluate the performance of the proposed methods. The accuracy of detecting the quality of patients’ murmurs is 73.6%. The F1-scores (precision and recall) for the murmurs of harsh and blowing are 76.8% (73.0%, 83.0%) and 67.8% (76.0%, 63.3%), respectively. The proposed methods have been thoroughly evaluated and have the potential to assist physicians with the diagnosis of cardiovascular diseases as well as explore the relationship between murmur quality and cardiovascular diseases in depth.

Augmenting Aquaculture Efficiency through Involutional Neural Networks and Self-Attention for Oplegnathus Punctatus Feeding Intensity Classification from Log Mel Spectrograms.

Understanding the feeding dynamics of aquatic animals is crucial for aquaculture optimization and ecosystem management. This paper proposes a novel framework for analyzing fish feeding behavior based on a fusion of spectrogram-extracted features and deep learning architecture. Raw audio waveforms are first transformed into Log Mel Spectrograms, and a fusion of features such as the Discrete Wavelet Transform, the Gabor filter, the Local Binary Pattern, and the Laplacian High Pass Filter, followed by a well-adapted deep model, is proposed to capture crucial spectral and spectral information that can help distinguish between the various forms of fish feeding behavior. The Involutional Neural Network (INN)-based deep learning model is used for classification, achieving an accuracy of up to 97% across various temporal segments. The proposed methodology is shown to be effective in accurately classifying the feeding intensities of Oplegnathus punctatus, enabling insights pertinent to aquaculture enhancement and ecosystem management. Future work may include additional feature extraction modalities and multi-modal data integration to further our understanding and contribute towards the sustainable management of marine resources.

Summarization for Enhanced Video Conference Collaboration

Abstract: Efficient collaboration during video conferences is crucial for team productivity. However, the absence of effective mechanisms to capture and preserve key insights often results in misunderstandings and reduced efficiency. The proposed initiative focuses on developing a web application to improve collaboration in video conferences by converting speech conversations into summarized text. Leveraging machine learning models, the system transcribes spoken content using whisper model and employs abstractive summarization using Pegasus X-Sum model. The Whisper architecture is implemented as an encoder-decoder Transformer. Input audio is split into 30-second chunks, converted into a log-Mel spectrogram, and then passed into an encoder. A decoder is trained to predict the corresponding text caption, intermixed with special tokens that direct the model to perform transcription. The goal is to empower users to effortlessly capture and retain crucial discussions, promoting better understanding and decision-making.

Operation and Productivity Monitoring from Sound Signal of Legacy Pipe Bending Machine via Convolutional Neural Network (CNN)

This study introduces a non-invasive approach to monitor operation and productivity of a legacy pipe bending machine in real-time based on a lightweight convolutional neural network (CNN) model and internal sound as input data. Various sensors were deployed to determine the optimal sensor type and placement, and labels for training and testing the CNN model were generated through the meticulous collection of sound data in conjunction with webcam videos. The CNN model, which was optimized through hyperparameter tuning via grid search and utilized feature extraction using Log-Mel spectrogram, demonstrated notable prediction accuracies in the test. However, when applied in a real-world manufacturing scenario, the model encountered a significant number of errors in predicting productivity. To navigate through this challenge and enhance the predictive accuracy of the system, a buffer algorithm using the inferences of CNN models was proposed. This algorithm employs a queuing method for continuous sound monitoring securing robust predictions, refines the interpretation of the CNN model inferences, and enhances prediction outcomes in actual implementation where accuracy of monitoring productivity information is crucial. The proposed lightweight CNN model alongside the buffer algorithm was successfully deployed on an edge computer, enabling real-time remote monitoring.

Leveraging Deep Learning for Fine-Grained Categorization of Parkinson's Disease Progression Levels through Analysis of Vocal Acoustic Patterns.

Speech impairments often emerge as one of the primary indicators of Parkinson's disease (PD), albeit not readily apparent in its early stages. While previous studies focused predominantly on binary PD detection, this research explored the use of deep learning models to automatically classify sustained vowel recordings into healthy controls, mild PD, or severe PD based on motor symptom severity scores. Popular convolutional neural network (CNN) architectures, VGG and ResNet, as well as vision transformers, Swin, were fine-tuned on log mel spectrogram image representations of the segmented voice data. Furthermore, the research investigated the effects of audio segment lengths and specific vowel sounds on the performance of these models. The findings indicated that implementing longer segments yielded better performance. The models showed strong capability in distinguishing PD from healthy subjects, achieving over 95% precision. However, reliably discriminating between mild and severe PD cases remained challenging. The VGG16 achieved the best overall classification performance with 91.8% accuracy and the largest area under the ROC curve. Furthermore, focusing analysis on the vowel /u/ could further improve accuracy to 96%. Applying visualization techniques like Grad-CAM also highlighted how CNN models focused on localized spectrogram regions while transformers attended to more widespread patterns. Overall, this work showed the potential of deep learning for non-invasive screening and monitoring of PD progression from voice recordings, but larger multi-class labeled datasets are needed to further improve severity classification.

A deep learning approach to dysarthric utterance classification with BiLSTM-GRU, speech cue filtering, and log mel spectrograms

A deep learning approach to dysarthric utterance classification with BiLSTM-GRU, speech cue filtering, and log mel spectrograms

Audio–visual representation learning for anomaly events detection in crowds

In recent years, anomaly events detection in crowd scenes attracts many researchers’ attentions, because of its importance to public safety. Existing methods usually exploit visual information to analyze whether any abnormal events have occurred due to only visual sensors are generally equipped in public places. However, when an abnormal event in crowds occurs, sound information may be discriminative to assist the crowd analysis system to determine whether there is an abnormality. Compared with vision information that is easily occluded, audio signals have a certain degree of penetration. Thus, this paper attempt to exploit multi-modal learning for modeling the audio and visual signals simultaneously. To be specific, we design a two-branch network to model different types of information. The first is a typical 3D CNN model to extract temporal appearance feature from video clips. The second is an audio CNN for encoding Log Mel-Spectrogram of audio signals. Finally, by fusing the above features, the more accurate prediction will be produced. We conduct the experiments on SHADE dataset, a synthetic audio–visual dataset in surveillance scenes, and find introducing audio signals effectively improves the performance of anomaly events detection and outperforms other state-of-the-art methods. Furthermore, we will release the code and the pre-trained models as soon as possible.

Deep Learning Based Classification of Underwater Acoustic Signals

The classification of underwater acoustic targets is a challenging task due to the intricacies involved in soundscape, huge amounts of background interference, and the varied features of underwater targets. Various approaches have been explored for classifying underwater targets based on acoustic signatures. The problem with these conventional methods lies in the extensive domain-specific knowledge they demand for effective feature engineering. Alternatively, the deep learning approach is of great appeal for assistance to manual sonar operators who mainly rely on their expertise for the classification of targets. This work leverages the approach of deep neural networks for underwater acoustic target categorization, aiming to analyze whether we can reduce the need for extensive domain knowledge required, by letting deep learning algorithms itself find the deep audio embeddings. This approach implements an audio classifier designed to classify acoustic targets with input as log Mel-spectrograms. To compare the performance of this classifier, we also implement a regular neural network for audio classification with MFCCs as input features. The overall results of the model are promising.

Make some Noise: Acoustic Classification of Manual Work Steps Towards Adaptive Assistance Systems

With 32 million people working in the European manufacturing sector, human work still plays a crucial role in industry. However, due to lot size one manufacturing and increased quality requirements, the complexity of products and processes is growing. Therefore, numerous approaches introduce adaptive assistance systems in assembly to support workers during complex work tasks and adapt their level of assistance to the current situation. To enable adaptivity, human action recognition must be incorporated into the consideration of context. Until now, research has focused on providing context to the machine through wearable sensors or cameras. However, wearable sensors hinder worker’s movements and cameras have difficulties distinguishing between work steps of high visual similarity. To mitigate these challenges, we present a new method to classify manual work steps only by their typical acoustic characteristics. The proposed method uses log-Mel spectrograms of work sounds fed into a convolutional neural network (CNN), thus learning their characteristic structure. Moreover, we present a new public dataset for the acoustic classification of manual work steps. The dataset includes typical sources of sounds in manufacturing, such as working with a bench grinder, cordless screwdriver, fling, or grabbing screws. Before feeding the data to the CNN, we apply various pre-processing and data augmentation techniques to increase generalisation capabilities. Our method can detect work steps with reliable accuracy while requiring less parameters than other techniques, proving that detecting work context through acoustics is possible and feasible.

EMG gesture signal analysis towards diagnosis of upper limb using dual-pathway convolutional neural network.

This research introduces a novel dual-pathway convolutional neural network (DP-CNN) architecture tailored for robust performance in Log-Mel spectrogram image analysis derived from raw multichannel electromyography signals. The primary objective is to assess the effectiveness of the proposed DP-CNN architecture across three datasets (NinaPro DB1, DB2, and DB3), encompassing both able-bodied and amputee subjects. Performance metrics, including accuracy, precision, recall, and F1-score, are employed for comprehensive evaluation. The DP-CNN demonstrates notable mean accuracies of 94.93 ± 1.71% and 94.00 ± 3.65% on NinaPro DB1 and DB2 for healthy subjects, respectively. Additionally, it achieves a robust mean classification accuracy of 85.36 ± 0.82% on amputee subjects in DB3, affirming its efficacy. Comparative analysis with previous methodologies on the same datasets reveals substantial improvements of 28.33%, 26.92%, and 39.09% over the baseline for DB1, DB2, and DB3, respectively. The DP-CNN's superior performance extends to comparisons with transfer learning models for image classification, reaffirming its efficacy. Across diverse datasets involving both able-bodied and amputee subjects, the DP-CNN exhibits enhanced capabilities, holding promise for advancing myoelectric control.

A Machine Anomalous Sound Detection Method Using the lMS Spectrogram and ES-MobileNetV3 Network

Unsupervised anomalous sound detection by machines holds significant importance within the realm of industrial automation. Currently, the task of machine-based anomalous sound detection in complex industrial settings is faced with issues such as the challenge of extracting acoustic feature information and an insufficient feature extraction capability within the detection network. To address these challenges, this study proposes a machine anomalous sound detection method using the lMS spectrogram and ES-MobileNetV3 network. Firstly, the log-Mel spectrogram feature and the SincNet spectrogram feature are extracted from the raw wave, and the new lMS spectrogram is formed after fusion, serving as network input features. Subsequently, based on the MobileNetV3 network, an improved detection network, ES-MobileNetV3, is proposed in this paper. This network incorporates the Efficient Channel Attention module and the SoftPool method, which collectively reduces the loss of feature information and enhances the feature extraction capability of the detection network. Finally, experiments are conducted on the dataset provided by DCASE 2020 Task 2. Our proposed method attained an averaged area under the receiver operating characteristic curve (AUC) of 96.67% and an averaged partial AUC (pAUC) of 92.38%, demonstrating superior detection performance compared to other advanced methods.

EnViTSA: Ensemble of Vision Transformer with SpecAugment for Acoustic Event Classification.

Recent successes in deep learning have inspired researchers to apply deep neural networks to Acoustic Event Classification (AEC). While deep learning methods can train effective AEC models, they are susceptible to overfitting due to the models' high complexity. In this paper, we introduce EnViTSA, an innovative approach that tackles key challenges in AEC. EnViTSA combines an ensemble of Vision Transformers with SpecAugment, a novel data augmentation technique, to significantly enhance AEC performance. Raw acoustic signals are transformed into Log Mel-spectrograms using Short-Time Fourier Transform, resulting in a fixed-size spectrogram representation. To address data scarcity and overfitting issues, we employ SpecAugment to generate additional training samples through time masking and frequency masking. The core of EnViTSA resides in its ensemble of pre-trained Vision Transformers, harnessing the unique strengths of the Vision Transformer architecture. This ensemble approach not only reduces inductive biases but also effectively mitigates overfitting. In this study, we evaluate the EnViTSA method on three benchmark datasets: ESC-10, ESC-50, and UrbanSound8K. The experimental results underscore the efficacy of our approach, achieving impressive accuracy scores of 93.50%, 85.85%, and 83.20% on ESC-10, ESC-50, and UrbanSound8K, respectively. EnViTSA represents a substantial advancement in AEC, demonstrating the potential of Vision Transformers and SpecAugment in the acoustic domain.

Sound-Based Real-Time Scrap Falling State Monitoring in the Shearing Process of Wide Heavy Plate Mill

The scrap falling state in the shearing process of a Wide Heavy Plate Mill depends on manual monitoring and judgment. The steel plate can only move forward after the scrap falls, otherwise, the equipment will be damaged. An experienced operator can use the sound they hear to estimate the falling state. To automatically recognize sound events, a scheme of sound data acquisition processing and recognition for monitoring the scrap state of the shearing process is proposed in this paper. Firstly, a pickup is installed at the monitoring point of the crop shear to collect the sound. Afterward, the sound data is processed and finally fed into the model for recognition. Since many sound segments in the field are not related to key actions, an endpoint detection technique based on a short-term energy threshold and the variable window length strategy is applied in the data processing section to obtain the sound segments to be identified. The Log-Mel spectrogram feature of the sound segment is then extracted. After a data filling to ensure a uniform data shape, the feature map is fed into a lightweight deep convolution neural network called MobileNetV2 to quickly identify sound events. For the shearing process events, the model recognition accuracy can reach 95% of the F1 score in a realistic noise environment and we can identify the scrap falling 0.3s after the action starts. Finally, the model was applied to the UrbanSound8K dataset and achieved 95% recognition accuracy, validating the performance of the method.