Sound Recognition Research Articles

A convolutional neural network to control sound level for air conditioning units in four different classroom conditions

Air conditioning units (ACUs) are widely used in educational areas like classrooms in order to ensure the occupant’s well-being and to control CO2 concentration (by ventilating using fresh outdoor air). However, the noise level of these ACUs can disrupt the learning environment. Consequently, we propose an approach based on machine learning that is able to distinguish between the different acoustic situations in a classroom, and to dynamically adjust the air volume flow accordingly. To this end, the present algorithm was trained with sound recordings in four different scenarios in a classroom. Both Mel spectrogram and Cochleagram are considered and applied for the task of training the convolutional neural networks (CNN) model, thus enhancing published works in the literature, which only considered the Mel spectrogram. Results show how the Cochleagram is pre-eminent to handle the CNN model training over the Mel spectrogram. Accordingly, we use the Cochleagram for the CNN model training, which is subsequently used for detecting the current situation in the room and adapting the ACU operation to this situation. The results show that running ACUs in high mode provides a learning environment with low CO2 concentration and high noise. Instead, controlling ACU based scenario predictions made by the CNN model provides a good learning environment with adequate CO2 concentration and acceptable noise level. Our results demonstrate the effectiveness of using sound classification as a trigger for ventilation control, with the CNN model achieving a good accuracy rate in sound recognition. This underscores the potential of integrating advanced machine learning techniques into building management systems to foster environments that adapt to the needs of their inhabitants automatically. The results of the present work are useful for improving the comfort of the occupants through dynamic ACUs adjustments based on acoustical situations.

Applying the Generative Model Integrated with the Diffusion Technique to Improve Virtual Sample Generation in Environmental Sound Classification

Applying the Generative Model Integrated with the Diffusion Technique to Improve Virtual Sample Generation in Environmental Sound Classification

Backdoor Attacks and Defenses Targeting Multi-Domain AI Models: A Comprehensive Review

Since the emergence of security concerns in artificial intelligence (AI), there has been significant attention devoted to the examination of backdoor attacks. Attackers can utilize backdoor attacks to manipulate model predictions, leading to significant potential harm. However, current research on backdoor attacks and defenses in both theoretical and practical fields still has many shortcomings. To systematically analyze these shortcomings and address the lack of comprehensive reviews, this paper presents a comprehensive and systematic summary of both backdoor attacks and defenses targeting multi-domain AI models. Simultaneously, based on the design principles and shared characteristics of triggers in different domains and the implementation stages of backdoor defense, this paper proposes a new classification method for backdoor attacks and defenses. We use this method to extensively review backdoor attacks in the fields of computer vision and natural language processing, and also examine the current applications of backdoor attacks in audio recognition, video action recognition, multimodal tasks, time series tasks, generative learning, and reinforcement learning, while critically analyzing the open problems of various backdoor attack techniques and defense strategies. Finally, this paper builds upon the analysis of the current state of AI security to further explore potential future research directions for backdoor attacks and defenses.

Neural Networks for Audio Classification: Multi-scale CNN-LSTM Approach to Animal Sound Recognition

Abstract. This research explores the application of neural networks, specifically CNN-LSTM models, for classifying sound signals from dogs, frogs, and cats, selected from the ESC-50 dataset. The sound data was preprocessed using Mel-frequency cepstral coefficients (MFCCs) and augmented through time stretching, pitch shifting, and noise addition to enhance model generalization in varied acoustic environments. We compared two deep learning models: a traditional CNN-LSTM and an improved version with multi-scale feature extraction, allowing for capturing both short-term and long-term sound patterns. Our findings show that the multi-scale CNN-LSTM architecture outperforms the traditional CNN-LSTM, achieving a test accuracy of 86.11% compared to 80.56%. These results highlight the effectiveness of multi-scale feature extraction for handling complex audio signals. This research offers valuable insights into bioacoustics and has broader applications in areas such as environmental sound monitoring, wildlife preservation, and animal behavior analysis.

Exercises to improve directional sound recognition by visually impaired users heading to the workplace

Moving is an integral part of everyday life and applies to accessing the workplace, attending a health centre or meeting friends. Determination of the source of sound may be important when a visually impaired person is moving around. This allows to navigate, but also provides important information about potential danger. Sound exercises have been prepared to support the ability to recognize the direction of sound and to learn about the problems associated with it. The research version of the app was prepared covering 12 situations related to three static and nine dynamic sources of sounds. All 10 participants in the exercises had a significant degree of visual impairment. An extended dataset obtained from three subjects without disabilities was also included. As a result, six exercises qualified for the final set of sound exercises, which allow to gain benefits from their implementation.

Sounds of Nature and Hearing Loss: A Call to Action.

The ability to monitor surrounding natural sounds and scenes is important for performing many activities in daily life and for overall well-being. Yet, unlike speech, perception of natural sounds and scenes is relatively understudied in relation to hearing loss, despite the documented restorative health effects. We present data from first-time hearing aid users describing "rediscovered" natural sounds they could now perceive with clarity. These data suggest that hearing loss not only diminishes recognition of natural sounds, but also limits people's awareness of the richness of their environment, thus limiting their connection to it. Little is presently known about the extent hearing aids can restore the perception of abundance, clarity, or intensity of natural sounds. Our call to action outlines specific steps to improve the experience of natural sounds and scenes for people with hearing loss-an overlooked aspect of their quality of life.

Heart Sound Classification Using Harmonic and Percussive Spectral Features from Phonocardiograms with a Deep ANN Approach

Cardiovascular diseases (CVDs) are a leading cause of mortality worldwide, with a particularly high burden in India. Non-invasive methods like Phonocardiogram (PCG) analysis capture the acoustic activity of the heart. This holds significant potential for the early detection and diagnosis of heart conditions. However, the complexity and variability of PCG signals pose considerable challenges for accurate classification. Traditional methods of PCG signal analysis, including time-domain, frequency-domain, and time-frequency domain techniques, often fall short in capturing the intricate details necessary for reliable diagnosis. This study introduces an innovative approach that leverages harmonic–percussive source separation (HPSS) to extract distinct harmonic and percussive spectral features from PCG signals. These features are then utilized to train a deep feed-forward artificial neural network (ANN), classifying heart conditions as normal or abnormal. The methodology involves advanced digital signal processing techniques applied to PCG recordings from the PhysioNet 2016 dataset. The feature set comprises 164 attributes, including the Chroma STFT, Chroma CENS, Mel-frequency cepstral coefficients (MFCCs), and statistical features. These are refined using the ROC-AUC feature selection method to ensure optimal performance. The deep feed-forward ANN model was rigorously trained and validated on a balanced dataset. Techniques such as noise reduction and outlier detection were used to improve model training. The proposed model achieved a validation accuracy of 93.40% with sensitivity and specificity rates of 82.40% and 80.60%, respectively. These results underscore the effectiveness of harmonic-based features and the robustness of the ANN in heart sound classification. This research highlights the potential for deploying such models in non-invasive cardiac diagnostics, particularly in resource-constrained settings. It also lays the groundwork for future advancements in cardiac signal analysis.

LightCardiacNet: light-weight deep ensemble network with attention mechanism for cardiac sound classification

Cardiovascular diseases (CVDs) account for about 32% of global deaths. While digital stethoscopes can record heart sounds, expert analysis is often lacking. To address this, we propose LightCardiacNet, an interpretable, lightweight ensemble neural network using Bi-Directional Gated Recurrent Units (Bi-GRU). It is trained on the PASCAL Heart Challenge and CirCor DigiScope datasets. Static network pruning enhances model sparsity for real-time deployment. We employ various data augmentation techniques to improve resilience to background noise. An ensemble of the two networks is constructed by employing a weighted average approach that combines the two light-weight attention Bi-GRU networks trained on different datasets, which outperforms several state-of-the-art networks achieving an accuracy of 99.8%, specificity of 99.6%, sensitivity of 95.2%, ROC-AUC of 0.974 and inference time of 17 ms on the PASCAL dataset, accuracy of 98.5%, specificity of 95.1%, sensitivity of 90.9%, ROC-AUC of 0.961 and inference time of 18 ms on the CirCor dataset, and an accuracy of 96.21%, sensitivity of 92.78%, specificity of 93.16%, ROC-AUC of 0.913 and inference time of 17.5 ms on real-world data. We adopt the SHAP algorithm to incorporate model interpretability and provide insights to make it clinically explainable and useful to healthcare professionals.

Hierarchical-Concatenate Fusion TDNN for sound event classification.

Semantic feature combination/parsing issue is one of the key problems in sound event classification for acoustic scene analysis, environmental sound monitoring, and urban soundscape analysis. The input audio signal in the acoustic scene classification is composed of multiple acoustic events, which usually leads to low recognition rate in complex environments. To address this issue, this paper proposes the Hierarchical-Concatenate Fusion(HCF)-TDNN model by adding HCF Module to ECAPA-TDNN model for sound event classification. In the HCF module, firstly, the audio signal is converted into two-dimensional time-frequency features for segmentation. Then, the segmented features are convolved one by one for improving the small receptive field in perceiving details. Finally, after the convolution is completed, the two adjacent parts are combined before proceeding with the next convolution for enlarging the receptive field in capturing large targets. Therefore, the improved model further enhances the scalability by emphasizing channel attention and efficient propagation and aggregation of feature information. The proposed model is trained and validated on the Urbansound8K dataset. The experimental results show that the proposed model can achieve the best classification accuracy of 95.83%, which is an approximate improvement of 5% (relatively) over the ECAPA-TDNN model.

Vision assistant for visually impaired individuals

Visually impaired individuals face significant challenges in accessing environmental information, social interactions, and safety in their daily lives. This article aims to increase the independence of visually impaired individuals and simplify their daily lives using an AI-supported image and sound recognition system. The developed prototype is a pair of glasses consisting of a Raspberry Pi 4 microprocessor, integrated headphones, touch controls, and a powerful battery management system. The electronic diagram includes the integration of a camera and microprocessor. The AI algorithms developed successfully perform tasks such as object recognition, text reading, and facial recognition. The article provides a comprehensive literature review and introduces the prototype to visually impaired users. Based on the feedback from the introduction, the prototype's ease of use and effectiveness were evaluated, and improvements were made based on user feedback. The algorithms were tested for accuracy, with the prototype achieving a 78% accuracy score, close to GPT-4's 82%. Additionally, tests with 20 users resulted in an 85% user satisfaction rate.

Towards the Development of the Clinical Decision Support System for the Identification of Respiration Diseases via Lung Sound Classification Using 1D-CNN.

Respiratory disorders are commonly regarded as complex disorders to diagnose due to their multi-factorial nature, encompassing the interplay between hereditary variables, comorbidities, environmental exposures, and therapies, among other contributing factors. This study presents a Clinical Decision Support System (CDSS) for the early detection of respiratory disorders using a one-dimensional convolutional neural network (1D-CNN) model. The ICBHI 2017 Breathing Sound Database, which contains samples of different breathing sounds, was used in this research. During pre-processing, audio clips were resampled to a uniform rate, and breathing cycles were segmented into individual instances of the lung sound. A One-Dimensional Convolutional Neural Network (1D-CNN) consisting of convolutional layers, max pooling layers, dropout layers, and fully connected layers, was designed to classify the processed clips into four categories: normal, crackles, wheezes, and combined crackles and wheezes. To address class imbalance, the Synthetic Minority Over-sampling Technique (SMOTE) was applied to the training data. Hyperparameters were optimized using grid search with k-fold cross-validation. The model achieved an overall accuracy of 0.95, outperforming state-of-the-art methods. Particularly, the normal and crackles categories attained the highest F1-scores of 0.97 and 0.95, respectively. The model's robustness was further validated through 5-fold and 10-fold cross-validation experiments. This research highlighted an essential aspect of diagnosing lung sounds through artificial intelligence and utilized the 1D-CNN to classify lung sounds accurately. The proposed advancement of technology shall enable medical care practitioners to diagnose lung disorders in an improved manner, leading to better patient care.

Heart sound classification algorithm based on bispectral feature extraction and convolutional neural networks

Cardiovascular disease (CVD) is one of the leading causes of death worldwide. Heart sound classification plays a key role in the early detection of CVD. The difference between normal and abnormal heart sounds is not obvious. In this paper, in order to improve the accuracy of the heart sound classification model, we propose a heart sound feature extraction method based on bispectral analysis and combine it with convolutional neural network (CNN) to classify heart sounds. The model can effectively suppress Gaussian noise by using bispectral analysis and can effectively extract the features of heart sound signals without relying on the accurate segmentation of heart sound signals. At the same time, the model combines with the strong classification performance of convolutional neural network and finally achieves the accurate classification of heart sound. According to the experimental results, the proposed algorithm achieves 0.910, 0.884 and 0.940 in terms of accuracy, sensitivity and specificity under the same data and experimental conditions, respectively. Compared with other heart sound classification algorithms, the proposed algorithm shows a significant improvement and strong robustness and generalization ability, so it is expected to be applied to the auxiliary detection of congenital heart disease.

Enhancing Situational Awareness with VAS-Compass Net for the Recognition of Directional Vehicle Alert Sounds.

People with hearing impairments often face increased risks related to traffic accidents due to their reduced ability to perceive surrounding sounds. Given the cost and usage limitations of traditional hearing aids and cochlear implants, this study aims to develop a sound alert assistance system (SAAS) to enhance situational awareness and improve travel safety for people with hearing impairments. We proposed the VAS-Compass Net (Vehicle Alert Sound-Compass Net), which integrates three lightweight convolutional neural networks: EfficientNet-lite0, MobileNetV3-Small, and GhostNet. Through employing a fuzzy ranking ensemble technique, our proposed model can identify different categories of vehicle alert sounds and directions of sound sources on an edge computing device. The experimental dataset consisted of images derived from the sounds of approaching police cars, ambulances, fire trucks, and car horns from various directions. The audio signals were converted into spectrogram images and Mel-frequency cepstral coefficient images, and they were fused into a complete image using image stitching techniques. We successfully deployed our proposed model on a Raspberry Pi 5 microcomputer, paired with a customized smartwatch to realize an SAAS. Our experimental results demonstrated that VAS-Compass Net achieved an accuracy of 84.38% based on server-based computing and an accuracy of 83.01% based on edge computing. Our proposed SAAS has the potential to significantly enhance the situational awareness, alertness, and safety of people with hearing impairments on the road.

Improved Patch-Mix Transformer and Contrastive Learning Method for Sound Classification in Noisy Environments

Improved Patch-Mix Transformer and Contrastive Learning Method for Sound Classification in Noisy Environments

Equipment Sounds’ Event Localization and Detection Using Synthetic Multi-Channel Audio Signal to Support Collision Hazard Prevention

Construction workplaces often face unforeseen collision hazards due to a decline in auditory situational awareness among on-foot workers, leading to severe injuries and fatalities. Previous studies that used auditory signals to prevent collision hazards focused on employing a classical beamforming approach to determine equipment sounds’ Direction of Arrival (DOA). No existing frameworks implement a neural network-based approach for both equipment sound classification and localization. This paper presents an innovative framework for sound classification and localization using multichannel sound datasets artificially synthesized in a virtual three-dimensional space. The simulation synthesized 10,000 multi-channel datasets using just fourteen single sound source audiotapes. This training includes a two-staged convolutional recurrent neural network (CRNN), where the first stage learns multi-label sound event classes followed by the second stage to estimate their DOA. The proposed framework achieves a low average DOA error of 30 degrees and a high F-score of 0.98, demonstrating accurate localization and classification of equipment near workers’ positions on the site.

Data Augmentation Method Using Room Transfer Function for Monitoring of Domestic Activities

Monitoring domestic activities helps us to understand user behaviors in indoor environments, which has garnered interest as it aids in understanding human activities in context-aware computing. In the field of acoustics, this goal has been achieved through studies employing machine learning techniques, which are widely used for classification tasks involving sound recognition and other objectives. Machine learning typically achieves better performance with large amounts of high-quality training data. Given the high cost of data collection, development datasets often suffer from imbalanced data or lack high-quality samples, leading to performance degradations in machine learning models. The present study aims to address this data issue through data augmentation techniques. Specifically, since the proposed method targets indoor activities in domestic activity detection, room transfer functions were used for data augmentation. The results show that the proposed method achieves a 0.59% improvement in the F1-Score (micro) from that of the baseline system for the development dataset. Additionally, test data including microphones that were not used during training achieved an F1-Score improvement of 0.78% over that of the baseline system. This demonstrates the enhanced model generalization performance of the proposed method on samples having different room transfer functions to those of the trained dataset.

An Overview of Software Sensor Applications in Biosystem Monitoring and Control

This review highlights the critical role of software sensors in advancing biosystem monitoring and control by addressing the unique challenges biological systems pose. Biosystems—from cellular interactions to ecological dynamics—are characterized by intrinsic nonlinearity, temporal variability, and uncertainty, posing significant challenges for traditional monitoring approaches. A critical challenge highlighted is that what is typically measurable may not align with what needs to be monitored. Software sensors offer a transformative approach by integrating hardware sensor data with advanced computational models, enabling the indirect estimation of hard-to-measure variables, such as stress indicators, health metrics in animals and humans, and key soil properties. This article outlines advancements in sensor technologies and their integration into model-based monitoring and control systems, leveraging the capabilities of Internet of Things (IoT) devices, wearables, remote sensing, and smart sensors. It provides an overview of common methodologies for designing software sensors, focusing on the modelling process. The discussion contrasts hypothetico-deductive (mechanistic) models with inductive (data-driven) models, illustrating the trade-offs between model accuracy and interpretability. Specific case studies are presented, showcasing software sensor applications such as the use of a Kalman filter in greenhouse control, the remote detection of soil organic matter, and sound recognition algorithms for the early detection of respiratory infections in animals. Key challenges in designing software sensors, including the complexity of biological systems, inherent temporal and individual variabilities, and the trade-offs between model simplicity and predictive performance, are also discussed. This review emphasizes the potential of software sensors to enhance decision-making and promote sustainability in agriculture, healthcare, and environmental monitoring.

Audio feature enhancement based on quaternion filtering and deep hashing

This paper aims to solve the problems of difficult convergence of audio model training, large data demand, and large dimensionality of storage space for audio-generated feature vectors. To this end, this paper proposes the use of quaternion Gabor filtering to suppress the background information of the spectrogram and reduce the interference of the data for the case of insufficient data alignment between audio data and image data after shifting the domain. In addition, different scales of window lengths and frame shifts are used to capture the connections between different vocal objects. To address the problem that the generated feature vectors are large dimensional, we use a deep hash module to map high-dimensional features to low-dimensional features and use a probability function to make the learned samples more consistent with the overall distribution. In the experimental evaluation, the proposed method was evaluated on the environmental sound classification dataset and the music genre classification dataset. The proposed method uses only a common backbone network and improves the accuracy of audio recognition.

Explosion Detection Using Smartphones: Ensemble Learning with the Smartphone High-Explosive Audio Recordings Dataset and the ESC-50 Dataset.

Explosion monitoring is performed by infrasound and seismoacoustic sensor networks that are distributed globally, regionally, and locally. However, these networks are unevenly and sparsely distributed, especially at the local scale, as maintaining and deploying networks is costly. With increasing interest in smaller-yield explosions, the need for more dense networks has increased. To address this issue, we propose using smartphone sensors for explosion detection as they are cost-effective and easy to deploy. Although there are studies using smartphone sensors for explosion detection, the field is still in its infancy and new technologies need to be developed. We applied a machine learning model for explosion detection using smartphone microphones. The data used were from the Smartphone High-explosive Audio Recordings Dataset (SHAReD), a collection of 326 waveforms from 70 high-explosive (HE) events recorded on smartphones, and the ESC-50 dataset, a benchmarking dataset commonly used for environmental sound classification. Two machine learning models were trained and combined into an ensemble model for explosion detection. The resulting ensemble model classified audio signals as either "explosion", "ambient", or "other" with true positive rates (recall) greater than 96% for all three categories.

An acoustic nanogenerator based on porous PDMS/ZnO/PVDF-TrFE structure for self-powered intermediated-frequency sound monitoring

Abstract Sound energy is widely present in natural environment, nowadays predominantly collected using voluminous and heavy rigid devices, limiting their portability. Till now, there are few reports about using flexible materials to collect sound energy, and it is usually susceptible to environmental humidity. This study combines flexible polymer polydimethylsiloxane (PDMS), piezoelectric material polyvinylidene difluoride trifluoroethylene (PVDF-TrFE), nano zinc oxide (ZnO), and conductive carbon nanotubes (CNTs) to fabricate a porous nano ZnO/PVDF-TrFE/CNTs PDMS sound-driven nanogenerator (PZPCP SNG). It amalgamates frictional and piezoelectric effects: its porous structure enables efficient collection of vibration and frictional energy generated by sound waves; the piezoelectric effect of PVDF-TrFE and nano ZnO facilitates the conversion of acoustic mechanical energy, collectively enhancing the generator's output. Experimental optimization yields the best production conditions, achieving optimal outputs of 481.1 mV (open-circuit voltage), 209.13 nA (short-circuit current) under 400 Hz/125 dB sound stimuli, with a surface power density of 9.1 µW/m² (volumetric power density of 2.28 mW/m³). And it can convert sound ranging from 63-3000 Hz. With hardware circuitry, up to 5 series-connected light-emitting diodes (LEDs) can be illuminated within the circuit, demonstrating a certain degree of sound recognition capability. The proposed PZPCP SNG design is simple, effective, and lightweight, enabling flexible and stable intermediated-frequency acoustic energy harvesting. Its hydrophobic structural characteristics render it adaptable to various humidity, presenting a new approach toward widespread low-power sound detection and self-powering applications.