Input Spectrogram Research Articles

Research on the classification of complex noise-mixed microseismic events based on machine vision

Event classification is important for accurately monitoring and warning against rockburst hazards using microseismic technology. Here, we propose an automatic classification method for microseismic events based on machine vision. The method uses Histogram of Oriented Gradient (HOG) integrated with Support Vector Machine (SVM) as the core model (HOG-SVM, HSVM) to classify microseismic events. First, the method uses as input spectrograms generated from microseismic event signals recorded in the field. Next, the HOG method is used to accurately extract the spectral feature information of the useful signals of microseismic events under the interference of noisy signal. Finally, the extracted feature data is used to train SVM, after the training is completed, the SVM is used to classify the microseismic events. The performance of the method for categorizing microseismic events was tested using multiple independent test sets built from data monitored in the field of a mine in Shandong Province. The results show that the method can effectively extract the spectral feature information of useful signals of microseismic events contaminated with noise, with good classification accuracy and robustness to noise. It classifies microseismic events with high accuracy and efficiency compared to well-performing classification methods based on seismic source parameters and typical depth models. The method can provide technical support for the effective classification of microseismic events in complex construction sites, especially in noisy deep underground construction environments.

Enhance micro-Doppler signatures-based human activity classification accuracy of FMCW radar using the threshold method

Nowadays, radar-based human activity classification is being widely adopted in healthcare systems due to its benefits in terms of personal privacy compliance, non-contact sensing, and being unaffected by weather conditions. This study proposes a threshold method in the pre-processing stage to improve human activity classification accuracy by determining the region of meaningful information (RMI) on the spectrogram. Initially, a mask function, which is created by a certain threshold value, is applied to the input spectrogram to highlight the RMI from the micro-Doppler (m-D) signatures. Only the highlighted RMI on the spectrogram is retained as input to the classifiers. Then, five Convolutional Neural Networks (CNNs) of varying complexity are employed to extract features, identify activities, and assess the effectiveness of the suggested approach. The experimental results demonstrate that the suggested approach has enhanced classification accuracy by up to 11% when compared to the original unprocessed dataset.

Estimating Classification Accuracy for Unlabeled Datasets Based on Block Scaling

This paper proposes an approach called block scaling quality (BSQ) for estimating the prediction accuracy of a deep network model. The basic operation perturbs the input spectrogram by multiplying all values within a block by , where is equal to 0 in the experiments. The ratio of perturbed spectrograms that have different prediction labels than the original spectrogram to the total number of perturbed spectrograms indicates how much of the spectrogram is crucial for the prediction. Thus, this ratio is inversely correlated with the accuracy of the dataset. The BSQ approach demonstrates satisfactory estimation accuracy in experiments when compared with various other approaches. When using only the Jamendo and FMA datasets, the estimation accuracy experiences an average error of 4.9% and 1.8%, respectively. Moreover, the BSQ approach holds advantages over some of the comparison counterparts. Overall, it presents a promising approach for estimating the accuracy of a deep network model.

Replay attack detection based on deformable convolutional neural network and temporal-frequency attention model

Abstract As an important identity authentication method, speaker verification (SV) has been widely used in many domains, e.g., mobile financials. At the same time, the existing SV systems are insecure under replay spoofing attacks. Toward a more secure and stable SV system, this article proposes a replay attack detection system based on deformable convolutional neural networks (DCNNs) and a time–frequency double-channel attention model. In DCNN, the positions of elements in the convolutional kernel are not fixed. Instead, they are modified by some trainable variable to help the model extract more useful local information from input spectrograms. Meanwhile, a time–frequency domino double-channel attention model is adopted to extract more effective distinctive features to collect valuable information for distinguishing genuine and replay speeches. Experimental results on ASVspoof 2019 dataset show that the proposed model can detect replay attacks accurately.

A Genetic Algorithm Approach to Automate Architecture Design for Acoustic Scene Classification

Convolutional neural networks (CNNs) have been widely used with remarkable success in the acoustic scene classification (ASC) task. However, the performance of these CNNs highly relies on their architectures, requiring a lot of effort and expertise to design CNNs suitable for the investigated problem. In this work, we propose an efficient genetic algorithm (GA) that aims to find optimized CNN architectures for the ASC task. The proposed algorithm uses frequency-dimension splitting of the input spectrograms in order to explore the architecture search space in sub-CNN models in addition to classical single-path CNNs. Specifically, this algorithm aims to find the best number of sub-CNNs in addition to their architectures to better capture the distinct features of the input spectrograms. The proposed GA is specifically designed for sound classification to suit the ASC task than many other GAs that optimize conventional single-path CNN architectures. Experimental results on three benchmark datasets demonstrate the effectiveness of the proposed method. Specifically, the proposed algorithm has achieved around 17.8%, 16%, and 17.2%, relative improvement in accuracy with respect to the baseline systems on the development datasets of DCASE2018-Task1A, DCASE2019-Task1A, and DCASE2020-Task1A, respectively.

Analysis of Deep Learning-Based Decision-Making in an Emotional Spontaneous Speech Task

In this work, we present an approach to understand the computational methods and decision-making involved in the identification of emotions in spontaneous speech. The selected task consists of Spanish TV debates, which entail a high level of complexity as well as additional subjectivity in the human perception-based annotation procedure. A simple convolutional neural model is proposed, and its behaviour is analysed to explain its decision-making. The proposed model slightly outperforms commonly used CNN architectures such as VGG16, while being much lighter. Internal layer-by-layer transformations of the input spectrogram are visualised and analysed. Finally, a class model visualisation is proposed as a simple interpretation approach whose usefulness is assessed in the work.

Hybrid Machine-Learning-Based Spectrum Sensing and Allocation With Adaptive Congestion-Aware Modeling in CR-Assisted IoV Networks

Unlicensed cognitive-radio (CR)-assisted Internet of Vehicles (IoV) users can access licensed providers’ radio spectrum and concurrently utilize the dedicated channel for data transmission in vehicular communication. Optimizing channel access in cognitive IoV networks can help maximize available spectrum resources. This article proposes a novel sensing and communication integrated framework, dubbed as the CR-assisted IoV network (CRAV-Net), using a cluster-based hybrid optimization approach with adaptive congestion-aware modeling for dynamic high-mobility vehicular networks in an urban city context. In CRAV-Net, intelligent hybrid learning spectrum agents are introduced, which perform spectrum sensing (SS) using a deep learning (DL) model. It dynamically learns the multilevel spatial and temporal graphical features from input spectrograms through layer-by-layer propagation. It efficiently predicts the spectrum occupancy in the primary spectrum, without a priori knowledge of the radio environment. Then, to assign the vacant channels to the secondary vehicles, a support vector machine classifier is trained based on several learning features, including the vehicle stay time, vehicle density, and network capacity, to select the optimal resource route. The proposed framework achieves an overall accuracy of 99.74% in SS using the custom data set, outperforming state of the art by 12.60% at −25-dB signal-to-noise ratio. In addition, it brings a performance gain of 0.81% in SS accuracy when evaluated on real-world signals. Furthermore, in optimal network node allocation, the proposed framework achieves a mean accuracy of 98.45%, outperforming the existing methods by 0.63% and 18.32% in terms of accuracy and allocation time, respectively.

Classification of Tetanus Severity in Intensive-Care Settings for Low-Income Countries Using Wearable Sensing.

Infectious diseases remain a common problem in low- and middle-income countries, including in Vietnam. Tetanus is a severe infectious disease characterized by muscle spasms and complicated by autonomic nervous system dysfunction in severe cases. Patients require careful monitoring using electrocardiograms (ECGs) to detect deterioration and the onset of autonomic nervous system dysfunction as early as possible. Machine learning analysis of ECG has been shown of extra value in predicting tetanus severity, however any additional ECG signal analysis places a high demand on time-limited hospital staff and requires specialist equipment. Therefore, we present a novel approach to tetanus monitoring from low-cost wearable sensors combined with a deep-learning-based automatic severity detection. This approach can automatically triage tetanus patients and reduce the burden on hospital staff. In this study, we propose a two-dimensional (2D) convolutional neural network with a channel-wise attention mechanism for the binary classification of ECG signals. According to the Ablett classification of tetanus severity, we define grades 1 and 2 as mild tetanus and grades 3 and 4 as severe tetanus. The one-dimensional ECG time series signals are transformed into 2D spectrograms. The 2D attention-based network is designed to extract the features from the input spectrograms. Experiments demonstrate a promising performance for the proposed method in tetanus classification with an F1 score of 0.79 ± 0.03, precision of 0.78 ± 0.08, recall of 0.82 ± 0.05, specificity of 0.85 ± 0.08, accuracy of 0.84 ± 0.04 and AUC of 0.84 ± 0.03.

Visual inspection of fault type and zone prediction in electrical grids using interpretable spectrogram-based CNN modeling

In electrical grids, fault diagnosis (fault type and fault location classifications) are critical due to their economic and important implications. Numerous smart grid applications have embraced data-driven methodologies. While the majority of the work in this topic has been on increasing the predicted accuracy of machine-learning model for fault diagnosis, one important aspect that has received less attention is the interpretability of these systems.We advocate for a complementary perspective. To represent faulty signals, we propose a spectrogram–convolutional neural network based representation of the electrical signals where pre-trained models such as GoogleNet and SqueezeNet are trivially used. We then perform multiple fault classification tasks and offer a visual interpretation of the collected findings. The suggested approach makes the model more transparent through the use of Gradient-weighted Class Activation Mapping (Grad-CAM), which visualizes regions in the input spectrogram that are more relevant for predictions, assisting the end-user in the understanding and interpreting the results. We explore the merits of the suggested technique in terms of increasing the transparency of the black-box machine learning system, which is a critical requirement for designing modernized smart grids.

Continuous Human Activity Recognition With Distributed Radar Sensor Networks and CNN–RNN Architectures

Unconstrained human activities recognition with a radar network is considered. A hybrid classifier combining both CNNs and RNNs for spatial-temporal pattern extraction is proposed. The two-dimensional CNNs (2D-CNNs) are first applied to the radar data to perform spatial feature extraction on the input spectrograms. Subsequently, gated recurrent units with bidirectional implementations are used to capture the long- and short-term temporal dependencies in the feature maps generated by the 2D-CNNs. Three NN-based data fusion methods were explored and compared to utilize the rich information provided by the different radar nodes. The performance of the proposed classifier was validated rigorously using the K-fold CV and L1PO method. Unlike competitive research, the dataset with continuous human activities with seamless inter-activity transitions that can occur at any time and unconstrained moving trajectories of the participants has been collected and used for evaluation purposes. Classification accuracy of about 90.8% is achieved for nine-class HAR by the proposed classifier with the halfway fusion method.

BirdNET: A deep learning solution for avian diversity monitoring

Variation in avian diversity in space and time is commonly used as a metric to assess environmental changes. Conventionally, such data were collected by expert observers, but passively collected acoustic data is rapidly emerging as an alternative survey technique. However, efficiently extracting accurate species richness data from large audio datasets has proven challenging. Recent advances in deep artificial neural networks (DNNs) have transformed the field of machine learning, frequently outperforming traditional signal processing techniques in the domain of acoustic event detection and classification. We developed a DNN, called BirdNET, capable of identifying 984 North American and European bird species by sound. Our task-specific model architecture was derived from the family of residual networks (ResNets), consisted of 157 layers with more than 27 million parameters, and was trained using extensive data pre-processing, augmentation, and mixup. We tested the model against three independent datasets: (a) 22,960 single-species recordings; (b) 286 h of fully annotated soundscape data collected by an array of autonomous recording units in a design analogous to what researchers might use to measure avian diversity in a field setting; and (c) 33,670 h of soundscape data from a single high-quality omnidirectional microphone deployed near four eBird hotspots frequented by expert birders. We found that domain-specific data augmentation is key to build models that are robust against high ambient noise levels and can cope with overlapping vocalizations. Task-specific model designs and training regimes for audio event recognition perform on-par with very complex architectures used in other domains (e.g., object detection in images). We also found that high temporal resolution of input spectrograms (short FFT window length) improves the classification performance for bird sounds. In summary, BirdNET achieved a mean average precision of 0.791 for single-species recordings, a F0.5 score of 0.414 for annotated soundscapes, and an average correlation of 0.251 with hotspot observation across 121 species and 4 years of audio data. By enabling the efficient extraction of the vocalizations of many hundreds of bird species from potentially vast amounts of audio data, BirdNET and similar tools have the potential to add tremendous value to existing and future passively collected audio datasets and may transform the field of avian ecology and conservation.

A convolutional recurrent neural network with attention framework for speech separation in monaural recordings

Most speech separation studies in monaural channel use only a single type of network, and the separation effect is typically not satisfactory, posing difficulties for high quality speech separation. In this study, we propose a convolutional recurrent neural network with an attention (CRNN-A) framework for speech separation, fusing advantages of two networks together. The proposed separation framework uses a convolutional neural network (CNN) as the front-end of a recurrent neural network (RNN), alleviating the problem that a sole RNN cannot effectively learn the necessary features. This framework makes use of the translation invariance provided by CNN to extract information without modifying the original signals. Within the supplemented CNN, two different convolution kernels are designed to capture information in both the time and frequency domains of the input spectrogram. After concatenating the time-domain and the frequency-domain feature maps, the feature information of speech is exploited through consecutive convolutional layers. Finally, the feature map learned from the front-end CNN is combined with the original spectrogram and is sent to the back-end RNN. Further, the attention mechanism is further incorporated, focusing on the relationship among different feature maps. The effectiveness of the proposed method is evaluated on the standard dataset MIR-1K and the results prove that the proposed method outperforms the baseline RNN and other popular speech separation methods, in terms of GNSDR (gloabl normalised source-to-distortion ratio), GSIR (global source-to-interferences ratio), and GSAR (gloabl source-to-artifacts ratio). In summary, the proposed CRNN-A framework can effectively combine the advantages of CNN and RNN, and further optimise the separation performance via the attention mechanism. The proposed framework can shed a new light on speech separation, speech enhancement, and other related fields.

A Hybrid Technique using CNN LSTM for Speech Emotion Recognition

Automatic speech emotion recognition is a very necessary activity for effective human-computer interaction. This paper is motivated by using spectrograms as inputs to the hybrid deep convolutional LSTM for speech emotion recognition. In this study, we trained our proposed model using four convolutional layers for high-level feature extraction from input spectrograms, LSTM layer for accumulating long-term dependencies and finally two dense layers. Experimental results on the SAVEE database shows promising performance. Our proposed model is highly capable as it obtained an accuracy of 94.26%.

Classification of Arrhythmia by Using Deep Learning with 2-D ECG Spectral Image Representation

The electrocardiogram (ECG) is one of the most extensively employed signals used in the diagnosis and prediction of cardiovascular diseases (CVDs). The ECG signals can capture the heart’s rhythmic irregularities, commonly known as arrhythmias. A careful study of ECG signals is crucial for precise diagnoses of patients’ acute and chronic heart conditions. In this study, we propose a two-dimensional (2-D) convolutional neural network (CNN) model for the classification of ECG signals into eight classes; namely, normal beat, premature ventricular contraction beat, paced beat, right bundle branch block beat, left bundle branch block beat, atrial premature contraction beat, ventricular flutter wave beat, and ventricular escape beat. The one-dimensional ECG time series signals are transformed into 2-D spectrograms through short-time Fourier transform. The 2-D CNN model consisting of four convolutional layers and four pooling layers is designed for extracting robust features from the input spectrograms. Our proposed methodology is evaluated on a publicly available MIT-BIH arrhythmia dataset. We achieved a state-of-the-art average classification accuracy of 99.11%, which is better than those of recently reported results in classifying similar types of arrhythmias. The performance is significant in other indices as well, including sensitivity and specificity, which indicates the success of the proposed method.

Online Spectrogram Inversion for Low-Latency Audio Source Separation

Audio source separation is usually achieved by estimating the short-time Fourier transform (STFT) magnitude of each source, and then applying a spectrogram inversion algorithm to retrieve time-domain signals. In particular, the multiple input spectrogram inversion (MISI) algorithm has been exploited successfully in several recent works. However, this algorithm suffers from two drawbacks, which we address in this paper. First, it has originally been introduced in a heuristic fashion: we propose here a rigorous optimization framework in which MISI is derived, thus proving the convergence of this algorithm. Besides, while MISI operates offline, we propose here an online version of MISI called oMISI, which is suitable for low-latency source separation, an important requirement for e.g., hearing aids applications. oMISI also allows one to use alternative phase initialization schemes exploiting the temporal structure of audio signals. Experiments conducted on a speech separation task show that oMISI performs as well as its offline counterpart, thus demonstrating its potential for real-time source separation.

InverSynth: Deep Estimation of Synthesizer Parameter Configurations From Audio Signals

Sound synthesis is a complex field that requires domain expertise. Manual tuning of synthesizer parameters to match a specific sound can be an exhaustive task, even for experienced sound engineers. In this paper, we introduce InverSynth - an automatic method for synthesizer parameters tuning to match a given input sound. InverSynth is based on strided convolutional neural networks and is capable of inferring the synthesizer parameters configuration from the input spectrogram and even from the raw audio. The effectiveness InverSynth is demonstrated on a subtractive synthesizer with four frequency modulated oscillators, envelope generator and a gater effect. We present extensive quantitative and qualitative results that showcase the superiority InverSynth over several baselines. Furthermore, we show that the network depth is an important factor that contributes to the prediction accuracy.

Blind source separation with optimal transport non-negative matrix factorization

Optimal transport as a loss for machine learning optimization problems has recently gained a lot of attention. Building upon recent advances in computational optimal transport, we develop an optimal transport non-negative matrix factorization (NMF) algorithm for supervised speech blind source separation (BSS). Optimal transport allows us to design and leverage a cost between short-time Fourier transform (STFT) spectrogram frequencies, which takes into account how humans perceive sound. We give empirical evidence that using our proposed optimal transport, NMF leads to perceptually better results than NMF with other losses, for both isolated voice reconstruction and speech denoising using BSS. Finally, we demonstrate how to use optimal transport for cross-domain sound processing tasks, where frequencies represented in the input spectrograms may be different from one spectrogram to another.

A Stereo Music Preprocessing Scheme for Cochlear Implant Users

Listening to music is still one of the more challenging aspects of using a cochlear implant (CI) for most users. Simple musical structures, a clear rhythm/beat, and lyrics that are easy to follow are among the top factors contributing to music appreciation for CI users. Modifying the audio mix of complex music potentially improves music enjoyment in CI users. A stereo music preprocessing scheme is described in which vocals, drums, and bass are emphasized based on the representation of the harmonic and the percussive components in the input spectrogram, combined with the spatial allocation of instruments in typical stereo recordings. The scheme is assessed with postlingually deafened CI subjects (N = 7) using pop/rock music excerpts with different complexity levels. The scheme is capable of modifying relative instrument level settings, with the aim of improving music appreciation in CI users, and allows individual preference adjustments. The assessment with CI subjects confirms the preference for more emphasis on vocals, drums, and bass as offered by the preprocessing scheme, especially for songs with higher complexity. The stereo music preprocessing scheme has the potential to improve music enjoyment in CI users by modifying the audio mix in widespread (stereo) music recordings. Since music enjoyment in CI users is generally poor, this scheme can assist the music listening experience of CI users as a training or rehabilitation tool.

Informed Source Separation Using Iterative Reconstruction

This paper presents a technique for Informed Source Separation (ISS) of a single channel mixture, based on the Multiple Input Spectrogram Inversion (MISI) phase estimation method. The reconstruction of the source signals is iterative, alternating between a time-frequency consistency enforcement and a re-mixing constraint. A dual resolution technique is also proposed, for sharper transients reconstruction. The two algorithms are compared to a state-of-the-art Wiener-based ISS technique, on a database of fourteen monophonic mixtures, with standard source separation objective measures. Experimental results show that the proposed algorithms outperform both this reference technique and the oracle Wiener filter by up to 3 dB in distortion, at the cost of a significantly heavier computation.

A system for tracking frequency-modulated bowhead whale calls on vector sensors in the presence of seismic exploration activity.

A group of software tools has been developed for automated detection and localization of bowhead whale sounds with arbitrary frequency-modulated tones. The software is a mixture of JAVA, UNIX C-shell scripts, and MATLAB components, and consists of six sequential components: an incoherent spectral band event detector that yields time and bearings of transient events; an interval estimator to remove weak airgun signals; a feature extractor that processes an input spectrogram to yield a 23-element feature vector; two feed-forward neural network classifiers that sequentially winnow non-biologic signals and pinnipeds; a linking procedure to combine signals detected on sensors several kilometers apart; and a localization method that uses a maximum-likelihood procedure for estimating a position from a series of bearings. The software has been applied to data collected between 2007 and 2009 from at least 35 vector sensor autonomous recorders deployed over a 280-km swath in the Beaufort Sea. The largest dataset (2008) logged 1.35 × 106 calls at 386 000 locations over 54 days across 40 recorders and took 24 days to compute using a single 2.66-GHz processor. Seven recorders consumed over 50% of the processing time. [Work supported by Shell Exploration and Production Company.]