Time-frequency Map Research Articles

Research on Classification and Recognition of Industrial Stainless Steel Welded Pipe Defects Based on Convolution Neural Network

For the classification and identification of industrial stainless steel welded pipe defects, a combined method of STFT and CNN based on vortex test is proposed. First, the collected original eddy current signal is STFT transformed to obtain a two-dimensional time-frequency map. Then, the two-dimensional time-frequency map was input into the two neural networks VGG-16 and GoogLeNet for model training, selecting a more accurate network model under the condition of the same learning rate. The trained network model is then classified using different learning rates. The results show that with the learning rate of 0.0001, the VGG-16 training model is better than the GoogLeNet training model, which has a certain reference significance for the classification and identification of defects in industrial stainless steel welded pipes.

TERFDA: Tensor Embedding RF Domain Adaptation for varying noise interference

Radio frequency machine learning (RFML) can be loosely termed as a field that machine learning (ML) and deep learning (DL) techniques to applications related to wireless communications. However, traditional RFML basically assume that the data of training set and test set are independent and identically distributed and only a large number of labeled data can train a classification model which can effectively classify test set data. In other words, without enough training samples, it is impossible to learn an automatic modulation classifier that performs well in varying noise interference environment. Feature-based transfer learning minimizes the distribution difference between historical modulated signal data and new data by learning similarity-maximizing feature spaces. Therefore, in this paper, Dynamic Distribution Adaptation (DDA) is adopted to address the above challenges. We propose a Tensor Embedding RF Domain Adaptation (TERFDA) approach, which learns the latent subspace of the tensors formed by the time–frequency maps of the signals, so that use the multi-dimensional domain information of the signals to jointly learn the shared feature subspace of the source domain and the target domain, then perform DDA in the shared subspace. The experimental results show that under the modulated signal data, compared with the state-of-the-art DA algorithm, TERFDA has less requirements on the number of samples and categories, and has superior performance for confrontation the varying noise interference between source domain and target domain.

ResNet-integrated very early bolt looseness monitoring based on intrinsic feature extraction of percussion sounds

Very early bolt looseness monitoring has been a challenge in the field of structural health monitoring. The authors have conducted a further study of the previous researches, with the aim of detecting very early bolt looseness conditions. The intrinsic features of vibro-acoustic signals contain the underlying dynamic characteristics denoting full range bolt looseness conditions. Correspondingly, this paper proposes a novel ResNet-integrated very early bolt looseness monitoring approach based on intrinsic feature extraction of percussion sounds. The intrinsic features of percussion-caused sound signals were extracted by variational mode decomposition (VMD), where the parameters of VMD were determined by grey wolf optimization algorithm. The optimal band-limited intrinsic mode functions were converted into two-dimensional time–frequency maps by continuous wavelet transform. The (red green blue) RGB images were adopted as the input of residual network (ResNet) to monitor very early bolt looseness conditions. The results and analysis illustrate the validity and superiority of the novel ResNet-integrated very early bolt looseness monitoring approach. The proposed approach in our researches provides a novel solution for very early bolt looseness monitoring in the field of structural health monitoring. The strategy can also be extended to other nonlinear signal processing-involved fields.

An Intelligent Method Based on Time–Frequency Analysis and Deep Learning Semantic Segmentation for Investigating the Electromagnetic Pulse Features of Engine Digital Controllers

A novel intelligent extraction method based on time–frequency analysis and deep learning semantic segmentation were proposed for detailing electromagnetic pulse (EMP) sensitive frequency bands of engine digital controllers. In this method, wavelet transformation was performed to convert EMP-coupled signals into image data sets. Two semantic segmentation networks, namely fully convolutional networks and DeepLabV3+, were used to train and verify the obtained image data sets. The performance of the two models was compared using the confusion matrix and Kappa coefficient. Threshold segmentation, binarization, and average pooling operations were sequentially performed on the feature map output of the networks, and EMP characteristic pixel percentage was calculated using the time–frequency map. The original image was segmented by using the result of the binarization of the feature map output by the last layer of the network as a mask. This technique is immune to the interference caused by noise pixels when calculating the spectrum index. The proposed method has improved the EMP feature extraction accuracy of the model, and the overall performance of the DeepLabV3+ network model is proper. It was performed to obtain a theoretical basis for subsequent targeted EMP protection.

Time-frequency fusion features-based incremental network for smartphone measured structural seismic response classification

Smartphone as a portable device can be used for public participatory structural seismic response acquisition. This data acquisition approach not only reduces the acquisition cost but also facilitates the possibility to obtain city-scale building seismic responses. In this paper, we propose a novel structural seismic response classification method using time–frequency fusion features-based incremental network, namely SeismicNet, that can automatically classify the signals acquired through smartphones into structural response under normal operation conditions (structural normal response) or structural response under strong ground motion (structural seismic response). In this method, the time-domain signals of structural response are converted into time–frequency maps, which will be fed into a lightweight convolutional network for the extraction of the initial fusion features. Subsequently, these features are mapped by random weights to dynamic nodes. This not only improves the capability of feature expression, but also allows incremental updating of the model. Finally, a simulated structural response dataset and some real structural seismic responses are used to verify the accuracy and practicability of the proposed method. Furthermore, the results showed that the proposed method is more than 84 times in training speed and 2 times in inference speed than some well-known deep learning networks (VGG16 and ResNet50).

A CNN-Transformer Hybrid Recognition Approach for sEMG-Based Dynamic Gesture Prediction

As a unique physiological electrical signal in the human body, surface electromyography (sEMG) signals always include human movement intention and muscle state. Through the collection of sEMG signals, different gestures can be effectively recognized. At present, the convolutional neural network (CNN) has been widely applied to different gesture recognition systems. However, due to its inherent limitations in global context feature extraction, it exists a certain shortcoming on high-precision prediction tasks. To solve this issue, a CNN-transformer hybrid recognition approach is proposed for high-precision dynamic gesture prediction. In addition, the continuous wavelet transform (CWT) is proposed for to acquire the time-frequency maps. To realize effective feature representation of local features from the time-frequency maps, an attention fusion block (AFB) is proposed to build the deep CNN network branch to effectively extract key channel information and spatial information from local features. Faced with the inherent limitations in global context feature extraction of CNNs, a transformer network branch is proposed to model the global relationship between pixels, called convolution and transformer (CAT) network branch. In addition, a multi-scale feature attention block (MFA) is proposed for effective feature aggregation of local features and global contexts by learning adaptive multi-scale features and suppressing irrelevant scale information. The experimental results on the established multi-channel sEMG signal time-frequency map dataset show that the proposed CNN transformer hybrid recognition network has competitive recognition performance compared with other state-of-the-art recognition networks, and the average recognition speed of each spectrogram on the test set is only 14.7ms. The proposed network can effectively improve network performance and identification efficiency.

A Cross-Scale Feature Aggregation Network Based on Channel–Spatial Attention for Human and Animal Identification of Life Detection Radar

This letter mainly considers environmental clutter problem in distinguishing between stationary humans and animals through-wall circumstances. Focusing on the challenges of object identification in the time-frequency map, we propose a cross-scale feature aggregation (CSFA) network based on channel-spatial attention, which can improve the identification accuracy of stationary human and animal. Specifically, life detection radar is utilized to collect data and time-frequency analysis method synchrosqueezing transform (SST) is used to suppress the signal noise and generate higher resolution time-frequency maps. In order to make full use of the target information, we use a feature pyramid network (FPN) to obtain multilevel feature information maps from time-frequency maps. After that, the CSFA module is utilized to extract detailed micro-Doppler feature information from feature maps. And we use deep Convolutional Neural Network (CNN) to classify humans from animals. Experimental results show that the proposed model has better performance in accuracy compared with the existing methods.

Remaining Useful Life Estimation of Bearing Using Deep Multiscale Window-Based Transformer

Accurate and reliable remaining useful life (RUL) estimation for bearings relies on the extraction of desired degradation features from vibration signals. Now, various feature extractors based on convolutional neural networks have been proposed for RUL prediction and achieved great success. However, these approaches focus on extracting degradation features from data in a single time frame but fail to capture the degradation information across multiple time frames, leading to degenerate estimation performance. In this study, we propose a novel multi-scale feature extractor called deep multi-scale window-based transformer (DMW-Trans) to overcome these limitations. To be more specific, a bunch of feature maps (FMs) are firstly generated with time-frequency maps (TFMs) at multiple time frames. Then, sequential layers equipped with window-based transformer blocks, i.e., WT-Blocks, and patch-merge modules are designed, where the former is used to extract features with multi-head self-attention windows while the latter is employed to fuse features from neighboring areas on TFMs to enlarge the attention fields from layer to layer. Through this hierarchical feature extraction process, multi-scale degradation related features can be effectively extracted. Lastly, multi-layer perceptron is applied to fuse the multi-scale features to estimate RUL. Due to the excellent multi-scale feature extraction capability, the proposed DMW-Trans can provide accurate and reliable RUL prediction results. Extensive experiments on two public run-to-failure bearing datasets demonstrate the superior performance of our proposed method compared to some state-of-the-art approaches.

An Underwater Acoustic Target Recognition Method Based on AMNet

Underwater acoustic target recognition is an important supporting technology for underwater information acquisition and countermeasure. Usually, ship radiated noise is covered by the underwater acoustic background and previous deep learning methods for this task rely on clear and effective acoustic features. We propose a novel network called AMNet to alleviate the problem in this paper. It consists of a multi-branch backbone network coupled with a convolutional attention network. The proposed network is able to obtain the internal features of radiated noise from the time-frequency map of the original data. The convolutional attention network adaptively selects the effective features by weighting them against the global information of the time-frequency map to assist the multi-branch backbone network in classification recognition. Experimental results demonstrate that our model achieves an overall accuracy of 99.4% (2.4% improvement) on the ShipsEar database.

An End-to-End Approach for Rigid-Body Target Micro-Doppler Analysis Based on the Asymmetrical Autoencoding Network

Micro-Doppler analysis of rigid-body target is significant for attitude estimation and recognition of space objects. The traditional micro-Doppler analysis method for rigid-body targets includes two sequential steps. First, the time-frequency analysis is performed on the radar echo data of the target, and then the micro-Doppler curve of each scattering center is separated and extracted from the time-frequency map. The second step depends on the micro motion model of the target. In the micro-Doppler analysis of real rigid-body target, there are some problems such as the mismatch of the micro motion model, the incidence angle dependence of the scattering center position, and the partial occlusion of the scattering center. Therefore, it is very difficult to correctly extract the micro-Doppler curves of multiple scatterers. In this paper, an end-to-end micro-Doppler analysis method for rigid-body target based on deep learning network is proposed, which can directly separate and extract the micro-Doppler curves of multiple scatterers from the target echo data. Specifically, an Asymmetrical AutoEncoding (A2E) network equipped with a Data Pre-Processing (DP2) module is developed to extract Time-Frequency Curves (TFCs) from radar echos. Considering the sparseness of Time-Frequency Distribution (TFDs), we then develop a novel Energy-Concentration Objective (ECO) function based on Min-Max game to enhance curves energy while suppress the background energy. In practice, measured TFDs are rarely annotated, it restricts the generalization capability of the A2E from the simulation to the measurement. To bridge the gap, two-fold modifications are finally constructed: i) we insert a partial-shared branch of decoder to reconstruct the TFD from the DP2 module; ii) we regularize the ECO function with a knowledge preservation based reconstruction bound to further capture the characteristics of measured echos in a semi-supervised way at test time to relieve the domain-shift problem.

DRFM-Based Jamming Signal Recognition Method Guided by Target Detection

Digital radio frequency memory (DRFM) based jammer equipped on target can generate multiple false targets by intercepting and forwarding the signal transmitted by radar, which lead to a serious threat to the performance of a radar system. Therefore, the recognition of radar jamming signal type is a key step of radar countermeasure. The existing jamming recognition methods do not take into account the factor of real-time performance. Since it uses the raw time-domain signals and complex network structures, which incurs dense computational complexity. To tackle this problem, this paper analyzes the time-frequency distribution maps of jamming signals after signal processing, and proposes a radar jamming recognition method based on the guidance of target detection. In the proposed method, firstly, based on the target detection information, the time-frequency image of the jamming region is obtained by performing short-time Fourier transformation (STFT); Secondly, a lightweight convolutional neural network (CNN) model was constructed to achieve jamming recognition. Finally, effectiveness of this method was evaluated using the simulated datasets with six typical radar jamming signals. The results show that the proposed method performed well in accuracy and efficiency, which is practical for real-time anti-jamming.

EEG-Channel-Temporal-Spectral-Attention Correlation for Motor Imagery EEG Classification.

In brain-computer interface (BCI) work, how correctly identifying various features and their corresponding actions from complex Electroencephalography (EEG) signals is a challenging technology. However, most current methods do not consider EEG feature information in spatial, temporal and spectral domains, and the structure of these models cannot effectively extract discriminative features, resulting in limited classification performance. To address this issue, we propose a novel text motor-imagery EEG discrimination method, namely wavelet-based temporal-spectral-attention correlation coefficient (WTS-CC), to simultaneously consider the features and their weighting in spatial, EEG-channel, temporal and spectral domains in this study. The initial Temporal Feature Extraction (iTFE) module extracts the initial important temporal features of MI EEG signals. The Deep EEG-Channel-attention (DEC) module is then proposed to automatically adjust the weight of each EEG channel according to its importance, thereby effectively enhancing more important EEG channels and suppressing less important EEG channels. Next, the Wavelet-based Temporal-Spectral-attention (WTS) module is proposed to obtain more significant discriminative features between different MI tasks by weighting features on two-dimensional time-frequency maps. Finally, a simple discrimination module is used for MI EEG discrimination. The experimental results indicate that the proposed text WTS-CC method can achieve promising discrimination performance that outperforms the state-of-the-art methods in terms of classification accuracy, Kappa coefficient, F1 score, and AUC on three public datasets.

Characterizing physiological high-frequency oscillations using deep learning

Objective. Intracranially-recorded interictal high-frequency oscillations (HFOs) have been proposed as a promising spatial biomarker of the epileptogenic zone. However, HFOs can also be recorded in the healthy brain regions, which complicates the interpretation of HFOs. The present study aimed to characterize salient features of physiological HFOs using deep learning (DL). Approach. We studied children with neocortical epilepsy who underwent intracranial strip/grid evaluation. Time-series EEG data were transformed into DL training inputs. The eloquent cortex (EC) was defined by functional cortical mapping and used as a DL label. Morphological characteristics of HFOs obtained from EC (ecHFOs) were distilled and interpreted through a novel weakly supervised DL model. Main results. A total of 63 379 interictal intracranially-recorded HFOs from 18 children were analyzed. The ecHFOs had lower amplitude throughout the 80–500 Hz frequency band around the HFO onset and also had a lower signal amplitude in the low frequency band throughout a one-second time window than non-ecHFOs, resembling a bell-shaped template in the time–frequency map. A minority of ecHFOs were HFOs with spikes (22.9%). Such morphological characteristics were confirmed to influence DL model prediction via perturbation analyses. Using the resection ratio (removed HFOs/detected HFOs) of non-ecHFOs, the prediction of postoperative seizure outcomes improved compared to using uncorrected HFOs (area under the ROC curve of 0.82, increased from 0.76). Significance. We characterized salient features of physiological HFOs using a DL algorithm. Our results suggested that this DL-based HFO classification, once trained, might help separate physiological from pathological HFOs, and efficiently guide surgical resection using HFOs.

Research on Hydraulic Power System Operation Status Diagnosis Technology Based on Hybrid CNN Model

Aiming at the problems that the features extracted from the traditional system operation state are not adaptive and the specific system operation state is difficult to match, a gearbox system operation state diagnosis method based on continuous wavelet transform (CWT) and two-dimensional convolutional neural network (CNN) is proposed. The method uses the continuous wavelet transform to construct the time-frequency map of the hydrodynamic system operating state signal, and uses it as the input to construct a convolutional neural network model, and forms a deep distributed system operating state feature expression through a multilayer convolutional pool. The structural parameters of each layer of the network are adjusted by the back propagation algorithm to establish an accurate mapping from the signal characteristics to the system operating state. In the experiments under different working conditions and different system operation states, the accuracy of system operation state recognition reaches 99.2%, which verifies the effectiveness of the method. Using this method of adaptively learning rich information in the signal can provide a basis for intelligent system operation state diagnosis.

The “SPectrogram Analysis and Cataloguing Environment” (SPACE) labelling tool

The SPectrogram Analysis and Cataloguing Environment (SPACE) tool is an interactive python tool designed to label radio emission features of interest in a time-frequency map (called “dynamic spectrum”). The program uses Matplotlib’s Polygon Selector widget to allow a user to select and edit an undefined number of vertices on top of the dynamic spectrum before closing the shape (polygon). Multiple polygons may be drawn on any spectrum, and the feature name along with the coordinates for each polygon vertex are saved into a “.json” file as per the “Time-Frequency Catalogue” (TFCat) format along with other data such as the feature id, observer name, and data units. This paper describes the first official stable release (version 2.0) of the tool.

Unsupervised cross-domain rolling bearing fault diagnosis based on time-frequency information fusion

In recent years, data-driven methods have been widely used in rolling bearing fault diagnosis with great success, which mainly relies on the same data distribution and massive labeled data. However, bearing equipment is in normal working state for most of the time and operates under variable operating conditions. This makes it difficult to obtain bearing data labels, and the distribution of the collected samples varies widely. To address these problems, an unsupervised cross-domain fault diagnosis method based on time-frequency information fusion is proposed in this paper. Firstly, wavelet packet decomposition and reconstruction are carried out on the bearing vibration signal, and the energy eigenvectors of each sub-band are extracted to obtain a 2-D time-frequency map of fault features. Secondly, an unsupervised cross-domain fault diagnosis model is constructed, the improved maximum mean discrepancy algorithm is used as the measurement standard, and the joint distribution distance is calculated with the help of pseudo-labels to reduce data distribution differences. Finally, the model is applied to the motor bearing for comparison and verification. The results demonstrate its high diagnosis accuracy and strong robustness.

Motor Imaging EEG Signal Recognition of ResNet18 Network Based on Deformable Convolution

EEG signals with a weak amplitude, complex background noise, randomness, significant individual differences, and small data volume lead to insufficient feature extraction and low classification accuracy. Spurred by these concerns, this paper proposes a motor imaging EEG signal classification method based on fusing the improved ResNet18 network with the deformable convolutional network (DCN). Specifically, the original signal’s local airspace characteristics are enhanced by the common spatial patterns (CSP), and the time-frequency domain characteristics are displayed using the short-time Fourier transform (STFT). Then, the signal is converted into a time-frequency map, where a deformable convolution is applied to capture the contour characteristics of the time-frequency map. This strategy solves the problems of traditional convolution related to hard rules, i.e., the convolutional kernel shape can only be a square or rectangular core and cannot be dynamically changed according to the recognition target, resulting in a low recognition rate, prohibiting the network from extracting hidden features and affording enhanced identification and classification. Experimental results demonstrate that our method attains an average classification accuracy on a two-classification and two four-classification motor imaging EEG signals of 90.30%, 86.50%, and 88.08%, respectively, which is much higher than current work, proving our method’s effectiveness.

EEG-Based Emotion Recognition Using Convolutional Recurrent Neural Network with Multi-Head Self-Attention

In recent years, deep learning has been widely used in emotion recognition, but the models and algorithms in practical applications still have much room for improvement. With the development of graph convolutional neural networks, new ideas for emotional recognition based on EEG have arisen. In this paper, we propose a novel deep learning model-based emotion recognition method. First, the EEG signal is spatially filtered by using the common spatial pattern (CSP), and the filtered signal is converted into a time–frequency map by continuous wavelet transform (CWT). This is used as the input data of the network; then the feature extraction and classification are performed by the deep learning model. We called this model CNN-BiLSTM-MHSA, which consists of a convolutional neural network (CNN), bi-directional long and short-term memory network (BiLSTM), and multi-head self-attention (MHSA). This network is capable of learning the time series and spatial information of EEG emotion signals in depth, smoothing EEG signals and extracting deep features with CNN, learning emotion information of future and past time series with BiLSTM, and improving recognition accuracy with MHSA by reassigning weights to emotion features. Finally, we conducted experiments on the DEAP dataset for sentiment classification, and the experimental results showed that the method has better results than the existing classification. The accuracy of high and low valence, arousal, dominance, and liking state recognition is 98.10%, and the accuracy of four classifications of high and low valence-arousal recognition is 89.33%.

Fault diagnosis of planetary gears based on intrinsic feature extraction and deep transfer learning

The planetary gearbox is a key transmission apparatus used to change speed and torque. The planetary gear is one of the most failure-prone components in a planetary gearbox. Due to the complexity of working environments, collected vibration signals contain a lot of noise and interference; fault characteristic frequencies are usually submerged or even lost. Thus, feature extraction from the vibration signal is beneficial to subsequent fault diagnosis. As a fault identification approach that has been increasingly popular in the field of fault diagnosis, deep learning requires a large number of samples to train the model. Insufficient samples lead to low diagnostic accuracy for deep learning models. This paper proposes a novel fault diagnosis approach for planetary gears based on intrinsic feature extraction and deep transfer learning. The original vibration signals are decomposed into a series of band-limited intrinsic mode functions (BLIMFs) by variational mode decomposition. BLIMF with the most apparent fault characteristics is selected to generate two-dimensional time-frequency maps by continuous wavelet transform. The preprocessed time-frequency maps are adopted as the input of the pretrained VGG16 model. The bottom layers are frozen, and the top layers are fine-tuned to achieve fault diagnosis for planetary gears. Applications to planetary gear datasets verify the superiority of the proposed method.

Diesel Engine Fault Diagnosis Method Based on Optimized VMD and Improved CNN

The safe operation of diesel engines performs a vital function in industrial production and life. Because diesel engines often work in harsh environmental conditions, they are prone to failure. Therefore, this paper proposes a fault analysis method based on a combination of optimized variational mode decomposition (VMD) and improved convolutional neural networks (CNN) to address the necessary need for preventive maintenance of diesel engines. The authentic vibration sign is first decomposed by using the (VMD) algorithm, then the greatest range of decomposition layers is decided by using scattering entropy and the useful components are preferentially chosen for reconstruction. The continuous wavelet transform (CWT) records preprocessing method is then delivered to radically change the noise-reduced vibration sign into a time-frequency map, which is fed into the CNN for model coaching and extraction of fault features. Finally, fault classification is realized by support vector machine (SVM) with excellent classification performance. Through preset fault experiments on diesel engines, it is established that the technique proposed in this paper can successfully identify fault states, and the classification accuracy is higher than alternative methods.