Residual Convolutional Network Research Articles

Identification of industrial exhaust based on an electronic nose with an interleaved grouped residual convolutional compression network

Identification of industrial exhaust based on an electronic nose with an interleaved grouped residual convolutional compression network

Polyline simplification using a region proposal network integrating raster and vector features

ABSTRACT Polyline simplification is crucial for cartography and spatial database management. In recent decades, various rule-based algorithms for vector polyline simplification have been proposed. However, most existing algorithms lack parameter self-adaptive capabilities and require repeated manual parameter adjustments when applied to different polylines. While deep-learning-based methods have recently been introduced for raster polyline image simplification, they cannot achieve end-to-end simplification when the input data and output results are vector polylines. This paper proposes a new deep-learning-based method for vector polyline simplification by integrating both the vector and raster features of the polyline. Specifically, a deep separable convolutional residual neural network was first used to extract the convolutional features of each image. Then, the region proposal network was modified to generate proposal boxes using vector coordinates, and these proposal boxes were used to locate the convolutional feature maps of bends. Finally, convolutional feature maps were fed into a binary classification network to identify unimportant vertices that should be omitted for vector polyline simplification. The experimental results indicated that the proposed method can exploit raster and vector features to achieve automatic and effective polyline simplification without prior map generalization knowledge and manual settings of rules and parameters. The polyline simplification results of the proposed method have a higher compression ratio of coordinate points and lower shape deformation and deviation than the results generated by the classic Wang and Müller (WM) algorithm and Support Vector Machine (SVM) based algorithm, which shows the potential of the proposed method for future applications in map generalization.

An Ensemble of Neural Networks for Moist Physics Processes, Its Generalizability and Stable Integration

AbstractWith the recent advances in data science, machine learning has been increasingly applied to convection and cloud parameterizations in global climate models (GCMs). This study extends the work of Han et al. (2020, https://doi.org/10.1029/2020MS002076) and uses an ensemble of 32‐layer deep convolutional residual neural networks, referred to as ResCu‐en, to emulate convection and cloud processes simulated by a superparameterized GCM, SPCAM. ResCu‐en predicts GCM grid‐scale temperature and moisture tendencies, and cloud liquid and ice water contents from moist physics processes. The surface rainfall is derived from the column‐integrated moisture tendency. The prediction uncertainty inherent in deep learning algorithms in emulating the moist physics is reduced by ensemble averaging. Results in 1‐year independent offline validation show that ResCu‐en has high prediction accuracy for all output variables, both in the current climate and in a warmer climate with +4K sea surface temperature. The analysis of different neural net configurations shows that the success to generalize in a warmer climate is attributed to convective memory and the 1‐dimensional convolution layers incorporated into ResCu‐en. We further implement a member of ResCu‐en into CAM5 with real world geography and run the neural‐network‐enabled CAM5 (NCAM) for 5 years without encountering any numerical integration instability. The simulation generally captures the global distribution of the mean precipitation, with a better simulation of precipitation intensity and diurnal cycle. However, there are large biases in temperature and moisture in high latitudes. These results highlight the importance of convective memory and demonstrate the potential for machine learning to enhance climate modeling.

Seismic Elastic Parameter Inversion via a FCRN and GRU Hybrid Network with Multi-Task Learning

Seismic elastic parameter inversion translates seismic data into subsurface structures and physical properties of formations. Traditional model-based inversion methods have limitations in retrieving complex geological structures. In recent years, deep learning methods have emerged as preferable alternatives. Nevertheless, inverting multiple elastic parameters using neural networks individually is computationally intensive and can lead to overfitting due to a shortage of labeled data in field applications. Multi-task learning can be employed to invert elastic parameters simultaneously. In this work, a hybrid network that leverages the fully convolutional residual network (FCRN) and the gated recurrent unit network (GRU) is designed for the simultaneous inversion of P-wave velocity and density from post-stack seismic data. The FCRN efficiently extracts local information from seismic data, while the GRU captures global dependency over time. To further improve the horizontal continuity and inversion stability, we use a multi-trace to single-trace (M2S) inversion strategy. Consequently, we name our proposed method the M2S multi-task FCRN and GRU hybrid network (M2S-MFCRGRU). Through anti-noise experiments and blind well tests, M2S-MFCRGRU exhibits superior anti-noise performance and generalization ability. Comprehensive experimental inversion results also showcase the excellent lateral continuity, vertical resolution, and stability of the M2S-MFCRGRU inversion results.

Rapid and flexible battery capacity estimation using random short-time charging segments based on residual convolutional networks

Monitoring the capacity of lithium-ion battery is a crucial task to ensure its safety and reliability during long-term use. However, conventional capacity estimation methods heavily rely on the specially designed operating conditions or durations of charge/discharge cycles, limiting their applications in real-world operations. To address such challenges, in this paper, a fast and flexible method is proposed to accurately estimate battery capacity based on a residual convolutional neural network using only small pieces of raw measurement data. And Bayesian optimization as well as network slimming are introduced to optimize and prune the network structure. Then, the proposed model is validated in two public battery degradation data sets, containing two types of batteries and six types of charging strategies in total. It's shown that the model is flexible enough to cope with the application scenarios of different charging strategies, different sampling frequencies and different voltage ranges starting at arbitrary initial SOCs, while still achieving fast and accurate capacity estimation. The mean absolute errors on 38 LFP batteries and 8 LCO batteries with a four-fold cross validation approach are all below 0.5% throughout the whole life of the batteries. Comprehensive case studies are also carried out to investigate the influence of capacity increment size, compression ratio of input data and the depth of network on the trade-offs between the model accuracy and its practicability for wide applications.

Combining radiomics and deep learning features of intra-tumoral and peri-tumoral regions for the classification of breast cancer lung metastasis and primary lung cancer with low-dose CT.

To investigate the performance of deep learning and radiomics features of intra-tumoral region (ITR) and peri-tumoral region (PTR) in the diagnosing of breast cancer lung metastasis (BCLM) and primary lung cancer (PLC) with low-dose CT (LDCT). We retrospectively collected the LDCT images of 100 breast cancer patients with lung lesions, comprising 60 cases of BCLM and 40 cases of PLC. We proposed a fusion model that combined deep learning features extracted from ResNet18-based multi-input residual convolution network with traditional radiomics features. Specifically, the fusion model adopted a multi-region strategy, incorporating the aforementioned features from both the ITR and PTR. Then, we randomly divided the dataset into training and validation sets using fivefold cross-validation approach. Comprehensive comparative experiments were performed between the proposed fusion model and other eight models, including the intra-tumoral deep learning model, the intra-tumoral radiomics model, the intra-tumoral deep-learning radiomics model, the peri-tumoral deep learning model, the peri-tumoral radiomics model, the peri-tumoral deep-learning radiomics model, the multi-region radiomics model, and the multi-region deep-learning model. The fusion model developed using deep-learning radiomics feature sets extracted from the ITR and PTR had the best classification performance, with the area under the curve of 0.913 (95% CI 0.840-0.960). This was significantly higher than that of the single region's radiomics model or deep learning model. The combination of radiomics and deep learning features was effective in discriminating BCLM and PLC. Additionally, the analysis of the PTR can mine more comprehensive tumor information.

Prediction of drug-protein interaction based on dual channel neural networks with attention mechanism.

The precise identification of drug-protein inter action (DPI) can significantly speed up the drug discovery process. Bioassay methods are time-consuming and expensive to screen for each pair of drug proteins. Machine-learning-based methods cannot accurately predict a large number of DPIs. Compared with traditional computing methods, deep learning methods need less domain knowledge and have strong data learning ability. In this study, we construct a DPI prediction model based on dual channel neural networks with an efficient path attention mechanism, called DCA-DPI. The drug molecular graph and protein sequence are used as the data input of the model, and the residual graph neural network and the residual convolution network are used to learn the feature representation of the drug and protein, respectively, to obtain the feature vector of the drug and the hidden vector of protein. To get a more accurate protein feature vector, the weighted sum of the hidden vector of protein is applied using the neural attention mechanism. In the end, drug and protein vectors are concatenated and input into the full connection layer for classification. In order to evaluate the performance of DCA-DPI, three widely used public data, Human, C.elegans and DUD-E, are used in the experiment. The evaluation metrics values in the experiment are superior to other relevant methods. Experiments show that our model is efficient for DPI prediction.

Radar Active Jamming Recognition under Open World Setting

To address the issue that conventional methods cannot recognize unknown patterns of radar jamming, this study adopts the idea of zero-shot learning (ZSL) and proposes an open world recognition method, RCAE-OWR, based on residual convolutional autoencoders, which can implement the classification of known and unknown patterns. In the supervised training phase, a residual convolutional autoencoder network structure is first constructed to extract the semantic information from a training set consisting solely of known jamming patterns. By incorporating center loss and reconstruction loss into the softmax loss function, a joint loss function is constructed to minimize the intra-class distance and maximize the inter-class distance in the jamming features. Moving to the unsupervised classification phase, a test set containing both known and unknown patterns is fed into the trained encoder, and a distance-based recognition method is utilized to classify the jamming signals. The results demonstrate that the proposed model not only achieves sufficient learning and representation of known jamming patterns but also effectively identifies and classifies unknown jamming signals. When the jamming-to-noise ratio (JNR) exceeds 10 dB, the recognition rate for seven known jamming patterns and two unknown jamming patterns is more than 92%.

FGRC-Net: A high-information interactive convolutional neural network for identifying ink spectral information

Modifying some keywords or numbers on documents to change the original intention is illegal. In some litigation cases, especially economic cases, there is often a need to examine the type of ink on documents. This paper proposes a nondestructive and accurate detection method for ink inspection. In this work, a high-information interactive, full group residual convolution network (FGRC-Net) is proposed and combined with spectral information to classify the inks of six different brands of pens. First, FGRC-Net realizes the information interaction among channels to effectively improve the feature extraction ability through multigroup convolution and multipath cascade. Meanwhile, to avoid feature degradation, the residual connection is introduced. Second, the ink spectral information of six pens based on the hyperspectral system is obtained. Finally, in comparing FGRC-Net and other deep learning methods, FGRC-Net shows the best classification performance with 98.38% accuracy, 98.53% precision, and 98.40% recall. The results show that combining FGRC-Net with a hyperspectral system is an effective detection method for handwritten ink classification. It also provides an effective detection method for handwritten document ink inspection. At the same time, the universality and validity of FGRC-Net in processing spectral information are verified by the comparison of multiple datasets.

Retrieving Ocean Surface Winds and Waves from Augmented Dual-Polarization Sentinel-1 SAR Data Using Deep Convolutional Residual Networks

Sea surface winds and waves are very important phenomena that exist in the air–sea boundary layer. With the advent of climate change, cascade effects are bringing more attention to these phenomena as warmer sea surface temperatures bring about stronger winds, thereby altering global wave conditions. Synthetic aperture radar (SAR) is a powerful sensor for high-resolution surface wind and wave observations and has accumulated large quantities of data. Furthermore, deep learning methods have been increasingly utilized in geoscience, especially the inversion of ocean information from SAR imagery. Here, we propose a method to invert various parameters of ocean surface winds and waves using Sentinel-1 SAR IW mode data. To ensure this method is more robust and scalable, we augmented the input data with dual-polarized SAR imagery, an incident angle, and a more constrained homogeneity test. This method adopts a deeper structure in order to retrieve more wind and wave parameters, and the use of residual networks can accelerate training convergence and improve regression accuracy. Using 1600 training samples filtered by a novel homogeneity test and with significant wave heights between 0 and 10 m, results from error parameters including the root mean square error (RMSE), scatter index (SI), and correlation coefficient (COR) show the great performance of this proposed method. The RMSE is 0.45 m, 0.76 s, and 1.90 m/s for the significant wave height, mean wave period, and wind speed, respectively. Furthermore, the temporal variation and spatial distribution of the estimates are consistent with China–France Oceanography Satellite (CFOSAT) observations, buoy measurements, WaveWatch3 regional model data, and ERA5 reanalysis data.

Hybrid photonic deep convolutional residual spiking neural networks for text classification.

Spiking neural networks (SNNs) offer powerful computation capability due to its event-driven nature and temporal processing. However, it is still limited to shallow structure and simple tasks due to the training difficulty. In this work, we propose a deep convolutional residual spiking neural network (DCRSNN) for text classification tasks. In the DCRSNN, the feature extraction is achieved via a convolution SNN with residual connection, using the surrogate gradient direct training technique. Classification is performed by a fully-connected network. We also suggest a hybrid photonic DCRSNN, in which photonic SNNs are used for classification with a converted training method. The accuracy of hard and soft reset methods, as well as three different surrogate functions, were evaluated and compared across four different datasets. Results indicated a maximum accuracy of 76.36% for MR, 91.03% for AG News, 88.06% for IMDB and 93.99% for Yelp review polarity. Soft reset methods used in the deep convolutional SNN yielded slightly better accuracy than their hard reset counterparts. We also considered the effects of different pooling methods and observation time windows and found that the convergence accuracy achieved by convolutional SNNs was comparable to that of convolutional neural networks under the same conditions. Moreover, the hybrid photonic DCRSNN also shows comparable testing accuracy. This work provides new insights into extending the SNN applications in the field of text classification and natural language processing, which is interesting for the resources-restrained scenarios.

Mathematical Analysis and Performance Evaluation of the GELU Activation Function in Deep Learning

Selecting the most suitable activation function is a critical factor in the effectiveness of deep learning models, as it influences their learning capacity, stability, and computational efficiency. In recent years, the Gaussian error linear unit (GELU) activation function has emerged as a dominant method, surpassing traditional functions such as the rectified linear unit (ReLU) in various applications. This study presents a rigorous mathematical investigation of the GELU activation function, exploring its differentiability, boundedness, stationarity, and smoothness properties in detail. In addition, we conduct an extensive experimental comparison of the GELU function against a broad range of alternative activation functions, utilizing a residual convolutional network trained on the CIFAR-10, CIFAR-100, and STL-10 datasets as the empirical testbed. Our results demonstrate the superior performance of GELU compared to other activation functions, establishing its suitability for a wide range of deep learning applications. This comprehensive study contributes to a more profound understanding of the underlying mathematical properties of GELU and provides valuable insights for practitioners aiming to select activation functions that optimally align with their specific objectives and constraints in deep learning.

Beyond Heart Murmur Detection: Automatic Murmur Grading From Phonocardiogram.

Murmurs are abnormal heart sounds, identified by experts through cardiac auscultation. The murmur grade, a quantitative measure of the murmur intensity, is strongly correlated with the patient's clinical condition. This work aims to estimate each patient's murmur grade (i.e., absent, soft, loud) from multiple auscultation location phonocardiograms (PCGs) of a large population of pediatric patients from a low-resource rural area. The Mel spectrogram representation of each PCG recording is given to an ensemble of 15 convolutional residual neural networks with channel-wise attention mechanisms to classify each PCG recording. The final murmur grade for each patient is derived based on the proposed decision rule and considering all estimated labels for available recordings. The proposed method is cross-validated on a dataset consisting of 3456 PCG recordings from 1007 patients using a stratified ten-fold cross-validation. Additionally, the method was tested on a hidden test set comprised of 1538 PCG recordings from 442 patients. The overall cross-validation performances for patient-level murmur gradings are 86.3% and 81.6% in terms of the unweighted average of sensitivities and F1-scores, respectively. The sensitivities (and F1-scores) for absent, soft, and loud murmurs are 90.7% (93.6%), 75.8% (66.8%), and 92.3% (84.2%), respectively. On the test set, the algorithm achieves an unweighted average of sensitivities of 80.4% and an F1-score of 75.8%. This study provides a potential approach for algorithmic pre-screening in low-resource settings with relatively high expert screening costs. The proposed method represents a significant step beyond detection of murmurs, providing characterization of intensity, which may provide an enhanced classification of clinical outcomes.

Fingerprint-Based mmWave Positioning System Aided by Reconfigurable Intelligent Surface

Reconfigurable intelligent surface (RIS) is a promising technique for millimeter wave (mmWave) positioning systems. In this paper, we consider multiple mobile users (MUs) positioning problem in the multiple-input multiple-output (MIMO) time-division duplex (TDD) mmWave systems aided by the RIS. We derive the expression for the space-time channel response vector (STCRV) as a novel type of fingerprint. The STCRV fingerprint comprises the channel characteristics of the cascaded channel, such as the angle of arrival (AOA) at the RIS and the time delay from the MU to the RIS, which are indicative of the position of the MU. To process the STCRV with more features, we propose a novel residual convolution network regression (RCNR) learning algorithm to output the estimated three-dimensional (3D) position of the MU with higher accuracy. Specifically, the RCNR learninng algorithm includes a data processing block to process the input STCRV, a normal convolution block to extract the features of STCRV, four residual convolution blocks to further extract the features and protect the integrity of the features, and a regression block to estimate the 3D position. Extensive simulation results are also presented to demonstrate that the proposed RCNR learning algorithm outperforms the traditional convolution neural network (CNN).

Hyperspectral remote sensing image classification based on residual generative Adversarial Neural Networks

Classification of hyperspectral images is an essential application of deep learning techniques. However, standard deep learning approaches require a large number of labelled samples for model training, and classification performance can be enhanced. In this study, we propose the use of residual generative adversarial networks for the classification of hyperspectral images based on deep learning Convolutional Neural Networks, a large number of labelled samples, and the high classification accuracy required for the classification task. The method is based on generative adversarial networks, with a 6-layer residual network containing up-sampling and convolutional layers replacing the inverse convolutional layer network structure of the generator to enhance the data generation capability, and an 18-layer residual convolutional network replacing the convolutional layer network structure of the discriminator to enhance the feature extraction capability. The hyperspectral image classification method with residual generation adversarial network enhances the information exchange between the shallow network and the deep network by adding a residual structure to the network, extracts the deep features of hyperspectral images and improves the accuracy of hyperspectral image classification. Extensive experiments on several benchmark hyperspectral datasets have shown that the method outperforms comparative methods by 0.7% to 22.3% on OA.

Prediction of gastrointestinal functional state based on myoelectric recordings utilizing a deep neural network architecture.

Functional and motility-related gastrointestinal (GI) disorders affect nearly 40% percent of the population. Disturbances of GI myoelectric activity have been proposed to play a significant role in these disorders. A significant barrier to usage of these signals in diagnosis and treatment is the lack of consistent relationships between GI myoelectric features and function. A potential cause of this issue is the use of arbitrary classification criteria, such as percentage of power in tachygastric and bradygastric frequency bands. Here we applied automatic feature extraction using a deep neural network architecture on GI myoelectric signals from free-moving ferrets. For each animal, we recorded during baseline control and feeding conditions lasting for 1 h. Data were trained on a 1-dimensional residual convolutional network, followed by a fully connected layer, with a decision based on a sigmoidal output. For this 2-class problem, accuracy was 90%, sensitivity (feeding detection) was 90%, and specificity (baseline detection) was 89%. By comparison, approaches using hand-crafted features (e.g., SVM, random forest, and logistic regression) produced an accuracy from 54% to 82%, sensitivity from 46% to 84% and specificity from 66% to 80%. These results suggest that automatic feature extraction and deep neural networks could be useful to assess GI function for comparing baseline to an active functional GI state, such as feeding. In future testing, the current approach could be applied to determine normal and disease-related GI myoelectric patterns to diagnosis and assess patients with GI disease.

Spatio-temporal correlation for simultaneous ultra-short-term wind speed prediction at multiple locations

Wind, as a fluid, has continuity in both space and time. Coupling spatial and temporal information to build prediction models can improve wind speed prediction accuracy. This paper proposes a method that predicts wind speed at multiple locations using both spatial and temporal data. Three deep learning models are introduced: Convolutional Residual Spatial-Temporal Long Short-Term Memory neural network (CoReSTL), Convolutional Spatial-Temporal-3D neural network (CoST-3), and Convolutional Spatial-Temporal Long Short-Term Memory neural network (CoST-L). These models combine Convolutional Long Short-Term Memory (ConvLSTM), Residual Network (ResNet), and 1 × 1 3D convolution to extract spatial and temporal correlations between multi-site wind speeds. The spatio-temporal prediction of wind fields under two terrains was carried out to screen out neural network models with higher accuracy. The results show that CoReSTL, CoST-3, and CoST-L all achieved better prediction results. However, the performance of the CoReSTL model was better than that of CoST-3 and CoST-L, with stronger applicability in complex real terrain.

A partial domain adaptation scheme based on weighted adversarial nets with improved CBAM for fault diagnosis of wind turbine gearbox

Most domain adaptation methods for fault diagnosis depend heavily on the precondition that the source and target domain have an identical label space, which is hard to be satisfied in industrial sites. Recently, many approaches have been developed to implement partial domain adaptation. However, most existing methods adopt classic convolutional neural network as the feature extractor, which limits the ability to learn discriminative representations from non-stationary vibration signals of wind turbine (WT) gearboxes. Moreover, the design of multiple subdomain adaptation will cause complex network structure with many source classes. To address these problems, this paper proposes a partial domain adaptation scheme based on weighted adversarial nets with improved convolutional block attention module (CBAM) for WT gearbox unsupervised fault diagnosis. In detail, a residual convolutional network combining the improved CBAM is designed to extract finer domain discriminative features for knowledge transfer. Meanwhile, a weighting mechanism based on the two-stage domain discriminator is designed to evaluate the contribution of each source sample, through which a simplified transfer network structure is constructed and the source samples unrelated to the target domain can be filtered. Furthermore, an adversarial transfer strategy is introduced to decrease the distribution discrepancy between domains, then the helpful diagnosis knowledge can be transferred. Experiments on two cases demonstrate the superiority and effectiveness of the proposed method compared with existing domain adaptation methods.

Temporal-spatial convolutional residual network for decoding attempted movement related EEG signals of subjects with spinal cord injury

Brain Computer Interface (BCI) offers a promising approach to restoring hand functionality for people with cervical spinal cord injury (SCI). A reliable classification of brain activities based on appropriate flexibility in feature extraction could enhance BCI systems performance. In the present study, based on convolutional layers with temporal-spatial, Separable and Depthwise structures, we develop Temporal-Spatial Convolutional Residual Network)TSCR-Net(and Temporal-Spatial Convolutional Iterative Residual Network)TSCIR-Net(structures to classify electroencephalogram (EEG) signals. Using EEG signals in five different hand movement classes of SCI people, we compare the effectiveness of TSCIR-Net and TSCR-Net models with some competitive methods. We use the bayesian hyperparameter optimization algorithm to tune the hyperparameters of compact convolutional neural networks. In order to show the high generalizability of the proposed models, we compare the results of the models in different frequency ranges. Our proposed models decoded distinctive characteristics of different movement efforts and obtained higher classification accuracy than previous deep neural networks. Our findings indicate that TSCIR-Net and TSCR-Net models fulfills a better classification accuracy of 71.11%, and 64.55% for EEG_All and 57.74%, and 67.87% for EEG_Low frequency data sets than the compared methods in the literature.

Integrated intelligent fault diagnosis approach of offshore wind turbine bearing based on information stream fusion and semi-supervised learning

Offshore wind turbines play a vital role in transferring wind energy to electricity, which could help relieve the energy crisis and improve the global climate. In general, offshore wind turbines are installed open sea to avoid the potential interruption of people’s daily life. In such kind of harsh operating environment, the wind turbine transmission system is prone to failure, especially for the rolling bearings. Therefore, it is crucial to conduct condition monitoring of rolling bearings to ensure the safe and efficient operation of offshore wind turbines. Intelligent fault diagnosis is a research hotspot for condition monitoring of rolling bearings. However, the existing intelligent fault diagnosis techniques have some limitations. For example, most of the existing techniques were developed based on single sensory data, which can lead to inaccurate and unstable diagnostic results. Moreover, most existing techniques implicitly assume that there are sufficient labeled samples for classifier training. This may not be the case for offshore wind turbines where the labeled samples are limited. To address the aforementioned issues, an intelligent fault diagnosis technique by integrating an information stream fusion and a semi-supervised learning approach is proposed in this study. In the proposed method, a coupled convolutional residual network is proposed to realize the information streams fusion, in which the vibration signal and acoustic emission signal are served as the inputs of the proposed network, and then a concatenation operation is used to fuse the features obtained from two information streams. Meanwhile, a semi-supervised learning approach is also proposed, which can utilize the labeled samples, the correctly predicted samples, and the unlabeled samples to improve diagnostic accuracy. The diagnostic result on the experimental offshore wind turbine bearing dataset demonstrates that the proposed method achieves the highest diagnostic accuracy compared to existing comparative methods.