Residual Connections Research Articles

An integrated approach for mechanical fault diagnosis using maximum mean square discrepancy representation and CNN-based mixed information fusion

Bearings are an essential component of modern industry and cross-domain diagnosis of bearings holds significant importance. However, in practical applications, issues such as insufficient training data and differences between equipment present challenges. Transfer learning has become one of the effective methods to address these problems. This article proposes a fault diagnosis method that combines maximum mean square discrepancy (MMSD) measurement with the convolutional neural network (CNN) with mixed information (MIXCNN). MIXCNN enhances spatial position discrimination through deep convolution and achieves cross-channel information interaction using traditional convolution. The introduction of residual connections reduces information loss, while increasing network depth focuses on highly distinguishable features. MMSD constructs a metric that comprehensively reflects the mean and variance information of data samples in the reproducing kernel Hilbert space, thereby enhancing domain confusion. Experimental results show that this method achieves high diagnostic accuracy in various transfer tasks, with a maximum accuracy of 99.29%, providing reliable support for bearing fault diagnosis.

Global and local feature extraction based on convolutional neural network residual learning for MR image denoising.

Given the different noise distribution information of global and local magnetic resonance (MR) images, this study aims to extend the current work on convolutional neural networks that preserve global structure and local details in MR image denoising tasks. This study proposed a parallel and serial network for denoising 3D MR images, called 3D-PSNet. We use the residual depthwise separable convolution block to learn the local information of the feature map, reduce the network parameters, and thus improve the training speed and parameter efficiency. In addition, we consider the feature extraction of the global image and utilize residual dilated convolution to process the feature map to expand the receptive field of the network and avoid the loss of global information. Finally, we combine both of them to form a parallel network. What's more, we integrate reinforced residual convolution blocks with dense connections to form serial network branches, which can remove redundant information and refine features to further obtain accurate noise information. The peak signal-to-noise ratio, structural similarity index measure, and root mean square error metrics of 3D-PSNet are as high as 47.79%, 99.81%, and 0.40%, respectively, achieving competitive denoising effect on three public datasets. The ablation experiments demonstrated the effectiveness of all the designed modules regarding all the evaluated metrics in both datasets. The proposed 3D-PSNet takes advantage of multi-scale receptive fields, local feature extraction and residual dense connections to more effectively restore the global structure and local fine features in MR images, and is expected to help doctors quickly and accurately diagnose patients' conditions.

TAFENet: A Two-Stage Attention-Based Feature-Enhancement Network for Strip Steel Surface Defect Detection

Strip steel serves as a crucial raw material in numerous industries, including aircraft and automobile manufacturing. Surface defects in strip steel can degrade the performance, quality, and appearance of industrial steel products. Detecting surface defects in steel strip products is challenging due to the low contrast between defects and background, small defect targets, as well as significant variations in defect sizes. To address these challenges, a two-stage attention-based feature-enhancement network (TAFENet) is proposed, wherein the first-stage feature-enhancement procedure utilizes an attentional convolutional fusion module with convolution to combine all four-level features and then strengthens the features of different levels via a residual spatial-channel attention connection module (RSC). The second-stage feature-enhancement procedure combines three-level features using an attentional self-attention fusion module and then strengthens the features using a RSC attention module. Experiments on the NEU-DET and GC10-DET datasets demonstrated that the proposed method significantly improved detection accuracy, thereby confirming the effectiveness and generalization capability of the proposed method.

RBP-DIP: Residual back projection with deep image prior for ill-posed CT reconstruction

The success of deep image prior (DIP) in a number of image processing tasks has motivated their application in image reconstruction problems in computed tomography (CT). In this paper, we introduce a residual back projection technique (RBP) that improves the performance of deep image prior framework in iterative CT reconstruction, especially when the reconstruction problem is highly ill-posed. The RBP-DIP framework uses an untrained U-net in conjunction with a novel residual back projection connection to minimize the objective function while improving reconstruction accuracy. In each iteration, the weights of the untrained U-net are optimized, and the output of the U-net in the current iteration is used to update the input of the U-net in the next iteration through the proposed RBP connection. The introduction of the RBP connection strengthens the regularization effects of the DIP framework in the context of iterative CT reconstruction leading to improvements in accuracy. Our experiments demonstrate that the RBP-DIP framework offers improvements over other state-of-the-art conventional IR methods, as well as pre-trained and untrained models with similar network structures under multiple conditions. These improvements are particularly significant in the few-view and limited-angle CT reconstructions, where the corresponding inverse problems are highly ill-posed and the training data is limited. Furthermore, RBP-DIP has the potential for further improvement. Most existing IR algorithms, pre-trained models, and enhancements applicable to the original DIP algorithm can also be integrated into the RBP-DIP framework.

Dental Caries Segmentation using Deformable Dense Residual Half U-Net for Teledentistry System

Clinical practitioners’ workload and challenges are significantly reduced by classifying, predicting, and localizing lesions or dental caries. In recent research, a high-reliability diagnostic system within deep learning models has been implemented in a clinical teledentistry system. In order to construct an efficient, precise, and lightweight deep learning architecture, it is dynamically structured. In this paper, we present an efficient, accurate, and lightweight deep learning architecture for augmenting spatial locations and improving the transformation modeling abilities of fixed-structure CNNs. Deformable Dense Residual (DDR) enhances the efficacy of the residual convolution block by optimizing its structure, thereby mitigating model redundancy and ameliorating the challenge of vanishing gradients encountered during the training stages. DDR Half U-Net presents notable advancements to the simplified U-Net framework across three pivotal domains: the encoder, decoder, and loss function. Specifically, the encoder integrates deformable convolutions, thereby enhancing the model's capacity to discern features of diverse scales and configurations. In the decoder, a sophisticated arrangement of dense residual connections facilitates the fusion of low-level and high-level features, contributing to comprehensive feature extraction. Moreover, the utilization of a weight-adaptive loss function ensures equitable consideration of both caries and non-caries samples, thereby promoting balanced optimization during training.

SRARDA: A lightweight adaptive residual dense attention generative adversarial network for image super-resolution

Image super-resolution (SR) is the task of inferring a high resolution (HR) image from one/multiple single low resolution (LR) input(s). Traditional networks are evaluated by pixel-level metrics such as Peak-Signal-to-Noise Ratio (PSNR) etc., which do not always align with human perception of image quality. They often produce excessively smooth images that lack high-frequency texture and appear unnatural. Therefore, in this paper, we propose a lightweight adaptive residual dense attention generative adversarial network (SRARDA) for image SR. Firstly, our generator adopts the residual in residual (RIR) structure but redesigns the basic module. By using dynamic residual connection (ARC) to dynamically adjust the importance of residual and main paths, we design a novel adaptive residual dense attention block (ARDAB) that enhances the feature extraction capability of the generator. In addition, we build a high-frequency filtering unit (HFU) to extract more high-frequency features from the LR space. Finally, to fully utilize the discriminator, we use WGAN to compute the difference between the HR image and the reconstructed image. Experiments demonstrate that SRARDA effectively addresses the issue of excessive smoothing in reconstructed images, while also enhancing visual quality.

Optic Nerve Sheath Ultrasound Image Segmentation Based on CBC-YOLOv5s

The diameter of the optic nerve sheath is an important indicator for assessing the intracranial pressure in critically ill patients. The methods for measuring the optic nerve sheath diameter are generally divided into invasive and non-invasive methods. Compared to the invasive methods, the non-invasive methods are safer and have thus gained popularity. Among the non-invasive methods, using deep learning to process the ultrasound images of the eyes of critically ill patients and promptly output the diameter of the optic nerve sheath offers significant advantages. This paper proposes a CBC-YOLOv5s optic nerve sheath ultrasound image segmentation method that integrates both local and global features. First, it introduces the CBC-Backbone feature extraction network, which consists of dual-layer C3 Swin-Transformer (C3STR) and dual-layer Bottleneck Transformer (BoT3) modules. The C3STR backbone’s multi-layer convolution and residual connections focus on the local features of the optic nerve sheath, while the Window Transformer Attention (WTA) mechanism in the C3STR module and the Multi-Head Self-Attention (MHSA) in the BoT3 module enhance the model’s understanding of the global features of the optic nerve sheath. The extracted local and global features are fully integrated in the Spatial Pyramid Pooling Fusion (SPPF) module. Additionally, the CBC-Neck feature pyramid is proposed, which includes a single-layer C3STR module and three-layer CReToNeXt (CRTN) module. During upsampling feature fusion, the C3STR module is used to enhance the local and global awareness of the fused features. During downsampling feature fusion, the CRTN module’s multi-level residual design helps the network to better capture the global features of the optic nerve sheath within the fused features. The introduction of these modules achieves the thorough integration of the local and global features, enabling the model to efficiently and accurately identify the optic nerve sheath boundaries, even when the ocular ultrasound images are blurry or the boundaries are unclear. The Z2HOSPITAL-5000 dataset collected from Zhejiang University Second Hospital was used for the experiments. Compared to the widely used YOLOv5s and U-Net algorithms, the proposed method shows improved performance on the blurry test set. Specifically, the proposed method achieves precision, recall, and Intersection over Union (IoU) values that are 4.1%, 2.1%, and 4.5% higher than those of YOLOv5s. When compared to U-Net, the precision, recall, and IoU are improved by 9.2%, 21%, and 19.7%, respectively.

Hybrid CNN-Mamba network for single-pixel imaging

Recent progress in single-pixel imaging (SPI) has exhibited remarkable performance using deep neural networks, e.g., convolutional neural networks (CNNs) and vision Transformers (ViTs). Nonetheless, it is challenging for existing methods to well model object image from single-pixel detections that have a long-range dependency, where CNNs are constrained by their local receptive fields, and ViTs suffer from high quadratic complexity of attention mechanism. Inspired by the Mamba architecture, known for its proficiency in handling long sequences and global contextual information with enhanced computational efficiency as state space models (SSMs), we propose a hybrid network of CNN and Mamba for SPI, named CMSPI. The proposed CMSPI integrates the local feature extraction capability of convolutional layers with the abilities of SSMs for efficiently capturing the long-range dependency, and the design of complementary split-concat structure, depthwise separable convolution, and residual connection enhance learning power of network model. Besides, CMSPI adopts a two-step training strategy, which makes reconstruction performance better and hardware-friendly. Simulations and real experiments demonstrate that CMSPI has higher imaging quality, lower memory consumption, and less computational burden than the state-of-the-art SPI methods.

Spectrum Sensing Method Based on STFT-RADN in Cognitive Radio Networks.

To address the common issues in traditional convolutional neural network (CNN)-based spectrum sensing algorithms in cognitive radio networks (CRNs), including inadequate signal feature representation, inefficient utilization of feature map information, and limited feature extraction capabilities due to shallow network structures, this paper proposes a spectrum sensing algorithm based on a short-time Fourier transform (STFT) and residual attention dense network (RADN). Specifically, the RADN model improves the basic residual block and introduces the convolutional block attention module (CBAM), combining residual connections and dense connections to form a powerful deep feature extraction structure known as residual in dense (RID). This significantly enhances the network's feature extraction capabilities. By performing STFT on the received signals and normalizing them, the signals are converted into time-frequency spectrograms as network inputs, better capturing signal features. The RADN is trained to extract abstract features from the time-frequency images, and the trained RADN serves as the final classifier for spectrum sensing. Experimental results demonstrate that the STFT-RADN spectrum sensing method significantly improves performance under low signal-to-noise ratio (SNR) conditions compared to traditional deep-learning-based methods. This method not only adapts to various modulation schemes but also exhibits high detection probability and strong robustness.

Evaluation of Fracturing Effect of Tight Reservoirs Based on Deep Learning.

The utilization of hydraulic fracturing technology is indispensable for unlocking the potential of tight oil and gas reservoirs. Understanding and accurately evaluating the impact of fracturing is pivotal in maximizing oil and gas production and optimizing wellbore performance. Currently, evaluation methods based on acoustic logging, such as orthogonal dipole anisotropy and radial tomography imaging, are widely used. However, when the fractures generated by hydraulic fracturing form a network-like pattern, orthogonal dipole anisotropy fails to accurately assess the fracturing effects. Radial tomography imaging can address this issue, but it is challenged by high manpower and time costs. This study aims to develop a more efficient and accurate method for evaluating fracturing effects in tight reservoirs using deep learning techniques. Specifically, the method utilizes dipole array acoustic logging curves recorded before and after fracturing. Manual labeling was conducted by integrating logging data interpretation results. An improved WGAN-GP was employed to generate adversarial samples for data augmentation, and fracturing effect evaluation was implemented using SE-ResNet, ResNet, and DenseNet. The experimental results demonstrated that ResNet with residual connections is more suitable for the dataset in this study, achieving higher accuracy in fracturing effect evaluation. The inclusion of the SE module further enhanced model accuracy by adaptively adjusting the weights of feature map channels, with the highest accuracy reaching 99.75%.

Recognizing Digital Ink Chinese Characters Written by International Students Using a Residual Network with 1-Dimensional Dilated Convolution

Due to the complex nature of Chinese characters, junior international students often encounter writing problems related to strokes, components, and their combinations when writing Chinese characters. Digital ink Chinese characters (DICCs) are obtained by sampling the writing trajectory of Chinese characters with a pen input device. DICCs contain rich information, such as the time and space of strokes and sampling points. Recognizing DICCs is crucial for evaluating and correcting writing errors and enhancing the quality of Chinese character teaching for international students. Here, the paper first employs a one-dimensional dilated convolution to digital ink Chinese character recognition (DICCR) and proposes a novel residual network with one-dimensional dilated convolution (1-D ResNetDC). The 1-D ResNetDC not only utilizes multi-scale convolution kernels, but also employs different dilation rates on a single-scale convolution kernel to obtain information from various ranges. Additionally, residual connections facilitate the training of deep one-dimensional convolutional neural networks. Moreover, the paper proposes a more expressive ten-dimensional feature representation that includes spatial, temporal, and writing direction information for each sampling point, thereby improving classification accuracy. Because the DICC dataset of international students is small and unbalanced, the 1-D ResNetDC is pre-trained on the published available dataset. The experiments demonstrate that our approach is effective and superior. This model features a compact architecture, a reduced number of parameters, and excellent scalability.

Short-term natural gas load forecasting based on EL-VMD-Transformer-ResLSTM

Due to changes in urban residents’ consumption habits and lifestyles, accurately predicting natural gas consumption has become increasingly important. To address this issue, this paper proposes a forecasting model that combines Ensemble Learning (EL), Variational Mode Decomposition (VMD), Transformer, and LSTM. First, XGBoost, CatBoost, and LightGBM are used as base learners in the ensemble learning framework, with the predictions generated by the ensemble model integrated into the original dataset. Next, the VMD method is employed to decompose the natural gas load sequence into several intrinsic mode functions (IMFs), effectively extracting the inherent features of the natural gas load sequence. Finally, the data is input into the Transformer-ResLSTM network for prediction. This network replaces the original Transformer decoder structure with an LSTM network and fully connected layers, creating a new decoder structure. Additionally, a residual connection mechanism is introduced in both the encoder of the Transformer network and the new decoder structure. Experimental results show that, compared to traditional models such as ARIMA, Transformer, GRU, and LSTM, the proposed hybrid model significantly improves prediction accuracy, reducing MSE by 92–98% and MAE by 74–83%. In summary, this method demonstrates significant potential and practical value in enhancing the accuracy of natural gas load forecasting.

Application and optimization of residual connection neural network in spacecraft thermal design

In thermal analysis modeling, the finite element method (FEM) is commonly used; however, it incurs high computational costs and complicates the global optimization of thermal parameters. To address these challenges, developing simplified surrogate models is crucial for enhancing analysis efficiency. Yet, constructing such models demands exceptional predictive accuracy, making conventional parameter adjustment methods inadequate for design needs. This paper introduces a novel Residual connection Neural Network model, called Res-NN, designed to approximate the CMOS finite element model. By employing residual connections, the Res-NN model significantly reduces fitting errors between networks, achieving a predictive accuracy of 94.6 %, which is 6.6 % higher than that of comparable Multi-Layer Perceptron (MLP) models. Moreover, Res-NN is over 100 times faster than traditional FEM in prediction speed, effectively circumventing the difficulties associated with parameter adjustments. To optimize the temperature fluctuations of the CMOS model, we utilized the Res-NN model as an iterative object within an optimization algorithm. Through experimental comparisons, we identified the PSO algorithm as the most effective option. The PSO optimization results demonstrated a chip temperature difference of 0.045 °C, with a simulation error of only 0.0649 °C, meeting design specifications and achieving a 61.7 % reduction in temperature variance compared to traditional thermal design method. This validates the superiority of the entire optimization process.

Efficient visual transformer transferring from neural ODE perspective

AbstractRecently, the Visual Image Transformer (ViT) has revolutionized various domains in computer vision. The transfer of pre‐trained ViT models on large‐scale datasets has proven to be a promising method for downstream tasks. However, traditional transfer methods introduce numerous additional parameters in transformer blocks, posing new challenges in learning downstream tasks. This article proposes an efficient transfer method from the perspective of neural Ordinary Differential Equations (ODEs) to address this issue. On the one hand, the residual connections in the transformer layers can be interpreted as the numerical integration of differential equations. Therefore, the transformer block can be described as two explicit Euler method equations. By dynamically learning the step size in the explicit Euler equation, a highly lightweight method for transferring the transformer block is obtained. On the other hand, a new learnable neural memory ODE block is proposed by taking inspiration from the self‐inhibition mechanism in neural systems. It increases the diversity of dynamical behaviours of the neurons to transfer the head block efficiently and enhances non‐linearity simultaneously. Experimental results in image classification demonstrate that the proposed approach can effectively transfer ViT models and outperform state‐of‐the‐art methods.

Comparative Assessment of U-Net-Based Deep Learning Models for Segmenting Microfractures and Pore Spaces in Digital Rocks

Summary Segmentation of high-resolution X-ray microcomputed tomography (µCT) images is crucial in digital rock physics (DRP), affecting the characterization and analysis of microscale phenomena in the porous media. The complexity of geological structures and nonideal scanning conditions pose significant challenges to conventional image segmentation approaches. Motivated by the recent increasing popularity of deep learning (DL) techniques in image processing, this work undertakes a comparative study of DL models, specifically U-Net and its variants, for segmenting multiple targets with distinguished features in digital rocks, including discrete fracture networks (DFNs), pore spaces, and solid rock. Particularly, DFNs have a smaller volumetric fraction over others, bringing in a substantial challenge of imbalanced segmentation. The primary focus is to evaluate the architecture and feature enhancement strategies of various DL models, including U-Net, attention U-Net, residual U-Net, U-Net++, and residual U-Net++. The models were designed as 2.5D, utilizing a central 2D image and its two adjacent upper and lower 2D images as input to provide a pseudo-3D context. In addition, because the ground truth of segmentation was unknown for real-world digital rocks, we created a benchmark data set following the inverse operations of segmentation. The data synthesis started from the label images (i.e., solid rock, pore spaces, and DFNs), followed by simulating partial volume blurring, adding random background noise, and introducing ring artifacts to mimic real raw X-ray µCT images. The data set, which included various rock types (i.e., sandstone and artificial data), scanning resolution, and magnitudes of noise and artifacts, was divided into training and testing data sets with a 90% and 10% ratio, respectively. Moreover, in addition to the conventional pixel-wise evaluation metrics, the physics-based metric of the lattice-Boltzmann method (LBM) simulated permeability provided more comprehensive assessments. The results demonstrated that the residual connections, nested architectures, and redesigned skip connections contribute to the model performance and give the residual U-Net++ the highest accuracy. The improvements were mainly on the boundaries and small targets, especially the DFNs, which dominate the interconnectivity and therefore affect the permeability greatly. This study also rigorously evaluated the efficiency and generalization of each model, demonstrating that the sophisticated architectures achieved excellent practicability and maintained robust performance on completely unseen data, ensuring their suitability for diverse and challenging DRP applications.

Self-supervised Entity Alignment Model Based on Joint Embedding

Abstract Entity alignment is an important problem for constructing Web-scale KGs. Self-supervised entity alignment models represented by SelfKG, proposed in 2022, have eliminated the essential requirement of accurate alignment: human labeling, which significantly saves resources. However, when SelfKG scales up to improve entity alignment accuracy, it introduces neighbor noise, leading to a decrease in alignment performance. This indicates that the ability of self-supervised models to aggregate and utilize knowledge graph information needs further exploration. To address this issue, we proposed the following approach: using a Graph Convolutional Network(GCN) to collect global entity and structural information, and a Graph Attention Network(GAT) to collect local neighborhood information from subgraphs. Subsequently, we introduced a novel fusion method to merge the two parts through weighted sum and residual connection, and completed self-supervised entity alignment using the Relative Similarity Metric(RSM) mechanism proposed by SelfKG. This model has both the ability to scale up and the flexibility to avoid neighbor noise, resulting in a significant improvement in performance.

Channel estimation of massive MIMO FSO communication system using deep attention residual U-Net

Channel estimation in massive-MIMO FSO systems is critical for ensuring reliable data transmission. However, conventional estimators offer limited benefits due to the computational difficulty of accurately estimating the channel. This paper presents a novel approach to estimate channels using an attention residual U-Net (ARU-Net) architecture which utilizes the advantages of both attention and residual connection. In the simulation, the channel matrix has been represented as a 2D image. The proposed model significantly outperforms traditional channel estimation methods and other deep learning models in terms of MSE (10−5 at 25 dB SNR), especially in atmospheric turbulence and other noises.

Phase unwrapping in digital holography based on SRDU-net

In digital holographic measurement, the hologram phase is extracted using an inverse tangent function, resulting in a wrapped phase that is constrained to be (-π,π]. However, the phase contains the height information of the object, which is crucial for accurate measurement of the object contour. Therefore, a digital holographic phase unwrapping method based on SRDU-Net (Separable-Residual-Dense-Inverted U-Net) is proposed in this paper. The SRDU-Net network is constructed by introducing depth-separable convolution, inverted residuals and dense blocks with U-Net as the network framework to achieve high-precision hologram phase recovery. This network adopts depth-separable convolution instead of traditional convolution, combines the inverse residual connection to construct a lightweight convolution structure, and defines dense blocks with this structure to form a lightweight grouped deep convolution network. Meanwhile, the Leaky ReLU activation function is used to introduce the learning rate mechanism to optimize the network parameters with Huber and MSE (mean square error) as the combined loss function. The simulated phase dataset is used to train the network, the trained network model is tested for speckle noise immunity, and phase unwrapping experiments are performed on the collected holograms of the test samples. The results show that SRDU-Net improves the SSIM by 0.1% and reduces the RMSE by 86% over Res-UNet (Residual-UNet). Therefore, the proposed method can realize high-precision recovery of digital holographic wrapped phases, and has a good robustness to phase unwrapping of holograms containing a high degree of speckle noise.

RGMeta: Enhancing Cold-Start Recommendations with a Residual Graph Meta-Embedding Model

Traditional recommendation models grapple with challenges such as the scarcity of similar user or item references and data sparsity, rendering the cold-start problem particularly formidable. Meta-learning has emerged as a promising avenue to address these issues, particularly in solving the item cold-start problem by generating meta-embeddings for new items as their initial ID embeddings. This approach has shown notable success in enhancing the accuracy of click-through rate predictions. However, prevalent meta-embedding models often focus solely on the attribute features of the item, neglecting crucial user information associated with it during the generation of initial ID embeddings for new items. This oversight hinders the exploitation of valuable user-related information to enhance the quality and accuracy of the initial ID embedding. To tackle this limitation, we introduce the residual graph meta-embedding model (RGMeta). RGMeta adopts a comprehensive approach by considering both the attribute features and target users of both old and new items. Through the integration of residual connections, the model effectively combines the representation information of the old neighbor items with the intrinsic features of the new item, resulting in an improved initial ID embedding generation. Experimental results demonstrate that RGMeta significantly enhances the performance of the cold-start phase, showcasing its effectiveness in overcoming challenges associated with sparse data and limited reference points. Our proposed model presents a promising step forward in leveraging both item attributes and user-related information to address cold-start problems in recommendation systems.

Simultaneous energy and mass calibration of large-radius jets with the ATLAS detector using a deep neural network

Abstract The energy and mass measurements of jets are crucial tasks for the Large Hadron Collider experiments. This paper presents a new calibration method to simultaneously calibrate these quantities for large-radius jets measured with the ATLAS detector using a deep neural network (DNN). To address the specificities of the calibration problem, special loss functions and training procedures are employed, and a complex network architecture, which includes feature annotation and residual connection layers, is used. The DNN-based calibration is compared to the standard numerical approach in an extensive series of tests. The DNN approach is found to perform significantly better in almost all of the tests and over most of the relevant kinematic phase space. In particular, it consistently improves the energy and mass resolutions, with a 30% better energy resolution obtained for transverse momenta p T > 500 GeV.