Shortcut Connections Research Articles

Second-order transformer network for video recognition

The video recognition community is undergoing a significant change in backbone shifting from CNNs to transformers. However, due to the temporal information existing in the video, vision transformers, which have been shown to be effective in image tasks, cannot model spatio-temporal structure in video recognition. In addition, current transformer-based video models exclusively utilize either class tokens or visual tokens for classification, failing to combine the merits of these two types of tokens. To solve the above problems, we propose a novel video second-order transformer network (ViSoT). As there are complex motion and appearance information in video, visual tokens of the last ViSoT layer are aggregated by cross-covariance pooling to model spatio-temporal information, which is combined with class tokens for classification. Meanwhile, a fast singular value power normalization is further introduced to achieve effective aggregation of visual tokens. For temporal modeling, before ViSoT block temporal convolutions are performed continuously in the early stage after the convolution stem via a short-cut connection. In ViSoT, token shift module and space–time attention are proposed for modeling temporal relations and spatio-temporal interaction across adjacent frames respectively. Validating ViSoT on four video benchmarks demonstrates its effectiveness and efficiency, comparable to or better than state-of-the-art methods with fewer parameters and GFLOPs.

Exploring the Impact of Additive Shortcuts in Neural Networks via Information Bottleneck-like Dynamics: From ResNet to Transformer.

Deep learning has made significant strides, driving advances in areas like computer vision, natural language processing, and autonomous systems. In this paper, we further investigate the implications of the role of additive shortcut connections, focusing on models such as ResNet, Vision Transformers (ViTs), and MLP-Mixers, given that they are essential in enabling efficient information flow and mitigating optimization challenges such as vanishing gradients. In particular, capitalizing on our recent information bottleneck approach, we analyze how additive shortcuts influence the fitting and compression phases of training, crucial for generalization. We leverage Z-X and Z-Y measures as practical alternatives to mutual information for observing these dynamics in high-dimensional spaces. Our empirical results demonstrate that models with identity shortcuts (ISs) often skip the initial fitting phase and move directly into the compression phase, while non-identity shortcut (NIS) models follow the conventional two-phase process. Furthermore, we explore how IS models are still able to compress effectively, maintaining their generalization capacity despite bypassing the early fitting stages. These findings offer new insights into the dynamics of shortcut connections in neural networks, contributing to the optimization of modern deep learning architectures.

Two-stage estimation of hourly diffuse solar radiation across China using end-to-end gradient boosting with sequentially boosted features

Diffuse solar radiation (DR) constitutes a vital component of solar energy reaching the surface of the Earth. The demand for extensive temporal and spatial coverage of DR data has intensified in the realms of solar energy harvesting, agriculture, and climate change. However, until now, long-term DR observations have only been available from 17 stations across mainland China. Consequently, there is a pressing need to estimate spatially continuous, high-temporal-resolution DR for large-scale regions in China. The current hindrance to DR estimations stems from the scarcity of stations equipped with DR observations. This study proposes a two-stage strategy to efficiently estimate seamless DR in 2019 at a national scale, leveraging both DR and total solar radiation (TR) observations from numerous stations. In the first stage, the approach generates virtual DR at TR stations by establishing a learned relationship between DR and TR observations. Subsequently, in the second stage, these virtual DR data, in conjunction with satellite and reanalysis datasets, are utilized to estimate national-scale DR. Additionally, a novel model, End-to-end Gradient Boosting with Shortcuts and Feature selection (EGB-SF), is introduced to estimate DR over China. One advantage of this model is its consideration of the impact of sequentially boosted features and their interactions. Embedded shortcut connections fully exploit the influence of existing features on newly introduced ones during the learning process. Beyond enhancing the accuracy of DR estimation, the EGB-SF algorithm can also elucidate the relative importance levels of input features to the model. Moreover, the two-stage strategy outperforms the method of estimating national DR using only DR observations, as evidenced by its superior spatial generalization abilities. Statistical evaluation, collaborative analysis with influencing factors, and comparisons with related products confirm the accuracy and spatial continuity of the DR estimations in this study. These results furnish reliable DR data across China for research in agriculture, climate, solar radiation, and related fields.

ResGAT: Residual Graph Attention Networks for molecular property prediction

Molecular property prediction is an important step in the drug discovery pipeline. Numerous computational methods have been developed to predict a wide range of molecular properties. While recent approaches have shown promising results, no single architecture can comprehensively address all tasks, making this area persistently challenging and requiring substantial time and effort. Beyond traditional machine learning and deep learning architectures for regular data, several deep learning architectures have been designed for graph-structured data to overcome the limitations of conventional methods. Utilizing graph-structured data in quantitative structure–activity relationship (QSAR) modeling allows models to effectively extract unique features, especially where connectivity information is crucial. In our study, we developed residual graph attention networks (ResGAT), a deep learning architecture for molecular graph-structured data. This architecture is a combination of graph attention networks and shortcut connections to address both regression and classification problems. It is also customizable to adapt to various dataset sizes, enhancing the learning process based on molecular patterns. When tested multiple times with both random and scaffold sampling strategies on nine benchmark molecular datasets, QSAR models developed using ResGAT demonstrated stability and competitive performance compared to state-of-the-art methods.

Application of Virtual Reality Technology Based on Convolutional Neural Network in Digital Media Art Creation

On the basis of the convolutional neural network based on the absolute value layer, a convolutional neural network based on multi-level parallel is proposed. This model adds shortcut connections between different convolutional layers of the convolutional neural network. The shortcut connections allow the feature map output from the shallower convolutional layers in the network to be directly used as branch information, which is similar to the deep convolutional layer. The feature maps are combined to obtain a feature map with more comprehensive information. By using the features of the shallow convolutional layer and the deep convolutional layer at the same time, the embedded traces it learns are more accurate. The final experimental results prove that the multi-level parallel convolutional neural network has a significant improvement in detection performance compared with the rich model art creation analysis algorithm, even when confronting the HILL art creation algorithm with an embedding rate of 0.4bpp.

Multitask Learning on Graph Convolutional Residual Neural Networks for Screening of Multitarget Anticancer Compounds.

Recently, various modern experimental screening pipelines and assays have been developed to find promising anticancer drug candidates. However, it is time-consuming and almost infeasible to screen an immense number of compounds for anticancer activity via experimental approaches. To partially address this issue, several computational advances have been proposed. In this study, we present iACP-GCR, a model based on multitask learning on graph convolutional residual neural networks with two types of shortcut connections, to identify multitarget anticancer compounds. In our architecture, the graph convolutional residual neural networks are shared by all the prediction tasks before being separately customized. The NCI-60 data set, one of the most reliable and well-known sources of experimentally verified compounds, was used to develop our model. From that data set, we collected and refined data about compounds screened across nine cancer types (panels), including breast, central nervous system, colon, leukemia, nonsmall cell lung, melanoma, ovarian, prostate, and renal, for model training and evaluation. The model performance evaluated on an independent test set shows that iACP-GCR surpasses the three advanced computational methods for multitask learning. The integration of two shortcut connection types in the shared networks also improves the prediction efficiency. We also deployed the model as a public web server to assist the research community in screening potential anticancer compounds.

A Lightweight Crop Pest Detection Method Based on Improved RTMDet

To address the issues of low detection accuracy and large model parameters in crop pest detection in natural scenes, this study improves the deep learning object detection model and proposes a lightweight and accurate method RTMDet++ for crop pest detection. First, the real-time object detection network RTMDet is utilized to design the pest detection model. Then, the backbone and neck structures are pruned to reduce the number of parameters and computation. Subsequently, a shortcut connection module is added to the classification and regression branches, respectively, to enhance its feature learning capability, thereby improving its accuracy. Experimental results show that, compared to the original model RTMDet, the improved model RTMDet++ reduces the number of parameters by 15.5%, the computation by 25.0%, and improves the mean average precision by 0.3% on the crop pest dataset IP102. The improved model RTMDet++ achieves a mAP of 94.1%, a precision of 92.5%, and a recall of 92.7% with 4.117M parameters and 3.130G computations, outperforming other object detection methods. The proposed model RTMDet++ achieves higher performance with fewer parameters and computations, which can be applied to crop pest detection in practice and aids in pest control research.

Research on facial expression recognition algorithm based on improved MobileNetV3

Aiming at the problem that face images are easily interfered by occlusion factors in uncontrollable environments, and the complex structure of traditional convolutional neural networks leads to low expression recognition rates, slow network convergence speed, and long network training time, an improved lightweight convolutional neural network is proposed for facial expression recognition algorithm. First, the dilation convolution is introduced into the shortcut connection of the inverted residual structure in the MobileNetV3 network to expand the receptive field of the convolution kernel and reduce the loss of expression features. Then, the channel attention mechanism SENet in the network is replaced by the two-dimensional (channel and spatial) attention mechanism SimAM introduced without parameters to reduce the network parameters. Finally, in the normalization operation, the Batch Normalization of the backbone network is replaced with Group Normalization, which is stable at various batch sizes, to reduce errors caused by processing small batches of data. Experimental results on RaFD, FER2013, and FER2013Plus face expression data sets show that the network reduces the training times while maintaining network accuracy, improves network convergence speed, and has good convergence effects.

A novel multivariate time series forecasting dendritic neuron model for COVID-19 pandemic transmission tendency

A novel coronavirus discovered in late 2019 (COVID-19) quickly spread into a global epidemic and, thankfully, was brought under control by 2022. Because of the virus’s unknown mutations and the vaccine’s waning potency, forecasting is still essential for resurgence prevention and medical resource management. Computational efficiency and long-term accuracy are two bottlenecks for national-level forecasting. This study develops a novel multivariate time series forecasting model, the densely connected highly flexible dendritic neuron model (DFDNM) to predict daily and weekly positive COVID-19 cases. DFDNM’s high flexibility mechanism improves its capacity to deal with nonlinear challenges. The dense introduction of shortcut connections alleviates the vanishing and exploding gradient problems, encourages feature reuse, and improves feature extraction. To deal with the rapidly growing parameters, an improved variation of the adaptive moment estimation (AdamW) algorithm is employed as the learning algorithm for the DFDNM because of its strong optimization ability. The experimental results and statistical analysis conducted across three Japanese prefectures confirm the efficacy and feasibility of the DFDNM while outperforming various state-of-the-art machine learning models. To the best of our knowledge, the proposed DFDNM is the first to restructure the dendritic neuron model’s neural architecture, demonstrating promising use in multivariate time series prediction. Because of its optimal performance, the DFDNM may serve as an important reference for national and regional government decision-makers aiming to optimize pandemic prevention and medical resource management. We also verify that DFDMN is efficiently applicable not only to COVID-19 transmission prediction, but also to more general multivariate prediction tasks. It leads us to believe that it might be applied as a promising prediction model in other fields.

The weighted multi-scale connections networks for macrodispersivity estimation

Macrodispersivity is critical for predicting solute behaviors with dispersive transport models. Conventional methods of estimating macrodispersivity usually need to solve flow equations and are time-consuming. Convolutional neural networks (CNN) have recently been proven capable of efficiently mapping the hydraulic conductivity field and macrodispersivity. However, the mapping accuracy still needs further improvement. In this paper, we present a new network shortcut connection style called weighted multi-scale connections (WMC) for convolutional neural networks to improve mapping accuracy. We provide empirical evidence showing that the WMC can improve the performance of CNN in macrodispersivity estimation by implementing the WMC in CNNs (CNN without short-cut connections, ResNet, and DenseNet), and evaluating them on datasets of macrodispersivity estimation. For the CNN without short-cut connections, the WMC can improve the estimating R2 by at least 3% on three datasets of conductivity fields. For ResNet18, the WMC improved the estimated R2 by an average of 2.5% on all three datasets. For ResNet34, the WMC improved the estimated R2 by an average of 5.6%. For ResNet50, the WMC improved the estimated R2 by an average of 16%. For ResNet101, the WMC improved the estimating R2 by an average of 30%. For DenseNets, the improved estimated R2 ranges from 0.5% to 5%. The WMC can strengthen feature propagation of different sizes and alleviate the vanishing-gradient issue. Moreover, it can be implemented to any CNN with down-sampling layers or blocks.

Whisper-SV: Adapting Whisper for low-data-resource speaker verification

Trained on 680,000 h of massive speech data, Whisper is a multitasking, multilingual speech foundation model demonstrating superior performance in automatic speech recognition, translation, and language identification. However, its applicability in speaker verification (SV) tasks remains unexplored, particularly in low-data-resource scenarios where labeled speaker data in specific domains are limited. To fill this gap, we propose a lightweight adaptor framework to boost SV with Whisper, namely Whisper-SV. Given that Whisper is not specifically optimized for SV tasks, we introduce a representation selection module to quantify the speaker-specific characteristics contained in each layer of Whisper and select the top-k layers with prominent discriminative speaker features. To aggregate pivotal speaker-related features while diminishing non-speaker redundancies across the selected top-k distinct layers of Whisper, we design a multi-layer aggregation module in Whisper-SV to integrate multi-layer representations into a singular, compacted representation for SV. In the multi-layer aggregation module, we employ convolutional layers with shortcut connections among different layers to refine speaker characteristics derived from multi-layer representations from Whisper. In addition, an attention aggregation layer is used to reduce non-speaker interference and amplify speaker-specific cues for SV tasks. Finally, a simple classification module is used for speaker classification. Experiments on VoxCeleb1, FFSVC, and IMSV datasets demonstrate that Whisper-SV achieves EER/minDCF of 2.22%/0.307, 6.14%/0.488, and 7.50%/0.582, respectively, showing superior performance in low-data-resource SV scenarios.

Bearing Fault Vibration Signal Denoising Based on Adaptive Denoising Autoencoder

Vibration signal analysis is regarded as a fundamental approach in diagnosing faults in rolling bearings, and recent advancements have shown notable progress in this domain. However, the presence of substantial background noise often results in the masking of these fault signals, posing a significant challenge for researchers. In response, an adaptive denoising autoencoder (ADAE) approach is proposed in this paper. The data representations are learned by the encoder through convolutional layers, while the data reconstruction is performed by the decoder using deconvolutional layers. Both the encoder and decoder incorporate adaptive shrinkage units to simulate denoising functions, effectively removing interfering information while preserving sensitive fault features. Additionally, dropout regularization is applied to sparsify the network and prevent overfitting, thereby enhancing the overall expressive power of the model. To further enhance ADAE’s noise resistance, shortcut connections are added. Evaluation using publicly available datasets under scenarios with known and unknown noise demonstrates that ADAE effectively enhances the signal-to-noise ratio in strongly noisy backgrounds, facilitating accurate diagnosis of faults in rolling bearings.

An attention mechanism and residual network based knowledge graph-enhanced recommender system

Recommender systems enhanced by a knowledge graph (KG) have attained widespread popularity and attention in recent years. However, traditional KG-based recommender systems encounter the challenge of gradient explosion as the network depth increases. Additionally, the abundance of unreliable paths in a KG has a detrimental impact on feature representation learning. In this article, we propose a KG-enhanced recommender system based on residual network and attention mechanism, which can capture high-order connectivity and long-range dependencies of the KG. Specifically, a resource allocation approach is employed to calculate the resource amount, which is subsequently utilized to evaluate the path reliability of the KG. After completing path extraction, we employ an attention mechanism to capture semantic correlations and structural information. To leverage the KG for enhancing recommender systems, we design a deep residual network with shortcut connections, effectively amalgamating advanced and abstract features using deep neural networks. The introduction of shortcut connections not only facilitates the fitting of residual mappings but also mitigates potential issues such as gradient explosion and convergence difficulties due to excessive network depth. Extensive experiments conducted on three standard datasets over baseline methods have demonstrated the superiority of our proposed recommender system.

Discovering physics-informed neural networks model for solving partial differential equations through evolutionary computation

In recent years, the researches about solving partial differential equations (PDEs) based on artificial neural network have attracted considerable attention. In these researches, the neural network models are usually designed depend on human experience or trial and error. Despite the emergence of several model searching methods, these methods primarily concentrate on optimizing the hyperparameters of fully connected neural network model based on the framework of physics-informed neural networks (PINNs), and the corresponding search spaces are relatively restricted, thereby limiting the exploration of superior models. This article proposes an evolutionary computation method aimed at discovering the PINNs model with higher approximation accuracy and faster convergence rate. In addition to searching the numbers of layers and neurons per hidden layer, this method concurrently explores the optimal shortcut connections between the layers and the novel parametric activation functions expressed by the binary trees. In evolution, the strategy about dynamic population size and training epochs (DPSTE) is adopted, which significantly increases the number of models to be explored and facilitates the discovery of models with fast convergence rate. In experiments, the performance of different models that are searched through Bayesian optimization, random search and evolution is compared in solving Klein–Gordon, Burgers, and Lamé equations. The experimental results affirm that the models discovered by the proposed evolutionary computation method generally exhibit superior approximation accuracy and convergence rate, and these models also show commendable generalization performance with respect to the source term, initial and boundary conditions, equation coefficient and computational domain. The corresponding code is available at https://github.com/MathBon/Discover-PINNs-Model.

Accurate Blood Pressure Measurement Using Smartphone's Built-in Accelerometer

Efficient blood pressure (BP) monitoring in everyday contexts stands as a substantial public health challenge that has garnered considerable attention from both industry and academia. Commercial mobile phones have emerged as a promising tool for BP measurement, benefitting from their widespread popularity, portability, and ease of use. Most mobile phone-based systems leverage a combination of the built-in camera and LED to capture photoplethysmography (PPG) signals, which can be used to infer BP by analyzing the blood flow characteristics. However, due to low Signal-to-Noise (SNR), various factors such as finger motion, improper finger placement, skin tattoos, or fluctuations in environmental lighting can distort the PPG signal. These distortions consequentially affect the performance of BP estimation. In this paper, we introduce a novel sensing system that utilizes the built-in accelerometer of a mobile phone to capture seismocardiography (SCG) signals, enabling accurate BP measurement. Our system surpasses previous mobile phone-based BP measurement systems, offering advantages such as high SNR, ease of use, and power efficiency. We propose a triple-stage noise reduction scheme, integrating improved complete ensemble empirical mode decomposition with adaptive noise (ICEEMDAN), recursive least squares (RLS) adaptive filter, and soft-thresholding, to effectively reconstruct high-quality heartbeat waveforms from initially contaminated raw SCG signals. Moreover, we introduce a data augmentation technique encompassing normalization coupled with temporal-sliding, effectively augmenting the diversity of the training sample set. To enable battery efficiency on smartphone, we propose a resource-efficient deep learning model that incorporates resource-efficient convolution, shortcut connections, and Huber loss. We conduct extensive experiments with 70 volunteers, comprising 35 healthy individuals and 35 individuals diagnosed with hypertension, under a user-independent setting. The excellent performance of our system demonstrates its capacity for robust and accurate daily BP measurement.

A novel reconstruction method for displacement missing data of arch dam via hierarchical clustering and deep learning

The absence of displacement monitoring data presents a challenge to real-time dam safety monitoring. This article introduces a novel method for reconstructing missing displacements, comprehensively taking into account the spatiotemporal correlation and causal effect mechanism of displacement. Firstly, adaptive weighted derivative dynamic time warping (AWDDTW) and adaptive weighted dynamic time warping (AWDTW), in conjunction with clustering algorithms, are proposed to extract spatiotemporal correlation among dam displacements. Secondly, by stacking residual blocks within the residual network and densifying the shortcut connections between residual blocks, the deep stacked residual network (DS-ResNet) is established to effectively capture the intricate mappings between displacements and various factors. Finally, using two arch dams, as examples, we simulated scenarios of continuous long-term and multiple short-term missing data to validate the new method proposed in this study. The results indicate that the proposed clustering algorithm can accurately compute the similarity between displacement sequences of different lengths and monitoring frequencies, thereby providing more precise displacement point partitioning results. Compared to models such as random forests, relevance vector machines, multilayer perceptron, ResNet and Transformer, the DS-ResNet demonstrates superior capability in identifying and extracting effective information from displacement sequences under both continuous long-term and multiple short-term missing data scenarios. Compared to methods that solely consider spatiotemporal correlation or causal effect mechanism of displacement, the proposed method can more comprehensively reflect the operating patterns of arch dams and more accurately reconstruct missing displacement values. The research presented herein offers new technical means and solution approaches for dam safety monitoring and operational management.

SingleS2R: Single sample driven Sim-to-Real transfer for Multi-Source Visual-Tactile Information Understanding using multi-scale vision transformers

Due to variations in light transmission and wear on the contact head, existing visual-tactile dataset building methods typically require a large amount of real-world data, making the dataset building process time-consuming and labor-intensive. Sim-to-Real learning has been proposed to realize Multi-Source Visual-Tactile Information Understanding (MSVTIU) in simulate and real environment, which can efficiently promote visual-tactile dataset building using simulation method for emerged robotic applications. However, the existing Sim-to-Real learning also requires more than 10,000 real data, while the corresponding data need to be re-collected when the sensor version changes. To address this challenge, we propose a powerful Sim-to-Real transfer for MSVTIU which requires only one single real-world tactile sample. To effectively extract features from the single real tactile sample, a multi-scale vision transformers-based Generative Adversarial Network (GAN) is proposed to address MSVTIU task under extremely limited data. We introduce a novel scale-dependent self-attention mechanism that allows attention layers to adapt their behavior at different stages of the generating process. In addition, we introduced a residual block for capturing contextual information between adjacent scales, which utilizes shortcut connections to fully preserve texture and structure information. We subsequently enhanced the model understanding of visual-tactile information using elastic transform and adaptive adversarial training strategy, both of which are designed specifically for MSVTIU. Experiments on two public datasets with diverse objects indicate that our Sim-to-Real transfer approach utilizing only one single real-world visual-tactile sample outperforms the state-of-the-art methods that requires tens of thousands of samples.

Automatic extraction of non-bifurcated lunar lineaments from Lunar Reconnaissance Orbiter (LRO) monochromatic images based on a Markov chain method

ABSTRACT Lineaments play a critical role in the study of lunar geological evolution. However, lineaments still need to be extracted manually which constrains lunar research. To fill this gap, a Markov chain-based methodology is introduced for automatically extracting vector-based non-bifurcated lineaments from images. The extracted lineaments are formed by successive nodes and line segments, where nodes denote pixel positions and line segments represented connecting lines between nodes. A modified U-net with residual shortcut connections and dilated convolutions is proposed based on the elongated shapes of lineaments to evaluate the probability of nodes belonging to lineaments. Four connection shape features are proposed to describe the connection shapes of nodes and a Gaussian Mixture Model is used to evaluate the probability of line segments belonging to the lineaments based on the proposed features. In the final stage, the lineament probabilities of both nodes and line segments to extract lineaments are considered in the Markov chain. Our method has experimented on a dataset containing 220 samples and 10-fold cross-validation was used to evaluate the performance. Both qualitative and quantitative results indicated that our method can effectively extract non-bifurcated lunar lineaments with arbitrary bending. The method sheds useful light on automatic lineament extraction.

ERNet : Enhanced ResNet for classification of breast histopathological images

Inspite of expeditious approaches in field of breast cancer, histopathological analysis is considered as gold standard in diagnosis of cancer. Researchers are working tremendously to automate the detection and analysis of breast histology images, which confess in improving the accuracy and also induce the mimisation of processing time. Deep learning models are providing greater contribution in solving several image classification tasks. In this paper we propose a model to classify breast histological images, which is redesigned from existing ResNet architecture that minimises model parameters and increase computational efficiency. This approach uses enhanced ResNet connection instead of identity shortcut connection used in ResNet architecture. We apply our proposed method on BreakHis dataset and achieve an accuracy around 95.92 %. The numerical results show that our proposed approach outperforms the previous methods with respect to sensitivity and accuracy.

Face recognition technology based on ResNet-50

Face recognition technology is progressively finding its place across diverse domains. In pursuit of enhancing the efficacy of face recognition systems, this study employs a ResNet-50 deep convolutional neural network. The dataset is meticulously gathered and processed via OpenCV, thus amplifying the precision and utility of face recognition. ResNet, an advanced convolutional neural network, incorporates the concept of residual connections, bridging convolutional layers through shortcut connections. These connections facilitate the addition of input to output, forming residual blocks. Consequently, ResNet-50 efficiently tackles the vanishing gradient issue, enabling the training of exceptionally deep networks. With 49 convolutional layers and a fully connected layer, ResNet-50 boasts a robust architecture. To emulate varying brightness conditions, post-collection image adjustments are applied randomly. This strategy curbs the impact of divergent lighting scenarios on recognition accuracy, bolstering the models practical applicability. Notably, experimental outcomes underscore the commendable performance of the trained ResNet-50 model in face recognition trials. This substantiates the broad-spectrum viability of face recognition technology in domains such as security surveillance, human-machine interaction, identity verification, and beyond.