Long-range Dependence Research Articles

Deep Learning Assisted Diagnosis of Chronic Obstructive Pulmonary Disease Based on a Local-to-Global Framework

To aid the diagnosis of chronic obstructive pulmonary disease (COPD), a local-to-global deep framework with group attentions and slice-aware loss is designed in this paper, which utilizes the chest CT sequences of the patients as the network input. To fully mine the medical hints submerged in the CT slices, two types of group attentions are designed to extract local–global features of the grouped slices. Specifically, in each group, a group local attention block (GLAB) and a group global attention block (GGAB) are designed to extract local features in the CT slices and long-range dependencies among the grouped slices. To alleviate the influence of different numbers of CT slices in the chest CT sequences for different patients, a slice-aware loss is proposed by incorporating a normalized coefficient into the cross-entropy loss. Experimental results indicate that the designed deep model performs a good COPD identification on a real COPD dataset with 96.08% accuracy, 94.12% sensitivity, 97.06% specificity, and 95.32% AUC, which is superior to some existing deep learning methods.

QAGA-Net: enhanced vision transformer-based object detection for remote sensing images

PurposeVision transformers (ViT) detectors excel in processing natural images. However, when processing remote sensing images (RSIs), ViT methods generally exhibit inferior accuracy compared to approaches based on convolutional neural networks (CNNs). Recently, researchers have proposed various structural optimization strategies to enhance the performance of ViT detectors, but the progress has been insignificant. We contend that the frequent scarcity of RSI samples is the primary cause of this problem, and model modifications alone cannot solve it.Design/methodology/approachTo address this, we introduce a faster RCNN-based approach, termed QAGA-Net, which significantly enhances the performance of ViT detectors in RSI recognition. Initially, we propose a novel quantitative augmentation learning (QAL) strategy to address the sparse data distribution in RSIs. This strategy is integrated as the QAL module, a plug-and-play component active exclusively during the model’s training phase. Subsequently, we enhanced the feature pyramid network (FPN) by introducing two efficient modules: a global attention (GA) module to model long-range feature dependencies and enhance multi-scale information fusion, and an efficient pooling (EP) module to optimize the model’s capability to understand both high and low frequency information. Importantly, QAGA-Net has a compact model size and achieves a balance between computational efficiency and accuracy.FindingsWe verified the performance of QAGA-Net by using two different efficient ViT models as the detector’s backbone. Extensive experiments on the NWPU-10 and DIOR20 datasets demonstrate that QAGA-Net achieves superior accuracy compared to 23 other ViT or CNN methods in the literature. Specifically, QAGA-Net shows an increase in mAP by 2.1% or 2.6% on the challenging DIOR20 dataset when compared to the top-ranked CNN or ViT detectors, respectively.Originality/valueThis paper highlights the impact of sparse data distribution on ViT detection performance. To address this, we introduce a fundamentally data-driven approach: the QAL module. Additionally, we introduced two efficient modules to enhance the performance of FPN. More importantly, our strategy has the potential to collaborate with other ViT detectors, as the proposed method does not require any structural modifications to the ViT backbone.

Retinal artery/vein vessel segmentation and measurement for hypertension based on multi-level edge-guided network

AbstractHypertension is a primary risk factor for the onset of cardiocerebrovascular diseases, leading to increased mortality. In the early stages of hypertension, changes in the diameter of the retinal arteries and veins are closely observed. Therefore, the automatic segmentation of these arteries and veins in retinal fundus images is crucial for hypertension monitoring. However, current deep learning based methods still fail to generate semantically consistent segmentation result, especially for narrow blood vessel segments. Moreover, the pixels near the vessel edge are prone to be missclassified by current methods, struggling to obtain edge-preserving segmentation results. These critical issues severely hinder downstream applications, which require accurate measurement of artery/vein diameter. First, to alleviate the issue of lacking semantic consistency for vessel segments, we propose a long-range spatial dependency modeling module to learn to model the long-range spatial dependency. On this basis, a multi-level edge guided spatial aggregation module is further presented to enhance the ability to accurately classify the edge pixels, generating edge-preserving results. Extensive experimental results on the widely used DRIVE dataset and our constructed dataset (HFC-50) show the superiority of the proposed model over state-of-the-art methods. Our method achieves balanced accuracy of 95.7% on DRIVE dataset, outperforming all state-of-art methods. Finally, in order to directly demonstrate the benefit of more accurate retinal artery/vein vessel segmentation for the measurement of the ratio of artery and vein, we quantitatively evaluate the computed artery/vein ratio for hypertension patients, with the artery/vein segmentation results generated by our proposed method.

Asymmetric Large Kernel Distillation Network for efficient single image super-resolution

Recently, significant advancements have been made in the field of efficient single-image super-resolution, primarily driven by the innovative concept of information distillation. This method adeptly leverages multi-level features to facilitate high-resolution image reconstruction, allowing for enhanced detail and clarity. However, many existing approaches predominantly emphasize the enhancement of distilled features, often overlooking the critical aspect of improving the feature extraction capabilities of the distillation module itself. In this paper, we address this limitation by introducing an asymmetric large-kernel convolution design. By increasing the size of the convolution kernel, we expand the receptive field, which enables the model to more effectively capture long-range dependencies among image pixels. This enhancement significantly improves the model's perceptual ability, leading to more accurate reconstructions. To maintain a manageable level of model complexity, we adopt a lightweight architecture that employs asymmetric convolution techniques. Building on this foundation, we propose the Lightweight Asymmetric Large Kernel Distillation Network (ALKDNet). Comprehensive experiments conducted on five widely recognized benchmark datasets-Set5, Set14, BSD100, Urban100, and Manga109-indicate that ALKDNet not only preserves efficiency but also demonstrates performance enhancements relative to existing super-resolution methods. The average PSNR and SSIM values show improvements of 0.10 dB and 0.0013, respectively, thereby achieving state-of-the art performance.

SSDSNet: A Dual-Stream State-Space Network for Efficient Image Denoising with Long-Range Dependency Modeling

Abstract In recent years, significant progress has been made in image denoising, driven by advancements in modern deep neural networks such as Convolutional Neural Networks
(CNNs) and Transformers. However, existing denoising models often face limitations due to inherent local inductive biases or the quadratic computational complexity of their architectures. Recently, Selective Structured State Space Models, exemplified by Mamba, have shown considerable potential in modeling long-range dependencies with linear complexity. Despite this promise, their application in low-level computer vision tasks remains underexplored. In this study, we introduce a straightforward yet effective model, the Dual-Stream State-Space Module (DSSM), specifically designed for image denoising. The DSSM serves as the core component of our model, combining convolutional operations with the
Mamba model to enhance the capability of capturing both local features and global context.
This dual-stream approach leverages the inherent structure of local image patches while effectively modeling long-range dependencies, resulting in feature representations tailored for denoising tasks. Extensive experiments demonstrate the effectiveness of our method. For instance, on the SIDD dataset, our model achieves performance on par with state-of-the-art methods, maintaining a global receptive field while operating with similar computational costs.

A Multi-Scale Convolutional Neural Network with Self-Knowledge Distillation for Bearing Fault Diagnosis

Efficient bearing fault diagnosis not only extends the operational lifespan of rolling bearings but also reduces unnecessary maintenance and resource waste. However, current deep learning-based methods face significant challenges, particularly due to the scarcity of fault data, which impedes the models’ ability to effectively learn parameters. Additionally, many existing methods rely on single-scale features, hindering the capture of global contextual information and diminishing diagnostic accuracy. To address these challenges, this paper proposes a Multi-Scale Convolutional Neural Network with Self-Knowledge Distillation (MSCNN-SKD) for bearing fault diagnosis. The MSCNN-SKD employs a five-stage architecture. Stage 1 uses wide-kernel convolution for initial feature extraction, while Stages 2 through 5 integrate a parallel multi-scale convolutional structure to capture both global contextual information and long-range dependencies. In the final two stages, a self-distillation process enhances learning by allowing deep-layer features to guide shallow-layer learning, improving performance, especially in data-limited scenarios. Extensive experiments on multiple datasets validate the model’s high diagnostic accuracy, computational efficiency, and robustness, demonstrating its suitability for real-time industrial applications in resource-limited environments.

Refined feature enhancement network for object detection

Convolutional neural networks-based object detection techniques have achieved positive performances. However, due to the limitations of local receptive field, some existing object detection methods cannot effectively capture global information in feature extraction phases, and thus lead to unsatisfactory detection performance. Moreover, the feature information extracted by the backbone network may be redundant. To alleviate these problems, in this paper we propose a refined feature enhancement network (RFENet) for object detection. Specifically, we first propose a feature enhancement module (FEM) to capture more global and local information from feature maps with certain long-range dependencies. We further propose a multi-branch dilated attention mechanism (MDAM) to refine the extracted features in a weighted form, which can select more important spatial and channel information and broaden the receptive field of the network. Finally, we validate RFENet on MS-COCO2017, PASCAL VOC2012, and PASCAL VOC07+12 datasets, respectively. Compared to the baseline network, our RFENet improves by 2.4 AP on MS-COCO2017 dataset, 3.4 mAP on PASCAL VOC2012 dataset, and 2.7 mAP on PASCAL VOC07+12 dataset. Extensive experiments show that our RFENet can perform competitively on different datasets. The code is available at https://github.com/object9detection/RFENet.

A Transfer Learning Model for Rolling Bearing Fault Diagnosis Based on Texture Loss Strategy and Nuclear Norm Regularization

Abstract In recent years, transfer learning (TL) approaches have seen extensive application in diagnosing bearing faults due to their exceptional performance. However, mechanical noise, equipment aging, and wear lead to notable disparities and differences in the multi-level feature distributions across the source and target domain signals. The issue is addressed by proposing a transfer learning model based on a texture loss strategy and nuclear norm regularization method. First, a Feature-Enhanced Network (MSRnet) is designed, which significantly improves the ability to capture local details and long-range dependencies by combining a multi-scale feature extraction module with a dilated residual module. Next, a texture loss strategy is proposed to align multi-scale features across domains by minimizing the Gram matrix of signal features. Finally, a nuclear norm regularization method is proposed to perform low-rank approximation on the signal matrix, facilitating the extraction of more robust feature data and mitigating the risk of overfitting. The experimental results demonstrate that the proposed method achieved an average accuracy of 98.58% on the University of Ottawa bearing fault dataset and 98.11% on the Jiangnan University (JNU) bearing dataset, surpassing eight other algorithms in bearing fault diagnosis.

Neural Machine Translation (NMT): Deep learning approaches through Neural Network Models

Abstract. This paper explores the significant advancements in Neural Machine Translation (NMT) models, focusing on the impact of different architectures, training methodologies, and optimization techniques on translation quality. The study contrasts the performance of Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and the Transformer model, highlighting the superior capabilities of the Transformer in handling long-range dependencies and providing contextually accurate translations. Key optimization techniques, such as learning rate scheduling, dropout regularization, and gradient clipping, are discussed in detail, emphasizing their roles in enhancing model performance and training efficiency. Furthermore, the paper presents a comparative analysis of NMT and traditional Statistical Machine Translation (SMT) systems, showcasing NMT's superior BLEU scores and fluency. The application of model distillation is also examined, demonstrating how smaller models can achieve high performance with reduced computational resources. These findings underscore the transformative potential of NMT in achieving state-of-the-art translation quality and efficiency.

Advancements in Image Recognition: Comparing CNNs and Vision Transformers

Abstract. This paper explores advancements in image recognition technologies, highlighting the shift from conventional methodologies to contemporary deep learning techniques, specifically focusing on Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). The study examines key architectures including CNNs, and various Transformer-based models, analyzing their performance evaluating their effectiveness in diverse tasks such as image classification, object detection, and facial recognition. The research highlights the strengths and limitations of CNNs and ViTs, focusing on their ability to handle complex and diverse datasets. A detailed comparative analysis is conducted, emphasizing performance metrics, robustness, and adaptability across different image recognition scenarios. The results reveal that while CNNs excel in traditional image processing tasks, ViTs demonstrate significant improvements in capturing long-range dependencies, thereby enhancing recognition accuracy in more complex contexts. This analysis offers critical perspectives on selecting and applying image recognition models, guiding future exploration and practical use in various industries. It underscores the impact of deep learning innovations on advancing image recognition capabilities and highlights potential directions for ongoing development in the field.

Multiscale attention-over-attention network for retinal disease recognition in OCT radiology images

Retinal disease recognition using Optical Coherence Tomography (OCT) images plays a pivotal role in the early diagnosis and treatment of conditions. However, the previous attempts relied on extracting single-scale features often refined by stacked layered attentions. This paper presents a novel deep learning-based Multiscale Feature Enhancement via a Dual Attention Network specifically designed for retinal disease recognition in OCT images. Our approach leverages the EfficientNetB7 backbone to extract multiscale features from OCT images, ensuring a comprehensive representation of global and local retinal structures. To further refine feature extraction, we propose a Pyramidal Attention mechanism that integrates Multi-Head Self-Attention (MHSA) with Dense Atrous Spatial Pyramid Pooling (DASPP), effectively capturing long-range dependencies and contextual information at multiple scales. Additionally, Efficient Channel Attention (ECA) and Spatial Refinement modules are introduced to enhance channel-wise and spatial feature representations, enabling precise localization of retinal abnormalities. A comprehensive ablation study confirms the progressive impact of integrated blocks and attention mechanisms that enhance overall performance. Our findings underscore the potential of advanced attention mechanisms and multiscale processing, highlighting the effectiveness of the network. Extensive experiments on two benchmark datasets demonstrate the superiority of the proposed network over existing state-of-the-art methods.

DeepAge: Harnessing Deep Neural Network for Epigenetic Age Estimation From DNA Methylation Data of Human Blood Samples

Accurate prediction of biological age from DNA methylation data is a critical endeavor in understanding the molecular mechanisms of aging and developing age-related disease interventions. Traditional epigenetic clocks rely on linear regression or basic machine learning models, which often fail to capture the complex, non-linear interactions within methylation data. This study introduces DeepAge, a novel deep learning framework utilizing Temporal Convolutional Networks (TCNs) to enhance the prediction of biological age from DNA methylation profiles using selected CpGs by a Dual-Correlation based apparoach. DeepAge leverages a sequence-based approach with dilated convolutions to effectively capture long-range dependencies between CpG sites, addressing the limitations of prior models by incorporating advanced network architectures including residual connections and dropout regularization. The dual correlation feature selection enhances our model's predictive capabilities by identifying the most age-relevant CpG sites. Our model outperforms existing epigenetic clocks across multiple datasets, offering significant improvements in accuracy and providing deeper insights into the epigenetic determinants of aging. The proposed method not only sets a new standard in age estimation but also highlights the potential of deep learning in biologically relevant feature extraction and interpretation, contributing to the broader field of computational biology and precision medicine.

DIRformer: A Novel Image Restoration Approach Based on U-shaped Transformer and Diffusion Models

Image restoration (IR) involves the retrieval of missing or damaged image information and represents a significant challenge in the field of visual reconstruction. Currently, U-Net based Diffusion Models (DMs) display favorable results when utilized for IR tasks. However, the Diffusion Model (DM) based on U-Net demonstrates shortcomings in capturing the global context for IR. To address this issue, we propose a Novel Image Restoration Approach Based on U-shaped Transformer and Diffusion Models (DIRformer). DIRformer enhances the modeling capacity for long-range dependencies within DMs. In particular, DIRformer replaces the traditional U-Net downsampling with Patch merging, dedicated to improving detail preservation, and replaces upsampling with Dual up-sample, strategically designed to alleviate checkerboard artifacts. Besides, as a lightweight and versatile transformer-based solution for IR, DIRformer incorporates time and degradation mapping into the transformer design, all while preserving the fundamental U-shaped structural framework. We assess the efficacy of DIRformer in a multi-tasking IR setting across four datasets. The experimental performance illustrates that DIRformer achieves competitive performance on distortion metrics, including PSNR and SSIM. Remarkably, our proposed approach is almost 25× smaller and 2× faster than the existing methods while achieving comparable high performance.

Study on the Evolutionary Characteristics of Post-Fire Forest Recovery Using Unmanned Aerial Vehicle Imagery and Deep Learning: A Case Study of Jinyun Mountain in Chongqing, China

Forest fires pose a significant threat to forest ecosystems, with severe impacts on both the environment and human society. Understanding the post-fire recovery processes of forests is crucial for developing strategies for species diversity conservation and ecological restoration and preventing further damage. The present study proposes applying the EAswin-Mask2former model based on semantic segmentation in deep learning using visible light band data to better monitor the evolution of burn areas in forests after fires. This model is an improvement of the classical semantic segmentation model Mask2former and can better adapt to the complex environment of burned forest areas. This model employs Swin-Transformer as the backbone for feature extraction, which is particularly advantageous for processing high-resolution images. It also includes the Contextual Transformer (CoT) Block to better capture contextual information capture and incorporates the Efficient Multi-Scale Attention (EMA) Block into the Efficiently Adaptive (EA) Block to enhance the model’s ability to learn key features and long-range dependencies. The experimental results demonstrate that the EAswin-Mask2former model can achieve a mean Intersection-over-Union (mIoU) of 76.35% in segmenting complex forest burn areas across different seasons, representing improvements of 3.26 and 0.58 percentage points, respectively, over the Mask2former models using ResNet and Swin-Transformer backbones, respectively. Moreover, this method surpasses the performance of the DeepLabV3+ and Segformer models by 4.04 and 1.75 percentage points, respectively. Ultimately, the proposed model offers excellent segmentation performance for both forest and burn areas and can effectively track the evolution of burned forests when combined with unmanned aerial vehicle (UAV) remote sensing images.

System and transcript dynamics of cells infected with severe acute respiratory syndrome virus 2 (SARS-CoV-2)

Statistical laws arise in many complex systems and can be explored to gain insights into their structure and behavior. Here, we investigate the dynamics of cells infected with severe acute respiratory syndrome virus 2 (SARS-CoV-2) at the system and individual gene levels; and demonstrate that the statistical frameworks used here are robust in spite of the technical noise associated with single-cell RNA sequencing (scRNA-seq) data. A biphasic fit to Taylor’s power law was observed, and it is likely associated with the larger sampling noise inherent to the measure of less expressed genes. The type of the distribution of the system, as assessed by Taylor’s parameters, varies along the course of infection in a cell type-dependent manner, but also sampling noise had a significant influence on Taylor’s parameters. At the individual gene level, we found that genes that displayed signals of punctual rank stability and/or long-range dependence behavior, as measured by Hurst exponents, were associated with translation, cellular respiration, apoptosis, protein-folding, virus processes, and immune response. Those genes were analyzed in the context of a protein-protein interaction network to find possible therapeutic targets.

Deep-Sea Fauna Segmentation: A Comparative Analysis of Convolutional and Vision Transformer Architectures at Lucky Strike Vent Field

Abstract. Due to recent technological developments, the acquisition and availability of deep-sea imagery has increased exponentially in the last years, leading to an increasing backlog in image annotation and processing, attributable to limited specialized human resources. In this work, we investigate the performance of well-established convolutional neural networks and Vision Transformer (ViT) based architectures, namely, DeepLabv3+ and UNETR, for the segmentation of fauna in deep-sea images. The dataset consists of images captured at the Lucky Strike Vent field, located on the mid-Atlantic ridge, of three edifices named Montsegur, White Castle, and Eiffel Tower. Our experimental investigation reveals that the Vision Transformer consistently outperforms the fully convolutional deep learning architecture, by approximately 14% in terms of F1-Score, demonstrating the effectiveness of ViTs in capturing intricate patterns and long-range dependencies present in deep-sea imagery. Our findings highlight the potential of ViTs as a promising approach for accurate semantic segmentation in challenging environmental contexts, paving the way for improved understanding and analysis of deep-sea ecosystems.

Local-Global Feature Adaptive Fusion Network for Building Crack Detection.

Cracks represent one of the most common types of damage in building structures and it is crucial to detect cracks in a timely manner to maintain the safety of the buildings. In general, tiny cracks require focusing on local detail information while complex long cracks and cracks similar to the background require more global features for detection. Therefore, it is necessary for crack detection to effectively integrate local and global information. Focusing on this, a local-global feature adaptive fusion network (LGFAF-Net) is proposed. Specifically, we introduce the VMamba encoder as the global feature extraction branch to capture global long-range dependencies. To enhance the ability of the network to acquire detailed information, the residual network is added as another local feature extraction branch, forming a dual-encoding network to enhance the performance of crack detection. In addition, a multi-feature adaptive fusion (MFAF) module is proposed to integrate local and global features from different branches and facilitate representative feature learning. Furthermore, we propose a building exterior wall crack dataset (BEWC) captured by unmanned aerial vehicles (UAVs) to evaluate the performance of the proposed method used to identify wall cracks. Other widely used public crack datasets are also utilized to verify the generalization of the method. Extensive experiments performed on three crack datasets demonstrate the effectiveness and superiority of the proposed method.

Two-Stream Modality-Based Deep Learning Approach for Enhanced Two-Person Human Interaction Recognition in Videos.

Human interaction recognition (HIR) between two people in videos is a critical field in computer vision and pattern recognition, aimed at identifying and understanding human interaction and actions for applications such as healthcare, surveillance, and human-computer interaction. Despite its significance, video-based HIR faces challenges in achieving satisfactory performance due to the complexity of human actions, variations in motion, different viewpoints, and environmental factors. In the study, we proposed a two-stream deep learning-based HIR system to address these challenges and improve the accuracy and reliability of HIR systems. In the process, two streams extract hierarchical features based on the skeleton and RGB information, respectively. In the first stream, we utilised YOLOv8-Pose for human pose extraction, then extracted features with three stacked LSM modules and enhanced them with a dense layer that is considered the final feature of the first stream. In the second stream, we utilised SAM on the input videos, and after filtering the Segment Anything Model (SAM) feature, we employed integrated LSTM and GRU to extract the long-range dependency feature and then enhanced them with a dense layer that was considered the final feature for the second stream module. Here, SAM was utilised for segmented mesh generation, and ImageNet was used for feature extraction from images or meshes, focusing on extracting relevant features from sequential image data. Moreover, we newly created a custom filter function to enhance computational efficiency and eliminate irrelevant keypoints and mesh components from the dataset. We concatenated the two stream features and produced the final feature that fed into the classification module. The extensive experiment with the two benchmark datasets of the proposed model achieved 96.56% and 96.16% accuracy, respectively. The high-performance accuracy of the proposed model proved its superiority.

AIFormer: Adaptive Interaction Transformer for 3D Point Cloud Understanding

Recently, significant advancements have been made in 3D point cloud analysis by leveraging transformer architecture in 3D space. However, it remains challenging to effectively implement local and global learning within irregular and sparse structures of 3D point clouds. This paper presents the Adaptive Interaction Transformer (AIFormer), a novel hierarchical transformer architecture designed to enhance 3D point cloud analysis by fusing local and global features through the adaptive interaction of features. Specifically, AIFormer mainly consists of several stacked AIFormer Blocks. Each AIFormer module employs the Local Relation Aggregation Module and the Global Context Aggregation Module, respectively, to extract local details of relationships within the reference point and long-range dependencies between reference points. Then, the local and global features are fused using the Adaptive Interaction Module for adaptive interaction to optimize the point representation. Additionally, the AIFormer Block further designs geometric relation functions and contextual relative semantic encoding to enhance local and global feature extraction capabilities, respectively. Extensive experiments on three popular 3D point cloud datasets verify that AIFormer achieves state-of-the-art or comparable performances. Our comprehensive ablation study further validates the effectiveness and soundness of the AIFormer design.

Robust multi-stage progressive autoencoder for hyperspectral anomaly detection

Recently, Autoencoders (AEs) have demonstrated remarkable performance in the field of hyperspectral anomaly detection, owing to their powerful capability in handling high-dimensional data. However, they often overlook the inherent global distribution characteristics and long-range dependencies in hyperspectral images (HSI). This oversight makes it challenging to accurately characterize and describe boundaries between different backgrounds and anomalies in complex HSI, thereby affecting detection accuracy. To address this issue, a robust multi-stage progressive autoencoder for hyperspectral anomaly detection (RMSAD) is proposed. Initially, a progressive multi-stage learning framework based on convolutional autoencoders is employed. This framework incrementally reveals and integrates deep contextual features along with their long-range dependencies in HSI, aiming to accurately characterize the background and anomalies. Subsequently, an innovative multi-scale fusion strategy is introduced at the intersections of each stage, reinforcing the learning and representation of background and global spatial details across multiple stages. Finally, by collectively extracting abnormal spatial information across stages, effectively reducing the tendency of autoencoders to reconstruct anomalies. This ensures the efficient restoration and replication of global textural details in HSI. The experimental results on the six HSI datasets demonstrate that the proposed RMSAD is superior to other state-of-the-art methods.