Long-range Model Research Articles

HMDA: A Hybrid Model with Multi-scale Deformable Attention for Medical Image Segmentation.

Transformers have been applied to medical image segmentation tasks owing to their excellent longrange modeling capability, compensating for the failure of Convolutional Neural Networks (CNNs) to extract global features. However, the standardized self-attention modules in Transformers, characterized by a uniform and inflexible pattern of attention distribution, frequently lead to unnecessary computational redundancy with high-dimensional data, consequently impeding the model's capacity for precise concentration on salient image regions. Additionally, achieving effective explicit interaction between the spatially detailed features captured by CNNs and the long-range contextual features provided by Transformers remains challenging. In this architecture, we propose a Hybrid Transformer and CNN architecture with Multi-scale Deformable Attention(HMDA), designed to address the aforementioned issues effectively. Specifically, we introduce a Multi-scale Spatially Adaptive Deformable Attention (MSADA) mechanism, which attends to a small set of key sampling points around a reference within the multi-scale features, to achieve better performance. In addition, we propose the Cross Attention Bridge (CAB) module, which integrates multi-scale transformer and local features through channelwise cross attention enriching feature synthesis. HMDA is validated on multiple datasets, and the results demonstrate the effectiveness of our approach, which achieves competitive results compared to the previous methods.

D-SAT: dual semantic aggregation transformer with dual attention for medical image segmentation

Objective. Medical image segmentation is significantly essential to assist clinicians in facilitating a quick and accurate diagnoses. However, most of the existing methods are still challenged by the loss of semantic information, blurred boundaries and the huge semantic gap between the encoder and decoder. Approach. To tackle these issues, a dual semantic aggregation transformer with dual attention is proposed for medical image segmentation. Firstly, the dual-semantic feature aggregation module is designed to build a bridge between convolutional neural network (CNN) and Transformer, effectively aggregating CNN’s local feature detail ability and Transformer’s long-range modeling ability to mitigate semantic information loss. Thereafter, the strip spatial attention mechanism is put forward to alleviate the blurred boundaries during encoding by constructing pixel-level feature relations across CSWin Transformer blocks from different spatial dimensions. Finally, a feature distribution gated attention module is constructed in the skip connection between the encoder and decoder to decrease the large semantic gap by filtering out the noise in low-level semantic information when fusing low-level and high-level semantic features during decoding. Main results. Comprehensive experiments conducted on abdominal multi-organ segmentation, cardiac diagnosis, polyp segmentation and skin lesion segmentation serve to validate the generalization and effectiveness of the proposed dual semantic aggregation transformer with dual attention (D-SAT). The superiority of D-SAT over current state-of-the-art methods is substantiated by both subjective and objective evaluations, revealing its remarkable performance in terms of segmentation accuracy and quality. Significance. The proposed method subtly preserves the local feature details and global context information in medical image segmentation, providing valuable support to improve diagnostic efficiency for clinicians and early disease control for patients. Code is available at https://github.com/Dxkm/D-SAT.

A novel attention-enhanced network for image super-resolution

The attention mechanism can significantly improve the performance of image super-resolution (SR) networks. In this paper, we propose a novel attention-enhanced network (AENet) to explore the long-range modeling ability. Specifically, we propose a Dual-attention Block (DAB), which consists of Large Kernel Convolution Decomposition Attention Block (LKDAB) and Channel Enhanced Structure Unit (CESU), to capture global and local information in spatial and channel dimension. For the purpose to extract multi-scale information at different granularity levels, we design Multi-scale Feature Adaptive Aggregation Block (MFAAB) to aggregate information of different scales. Finally, in order to summarize more reasonable information, we introduce a Reconstruction Sub-block (RSB) in the tail of the network. Extensive experiments on four benchmark datasets demonstrate that our AENet outperforms compared with other state-of-the-art SR models.

Entanglement entropy and topological properties in a long-range non-Hermitian Su–Schrieffer–Heeger model

We employ entanglement entropy to investigate the topological properties of the extended long-range non-Hermitian Su–Schrieffer–Heeger (SSH) model. We show how long-range hopping, nearest-neighbor hopping, and the non-Hermitian term influence the entanglement entropy in the SSH model, which serves as a valuable tool for examining the topological features of the SSH model. Our research reveals a remarkable phenomenon: entanglement entropy exhibits singular behavior in specific regions that precisely align with the spatial locations of topological phase transitions in the extended long-range SSH model. These findings are further supported by the application of winding numbers and energy spectra, offering comprehensive evidence for the topological aspects of both Hermitian and non-Hermitian SSH models. As a result, the singularity in entanglement entropy provides a robust framework for characterizing topological phase transitions.

SCA-Former: transformer-like network based on stream-cross attention for medical image segmentation

Objective. Deep convolutional neural networks (CNNs) have been widely applied in medical image analysis and achieved satisfactory performances. While most CNN-based methods exhibit strong feature representation capabilities, they face challenges in encoding long-range interaction information due to the limited receptive fields. Recently, the Transformer has been proposed to alleviate this issue, but its cost is greatly enlarging the model size, which may inhibit its promotion. Approach. To take strong long-range interaction modeling ability and small model size into account simultaneously, we propose a Transformer-like block-based U-shaped network for medical image segmentation, dubbed as SCA-Former. Furthermore, we propose a novel stream-cross attention (SCA) module to enforce the network to focus on finding a balance between local and global representations by extracting multi-scale and interactive features along spatial and channel dimensions. And SCA can effectively extract channel, multi-scale spatial, and long-range information for a more comprehensive feature representation. Main results. Experimental results demonstrate that SCA-Former outperforms the current state-of-the-art (SOTA) methods on three public datasets, including GLAS, ISIC 2017 and LUNG. Significance. This work exhibits a promising method to enhance the feature representation of convolutional neural networks and improve segmentation performance.

Global semantic-guided network for saliency prediction

The human visual system effectively analyzes scenes based on local, global and semantic properties. Deep learning-based saliency prediction models adopted two-stream networks, leveraged prior knowledge of global semantics, or added long-range dependency modeling structures like transformers to incorporate global saliency information. However, they either brought high complexity to learning local and global features or neglected the design for enhancing local features. In this paper, we propose a Global Semantic-Guided Network (GSGNet), which first enriches global semantics through a modified transformer block and then incorporates semantic information into visual features from local and global perspectives in an efficient way. Multi-head self-attention in transformers captures global features, but lacks information communication within and between feature subspaces (heads) when computing the similarity matrix. To learn global representations and enhance interactions of the subspaces, we propose a Channel-Squeeze Spatial Attention (CSSA) module to emphasize channel-relevant information in a compression manner and learn global spatial relationships. To better fuse local and global contextual information, we propose a hybrid CNN-Transformer block called local–global fusion block (LGFB) for aggregating semantic features simply and efficiently. Experimental results on four public datasets demonstrate that our model achieves compelling performance compared with the state-of-the-art saliency prediction models on various evaluation metrics.

Corner-to-Center long-range context model for efficient learned image compression

In the framework of learned image compression, the context model plays a pivotal role in capturing the dependencies among latent representations. To reduce the decoding time resulting from the serial autoregressive context model, the parallel context model has been proposed as an alternative that necessitates only two passes during the decoding phase, thus facilitating efficient image compression in real-world scenarios. However, performance degradation occurs due to its incomplete casual context. To tackle this issue, we conduct an in-depth analysis of the performance degradation observed in existing parallel context models, focusing on two aspects: the Quantity and Quality of information utilized for context prediction and decoding. Based on such analysis, we propose the Corner-to-Center transformer-based Context Model (C3M) designed to enhance context and latent predictions and improve rate–distortion performance. Specifically, we leverage the logarithmic-based prediction order to predict more context features from corner to center progressively. In addition, to enlarge the receptive field in the analysis and synthesis transformation, we use the Long-range Crossing Attention Module (LCAM) in the encoder/decoder to capture the long-range semantic information by assigning the different window shapes in different channels. Extensive experimental evaluations show that the proposed method is effective and outperforms the state-of-the-art parallel methods. Finally, according to the subjective analysis, we suggest that improving the detailed representation in transformer-based image compression is a promising direction to be explored.

A lightweight vision transformer with symmetric modules for vision tasks

Transformer-based networks have demonstrated their powerful performance in various vision tasks. However, these transformer-based networks are heavyweight and cannot be applied to edge computing (mobile) devices. Despite that the lightweight transformer network has emerged, several problems remain, i.e., weak feature extraction ability, feature redundancy, and lack of convolutional inductive bias. To address these three problems, we propose a lightweight visual transformer (Symmetric Former, SFormer), which contains two novel modules (Symmetric Block and Symmetric FFN). Specifically, we design Symmetric Block to expand feature capacity inside the module and enhance the long-range modeling capability of attention mechanism. To increase the compactness of the model and introduce inductive bias, we introduce convolutional cheap operations to design Symmetric FFN. We compared the SFormer with existing lightweight transformers on several vision tasks. Remarkably, on the image recognition task of ImageNet [13], SFormer gains 1.2% and 1.6% accuracy improvements compared to PVTv2-b0 and Swin Transformer, respectively. On the semantic segmentation task of ADE20K [64], SFormer delivers performance improvements of 0.2% and 0.7% compared to PVTv2-b0 and Swin Transformer, respectively. On the cityscapes dataset [11], SFormer delivers performance improvements of 2.5% and 4.2% compared to PVTv2-b0 and Swin Transformer, respectively. The code is open-source and available at: https://github.com/ISCLab-Bistu/Symmetric_Former.git.

Reciprocal transformer for hyperspectral and multispectral image fusion

Hyperspectral and multispectral (HS–MS) image fusion aims to reconstruct high-resolution hyperspectral images from low-resolution hyperspectral images and high-resolution multispectral images. While convolutional neural networks have been widely used for HS–MS fusion, their potential is curtailed due to the limited receptive field of each neuron, resulting in inadequate long-range modeling capabilities. Although several Transformer-based HS–MS fusion methods have been proposed, most of them often fail to fully integrate and coordinate the data from the two modalities (i.e., hyperspectral images and multispectral images). Such ineffective interactions significantly compromise the quality of the reconstructed hyperspectral images. In this paper, we introduce a novel reciprocal fusion strategy called the dual cross Transformer-based fusion (DCTransformer) for HS–MS fusion. The model excels in capturing the interplay between different modalities by utilizing directional pairwise multi-head cross-attention, which facilitates the transfer of information between modalities. Additionally, we incorporate a Swin Transformer block post cross-attention to enhance the self-attention within the context. Extensive experiments show that our DCTransformer performs favorably against other recent works on both simulation HSI datasets and real HSI datasets. The source code and pre-trained models are availabe at https://github.com/qingma2016/DCTransformer.

IoUformer: Pseudo-IoU prediction with transformer for visual tracking

Siamese tracking has witnessed tremendous progress in tracking paradigm. However, its default box estimation pipeline still faces a crucial inconsistency issue, namely, the bounding box decided by its classification score is not always best overlapped with the ground truth, thus harming performance. To this end, we explore a novel simple tracking paradigm based on the intersection over union (IoU) value prediction. To first bypass this inconsistency issue, we propose a concise target state predictor termed IoUformer, which instead of default box estimation pipeline directly predicts the IoU values related to tracking performance metrics. In detail, it extends the long-range dependency modeling ability of transformer to jointly grasp target-aware interactions between target template and search region, and search sub-region interactions, thus neatly unifying global semantic interaction and target state prediction. Thanks to this joint strength, IoUformer can predict reliable IoU values near-linear with the ground truth, which paves a safe way for our new IoU-based siamese tracking paradigm. Since it is non-trivial to explore this paradigm with pleased efficacy and portability, we offer the respective network components and two alternative localization ways. Experimental results show that our IoUformer-based tracker achieves promising results with less training data. For its applicability, it still serves as a refinement module to consistently boost existing advanced trackers.

GP-Net: Image Manipulation Detection and Localization via Long-Range Modeling and Transformers

With the rise of image manipulation techniques, an increasing number of individuals find it easy to manipulate image content. Undoubtedly, this presents a significant challenge to the integrity of multimedia data, thereby fueling the advancement of image forgery detection research. A majority of current methods employ convolutional neural networks (CNNs) for image manipulation localization, yielding promising outcomes. Nevertheless, CNN-based approaches possess limitations in establishing explicit long-range relationships. Consequently, addressing the image manipulation localization task necessitates a solution that adeptly builds global context while preserving a robust grasp of low-level details. In this paper, we propose GPNet to address this challenge. GPNet combines Transformer and CNN in parallel which can build global dependency and capture low-level details efficiently. Additionally, we devise an effective fusion module referred to as TcFusion, which proficiently amalgamates feature maps generated by both branches. Thorough extensive experiments conducted on diverse datasets showcase that our network outperforms prevailing state-of-the-art manipulation detection and localization approaches.

Blind single-image-based thin cloud removal using a cloud perception integrated fast Fourier convolutional network

Remote sensing images are frequently contaminated by clouds that often degrade the performance of subsequent applications. Cloud removal, therefore, is a standard step in remote sensing image preprocessing, and single-image-based thin cloud removal is a well-established area of research. Existing single-image-based thin cloud removal methods however, lack the capacity for simultaneous executions of efficient long-range modeling and physical attribute consideration. To extend this work and fill this gap, a novel blind single-image-based thin cloud removal method, called cloud perception integrated fast Fourier convolutional network (CP-FFCN), was designed and implemented. The CP-FFCN consists of two modules: the cloud perception module (CPM) and a fast Fourier convolution (FFC)-conducted reconstruction module (FFCN). The CPM uses a frequency spatial attention mechanism to realize long-range modeling of clouds, globally detect them in the cloudy image. It helps the CP-FFCN remove the clouds without external prior knowledge of the cloud distribution. The reconstruction module was designed with an FFC-conducted U-Net architecture to recover the clean images from cloudy scenarios, guided by the locations of clouds as detected by the CPM. In addition, the FFC blocks deployed in the encoder and decoder components in the U-Net architecture selectively learn the attributes of clouds and fogs from the frequency spectrograms to remove the clouds and reconstruct the underlying ground objects. The CP-FFCN selectively learns the frequency features for adequate cloud separation and at the same time efficiently models the long-range information for comprehensive scenario reconstruction with the help of these two modules. We adopted the Google Earth data and Landsat-8 imagery to train the CP-FFCN model and evaluate it on simulated and naturally occurring cloudy scenarios. The visual outcomes illustrate that the proposed CP-FFCN successfully removes thin and small-scale thick clouds with complex ground object scenarios, without external cloud masks and additional reference data. The quantitative analyses further demonstrate the higher effectiveness of the CP-FFCN when compared with several other state-of-the-art thin cloud removal methods, yielding a PSNR value over 39.24 and a SSIM value over 0.98 on the Landsat 8 images.

STDFormer: Spatial-Temporal Motion Transformer for Multiple Object Tracking

Mainstream multi-object tracking methods exploit appearance information and/or motion information to achieve interframe association. However, dealing with similar appearance and occlusion is a challenge for appearance information, while motion information is limited by linear assumptions and is prone to failure in nonlinear motion patterns. In this work, we disregard appearance clues and propose a pure motion tracker to address the above issues. It dexterously utilizes Transformer to estimate complex motion and achieves high-performance tracking with low computing resources. Furthermore, contrastive learning is introduced to optimize feature representation for robust association. Specifically, we first exploit the long-range modeling capability of Transformer to mine intention information in temporal motion and decision information in spatial interaction and introduce prior detection to constrain the range of motion estimation. Then, we introduce contrastive learning as an auxiliary task to extract reliable motion features to compute affinity and introduce bidirectional matching to improve the affinity computation distribution. In addition, given that both tasks are dedicated to narrowing the embedding distance between the motion features of the tracked object and the detection features, we design a joint-motion-and-association framework to unify the above two tasks in one framework for optimization. The experimental results achieved with three benchmark datasets, MOT17, MOT20 and DanceTrack, verify the effectiveness of our proposed method. Compared with state-of-the-art methods, the proposed STDFormer sets a new state-of-the-art on DanceTrack and achieves competitive performance on MOT17 and MOT20. This demonstrates the advantage of our method in handling associations under similar appearance, occlusion or nonlinear motion. At the same time, the significant advantages of the proposed method over Transformer-based and contrastive learning-based methods suggest a new direction for the application of Transformer and contrastive learning in MOT. In addition, to verify the generalization of STDFormer in unmanned aerial vehicle (UAV) videos, we also evaluate STDFormer on VisDrone2019. The results show that STDFormer achieves state-of-the-art performance on VisDrone2019, which proves that it can handle small-scale object associations in UAV videos well. The code is available at https://github.com/Xiaotong-Zhu/STDFormer.

Phase classification in the long-range Harper model using machine learning

Phase classification in the long-range Harper model using machine learning

Phase ordering dynamics of the random-field long-range Ising model in one dimension.

We investigate the influence of long-range (LR) interactions on the phase ordering dynamics of the one-dimensional random-field Ising model (RFIM). Unlike the usual RFIM, a spin interacts with all other spins through a ferromagnetic coupling that decays as r^{-(1+σ)}, where r is the distance between two spins. In the absence of LR interactions, the size of coarsening domains R(t) exhibits a crossover from pure system behavior R(t)∼t^{1/2} to an asymptotic regime characterized by logarithmic growth: R(t)∼(lnt)^{2}. The LR interactions affect the preasymptotic regime, which now exhibits ballistic growth R(t)∼t, followed by σ-dependent growth R(t)∼t^{1/(1+σ)}. Additionally, the LR interactions also affect the asymptotic logarithmic growth, which becomes R(t)∼(lnt)^{α(σ)} with α(σ)<2. Thus, LR interactions lead to faster growth than for the nearest-neighbor system at short times. Unexpectedly, this driving force causes a slowing down of the dynamics (α<2) in the asymptotic logarithmic regime. This is explained in terms of a nontrivial competition between the pinning force caused by the random field and the driving force introduced by LR interactions. We also study the spatial correlation function and the autocorrelation function of the magnetization field. The former exhibits superuniversality for all σ, i.e., a scaling function that is independent of the disorder strength. The same holds for the autocorrelation function when σ<1, whereas a signature of the violation of superuniversality is seen for σ>1.

Finite-temperature critical behaviors in 2D long-range quantum Heisenberg model

The Mermin-Wagner theorem states that spontaneous continuous symmetry breaking is prohibited in systems with short-range interactions at spatial dimension D ≤ 2. For long-range interactions with a power-law form (1/rα), the theorem further forbids ferromagnetic or antiferromagnetic order at finite temperature when α ≥ 2D. However, the situation for α ∈ (2, 4) at D = 2 is not covered by the theorem. To address this, we conduct large-scale quantum Monte Carlo simulations and field theoretical analysis. Our findings show spontaneous breaking of SU(2) symmetry in the ferromagnetic Heisenberg model with 1/rα-form long-range interactions at D = 2. We determine critical exponents through finite-size analysis for α < 3 (above the upper critical dimension with Gaussian fixed point) and 3 ≤ α < 4 (below the upper critical dimension with non-Gaussian fixed point). These results reveal new critical behaviors in 2D long-range Heisenberg models, encouraging further experimental studies of quantum materials with long-range interactions beyond the Mermin-Wagner theorem’s scope.

TransDose: Transformer-based radiotherapy dose prediction from CT images guided by super-pixel-level GCN classification.

Radiotherapy is a mainstay treatment for cancer in clinic. An excellent radiotherapy treatment plan is always based on a high-quality dose distribution map which is produced by repeated manual trial-and-errors of experienced experts. To accelerate the radiotherapy planning process, many automatic dose distribution prediction methods have been proposed recently and achieved considerable fruits. Nevertheless, these methods require certain auxiliary inputs besides CT images, such as segmentation masks of the tumor and organs at risk (OARs), which limits their prediction efficiency and application potential. To address this issue, we design a novel approach named as TransDose for dose distribution prediction that treats CT images as the unique input in this paper. Specifically, instead of inputting the segmentation masks to provide the prior anatomical information, we utilize a super-pixel-based graph convolutional network (GCN) to extract category-specific features, thereby compensating the network for the necessary anatomical knowledge. Besides, considering the strong continuous dependency between adjacent CT slices as well as adjacent dose maps, we embed the Transformer into the backbone, and make use of its superior ability of long-range sequence modeling to endow input features with inter-slice continuity message. To our knowledge, this is the first network that specially designed for the task of dose prediction from only CT images without ignoring necessary anatomical structure. Finally, we evaluate our model on two real datasets, and extensive experiments demonstrate the generalizability and advantages of our method.

X-ray Detection of Prohibited Item Method Based on Dual Attention Mechanism

Prohibited item detection plays a significant role in ensuring public safety, as the timely and accurate identification of prohibited items ensures the safety of lives and property. X-ray transmission imaging technology is commonly employed for prohibited item detection in public spaces, producing X-ray images of luggage to visualize their internal contents. However, challenges such as multiple object overlapping, varying angles, loss of details, and small targets in X-ray transmission imaging pose significant obstacles to prohibited item detection. Therefore, a dual attention mechanism network (DAMN) for X-ray prohibited item detection is proposed. The DAMN consists of three modules, i.e., spatial attention, channel attention, and dependency relationship optimization. A long-range dependency model is achieved by employing a dual attention mechanism with spatial and channel attention, effectively extracting feature information. Meanwhile, the dependency relationship module is integrated to address the shortcomings of traditional convolutional networks in terms of short-range correlations. We conducted experiments comparing the DAMN with several existing algorithms on datasets containing 12 categories of prohibited items, including firearms and knives. The results show that the DAMN has a good performance, particularly in scenarios involving small object detection, detail loss, and target overlap under complex conditions. Specifically, the detection average precision of the DAMN reaches 63.8%, with a segmentation average precision of 54.7%.

Window Token Transformer: Can learnable window token help window-based transformer build better long-range interactions?

Compared with the vanilla transformer, the window-based transformer offers a better trade-off between accuracy and efficiency. Although the window-based transformer has made great progress, its long-range modeling capabilities are limited due to the size of the local window and the window connection scheme. To address this problem, we propose a novel Window Token Transformer (WTT). The core mechanism of WTT is the addition of a window token for summarizing window information in each local window. We refer to this type of token interaction as Window Token Attention. These window tokens will interact spatially with the tokens in each window to enable long-range modeling. In order to preserve the hierarchical design of the window-based transformer, we design Feature Inheritance Module (FIM) in each phase of WTT to deliver the local window information from the previous phase to the window token in the next phase. In addition, we have designed a Global–Local Feedforward Network (GLFFN) in WTT, which can enhance the local awareness of the network while preserving the global awareness. Extensive experiments have shown that our WTT achieves competitive results with low parameters in image classification and downstream tasks.

Component-aware anomaly detection framework for adjustable and logical industrial visual inspection

Existing industrial visual anomaly detection models can be broadly categorized into two paradigms, i.e., the paradigm based on image reconstruction and the paradigm based on implicit feature modeling. While both of them have demonstrated great performance in detecting low-level structural anomalies, their limited adjustability and ability to detect high-level logical anomalies hinder their broader use in industrial visual inspection. To solve the above issues, this paper proposes a novel paradigm, that is, Component-aware Anomaly Detection (ComAD), which can simultaneously achieve adjustable and logical anomaly detection. Specifically, it segments the original image into diverse components, enabling model adjustability based on components’ significance and long-range modeling capabilities based on components’ metrological characteristics. We demonstrate that the proposed paradigm can be realized by a lightweight and nearly training-free framework. ComAD is evaluated on real-world scenarios to show its adjustability and logical anomaly detection capabilities. It achieves state-of-the-art performance on the challenging MVTec LOCO AD and CAD-SD benchmarks. Code is available at: https://github.com/liutongkun/ComAD.