Multi-scale Contextual Information Research Articles

DiagSWin: A multi-scale vision transformer with diagonal-shaped windows for object detection and segmentation

Recently, Vision Transformer and its variants have demonstrated remarkable performance on various computer vision tasks, thanks to its competence in capturing global visual dependencies through self-attention. However, global self-attention suffers from high computational cost due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection and semantic segmentation). Many recent works have attempted to reduce the cost by applying fine-grained local attention, but these approaches cripple the long-range modeling power of the original self-attention mechanism. Furthermore, these approaches usually have similar receptive fields within each layer, thus limiting the ability of each self-attention layer to capture multi-scale features, resulting in performance degradation when handling images with objects of different scales. To address these issues, we develop the Diagonal-shaped Window (DiagSWin) attention mechanism for modeling attentions in diagonal regions at hybrid scales per attention layer. The key idea of DiagSWin attention is to inject multi-scale receptive field sizes into tokens: before computing the self-attention matrix, each token attends its closest surrounding tokens at fine granularity and the tokens far away at coarse granularity. This mechanism is able to effectively capture multi-scale context information while reducing computational complexity. With DiagSwin attention, we present a new variant of Vision Transformer models, called DiagSWin Transformers, and demonstrate their superiority in extensive experiments across various tasks. Specifically, the DiagSwin Transformer with a large size achieves 84.4% Top-1 accuracy and outperforms the SOTA CSWin Transformer on ImageNet with 40% fewer model size and computation cost. When employed as backbones, DiagSWin Transformers achieve significant improvements over the current SOTA modules. In addition, our DiagSWin-Base model yields 51.1 box mAP and 45.8 mask mAP on COCO for object detection and segmentation, and 52.3 mIoU on the ADE20K for semantic segmentation.

BMSeNet: Multiscale Context Pyramid Pooling and Spatial Detail Enhancement Network for Real-Time Semantic Segmentation.

Most real-time semantic segmentation networks use shallow architectures to achieve fast inference speeds. This approach, however, limits a network's receptive field. Concurrently, feature information extraction is restricted to a single scale, which reduces the network's ability to generalize and maintain robustness. Furthermore, loss of image spatial details negatively impacts segmentation accuracy. To address these limitations, this paper proposes a Multiscale Context Pyramid Pooling and Spatial Detail Enhancement Network (BMSeNet). First, to address the limitation of singular semantic feature scales, a Multiscale Context Pyramid Pooling Module (MSCPPM) is introduced. By leveraging various pooling operations, this module efficiently enlarges the receptive field and better aggregates multiscale contextual information. Moreover, a Spatial Detail Enhancement Module (SDEM) is designed, to effectively compensate for lost spatial detail information and significantly enhance the perception of spatial details. Finally, a Bilateral Attention Fusion Module (BAFM) is proposed. This module leverages pixel positional correlations to guide the network in assigning appropriate weights to the features extracted from the two branches, effectively merging the feature information of both branches. Extensive experiments were conducted on the Cityscapes and CamVid datasets. Experimental results show that the proposed BMSeNet achieves a good balance between inference speed and segmentation accuracy, outperforming some state-of-the-art real-time semantic segmentation methods.

G-Net: Implementing an enhanced brain tumor segmentation framework using semantic segmentation design.

A fundamental computer vision task called semantic segmentation has significant uses in the understanding of medical pictures, including the segmentation of tumors in the brain. The G-Shaped Net architecture appears in this context as an innovative and promising design that combines components from many models to attain improved accuracy and efficiency. In order to improve efficiency, the G-Shaped Net architecture synergistically incorporates four fundamental components: the Self-Attention, Squeeze Excitation, Fusion, and Spatial Pyramid Pooling block structures. These factors work together to improve the precision and effectiveness of brain tumor segmentation. Self-Attention, a crucial component of G-Shaped architecture, gives the model the ability to concentrate on the image's most informative areas, enabling accurate localization of tumor boundaries. By adjusting channel-wise feature maps, Squeeze Excitation completes this by improving the model's capacity to capture fine-grained information in the medical pictures. Since the G-Shaped model's Spatial Pyramid Pooling component provides multi-scale contextual information, the model is capable of handling tumors of various sizes and complexity levels. Additionally, the Fusion block architectures combine characteristics from many sources, enabling a thorough comprehension of the image and improving the segmentation outcomes. The G-Shaped Net architecture is an asset for medical imaging and diagnostics and represents a substantial development in semantic segmentation, which is needed more and more for accurate brain tumor segmentation.

DEAF-Net: Detail-Enhanced Attention Feature Fusion Network for Retinal Vessel Segmentation.

Retinal vessel segmentation is crucial for the diagnosis of ophthalmic and cardiovascular diseases. However, retinal vessels are densely and irregularly distributed, with many capillaries blending into the background, and exhibit low contrast. Moreover, the encoder-decoder-based network for retinal vessel segmentation suffers from irreversible loss of detailed features due to multiple encoding and decoding, leading to incorrect segmentation of the vessels. Meanwhile, the single-dimensional attention mechanisms possess limitations, neglecting the importance of multidimensional features. To solve these issues, in this paper, we propose a detail-enhanced attention feature fusion network (DEAF-Net) for retinal vessel segmentation. First, the detail-enhanced residual block (DERB) module is proposed to strengthen the capacity for detailed representation, ensuring that intricate features are efficiently maintained during the segmentation of delicate vessels. Second, the multidimensional collaborative attention encoder (MCAE) module is proposed to optimize the extraction of multidimensional information. Then, the dynamic decoder (DYD) module is introduced to preserve spatial information during the decoding process and reduce the information loss caused by upsampling operations. Finally, the proposed detail-enhanced feature fusion (DEFF) module composed of DERB, MCAE and DYD modules fuses feature maps from both encoding and decoding and achieves effective aggregation of multi-scale contextual information. The experiments conducted on the datasets of DRIVE, CHASEDB1, and STARE, achieving Sen of 0.8305, 0.8784, and 0.8654, and AUC of 0.9886, 0.9913, and 0.9911 on DRIVE, CHASEDB1, and STARE, respectively, demonstrate the performance of our proposed network, particularly in the segmentation of fine retinal vessels.

A lightweight self-supervised learning segmentation model for variable and complex high-resolution remote sensing images

The complexity and variability of high-resolution remote sensing data, such as high intra-class variability and inter-class similarity, pose significant challenges to model segmentation. To address the problem, this paper constructs a lightweight self-supervised learning model for multi-label segmentation in the form of phased learning of multi-scale features. The model adopts axial depthwise separable convolutions to reduce computational complexity and enhance feature representation, utilizes dilated rates to capture large-scale and multi-scale contextual information for long-distance feature extraction, and incorporates convolution kernels of varying sizes to acquire both local and global feature information for the improved ability of learning feature representation. The experimental results show that our model achieves competitive performance and has smaller weight parameters, memory usage, and lower computational complexity compared with existing classical models that rely on large-scale weight parameters. Additionally, our ablation study delves into the encountered design issues to elucidate the rationality of our approach.

Self-Supervised Monocular Depth Estimation for Endoscopic Imaging.

Endoscopy holds a pivotal role in the early detection and treatment of diverse diseases, with artificial intelligence (AI)-assisted methods increasingly gaining prominence in disease screening. Among them, the depth estimation from endoscopic sequences is crucial for a spectrum of AI-assisted surgical techniques. However, the development of endoscopic depth estimation algorithms presents a formidable challenge due to the unique environmental intricacies and constraints within the dataset. This paper proposes a self-supervised depth estimation network to comprehensively explore the brightness changes in endoscopic images, and fuse different features at multiple levels to achieve an accurate prediction of endoscopic depth. First, a FlowNet is designed to evaluate the brightness changes of adjacent frames by calculating the multi-scale structural similarity. Second, a feature fusion module is presented to capture multi-scale contextual information. Experiments show that the average accuracy of the algorithm is 97.03% in the Stereo Correspondence and Reconstruction of Endoscopic Data (SCARED dataset). Based on the training parameters of the SCARED dataset, the algorithm achieves superior performance on the other two datasets (EndoSLAM and KVASIR dataset), indicating that the algorithm has good generalization performance.

Efficient SpineUNetX for X-ray: A spine segmentation network based on ConvNeXt and UNet

Accurate localization and delineation of vertebrae are crucial for diagnosing and treating spinal disorders. To achieve this, we propose an efficient X-ray full-spine vertebra instance segmentation method based on an enhanced U-Net architecture. Several key improvements have been made: the ConvNeXt encoder is employed to effectively capture complex features, and IFE feature extraction is introduced in the skip connections to focus on texture-rich and edge-clear clues. The CBAM attention mechanism is used in the bottleneck to integrate coarse and fine-grained semantic information. The decoder employs a residual structure combined with skip connections to achieve multi-scale contextual information and feature fusion. Our method has been validated through experiments on anterior-posterior and lateral spinal segmentation, demonstrating robust feature extraction and precise semantic segmentation capabilities. It effectively handles various spinal disorders, including scoliosis, vertebral wedging, lumbar spondylolisthesis and spondylolysis. This segmentation foundation enables rapid calibration of vertebral parameters and the computation of relevant metrics, providing valuable references and guidance for advancements in medical imaging.

REDef-DETR: real-time and efficient DETR for industrial surface defect detection

Abstract Industrial surface defect detection is an important part of industrial production, which aims to identify and detecting various defects on the surface of product to ensure quality and meet customer requirements. With the development of deep learning and image processing technologies, the surface defect detection methods based on computer vision has become the mainstream method. However, the prevalent convolutional neural network-based defect detection methods also have many problems. For example, these methods rely on post-processing of Non-Maximum Suppression and have poor detection ability for small targets, which affects the speed and accuracy of surface defect detection in industrial scenarios. Therefore, we propose a novel DEtection TRansformer-based surface defect detection method. Firstly, we propose a Multi-scale Contextual Information Dilated module and fuse it into the backbone. The module is mainly composed of large kernel convolutions, which aims to expand the receptive field of the model, thus reducing the leakage rate of the model. Moreover, we design an efficient encoder which mainly contains two important modules, namely feature enhancement based on cascaded group attention module and efficient feature fusion module based on content-aware. The former module effectively enhances the high-level semantic information extracted by the backbone, thus enabling the model to better interpret features, and it can improve the problem of high computational cost of transformer encoder, thus increasing the detection speed. The latter module performs multi-scale feature fusion across the feature information of various scales, thus improving the detection accuracy of the model for small-size defects. Experimental results show that the proposed method achieves 80.6%mAP and 80.3FPS on NEU-DET, and 98.0%mAP and 79.4FPS on PCB-DET. Our proposed method exhibits excellent detection performance and achieves real-time and efficient surface defect detection capability to meet the needs of industrial surface defect detection.

DFCAFNet: Dual-feature co-attentive fusion network for diabetic retinopathy grading

Diabetic retinopathy (DR) grading is a complicated task characterized by inter-class variations among several categories, the subtle detection of tiny lesions, and uneven data distribution. To achieve precise DR grading, extracting distinct features that can effectively capture minor visual variations in lesions is necessary. The constraints of conventional convolutional neural networks (CNNs) are evident in their difficulty in detecting tiny lesions, which is further enhanced by their biased focus due to an uneven distribution of data. Furthermore, comparable color and texture attributes among different classes add to the complexity of achieving accurate grading results. This paper presents a new multi-scale attention fusion network (MS-AFNet) incorporating multi-scale contextual information. By successfully combining local and global features, the network improves the diagnostic accuracy of the model in handling inter-class variances. A concurrent attention module (CAM) with multiple attention blocks enhances the ability to distinguish information by capturing complex features within the lesions through numerous attention mechanisms. This novel approach greatly improves the model’s capacity to distinguish nuanced features in minor lesions, resulting in more precise diagnostic outcomes. A dual-feature co-attentive fusion network (DFCAFNet) is proposed by combining CAM and MS-AFNet, which capture spatial patterns and highlight inter-channel interactions to detect complex patterns and enhance accuracy. An extensive assessment of two datasets on DR demonstrates the exceptional accuracy and computational efficiency of DFCAFNet.

Multi-scale Segformer: image segmentation algorithm for electric vehicles in fire accident

ABSTRACT Electric vehicles (EVs) have played a significant role in sustainability, and EVs fire accidents have raised doubts in recent years. To solve the mobile analytical equipment limitation in EVs fire accident and help staff receive prompt results at on-spot inspection, we provide a lightweight but accurate Transformer that can ideally adapt to the mobile environment. First, we built on the simple Segformer and extended it to aggregate the representations of amorphous objects, such as fire traces, in image recognition. Second, we used shunt-based self-attention (SSA) to enhance the model for capturing multi-scale contextual information and help distinguish the deformed level of EVs after combustion. Third, we redesigned a simple multi-level information aggregation (MIA) decoder to obtain the relationship between pixels in the channel dimensions by a weighted aggregation. Furthermore, to foster image trace recognition, we put forwards and evaluated the accuracy of models on electric vehicle fire traces (EVFTrace), a dataset of images of burnt EVs. On EVFTrace, the mean intersection over union (mIoU) achieves 72.24%. The float point operations (Flops) and parameters (Params) achieve 114.83 G and 89.5 M. Our model shows excellent efficiency and accuracy for burnt EVs segmentation tasks.

Coreference Resolution Based on High-Dimensional Multi-Scale Information.

Coreference resolution is a key task in Natural Language Processing. It is difficult to evaluate the similarity of long-span texts, which makes text-level encoding somewhat challenging. This paper first compares the impact of commonly used methods to improve the global information collection ability of the model on the BERT encoding performance. Based on this, a multi-scale context information module is designed to improve the applicability of the BERT encoding model under different text spans. In addition, improving linear separability through dimension expansion. Finally, cross-entropy loss is used as the loss function. After adding BERT and span BERT to the module designed in this article, F1 increased by 0.5% and 0.2%, respectively.

A context-ensembled refinement network for image segmentation of coated fuel particles

The thickness of the coating layer of coated fuel particles plays a vital role in the reliable operation of a high-temperature gas-cooled reactor. Coating thickness remains extremely challenging to measure it efficiently, accurately, and robotically. The existing methods struggle with poorly refined segmentation results and easy adhesion of boundaries, complicating the accurate image segmentation of coating layers, which is fundamental to achieving the measurement goal. To overcome these issues, we designed the Context-Ensembled Refinement Network. This network refines the segmentation results to accurately extract different coating regions, leveraging its context fusion module and boundary refinement module to effectively utilize multiscale contextual information and provide strong boundary refinement capability. Experimental results demonstrate that our proposed method outperforms recent state-of-the-art methods, achieving an mIoU value of 95.8 %, and holds promising potential for automatic measurement of coating thickness.

Lightweight multi-scale dynamic selection network for medical image segmentation

Medical image segmentation technology can assist doctors in accurately diagnosing and treating diseases. However, the diversity of segmented targets in scale and shape, as well as the complexity of the background environment, pose challenges for existing segmentation methods. In this context, we propose two effective deep neural networks, the multi-scale dynamic selection network (MDSNet) and its lightweight version MDSNet-Light for medical image segmentation. In MDSNet, we design a multi-scale dynamic selection module to capture multi-scale contextual information and dynamically adjust the acceptance domain of the feature map to achieve feature fusion at the optimal scale. We introduce PixelShuffle and PixelUnshuffle as sampling methods to effectively alleviate information loss issues for small-scale objects caused by pooling. Experimental results show that MDSNet outperforms eight existing superior neural networks in the test data of the publicly available dataset LIDC-IDRI on four evaluation indicators. And MDSNet-Light achieves a good balance between network performance and computational complexity, providing an option for applications in limited computing resource scenarios.

Incorporating convolutional and transformer architectures to enhance semantic segmentation of fine-resolution urban images

ABSTRACT Though convolutional neural networks (CNN) exhibit promise in image semantic segmentation, they have limitations in capturing global context information, resulting in inaccuracies in segmenting small object features and object boundaries. This study introduces a hybrid network, ICTANet, which incorporate convolutional and Transformer architectures to improve the segmentation performance of fine-resolution remote sensing urban imagery. The ICTANet model is essentially a Transformer-based encoder-decoder structure. The dual-encoder architecture, which combines CNN and Swin Transformer modules, is designed to extract both global and local detail information. The feature information at various stages is collected by the Feature Extraction and Fusion modules (FEF), enabling multi-scale contextual information fusion. In addition, an Auxiliary Boundary Detection (ABD) module is introduced at the end of the decoder to enhance the model’s ability to capture object boundary information. Numerous ablation experiments have been conducted to demonstrate the efficacy of various components within the network. The testing results have proven that the proposed model can achieve satisfactory performance on the ISPRS Vaihingen and Potsdam dataset, with overall accuracies of 91.9% and 92.0%, respectively. Simultaneously, the proposed model is also compared to the current state-of-the-art methods, exhibiting competitive performance, particularly in the segmentation of diminutive objects like cars and trees.

GMAlignNet: multi-scale lightweight brain tumor image segmentation with enhanced semantic information consistency

Although the U-shaped architecture, represented by UNet, has become a major network model for brain tumor segmentation, the repeated convolution and sampling operations can easily lead to the loss of crucial information. Additionally, directly fusing features from different levels without distinction can easily result in feature misalignment, affecting segmentation accuracy. On the other hand, traditional convolutional blocks used for feature extraction cannot capture the abundant multi-scale information present in brain tumor images. This paper proposes a multi-scale feature-aligned segmentation model called GMAlignNet that fully utilizes Ghost convolution to solve these problems. Ghost hierarchical decoupled fusion unit and Ghost hierarchical decoupled unit are used instead of standard convolutions in the encoding and decoding paths. This transformation replaces the holistic learning of volume structures by traditional convolutional blocks with multi-level learning on a specific view, facilitating the acquisition of abundant multi-scale contextual information through low-cost operations. Furthermore, a feature alignment unit is proposed that can utilize semantic information flow to guide the recovery of upsampled features. It performs pixel-level semantic information correction on misaligned features due to feature fusion. The proposed method is also employed to optimize three classic networks, namely DMFNet, HDCNet, and 3D UNet, demonstrating its effectiveness in automatic brain tumor segmentation. The proposed network model was applied to the BraTS 2018 dataset, and the results indicate that the proposed GMAlignNet achieved Dice coefficients of 81.65%, 90.07%, and 85.16% for enhancing tumor, whole tumor, and tumor core segmentation, respectively. Moreover, with only 0.29 M parameters and 26.88G FLOPs, it demonstrates better potential in terms of computational efficiency and possesses the advantages of lightweight. Extensive experiments on the BraTS 2018, BraTS 2019, and BraTS 2020 datasets suggest that the proposed model exhibits better potential in handling edge details and contour recognition.

STF-Net: sparsification transformer coding guided network for subcortical brain structure segmentation.

Subcortical brain structure segmentation plays an important role in the diagnosis of neuroimaging and has become the basis of computer-aided diagnosis. Due to the blurred boundaries and complex shapes of subcortical brain structures, labeling these structures by hand becomes a time-consuming and subjective task, greatly limiting their potential for clinical applications. Thus, this paper proposes the sparsification transformer (STF) module for accurate brain structure segmentation. The self-attention mechanism is used to establish global dependencies to efficiently extract the global information of the feature map with low computational complexity. Also, the shallow network is used to compensate for low-level detail information through the localization of convolutional operations to promote the representation capability of the network. In addition, a hybrid residual dilated convolution (HRDC) module is introduced at the bottom layer of the network to extend the receptive field and extract multi-scale contextual information. Meanwhile, the octave convolution edge feature extraction (OCT) module is applied at the skip connections of the network to pay more attention to the edge features of brain structures. The proposed network is trained with ahybrid loss function. The experimental evaluation on two public datasets: IBSR and MALC, shows outstanding performance in terms of objective and subjective quality.

An IIoT Temporal Data Anomaly Detection Method Combining Transformer and Adversarial Training

The existing Industrial Internet of Things (IIoT) temporal data analysis methods often suffer from issues such as information loss, difficulty balancing spatial and temporal features, and being affected by training data noise, which can lead to varying degrees of reduced model accuracy. Therefore, a new anomaly detection method was proposed, which integrated Transformer and adversarial training. Firstly, a bidirectional spatiotemporal feature extraction module was constructed by combining Graph Attention Networks (GAT) and Bidirectional Gated Recurrent Unit (BiGRU), which can simultaneously extract spatial and temporal features. Then, by combining multi-scale convolution with Long Short-Term Memory (LSTM), multi-scale contextual information was captured. Finally, an improved Transformer was used to fuse multi-dimensional features, combined with an adversarial-trained variational autoencoder to calculate the anomalies of the input data. This method outperforms other comparison models by conducting experiments on four publicly available datasets.

Multi-Scale Contextual Swin Transformer for Crop Image Segmentation

Combining UAV-based remote sensing with deep learning for image segmentation is a particularly innovative and effective technology in modern agriculture. This approach allows for detailed and precise analysis of agricultural fields, such as crop monitoring, yield prediction, and irrigation management, enhancing decision-making and farm management practices. Inspired by the recent advancements of Transformers in computer vision, this paper introduces the Multi-Scale Contextual Swin Transformer (MSC-Swin), a novel model for precise segmentation of UAV crop images. MSC-Swin innovatively combines a Swin Transformer architecture for detailed feature extraction with pooling operations to utilize multi-scale contextual information. Our extensive experimentation demonstrates that MSC-Swin not only achieves state-of-the-art performance on the Barley Remote Sensing dataset, with a record mIoU of 86.4% on the test set, but also exhibits robustness and excellent generalizability.

Multi-scale deep residual shrinkage networks with a hybrid attention mechanism for rolling bearing fault diagnosis

The fault diagnosis of rolling bearings based on deep networks is hindered by the unexpected noise involved with accessible vibration signals and global information abatement in deepened networks. To combat the degradation, a multi-scale deep residual shrinkage network with a hybrid attention mechanism (MH-DRSN) is proposed in this paper. First, a spatial domain attention mechanism is introduced into the residual shrinkage module to represent the distance dependence of the feature maps. Then, a hybrid attention mechanism considering both the inner-channeled and cross-channeled characteristics is constructed. Through the comprehensive evaluation of the feature map, it provides a soft threshold for the activation function and realizes the feature-map selection adaptively. Second, the dilated convolution with different dilation rates is implemented for multi-scale context information extraction. Through the feature combination of the DRSN and the dilated convolution, the global information of the rolling bearing fault is strengthened and preserved as the fault diagnosis network is deepened. Finally, the performance of the proposed fault-diagnosis model is validated on the dataset from Case Western Reserve University (CWRU). The experimental results show that, compared with common convolution neural networks, the proposed neural diagnosis model provides a higher identification accuracy and better robustness under noise interference.

基于DeepLabV3的复杂果园环境下樱桃分割与识别

Aiming at the problems of less research on cherry segmentation and identification, with slow recognition speed and low classification accuracy in agricultural products, a method based on DeepLabV3 was proposed to realize the rapid segmentation and identification of cherry in complex orchard environment. Complex environment mainly includes front lighting, back lighting, cloudy and rainy days, single fruit, multi fruit, fruit overlap, and branch and leaf occlusion. This model proposed the Atrous Spatial Pyramid Pooling (ASPP) module to effectively extract multi-scale contextual information, and solved the problem of target segmentation at multiple scales. The obtained data was divided into training, validation and testing sets in 7:1:2 ratios, and the residual network 50 (ResNet50) was selected as backbone of the DeepLabV3. Experimental results show that the algorithm in this paper can segment cherry quickly and accurately, the mean intersection over union (MIoU) was 91.06%, the mean pixel accuracy (MPA) was 93.05%, and the kappa coefficient was 0.89, which was better than fully convolutional networks (FCN), SegNet, DeepLabV1 and DeepLabV2. It is demonstrated that this study can provide technical support for intelligent segmentation of agricultural products.