Multi-scale Contextual Information Research Articles

A context-ensembled refinement network for image segmentation of coated fuel particles

The thickness of the coating layer of coated fuel particles plays a vital role in the reliable operation of a high-temperature gas-cooled reactor. Coating thickness remains extremely challenging to measure it efficiently, accurately, and robotically. The existing methods struggle with poorly refined segmentation results and easy adhesion of boundaries, complicating the accurate image segmentation of coating layers, which is fundamental to achieving the measurement goal. To overcome these issues, we designed the Context-Ensembled Refinement Network. This network refines the segmentation results to accurately extract different coating regions, leveraging its context fusion module and boundary refinement module to effectively utilize multiscale contextual information and provide strong boundary refinement capability. Experimental results demonstrate that our proposed method outperforms recent state-of-the-art methods, achieving an mIoU value of 95.8 %, and holds promising potential for automatic measurement of coating thickness.

Lightweight multi-scale dynamic selection network for medical image segmentation

Medical image segmentation technology can assist doctors in accurately diagnosing and treating diseases. However, the diversity of segmented targets in scale and shape, as well as the complexity of the background environment, pose challenges for existing segmentation methods. In this context, we propose two effective deep neural networks, the multi-scale dynamic selection network (MDSNet) and its lightweight version MDSNet-Light for medical image segmentation. In MDSNet, we design a multi-scale dynamic selection module to capture multi-scale contextual information and dynamically adjust the acceptance domain of the feature map to achieve feature fusion at the optimal scale. We introduce PixelShuffle and PixelUnshuffle as sampling methods to effectively alleviate information loss issues for small-scale objects caused by pooling. Experimental results show that MDSNet outperforms eight existing superior neural networks in the test data of the publicly available dataset LIDC-IDRI on four evaluation indicators. And MDSNet-Light achieves a good balance between network performance and computational complexity, providing an option for applications in limited computing resource scenarios.

Incorporating convolutional and transformer architectures to enhance semantic segmentation of fine-resolution urban images

ABSTRACT Though convolutional neural networks (CNN) exhibit promise in image semantic segmentation, they have limitations in capturing global context information, resulting in inaccuracies in segmenting small object features and object boundaries. This study introduces a hybrid network, ICTANet, which incorporate convolutional and Transformer architectures to improve the segmentation performance of fine-resolution remote sensing urban imagery. The ICTANet model is essentially a Transformer-based encoder-decoder structure. The dual-encoder architecture, which combines CNN and Swin Transformer modules, is designed to extract both global and local detail information. The feature information at various stages is collected by the Feature Extraction and Fusion modules (FEF), enabling multi-scale contextual information fusion. In addition, an Auxiliary Boundary Detection (ABD) module is introduced at the end of the decoder to enhance the model’s ability to capture object boundary information. Numerous ablation experiments have been conducted to demonstrate the efficacy of various components within the network. The testing results have proven that the proposed model can achieve satisfactory performance on the ISPRS Vaihingen and Potsdam dataset, with overall accuracies of 91.9% and 92.0%, respectively. Simultaneously, the proposed model is also compared to the current state-of-the-art methods, exhibiting competitive performance, particularly in the segmentation of diminutive objects like cars and trees.

GMAlignNet: multi-scale lightweight brain tumor image segmentation with enhanced semantic information consistency

Although the U-shaped architecture, represented by UNet, has become a major network model for brain tumor segmentation, the repeated convolution and sampling operations can easily lead to the loss of crucial information. Additionally, directly fusing features from different levels without distinction can easily result in feature misalignment, affecting segmentation accuracy. On the other hand, traditional convolutional blocks used for feature extraction cannot capture the abundant multi-scale information present in brain tumor images. This paper proposes a multi-scale feature-aligned segmentation model called GMAlignNet that fully utilizes Ghost convolution to solve these problems. Ghost hierarchical decoupled fusion unit and Ghost hierarchical decoupled unit are used instead of standard convolutions in the encoding and decoding paths. This transformation replaces the holistic learning of volume structures by traditional convolutional blocks with multi-level learning on a specific view, facilitating the acquisition of abundant multi-scale contextual information through low-cost operations. Furthermore, a feature alignment unit is proposed that can utilize semantic information flow to guide the recovery of upsampled features. It performs pixel-level semantic information correction on misaligned features due to feature fusion. The proposed method is also employed to optimize three classic networks, namely DMFNet, HDCNet, and 3D UNet, demonstrating its effectiveness in automatic brain tumor segmentation. The proposed network model was applied to the BraTS 2018 dataset, and the results indicate that the proposed GMAlignNet achieved Dice coefficients of 81.65%, 90.07%, and 85.16% for enhancing tumor, whole tumor, and tumor core segmentation, respectively. Moreover, with only 0.29 M parameters and 26.88G FLOPs, it demonstrates better potential in terms of computational efficiency and possesses the advantages of lightweight. Extensive experiments on the BraTS 2018, BraTS 2019, and BraTS 2020 datasets suggest that the proposed model exhibits better potential in handling edge details and contour recognition.

STF-Net: sparsification transformer coding guided network for subcortical brain structure segmentation.

Subcortical brain structure segmentation plays an important role in the diagnosis of neuroimaging and has become the basis of computer-aided diagnosis. Due to the blurred boundaries and complex shapes of subcortical brain structures, labeling these structures by hand becomes a time-consuming and subjective task, greatly limiting their potential for clinical applications. Thus, this paper proposes the sparsification transformer (STF) module for accurate brain structure segmentation. The self-attention mechanism is used to establish global dependencies to efficiently extract the global information of the feature map with low computational complexity. Also, the shallow network is used to compensate for low-level detail information through the localization of convolutional operations to promote the representation capability of the network. In addition, a hybrid residual dilated convolution (HRDC) module is introduced at the bottom layer of the network to extend the receptive field and extract multi-scale contextual information. Meanwhile, the octave convolution edge feature extraction (OCT) module is applied at the skip connections of the network to pay more attention to the edge features of brain structures. The proposed network is trained with ahybrid loss function. The experimental evaluation on two public datasets: IBSR and MALC, shows outstanding performance in terms of objective and subjective quality.

An IIoT Temporal Data Anomaly Detection Method Combining Transformer and Adversarial Training

The existing Industrial Internet of Things (IIoT) temporal data analysis methods often suffer from issues such as information loss, difficulty balancing spatial and temporal features, and being affected by training data noise, which can lead to varying degrees of reduced model accuracy. Therefore, a new anomaly detection method was proposed, which integrated Transformer and adversarial training. Firstly, a bidirectional spatiotemporal feature extraction module was constructed by combining Graph Attention Networks (GAT) and Bidirectional Gated Recurrent Unit (BiGRU), which can simultaneously extract spatial and temporal features. Then, by combining multi-scale convolution with Long Short-Term Memory (LSTM), multi-scale contextual information was captured. Finally, an improved Transformer was used to fuse multi-dimensional features, combined with an adversarial-trained variational autoencoder to calculate the anomalies of the input data. This method outperforms other comparison models by conducting experiments on four publicly available datasets.

Multi-Scale Contextual Swin Transformer for Crop Image Segmentation

Combining UAV-based remote sensing with deep learning for image segmentation is a particularly innovative and effective technology in modern agriculture. This approach allows for detailed and precise analysis of agricultural fields, such as crop monitoring, yield prediction, and irrigation management, enhancing decision-making and farm management practices. Inspired by the recent advancements of Transformers in computer vision, this paper introduces the Multi-Scale Contextual Swin Transformer (MSC-Swin), a novel model for precise segmentation of UAV crop images. MSC-Swin innovatively combines a Swin Transformer architecture for detailed feature extraction with pooling operations to utilize multi-scale contextual information. Our extensive experimentation demonstrates that MSC-Swin not only achieves state-of-the-art performance on the Barley Remote Sensing dataset, with a record mIoU of 86.4% on the test set, but also exhibits robustness and excellent generalizability.

Multi-scale deep residual shrinkage networks with a hybrid attention mechanism for rolling bearing fault diagnosis

The fault diagnosis of rolling bearings based on deep networks is hindered by the unexpected noise involved with accessible vibration signals and global information abatement in deepened networks. To combat the degradation, a multi-scale deep residual shrinkage network with a hybrid attention mechanism (MH-DRSN) is proposed in this paper. First, a spatial domain attention mechanism is introduced into the residual shrinkage module to represent the distance dependence of the feature maps. Then, a hybrid attention mechanism considering both the inner-channeled and cross-channeled characteristics is constructed. Through the comprehensive evaluation of the feature map, it provides a soft threshold for the activation function and realizes the feature-map selection adaptively. Second, the dilated convolution with different dilation rates is implemented for multi-scale context information extraction. Through the feature combination of the DRSN and the dilated convolution, the global information of the rolling bearing fault is strengthened and preserved as the fault diagnosis network is deepened. Finally, the performance of the proposed fault-diagnosis model is validated on the dataset from Case Western Reserve University (CWRU). The experimental results show that, compared with common convolution neural networks, the proposed neural diagnosis model provides a higher identification accuracy and better robustness under noise interference.

基于DeepLabV3的复杂果园环境下樱桃分割与识别

Aiming at the problems of less research on cherry segmentation and identification, with slow recognition speed and low classification accuracy in agricultural products, a method based on DeepLabV3 was proposed to realize the rapid segmentation and identification of cherry in complex orchard environment. Complex environment mainly includes front lighting, back lighting, cloudy and rainy days, single fruit, multi fruit, fruit overlap, and branch and leaf occlusion. This model proposed the Atrous Spatial Pyramid Pooling (ASPP) module to effectively extract multi-scale contextual information, and solved the problem of target segmentation at multiple scales. The obtained data was divided into training, validation and testing sets in 7:1:2 ratios, and the residual network 50 (ResNet50) was selected as backbone of the DeepLabV3. Experimental results show that the algorithm in this paper can segment cherry quickly and accurately, the mean intersection over union (MIoU) was 91.06%, the mean pixel accuracy (MPA) was 93.05%, and the kappa coefficient was 0.89, which was better than fully convolutional networks (FCN), SegNet, DeepLabV1 and DeepLabV2. It is demonstrated that this study can provide technical support for intelligent segmentation of agricultural products.

MSGAT: Multi-scale gated axial reverse attention transformer network for medical image segmentation

Medical image segmentation can help doctors accurately identify and locate various structures, tissues, or lesions in the patient's body, providing an important basis for clinical diagnosis. However, the segmentation of medical images is seriously affected by the complexity of the structures in the images, the variety of morphologies, the uneven contrast, and the blurring of the boundaries between the target tissues and the background. Therefore, we propose a Multiscale Gated Axial Reverse Attention Transformer (MSGAT) for medical image segmentation. First, we use the Pyramid Vision Transformer (PVT) as the backbone network to extract the high-level features and introduce the Channel Received Field Block (CRFB) to filter the interfering information in the high-level features, and then we augment and aggregate the high-level features separately. Specifically, on the one hand, the Multiscale Feature Enhancement Module (MSFEM) is used to obtain multiscale contextual information to enhance the positional information of the target, and on the other hand, Multiscale Parallel Partial Decoder (MSPPD) is employed to aggregate the high-level feature maps to generate rough segmentation maps. Finally, the rough segmentation map is used to guide the gated axial reverse attention (GARA) combined with the position information in the enhanced feature map to refine the segmentation boundaries and gradually mine the detailed information of the target. MSGAT is evaluated against state-of-the-art methods on skin cancer (ISIC2016, PH2), polyp (Kvair), and gland (GlaS) datasets. Results demonstrate MSGAT's superiority, with an average improvement of 6.19% and 9.35% in mIoU and DSC metrics, respectively, compared to existing methods.

FNeXter: A Multi-Scale Feature Fusion Network Based on ConvNeXt and Transformer for Retinal OCT Fluid Segmentation.

The accurate segmentation and quantification of retinal fluid in Optical Coherence Tomography (OCT) images are crucial for the diagnosis and treatment of ophthalmic diseases such as age-related macular degeneration. However, the accurate segmentation of retinal fluid is challenging due to significant variations in the size, position, and shape of fluid, as well as their complex, curved boundaries. To address these challenges, we propose a novel multi-scale feature fusion attention network (FNeXter), based on ConvNeXt and Transformer, for OCT fluid segmentation. In FNeXter, we introduce a novel global multi-scale hybrid encoder module that integrates ConvNeXt, Transformer, and region-aware spatial attention. This module can capture long-range dependencies and non-local similarities while also focusing on local features. Moreover, this module possesses the spatial region-aware capabilities, enabling it to adaptively focus on the lesions regions. Additionally, we propose a novel self-adaptive multi-scale feature fusion attention module to enhance the skip connections between the encoder and the decoder. The inclusion of this module elevates the model's capacity to learn global features and multi-scale contextual information effectively. Finally, we conduct comprehensive experiments to evaluate the performance of the proposed FNeXter. Experimental results demonstrate that our proposed approach outperforms other state-of-the-art methods in the task of fluid segmentation.

SwinPA-Net: Swin Transformer-Based Multiscale Feature Pyramid Aggregation Network for Medical Image Segmentation.

The precise segmentation of medical images is one of the key challenges in pathology research and clinical practice. However, many medical image segmentation tasks have problems such as large differences between different types of lesions and similar shapes as well as colors between lesions and surrounding tissues, which seriously affects the improvement of segmentation accuracy. In this article, a novel method called Swin Pyramid Aggregation network (SwinPA-Net) is proposed by combining two designed modules with Swin Transformer to learn more powerful and robust features. The two modules, named dense multiplicative connection (DMC) module and local pyramid attention (LPA) module, are proposed to aggregate the multiscale context information of medical images. The DMC module cascades the multiscale semantic feature information through dense multiplicative feature fusion, which minimizes the interference of shallow background noise to improve the feature expression and solves the problem of excessive variation in lesion size and type. Moreover, the LPA module guides the network to focus on the region of interest by merging the global attention and the local attention, which helps to solve similar problems. The proposed network is evaluated on two public benchmark datasets for polyp segmentation task and skin lesion segmentation task as well as a clinical private dataset for laparoscopic image segmentation task. Compared with existing state-of-the-art (SOTA) methods, the SwinPA-Net achieves the most advanced performance and can outperform the second-best method on the mean Dice score by 1.68%, 0.8%, and 1.2% on the three tasks, respectively.

An edge information fusion perception network for curtain wall frames segmentation

Automatically detection of curtain wall frames is a crucial part of building automation. However, the current curtain wall installation robots mainly rely on manual assistance for positioning, resulting in low efficiency and inaccuracy. To address these drawbacks, we propose an edge information fusion perception network (EIFP-Net) for RGB-D curtain wall frames detection. Specifically, a cross-modal context fusion module (CCFM) allows fusing of RGB and depth features to enhance cross-modal information complementation and capture multi-scale context information simultaneously. In addition, an edge features sensing module (EFSM) is derived from the RGB branch, which is committed to realizing edge feature extraction. Within this module, a differential enhancement module (DEM) is introduced to enhance the edge information. Finally, a multi-scale progressive refinement decoder (MPRD) is designed to refine the features, utilizing the edge information as a guide to capture comprehensive features. The experimental results on the constructed curtain wall frame dataset show that the proposed EIFP-Net performs better than the state-of-the-art detection models, achieving 85.94 %, 88.29 %, 96.84 %, 86.81 %, and 87.10 % on the five evaluation metrics of precision, recall, accuracy, mIoU, and F1-score, respectively.

Improved MRI Brain Tumor Segmentation Method Based on UNet++

Brain tumors are one of the most common tumors with high mortality. Accurate detection and quantitative evaluation of brain tumors are necessary for patient treatment. A reliable and effective method for automatic segmentation of brain tumors by magnetic resonance imaging (MRI) plays a crucial role in patient treatment. In recent years, the Unet++ network structure has shown better performance in image segmentation. However, it is deficient in small-scale feature extraction and segmentation accuracy. In this study, the AC-ResUnet++ network was proposed to address the problems of Unet++ network mentioned above, which combined residual module, convolutional block attention module (CBAM) and Atrous spatial pyramidal pooling (ASPP). The residual module can solve the problem of gradient explosion in the deep network, CBAM can focus on local feature information and enhance the feature learning ability of the network for the target range, and ASPP module can generate different receptive fields and obtain multi-scale contextual feature information. The BraTS 2018 and BraTS 2019 datasets were used to test the proposed network. The Dice similarity coefficient of AC-ResUnet++ in whole tumor, tumor core and enhanced tumor are 0.8521, 0.8696 and 0.7868, respectively. Compared with Unet++ network, the Dice similarity coefficient increased by 1.22%, 5.08% and 2.03% on the baselines. The comprehensive results demonstrate that AC-ResUnet++ performs more effectively in improving the segmentation accuracy of brain tumor.

MpMsCFMA-Net: Multi-path Multi-scale Context Feature Mixup and Aggregation Network for medical image segmentation

Automatic and accurate medical image segmentation is a crucial step for clinical diagnosis and treatment planning of diseases. The advanced convolutional neural network (CNN) approaches based on the encoder–decoder structure have achieved state-of-the-art performances in many different medical image segmentation tasks. However, existing networks have insufficient capability of extracting the context information in each encoding stage, so they cannot effectively perceive multi-scale objects in images. In addition, the continuous down-sampling and convolution operations in the encoding stage lead to much loss of the detailed information, resulting in poor segmentation performance. In this paper, we propose a Multi-path Multi-scale Context Feature MixUp and Aggregation Network (named MpMsCFMA-Net) which fuses and aggregates multi-path features with multi-scale context information to address these issues. Based on the encoder–decoder structure, we first design the encoder to encode the semantic and detailed information of input images and introduce multi-scale context extraction module in each encoding stage. Furthermore, we design multiple features mixup module between the encoder and the decoder, aiming at providing different levels of global context information for the decoder by reconstructing skip-connection. Finally, we introduce the decoder with deeper features aggregation to better fuse multi-scale context information across layers. Experimental results on four public medical image datasets confirm that our proposed network achieves promising results and outperforms other state-of-the-art methods in most of evaluation metrics. The source code will be publicly available at https://github.com/tricksterANDthug/MpMsCFMA-Net.

STCNet: Alternating CNN and improved transformer network for COVID-19 CT image segmentation

Since the emergence of the Corona Virus Disease in 2019 (COVID-19), it has become a serious health problem affecting the human respiratory system. At present, automatic segmentation of lung infection areas from Computed Tomography has been playing a crucial role in the diagnosis of this disease because of its ability to perform pathological studies based on the lung infection areas. However, due to the lung infection areas scattered distribution, the existing segmentation methods generally have the problems of missing and incomplete segmentation. The Convolutional Neural Network (CNN)-based approaches generally lack the ability to model explicit long-range relation, and the transformer-based methods are not conducive to capturing the detailed boundaries of infected areas. Whereas the infected regions of the coronavirus images are scattered and boundary information plays an important role, both the boundaries and the global infected areas need to be taken into account. Therefore, we propose a novel coronavirus image segmentation network alternately using Swin transformer and CNN (STCNet). Firstly, to enable network to capture richer features, the ReSwin transformer block is proposed and added after each level of convolution block in the encoder-decoder. Secondly, to effectively retain the infected areas boundary information, the skip connection cross attention module is used to provide spatial information to each decoder. And through the fine-tuned scale-aware pyramid fusion module to fuse multi-scale context information. Experimental results show that STCNet at can achieve state-of-the-art performance on two coronavirus segmentation datasets, with Dice achieves 79.92 % and 82.78 %, respectively. Our code is available at https://github.com/sineagles/STCNet.

A Parallel Convolutional Network Based on Spiking Neural Systems.

Deep convolutional neural networks have shown advanced performance in accurately segmenting images. In this paper, an SNP-like convolutional neuron structure is introduced, abstracted from the nonlinear mechanism in nonlinear spiking neural P (NSNP) systems. Then, a U-shaped convolutional neural network named SNP-like parallel-convolutional network, or SPC-Net, is constructed for segmentation tasks. The dual-convolution concatenate (DCC) and dual-convolution addition (DCA) network blocks are designed, respectively, in the encoder and decoder stages. The two blocks employ parallel convolution with different kernel sizes to improve feature representation ability and make full use of spatial detail information. Meanwhile, different feature fusion strategies are used to fuse their features to achieve feature complementarity and augmentation. Furthermore, a dual-scale pooling (DSP) module in the bottleneck is designed to improve the feature extraction capability, which can extract multi-scale contextual information and reduce information loss while extracting salient features. The SPC-Net is applied in medical image segmentation tasks and is compared with several recent segmentation methods on the GlaS and CRAG datasets. The proposed SPC-Net achieves 90.77% DICE coefficient, 83.76% IoU score and 83.93% F1 score, 86.33% ObjDice coefficient, 135.60 Obj-Hausdorff distance, respectively. The experimental results show that the proposed model can achieve good segmentation performance.

Integrating image and gene-data with a semi-supervised attention model for prediction of KRAS gene mutation status in non-small cell lung cancer.

KRAS is a pathogenic gene frequently implicated in non-small cell lung cancer (NSCLC). However, biopsy as a diagnostic method has practical limitations. Therefore, it is important to accurately determine the mutation status of the KRAS gene non-invasively by combining NSCLC CT images and genetic data for early diagnosis and subsequent targeted therapy of patients. This paper proposes a Semi-supervised Multimodal Multiscale Attention Model (S2MMAM). S2MMAM comprises a Supervised Multilevel Fusion Segmentation Network (SMF-SN) and a Semi-supervised Multimodal Fusion Classification Network (S2MF-CN). S2MMAM facilitates the execution of the classification task by transferring the useful information captured in SMF-SN to the S2MF-CN to improve the model prediction accuracy. In SMF-SN, we propose a Triple Attention-guided Feature Aggregation module for obtaining segmentation features that incorporate high-level semantic abstract features and low-level semantic detail features. Segmentation features provide pre-guidance and key information expansion for S2MF-CN. S2MF-CN shares the encoder and decoder parameters of SMF-SN, which enables S2MF-CN to obtain rich classification features. S2MF-CN uses the proposed Intra and Inter Mutual Guidance Attention Fusion (I2MGAF) module to first guide segmentation and classification feature fusion to extract hidden multi-scale contextual information. I2MGAF then guides the multidimensional fusion of genetic data and CT image data to compensate for the lack of information in single modality data. S2MMAM achieved 83.27% AUC and 81.67% accuracy in predicting KRAS gene mutation status in NSCLC. This method uses medical image CT and genetic data to effectively improve the accuracy of predicting KRAS gene mutation status in NSCLC.

Usage of an improved YOLOv5 for steel surface defect detection

Abstract The one-stage YOLOv5 steel surface defect detection has issues such as slow operation speed, loss of defect location and semantic information of small targets, and inadequate extraction of defect features. This study proposed a defect detection algorithm with improved YOLOv5 to solve these issues. The proposed algorithm used the slim-neck layer built by three new modules instead of the neck layer in YOLOv5s to achieve a lightweight network model. In addition, the spatial perception self-attention mechanism was introduced to enhance the feature extraction capability of the initial convolutional layer without limiting the input size. The improved Atrous Spatial Pyramid Pooling was added to expand the perceptual field and capture multiscale contextual information while preventing local information loss and enhancing the relevance of long-range information. The experimental results showed that the improved YOLOv5 algorithm has a reduced model volume, significantly higher detection accuracy and speed than the traditional algorithm, and the ability to detect steel surface defects quickly and accurately.

A Building Extraction Method for High-Resolution Remote Sensing Images with Multiple Attentions and Parallel Encoders Combining Enhanced Spectral Information

Accurately extracting pixel-level buildings from high-resolution remote sensing images is significant for various geographical information applications. Influenced by different natural, cultural, and social development levels, buildings may vary in shape and distribution, making it difficult for the network to maintain a stable segmentation effect of buildings in different areas of the image. In addition, the complex spectra of features in remote sensing images can affect the extracted details of multi-scale buildings in different ways. To this end, this study selects parts of Xi’an City, Shaanxi Province, China, as the study area. A parallel encoded building extraction network (MARS-Net) incorporating multiple attention mechanisms is proposed. MARS-Net builds its parallel encoder through DCNN and transformer to take advantage of their extraction of local and global features. According to the different depth positions of the network, coordinate attention (CA) and convolutional block attention module (CBAM) are introduced to bridge the encoder and decoder to retain richer spatial and semantic information during the encoding process, and adding the dense atrous spatial pyramid pooling (DenseASPP) captures multi-scale contextual information during the upsampling of the layers of the decoder. In addition, a spectral information enhancement module (SIEM) is designed in this study. SIEM further enhances building segmentation by blending and enhancing multi-band building information with relationships between bands. The experimental results show that MARS-Net performs better extraction results and obtains more effective enhancement after adding SIEM. The IoU on the self-built Xi’an and WHU building datasets are 87.53% and 89.62%, respectively, while the respective F1 scores are 93.34% and 94.52%.