Local Detailed Features Research Articles

MABDT: Multi-scale attention boosted deformable transformer for remote sensing image dehazing

Owing to the heterogeneous spatial distribution and non-uniform morphological characteristics of haze in remote sensing images (RSIs), conventional dehazing algorithms struggle to precisely recover the fine-grained details of terrestrial objects. To address this issue, a novel multi-scale attention boosted deformable Transformer (MABDT) tailored for RSI dehazing is proposed. This framework synergizes the multi-receptive field features elicited by convolutional neural network (CNN) with the long-term dependency features derived from Transformer, which facilitates a more adept restitution of texture and intricate detail information within RSIs. Firstly, spatial attention deformable convolution is introduced for computation of multi-head self-attention in the Transformer block, particularly in addressing complex haze scenarios encountered in RSIs. Subsequently, a multi-scale attention feature enhancement (MAFE) block is designed, tailored to capture local and multi-level detailed information features using multi-receptive field convolution operations, thereby accommodating non-uniform haze. Finally, a multi-level feature complementary fusion (MFCF) block is proposed, leveraging both shallow and deep features acquired from all encoding layers to augment each level of reconstructed image. The dehazing performance is evaluated on 6 open-source datasets, and quantitative and qualitative experimental results demonstrate the advancements of the proposed method in both metrical scores and visual quality. The source code is available at https://github.com/ningjin00/MABDT.

A Person Re-Identification Method for Fusing Convolutional Attention and Transformer Architecture

Rapid and accurate localization of target person via person re-identification is one of the important methods in intelligent security systems,but faces great challenges due to complex environmental changes. In order to build a model applicable to various complex scenes, deep learning-based person re-identification models have become a key research direction for researchers. The existing convolutional neural network-based or transformer-based deep learning models are still have certain shortcomings in the extraction of local details and are not able to cope well with realistic complex scenes. To address this problem, a method fusing convolutional neural network and transformer (FCAT) is proposed to enhance the transformer's attention to local detail information. This method mainly improves the transformer's ability to extract local detail features indirectly by embedding convolutional space attention and channel attention respectively to enhance the attention to important regions and important channel features in the image. Comparative ablation experiments on three publicly available person re-identification datasets demonstrate that FCAT achieves comparable results to existing methods on non-occluded datasets, and significantly improves performance on occluded datasets. The proposed model is also has a lightweight effect, and the inference speed is improved without increasing the amount of outer computation and model parameters.

ConvMambaSR: Leveraging State-Space Models and CNNs in a Dual-Branch Architecture for Remote Sensing Imagery Super-Resolution

Deep learning-based super-resolution (SR) techniques play a crucial role in enhancing the spatial resolution of images. However, remote sensing images present substantial challenges due to their diverse features, complex structures, and significant size variations in ground objects. Moreover, recovering lost details from low-resolution remote sensing images with complex and unknown degradations, such as downsampling, noise, and compression, remains a critical issue. To address these challenges, we propose ConvMambaSR, a novel super-resolution framework that integrates state-space models (SSMs) and Convolutional Neural Networks (CNNs). This framework is specifically designed to handle heterogeneous and complex ground features, as well as unknown degradations in remote sensing imagery. ConvMambaSR leverages SSMs to model global dependencies, activating more pixels in the super-resolution task. Concurrently, it employs CNNs to extract local detail features, enhancing the model’s ability to capture image textures and edges. Furthermore, we have developed a global–detail reconstruction module (GDRM) to integrate diverse levels of global and local information efficiently. We rigorously validated the proposed method on two distinct datasets, RSSCN7 and RSSRD-KQ, and benchmarked its performance against state-of-the-art SR models. Experiments show that our method achieves SOTA PSNR values of 26.06 and 24.29 on these datasets, respectively, and is visually superior, effectively addressing a variety of scenarios and significantly outperforming existing methods.

Identifying rice field weeds from unmanned aerial vehicle remote sensing imagery using deep learning

BackgroundRice field weed object detection can provide key information on weed species and locations for precise spraying, which is of great significance in actual agricultural production. However, facing the complex and changing real farm environments, traditional object detection methods still have difficulties in identifying small-sized, occluded and densely distributed weed instances. To address these problems, this paper proposes a multi-scale feature enhanced DETR network, named RMS-DETR. By adding multi-scale feature extraction branches on top of DETR, this model fully utilizes the information from different semantic feature layers to improve recognition capability for rice field weeds in real-world scenarios.MethodsIntroducing multi-scale feature layers on the basis of the DETR model, we conduct a differentiated design for different semantic feature layers. The high-level semantic feature layer adopts Transformer structure to extract contextual information between barnyard grass and rice plants. The low-level semantic feature layer uses CNN structure to extract local detail features of barnyard grass. Introducing multi-scale feature layers inevitably leads to increased model computation, thus lowering model inference speed. Therefore, we employ a new type of Pconv (Partial convolution) to replace traditional standard convolutions in the model.ResultsCompared to the original DETR model, our proposed RMS-DETR model achieved an average recognition accuracy improvement of 3.6% and 4.4% on our constructed rice field weeds dataset and the DOTA public dataset, respectively. The average recognition accuracies reached 0.792 and 0.851, respectively. The RMS-DETR model size is 40.8 M with inference time of 0.0081 s. Compared with three classical DETR models (Deformable DETR, Anchor DETR and DAB-DETR), the RMS-DETR model respectively improved average precision by 2.1%, 4.9% and 2.4%.DiscussionThis model is capable of accurately identifying rice field weeds in complex real-world scenarios, thus providing key technical support for precision spraying and management of variable-rate spraying systems.

PSF-C-Net: A Counterfactual Deep Learning Model for Person Re-Identification Based on Random Cropping Patch and Shuffling Filling

In the task of person re-identification (re-ID), capturing the long-range dependency of instances is crucial for accurate identification. The existing methods excel at extracting local features but often overlook the global information of instance images. To address this limitation, we propose a convolution-based counterfactual learning framework, called PSF-C-Net, to focus on global information rather than local detailed features. PSF-C-Net adopts a parameter-sharing dual-path structure to perform counterfactual operations in the prediction space. It takes both the actual instance image and a counterfactual instance image that disrupts the contextual relationship as the input. The counterfactual framework enables the interpretable modeling of global features without introducing additional parameters. Additionally, we propose a novel method for generating counterfactual instance images, which effectively constructs an explicit counterfactual space, to reliably implement counterfactual strategies. We have conducted extensive experiments to evaluate the performance of PSF-C-Net on the Market-1501 and Duke-MTMC-reID datasets. The results demonstrate that PSF-C-Net achieves state-of-the-art performance.

Enhancing precision of defect 3D reconstruction in metal ultrasonic testing through point cloud completion

During the ultrasound multi-layer focused scanning inspection process, the sequential images characterizing the defect morphology suffer from inter-layer contour information loss, which causes a reduction in the accuracy of defect 3D reconstruction, subsequently leading to errors in the characterization of the defect size and morphology. In order to address the above issues, a new method based on the Attention-based Residual Completion Network (ARC) is proposed for ultrasonic point cloud completion to characterize metal defects. Firstly, the ARC network extracts global contour morphological features and local edge detail features from the ultrasonic point cloud through consecutive residual convolutions. Subsequently, the two sets of features are concatenated and finally fed into a decoder based on self-attention, realizing the reconstruction of lost contour information and enhancing the 3D reconstruction accuracy of defects. In the experiment, an ultrasonic microscope was used to inspect actual steel plates. The internal defects were then completed using the ARC network, and the completion results were compared with the metallographic images of the defects. The experiment results indicated that, after completion, the characterization accuracy of defect morphology and sizes is enhanced by an average of 10.31 %. The ARC network provides a novel method for high-precision 3D characterization of internal defects in metal materials.

FDB-Net: Fusion double branch network combining CNN and transformer for medical image segmentation.

The rapid development of deep learning techniques has greatly improved the performance of medical image segmentation, and medical image segmentation networks based on convolutional neural networks and Transformer have been widely used in this field. However, due to the limitation of the restricted receptive field of convolutional operation and the lack of local fine information extraction ability of the self-attention mechanism in Transformer, the current neural networks with pure convolutional or Transformer structure as the backbone still perform poorly in medical image segmentation. In this paper, we propose FDB-Net (Fusion Double Branch Network, FDB-Net), a double branch medical image segmentation network combining CNN and Transformer, by using a CNN containing gnConv blocks and a Transformer containing Varied-Size Window Attention (VWA) blocks as the feature extraction backbone network, the dual-path encoder ensures that the network has a global receptive field as well as access to the target local detail features. We also propose a new feature fusion module (Deep Feature Fusion, DFF), which helps the image to simultaneously fuse features from two different structural encoders during the encoding process, ensuring the effective fusion of global and local information of the image. Our model achieves advanced results in all three typical tasks of medical image segmentation, which fully validates the effectiveness of FDB-Net.

Attention-Based Two-Branch Hybrid Fusion Network for Medical Image Segmentation

Accurate segmentation of medical images is vital for disease detection and treatment. Convolutional Neural Networks (CNN) and Transformer models are widely used in medical image segmentation due to their exceptional capabilities in image recognition and segmentation. However, CNNs often lack an understanding of the global context and may lose spatial details of the target, while Transformers struggle with local information processing, leading to reduced geometric detail of the target. To address these issues, this research presents a Global-Local Fusion network model (GLFUnet) based on the U-Net framework and attention mechanisms. The model employs a dual-branch network that utilizes ConvNeXt and Swin Transformer to simultaneously extract multi-level features from pathological images. It enhances ConvNeXt’s local feature extraction with spatial and global attention up-sampling modules, while improving Swin Transformer’s global context dependency with channel attention. The Attention Feature Fusion module and skip connections efficiently merge local detailed and global coarse features from CNN and Transformer branches at various scales. The fused features are then progressively restored to the original image resolution for pixel-level prediction. Comprehensive experiments on datasets of stomach and liver cancer demonstrate GLFUnet’s superior performance and adaptability in medical image segmentation, holding promise for clinical analysis and disease diagnosis.

Enhancing instance segmentation: Leveraging multiscale feature fusion and attention mechanisms for automated fish weight estimation

The accurate estimation of fish weight relies on the crucial parameter of individual fish contour features. While instance segmentation proves effective in extracting fish contours, challenges arise from diverse fish postures and reduced image sharpness underwater. Current instance segmentation methods often struggle to effectively balance global and local detailed features, which can result in inaccurate positioning of contour keypoints and consequently limit the accuracy of fish weight estimation. To overcome this, our study introduces a novel instance segmentation network tailored for precise fish contour extraction. The proposed approach incorporates multi-scale feature fusion and an attention mechanism based on the Segmenting Objects by Locations (SOLO) network, referred to as SOLO-MFFA. This paper designs a multi-scale context aggregation module to integrate features with a wider range of receptive fields, augmenting the model's capability to comprehend both local features and global information. At the same time, the introduction of a mixed-domain attention mechanism emphasizes more critical channel features and simultaneously improves the localization accuracy of contour points. Compared with SOLO and its improved model CAM-SOLO on the fish instance segmentation dataset, SOLO-MFFA demonstrated an effective improvement, with a 4.3% and 1.6% increase in mAP (mean Average Precision), respectively. The Decoupled-SOLO-MFFA achieved higher mAP. The visualization results also demonstrate that the contour features extracted in this paper are smoother and more accurately positioned. Additionally, in comparison to other well-known instance segmentation networks, our method has demonstrated significant improvements in both qualitative and quantitative evaluations. Furthermore, the integration of contour features derived from Decoupled-SOLO-MFFA, along with binocular vision, was utilized for the precise estimation of fish perimeter and weight. The findings reveal a strong correlation between the perimeter calculated by Decoupled-SOLO-MFFA and the actual weight, with a notably reduced error in weight estimation. Compared to previous methods, RMSE, MAE, and MAPE of the linear model constructed in this paper decreased by 3.92, 3.19, and 1.4%.

Multi-scale window transformer for cervical cytopathology image recognition

Cervical cancer is a major global health issue, particularly in developing countries where access to healthcare is limited. Early detection of pre-cancerous lesions is crucial for successful treatment and reducing mortality rates. However, traditional screening and diagnostic processes require cytopathology doctors to manually interpret a huge number of cells, which is time-consuming, costly, and prone to human experiences. In this paper, we propose a Multi-scale Window Transformer (MWT) for cervical cytopathology image recognition. We design multi-scale window multi-head self-attention (MW-MSA) to simultaneously integrate cell features of different scales. Small window self-attention is used to extract local cell detail features, and large window self-attention aims to integrate features from smaller-scale window attention to achieve window-to-window information interaction. Our design enables long-range feature integration but avoids whole image self-attention (SA) in ViT or twice local window SA in Swin Transformer. We find convolutional feed-forward networks (CFFN) are more efficient than original MLP-based FFN for representing cytopathology images. Our overall model adopts a pyramid architecture. We establish two multi-center cervical cell classification datasets of two-category 192,123 images and four-category 174,138 images. Extensive experiments demonstrate that our MWT outperforms state-of-the-art general classification networks and specialized classifiers for cytopathology images in the internal and external test sets. The results on large-scale datasets prove the effectiveness and generalization of our proposed model. Our work provides a reliable cytopathology image recognition method and helps establish computer-aided screening for cervical cancer. Our code is available at https://github.com/nmyz669/MWT, and our web service tool can be accessed at https://huggingface.co/spaces/nmyz/MWTdemo.

Computer aided diagnosis of diabetic retinopathy based on multi-view joint learning

Diabetic retinopathy (DR) is a kind of ocular complication of diabetes, and its degree grade is an essential basis for early diagnosis of patients. Manual diagnosis is a long and expensive process with a specific risk of misdiagnosis. Computer-aided diagnosis can provide more accurate and practical treatment recommendations. In this paper, we propose a multi-view joint learning DR diagnostic model called RT2Net, which integrates the global features of fundus images and the local detailed features of vascular images to reduce the limitations of single fundus image learning. Firstly, the original image is preprocessed using operations such as contrast-limited adaptive histogram equalization, and the vascular structure of the extracted DR image is segmented. Then, the vascular image and fundus image are input into two branch networks of RT2Net for feature extraction, respectively, and the feature fusion module adaptively fuses the feature vectors’ output from the branch networks. Finally, the optimized classification model is used to identify the five categories of DR. This paper conducts extensive experiments on the public datasets EyePACS and APTOS 2019 to demonstrate the method’s effectiveness. The accuracy of RT2Net on the two datasets reaches 88.2% and 85.4%, and the area under the receiver operating characteristic curve (AUC) is 0.98 and 0.96, respectively. The excellent classification ability of RT2Net for DR can significantly help patients detect and treat lesions early and provide doctors with a more reliable diagnosis basis, which has significant clinical value for diagnosing DR.

Hierarchical Patch Aggregation Transformer for Motion Deblurring

The encoder-decoder framework based on Transformer components has become a paradigm in the field of image deblurring architecture design. In this paper, we critically revisit this approach and find that many current architectures severely focus on limited local regions during the feature extraction stage. These designs compromise the feature richness and diversity of the encoder-decoder framework, leading to bottlenecks in performance improvement. To address these deficiencies, a novel Hierarchical Patch Aggregation Transformer architecture (HPAT) is proposed. In the initial feature extraction stage, HPAT combines Axis-Selective Transformer Blocks with linear complexity and is supplemented by an adaptive hierarchical attention fusion mechanism. These mechanisms enable the model to effectively capture the spatial relationships between features and integrate features from different hierarchical levels. Then, we redesign the feedforward network of the Transformer block in the encoder-decoder structure and propose the Fused Feedforward Network. This effective aggregation enhances the ability to capture and retain local detailed features. We evaluate HPAT through extensive experiments and compare its performance with baseline methods on public datasets. Experimental results show that the proposed HPAT model achieves state-of-the-art performance in image deblurring tasks.

Construction site environment helmet detection method based on multi-scale context and attention fusion

Construction site environment helmet detection is of great significance for protecting workers’ lives and realizing the automation of safety management. Aiming at the current object detection methods for the complex construction site environment in the small-scale helmet object detection ability is insufficient. This paper proposes a construction site environment helmet detection method based on multi-scale context and attention fusion. The method is able to aggregate the multi-scale contextual semantics of deep image features through the proposed multi-scale context module and expand the receptive field in order to improve the network’s discriminative learning ability for small-scale helmet objects. Meanwhile, the proposed attention feature fusion module dynamically fuses features from shallow features and network decoding features to enhance the network’s ability to learn the expression of global feature dependencies and local spatial detail features of helmet objects, and further improve the network’s detection precision of helmet objects. The experimental results show that on the constructed safety helmet wearing dataset, the proposed method in this paper has good detection effect and balanced detection speed compared with the existing mainstream object detection methods.

DHFNet: Decoupled Hierarchical Fusion Network for RGB-T dense prediction tasks

The fusion of RGB and thermal data for dense prediction tasks has been demonstrated to be an effective and robust approach in autonomous driving. Nevertheless, the challenge lies in fusing features from different modalities, as a simple fusion strategy may lead to redundant or conflicting semantic information. In this paper, we delve into the hierarchical connections between multi-modal features and propose a novel fusion paradigm termed DHFNet, which decouples multi-modal features into similar global long-distance and discrepant local detail features for hierarchical feature fusion. At each fusion stage, a Lightweight Global Self-attention (LGSA) module is designed to decouple the global long-distance features with cheap computational complexity cost, and a Cross Modal Long-distance Feature Fusion (CMLFF) module is designed to eliminate redundant features by facilitating information interaction between different modalities. During the process of decoupling local detail features, a Cross Modal Deformable Convolution (CMDC) module is proposed to dynamically extract effective local features and capture misaligned features between different modalities. Finally, the fused global long-distance and local detail features are recoupled to achieve efficient hierarchical fusion. The results on RGB-T semantic segmentation and object detection tasks demonstrate the effectiveness of proposed method. The code will be available at: https://github.com/donggaomu/DHFNet.

MF-Net: Multiple-feature extraction network for breast lesion segmentation in ultrasound images

Objective:Breast lesion segmentation in ultrasound images is of great significance for qualitative breast lesions. However, blurred lesion boundaries, irregular lesion shapes, and similar intensity distributions between lesion and background bring challenges to accurately segmenting breast lesions. Recently, several U-Net-based variants and transformer-based networks have been applied in breast lesion segmentation. Nevertheless, these methods have three limitations: (1) Introducing mass attention mechanisms and complex operations, (2) Ignoring the capability of extracting local detail features, and (3) Ignoring fusing different global semantic information. Approach:To alleviate these challenges, we propose a novel multiple-feature extraction network named MF-Net. The core designs are as follows: (1) A transformer-based auxiliary bi-encoder models long-range dependencies, (2) A multiple-feature extraction module excavates the robust local features, and (3) A global feature enhancement module integrates different global context information. Main results:We conducted extensive experiments on three public breast lesion datasets and compared our method with twelve state-of-the-art methods. The results of comparison experiments indicate that our model achieves a 3.16% improvement in boundary accuracy with a slight lead in segmentation accuracy. Similarly, when the boundary accuracy is close, our model also gets a 3.92% improvement in segmentation accuracy. In the generalization experiments, our network also demonstrates varying degrees of superiority in segmentation accuracy and boundary precision. Significance:Our network mainly has three advantages: (1) It achieves outstanding performance in the precise segmentation of the breast lesions, (2) It focuses on exploring details such as irregular morphology and blurred boundaries of breast lesions, providing valuable insights for further clinical research, and (3) It improves performance without decreasing inference speed through the introduction of critical components.

A lightweight contour detection network inspired by biology

In recent years, the field of bionics has attracted the attention of numerous scholars. Some models combined with biological vision have achieved excellent performance in computer vision and image processing tasks. In this paper, we propose a new bio-inspired lightweight contour detection network (BLCDNet) by combining parallel processing mechanisms of bio-visual information with convolutional neural networks. The backbone network of BLCDNet simulates the parallel pathways of ganglion cell–lateral geniculate nucleus and primary visual cortex (V1) area, realizing parallel processing and step-by-step extraction of input information, effectively extracting local features and detailed features in images, and thus improving the overall performance of the model. In addition, we design a depth feature extraction module combining depth separable convolution and residual connection in the decoding network to integrate the output of the backbone network, which further improves the performance of the model. We conducted a large number of experiments on BSDS500 and NYUD datasets, and the experimental results show that the BLCDNet proposed in this paper achieves the best performance compared with traditional methods and previous biologically inspired contour detection methods. In addition, BLCDNet still outperforms some VGG-based contour detection methods without pre-training and with fewer parameters, and it is competitive among all of them. The research in this paper also provides a new idea for the combination of biological vision and convolutional neural networks.

PCTDepth: Exploiting Parallel CNNs and Transformer via Dual Attention for Monocular Depth Estimation

Monocular depth estimation (MDE) has made great progress with the development of convolutional neural networks (CNNs). However, these approaches suffer from essential shortsightedness due to the utilization of insufficient feature-based reasoning. To this end, we propose an effective parallel CNNs and Transformer model for MDE via dual attention (PCTDepth). Specifically, we use two stream backbones to extract features, where ResNet and Swin Transformer are utilized to obtain local detail features and global long-range dependencies, respectively. Furthermore, a hierarchical fusion module (HFM) is designed to actively exchange beneficial information for the complementation of each representation during the intermediate fusion. Finally, a dual attention module is incorporated for each fused feature in the decoder stage to improve the accuracy of the model by enhancing inter-channel correlations and focusing on relevant spatial locations. Comprehensive experiments on the KITTI dataset demonstrate that the proposed model consistently outperforms the other state-of-the-art methods.

Dual fusion network for semantic segmentation of point clouds

The current networks for point cloud segmentation fail to fully consider the connections between points in each neighborhood, which makes the final segmentation capability underperform. To obtain richer local relationships between points, this paper proposes a network called GDFNet (Graph Convolution-based Dual Fusion Network), which dynamically mines the correlations between points in each layer of the network. GDFNet is divided into two parts: a density-sensitive assignment module and a spatial dynamic convolution module. The density-sensitive assignment module treats the traditional convolution kernel as a nonlinear function consisting of a weight and a density function applicable to the local coordinates of 3D points. For a given point, the weight function is learned by a multilayer perceptron network, and the density function is learned by kernel density estimation. In this way, the density features of each point are fitted, thus improving the inherent defect of the original point cloud with non-uniform density. The spatial dynamic convolution module groups the original feature channels and then uses gated processing units to enrich the feature representation of the input points. At the same time, dynamic convolution is embedded in the Euclidean space to obtain the structural information of each point and further supplement the local detail features. Extensive experiments demonstrate the effectiveness and superiority of our method on S3DIS and ScanNetv2 datasets.

Dual-stream pyramid and attention network for stereo matching

In order to accurately estimate the disparity of ill-posed regions, such as weak texture and occlusion regions, we propose DSPANet, a stereo matching network that incorporates a dual-stream pyramid module and a channel and spatial attention module. The dual-stream pyramid module captures numerous complementary features from different layers by utilizing multi-resolution inputs and feature extraction blocks. This approach enables the learning of local detailed features at various scales. These features at various scales are then combined to calculate the stereo matching cost. By incorporating channel and spatial attention module into the feature extraction process to obtain richer and more concise contextual information, the matching cost can be constructed more accurately, which provides powerful conditions for subsequent cost aggregation. In the cost aggregation stage, we utilize the stacked hourglass module for both encoding and decoding. Additionally, we incorporate 3D global attention upsampling during the decoding stage, which enables high-level features to provide guidance information to low-level features in a simple way. We evaluate our proposed method on the Scene Flow dataset, as well as the KITTI2012 and KITTI2015 datasets. The experimental results demonstrate that our DSPANet achieves superior performance and effectively enhances the matching results in ill-posed regions. Our code has been implemented using PyTorch and will be released after paper publication at https://github.com/Shi-LiQing/DSPANet.

A Deep Learning Method for Building Extraction from Remote Sensing Images by Fuzing Local and Global Features

As important disaster-bearing bodies, buildings are the focus of attention in seismic disaster risk assessment and emergency rescue. It is of great practical significance to extract buildings quickly and accurately with complex textures and variable scales and shapes from high-resolution remote sensing images. We proposed an improved TransUnet model based on multiscale grouped convolution and attention named MATUnet to retain more local detail features and enhance the representation ability of global features, while reducing the network parameters. We designed the multiscale grouped convolutional feature extraction module with attention (GAM) to enhance the representation of detailed features. The convolutional positional encoding module (PEG) was added to redetermine the number of transformer, it solved the problem of local feature information loss and the difficulty of convergence of the network. The channel attention module (CAM) of the decoder enhanced the salient information of the features and solved the problem of information redundancy after feature fusion. We experimented through MATUnet on the WHU building dataset and Massachusetts dataset. MATUnet achieved the best IOU results of 92.14% and 83.22%, respectively, and achieved better than the other generalized and state-of-the-art networks under the same conditions. We also have achieved good segmentation results on the GF2 Xichang building dataset.