Multi-level Feature Aggregation Research Articles

BMFA-Net: Boundary Constraint Multi-Level Feature Aggregation Framework for Precise Polyp Segmentation

BMFA-Net: Boundary Constraint Multi-Level Feature Aggregation Framework for Precise Polyp Segmentation

EWT: Efficient Wavelet-Transformer for single image denoising

Transformer-based image denoising methods have shown remarkable potential but suffer from high computational cost and large memory footprint due to their linear operations for capturing long-range dependencies. In this work , we aim to develop a more resource-efficient Transformer-based image denoising method that maintains high performance. To this end, we propose an Efficient Wavelet Transformer (EWT), which incorporates a Frequency-domain Conversion Pipeline (FCP) to reduce image resolution without losing critical features, and a Multi-level Feature Aggregation Module (MFAM) with a Dual-stream Feature Extraction Block (DFEB) to harness hierarchical features effectively. EWT achieves a faster processing speed by over 80% and reduces GPU memory usage by more than 60% compared to the original Transformer, while still delivering denoising performance on par with state-of-the-art methods. Extensive experiments show that EWT significantly improves the efficiency of Transformer-based image denoising, providing a more balanced approach between performance and resource consumption.

Multilevel feature aggregation and enhancement network for remote sensing change detection

Multilevel feature aggregation and enhancement network for remote sensing change detection

Attention guided multi-level feature aggregation network for camouflaged object detection

Camouflaged object detection (COD) aims to identify objects that are visually blended into their highly similar surroundings, which is an extremely complex and challenging visual task in real-world scenarios, and has recently attracted increasing research interest in the field of computer vision due to its valuable applications. The existing deep learning based methods of COD have the following problems: 1) the ambiguous boundary of the camouflaged objects in prediction map, 2) an inaccurate detection of the camouflaged object with accurate and complete structure details. To this end, an attention guided multi-level feature aggregation network is proposed for this task, which is based on three key designs. First, by embedding spatial pyramid attention (SPA) in ResNet-like backbone network, better multi-level features are extracted to identify the camouflaged objects with complete internal details. Second, an edge context module is designed to make full use of edge information, which highlights camouflaged object structure and generates accurate edge localization of COD. Third, the feature aggregation module based on local attention is used to fuse these enhanced multi-level features and edge context cues, which consists of two major components: the iterative Attentional Feature Fusion (iAFF) module and the Dual-branch Global Context Module (DGCM). Compared with the existing 16 state-of-the-art methods, extensive experiments on four widely-used benchmark datasets under four authoritative evaluation metrics illustrate that the proposed method is very beneficial to the COD task.

Two-stream Multi-level Dynamic Point Transformer for Two-person Interaction Recognition

As a fundamental aspect of human life, two-person interactions contain meaningful information about people’s activities, relationships, and social settings. Human action recognition serves as the foundation for many smart applications, with a strong focus on personal privacy. However, recognizing two-person interactions poses more challenges due to increased body occlusion and overlap compared to single-person actions. In this paper, we propose a point cloud-based network named Two-stream Multi-level Dynamic Point Transformer for two-person interaction recognition. Our model addresses the challenge of recognizing two-person interactions by incorporating local-region spatial information, appearance information, and motion information. To achieve this, we introduce a designed frame selection method named Interval Frame Sampling (IFS), which efficiently samples frames from videos, capturing more discriminative information in a relatively short processing time. Subsequently, a frame features learning module and a two-stream multi-level feature aggregation module extract global and partial features from the sampled frames, effectively representing the local-region spatial information, appearance information, and motion information related to the interactions. Finally, we apply a transformer to perform self-attention on the learned features for the final classification. Extensive experiments are conducted on two large-scale datasets, the interaction subsets of NTU RGB+D 60 and NTU RGB+D 120. The results show that our network outperforms state-of-the-art approaches in most standard evaluation settings.

Hybrid CNN-Transformer Network with Circular Feature Interaction for Acute Ischemic Stroke Lesion Segmentation on Non-contrast CT Scans.

Lesion segmentation is a fundamental step for the diagnosis of acute ischemic stroke (AIS). Non-contrast CT (NCCT) is still a mainstream imaging modality for AIS lesion measurement. However, AIS lesion segmentation on NCCT is challenging due to low contrast, noise and artifacts. To achieve accurate AIS lesion segmentation on NCCT, this study proposes a hybrid convolutional neural network (CNN) and Transformer network with circular feature interaction and bilateral difference learning. It consists of parallel CNN and Transformer encoders, a circular feature interaction module, and a shared CNN decoder with a bilateral difference learning module. A new Transformer block is particularly designed to solve the weak inductive bias problem of the traditional Transformer. To effectively combine features from CNN and Transformer encoders, we first design a multi-level feature aggregation module to combine multi-scale features in each encoder and then propose a novel feature interaction module containing circular CNN-to-Transformer and Transformer-to-CNN interaction blocks. Besides, a bilateral difference learning module is proposed at the bottom level of the decoder to learn the different information between the ischemic and contralateral sides of the brain. The proposed method is evaluated on three AIS datasets: the public AISD, a private dataset and an external dataset. Experimental results show that the proposed method achieves Dices of 61.39% and 46.74% on the AISD and the private dataset, respectively, outperforming 17 state-of-the-art segmentation methods. Besides, volumetric analysis on segmented lesions and external validation results imply that the proposed method is potential to provide support information for AIS diagnosis.

MFAE: Multilevel Feature Aggregation Enhanced Drug‐Target Affinity Prediction for Drug Repurposing Against Colorectal Cancer

Colorectal cancer (CRC), a leading cause of cancer‐related deaths globally, demands innovative therapeutic strategies to improve patient outcomes. Drug repurposing, identifying new uses for existing drugs, provides a cost‐effective solution. To this end, this study constructs the first drug‐target affinity dataset specifically for two novel therapeutic targets for CRC, P2X4 and mTOR, and designs a new deep learning‐based multilevel feature aggregation enhanced (MFAE) model. The model implements hierarchical feature extraction and multilevel feature aggregation and enhancement to simulate complex drug‐target interactions. Evaluations using fivefold cross‐validation on the collected CRC dataset showcase MFAE's superior predictive accuracy. Fine‐tuning of the model on external experimental data further enhances its performance, with a concordance index of 0.930, a determination coefficient of 0.782, and a mean squared error of 0.191. Ablation studies further highlight the key role of the group‐wise feature enhancement mechanism and ensemble learning strategy in enhancing the model's performance. Virtual screening of the Food and Drug Administration‐approved drugs identifies Ponatinib and Talazoparib as potential repurposing candidates. Despite limitations in experimental validation, this study establishes an innovative computational framework designed for CRC drug discovery. Overall, this research offers a valuable perspective on leveraging computational approaches for precision oncology.

RMLANet: Random Multi-Level Attention Network for Shadow Detection and Removal

This paper addresses the problem of shadow detection and shadow removal from a single image. Despite awareness of utilizing both local and global contexts, previous works only aggregate features level by level in a coarse-to-fine manner. To overcome this problem, we present RMLANet, a novel Random Multi-Level Attention Network. To be specific, we first design a shuffled multi-level feature aggregation module to fuse the multi-level features and the guiding features using the self-attention mechanism. Nevertheless, the computational complexity of dense self-attention is unaffordable when processing high-resolution inputs. We argue that dense attention between any pixel pair is unnecessary due to the local consistency in images. Then we further propose a sparse attention mechanism to reduce the number of attention pairs, which greatly reduces the computational complexity. Through extensive experiments on four shadow detection and three shadow removal benchmark datasets, our proposed RMLANet achieves superior performance over current state-of-the-art approaches for both shadow detection and shadow removal. Codes are publicly available at https://github.com/LeipingJie/RMLANet.

Image Inpainting Based on Multi-Level Feature Aggregation Network for Future Internet

(1) Background: In the future Internet era, clarity and structural rationality are important factors in image inpainting. Currently, image inpainting techniques based on generative adversarial networks have made great progress; however, in practical applications, there are still problems of unreasonable or blurred inpainting results for high-resolution images and images with complex structures. (2) Methods: In this work, we designed a lightweight multi-level feature aggregation network that extracts features from convolutions with different dilation rates, enabling the network to obtain more feature information and recover more reasonable missing image content. Fast Fourier convolution was designed and used in the generative network, enabling the generator to consider the global context at a shallow level, making it easier to perform high-resolution image inpainting tasks. (3) Results: The experiment shows that the method designed in this paper performs well in geometrically complex and high-resolution image inpainting tasks, providing a more reasonable and clearer inpainting image. Compared with the most advanced image inpainting methods, our method outperforms them in both subjective and objective evaluations. (4) Conclusions: The experimental results indicate that the method proposed in this paper has better clarity and more reasonable structural features.

Using a dual-stream attention neural network to characterize mild cognitive impairment based on retinal images

Mild cognitive impairment (MCI) is a critical transitional stage between normal cognition and dementia, for which early detection is crucial for timely intervention. Retinal imaging has been shown as a promising potential biomarker for MCI. This study aimed to develop a dual-stream attention neural network to classify individuals with MCI based on multi-modal retinal images. Our approach incorporated a cross-modality fusion technique, a variable scale dense residual model, and a multi-classifier mechanism within the dual-stream network. The model utilized a residual module to extract image features and employed a multi-level feature aggregation method to capture complex context information. Self-attention and cross-attention modules were utilized at each convolutional layer to fuse features from optical coherence tomography (OCT) and fundus modalities, resulting in multiple output losses. The neural network was applied to classify individuals with MCI, Alzheimer's disease, and control participants with normal cognition. Through fine-tuning the pre-trained model, we classified community-dwelling participants into two groups based on cognitive impairment test scores. To identify retinal imaging biomarkers associated with accurate prediction, we used the Gradient-weighted Class Activation Mapping technique. The proposed method achieved high precision rates of 84.96% and 80.90% in classifying MCI and positive test scores for cognitive impairment, respectively. Notably, changes in the optic nerve head on fundus photographs or OCT images among patients with MCI were not used to discriminate patients from the control group. These findings demonstrate the potential of our approach in identifying individuals with MCI and emphasize the significance of retinal imaging for early detection of cognitive impairment.

A Salient Object Detection Method Based on Boundary Enhancement.

Visual saliency refers to the human's ability to quickly focus on important parts of their visual field, which is a crucial aspect of image processing, particularly in fields like medical imaging and robotics. Understanding and simulating this mechanism is crucial for solving complex visual problems. In this paper, we propose a salient object detection method based on boundary enhancement, which is applicable to both 2D and 3D sensors data. To address the problem of large-scale variation of salient objects, our method introduces a multi-level feature aggregation module that enhances the expressive ability of fixed-resolution features by utilizing adjacent features to complement each other. Additionally, we propose a multi-scale information extraction module to capture local contextual information at different scales for back-propagated level-by-level features, which allows for better measurement of the composition of the feature map after back-fusion. To tackle the low confidence issue of boundary pixels, we also introduce a boundary extraction module to extract the boundary information of salient regions. This information is then fused with salient target information to further refine the saliency prediction results. During the training process, our method uses a mixed loss function to constrain the model training from two levels: pixels and images. The experimental results demonstrate that our salient target detection method based on boundary enhancement shows good detection effects on targets of different scales, multi-targets, linear targets, and targets in complex scenes. We compare our method with the best method in four conventional datasets and achieve an average improvement of 6.2% on the mean absolute error (MAE) indicators. Overall, our approach shows promise for improving the accuracy and efficiency of salient object detection in a variety of settings, including those involving 2D/3D semantic analysis and reconstruction/inpainting of image/video/point cloud data.

FrMLNet: Framelet-Based Multilevel Network for Pansharpening.

Most modern satellites can provide two types of images: 1) panchromatic (PAN) image and 2) multispectral (MS) image. The former has high spatial resolution and low spectral resolution, while the latter has high spectral resolution and low spatial resolution. To obtain images with both high spectral and spatial resolution, pansharpening has emerged to fuse the spatial information of the PAN image and the spectral information of the MS image. However, most pansharpening methods fail to preserve spatial and spectral information simultaneously. In this article, we propose a framelet-based convolutional neural network (CNN) for pansharpening which makes it possible to pursue both high spectral and high spatial resolution. Our network consists of three subnetworks: 1) feature embedding net; 2) feature fusion net; and 3) framelet prediction net. Different from conventional CNN methods directly inferring high-resolution MS images, our approach learns to predict their framelet coefficients from available PAN and MS images. The introduction of multilevel feature aggregation and hybrid residual connection makes full use of spatial information of PAN image and spectral information of MS image. Quantitative and qualitative experiments at reduced- and full-resolution demonstrate that the proposed method achieves more appealing results than other state-of-the-art pansharpening methods. The source code and trained models are available at https://github.com/TingMAC/FrMLNet.

Multi-level learning counting via pyramid vision transformer and CNN

Severe scale variation has become a challenging issue for hindering the improvement of accuracy in crowd counting task. To tackle the problem, we propose a Pyramid Transformer CNN Network (PTCNet), an effective combination of the transformer and the CNN, which possesses both the global receptive fields and the locality to deal with the severe scale variation problems and boost the prediction accuracy. Firstly, we utilize the pyramid vision transformer to extract multi-level global context information of the crowd, aiming at different head scales. And then, the multi-level information is fully fused in the multi-level feature aggregation module where detailed crowd characteristics from all feature spaces are preserved to be further processed. Finally, we design a multi-branch regression head to enrich the crowd features for strong representations and regress the density maps. Extensive experiments on challenging datasets with complex scenarios and multiple scales demonstrate the effectiveness of the our method. The proposed method achieves competitive results comparing with the state-of-the-art approaches and achieves state-of-the-art results(MAE:51.7, RMSE:79.6) on ShanghaiTech Part_A dataset.

Rethinking text rectification for scene text recognition

Existing scene text recognition methods have incorporated text rectification to lessen text irregularity in images for accurate recognition. Previous text rectification methods aim to convert an irregular text image into a regular form, making it easier to be recognized. In this study, we explore text rectification for text recognition and discover the issues: performance degradation of the recognition network and the unreliable situation of text rectification, which are ignored by all previous works. Therefore, we rethink what is causing two issues, and propose a rectification-based text recognition network to mitigate the above issues. The proposed network consists of text rectification and text recognition, and designs a multi-level feature aggregation module to enhance feature learning for character representation. Concretely, we devise a mixed batch training strategy to address the performance degradation of the recognition network, and design a confidence decoding scheme to avoid the unreliable situation of text rectification. Extensive ablation studies verified the positive role of the feature aggregation module in feature learning and the effectiveness of the proposed training strategy and decoding scheme in addressing the issues. Experimental results outperform the state-of-the-art results on public benchmarks.

Quality-Driven Dual-Branch Feature Integration Network for Video Salient Object Detection

Video salient object detection has attracted growing interest in recent years. However, some existing video saliency models often suffer from the inappropriate utilization of spatial and temporal cues and the insufficient aggregation of different level features, leading to remarkable performance degradation. Therefore, we propose a quality-driven dual-branch feature integration network majoring in the adaptive fusion of multi-modal cues and sufficient aggregation of multi-level spatiotemporal features. Firstly, we employ the quality-driven multi-modal feature fusion (QMFF) module to combine the spatial and temporal features. Particularly, the quality scores estimated from each level’s spatial and temporal cues are not only used to weigh the two modal features but also to adaptively integrate the coarse spatial and temporal saliency predictions into the guidance map, which further enhances the two modal features. Secondly, we deploy the dual-branch-based multi-level feature aggregation (DMFA) module to integrate multi-level spatiotemporal features, where the two branches including the progressive decoder branch and the direct concatenation branch sufficiently explore the cooperation of multi-level spatiotemporal features. In particular, in order to provide an adaptive fusion for the outputs of the two branches, we design the dual-branch fusion (DF) unit, where the channel weight of each output can be learned jointly from the two outputs. The experiments conducted on four video datasets clearly demonstrate the effectiveness and superiority of our model against the state-of-the-art video saliency models.

MashFormer: A Novel Multiscale Aware Hybrid Detector for Remote Sensing Object Detection

Object detection is a critical and demanding topic in the subject of processing satellite and airborne images. The targets acquired in remote sensing imagery are at various sizes, and the backgrounds are complicated, which makes object detection extremely challenging. We address these aforementioned issues in this paper by introducing the MashFormer, an innovative multi-scale aware CNN and Transformer integrated hybrid detector. Specifically, MashFormer employs the transformer block to complement the convolutional neural network (CNN) based feature extraction backbone, which could obtain the relationships between long-range features and enhance the representative ability in complex background scenarios. With the intention of improving the detection performance for objects with multi-scale characteristic, since in remote sensing scenarios, the size of object varies greatly. A multi-level feature aggregation component, incoperate with a cross-level feature alignment module is designed to alleviate the semantic discrepancy between features from shallow and deep layers. To verify the effectiveness of the suggested MashFormer, comparative experiments are carried out with other cutting-edge methodologies using the publicly available High Resolution Remote Sensing Detection (HRRSD) and Northwestern Polytechnical University (NWPU) VHR-10 datasets. The experimental findings confirm the effectiveness and superiority of our suggested model by indicating that our approach has greater mean average precision (mAP) than the other methodologies.

A context-aware progressive attention aggregation network for fabric defect detection

Fabric defect detection plays a critical role for measuring quality control in the textile manufacturing industry. Deep learning-based saliency models can quickly spot the most interesting regions that attract human attention from the complex background, which have been successfully applied in fabric defect detection. However, most of the previous methods mainly adopted multi-level feature aggregation yet ignored the complementary relationship among different features, and thus resulted in poor representation capability for the tiny and slender defects. To remedy these issues, we propose a novel saliency-based fabric defect detection network, which can exploit the complementary information between different layers to enhance the representation features ability and discrimination of defects. Specifically, a multi-scale feature aggregation unit (MFAU) is proposed to effectively characterize the multi-scale contextual features. Besides, a feature fusion refinement module (FFR) composed of an attention fusion unit (AFU) and an auxiliary refinement unit (ARU) is designed to exploit complementary important information and further refine the input features for enhancing the discriminative ability of defect features. Finally, a multi-level deep supervision (MDS) is adopted to guide the model to generate more accurate saliency maps. Under different evaluation metrics, our proposed method outperforms most state-of-the-art methods on our developed fabric datasets.

MobileSal: Extremely Efficient RGB-D Salient Object Detection.

The high computational cost of neural networks has prevented recent successes in RGB-D salient object detection (SOD) from benefiting real-world applications. Hence, this article introduces a novel network, MobileSal, which focuses on efficient RGB-D SOD using mobile networks for deep feature extraction. However, mobile networks are less powerful in feature representation than cumbersome networks. To this end, we observe that the depth information of color images can strengthen the feature representation related to SOD if leveraged properly. Therefore, we propose an implicit depth restoration (IDR) technique to strengthen the mobile networks' feature representation capability for RGB-D SOD. IDR is only adopted in the training phase and is omitted during testing, so it is computationally free. Besides, we propose compact pyramid refinement (CPR) for efficient multi-level feature aggregation to derive salient objects with clear boundaries. With IDR and CPR incorporated, MobileSal performs favorably against state-of-the-art methods on six challenging RGB-D SOD datasets with much faster speed (450fps for the input size of 320×320) and fewer parameters (6.5M). The code is released at https://mmcheng.net/mobilesal.

Dual-Branch Multi-Level Feature Aggregation Network for Pansharpening

Dear Editor, In pansharpening task, the most existing deep-learning-based pan-sharpening methods fail to fully utilize the different level features, inevitably leading to spectral or spatial distortions. To address this challenge, in this letter, we propose a dual-branch multi-level feature aggregation network for pansharpening (DMFANet). The experimental results on the WorldView-II (WV-II) and QuickBird (QB) dataset confirmed the notable superiority of our method over the current state-of-the-art methods from quantitative and qualitative point of view. The source code is available at https://github.com/Gui-Cheng/DMFANet.

Enabling modality interactions for RGB-T salient object detection

Most existing RGB and thermal (RGB-T) salient object detection (SOD) techniques focus on investing multi-modality feature fusion strategies for capturing cross-modality complementary information within RGB and thermal images. However, most of these strategies do not allow explicitly extracting the interactions among the features of different modalities, thus leading to insignificant cross-modality complementary information exploitation. In this paper, we propose a novel RGB-T SOD model that alleviates this issue by leveraging a modality-aware and scale-aware feature fusion module. Such a module is capable of capturing the cross-modality complementary information by exploiting the interactions of single-modality features across modalities and the interactions of multi-modality features across scales. A stage-wise feature aggregation module is also proposed to thoroughly exploit the cross-level complementary information and reduce their redundancies for generating accurate saliency maps with sharp boundaries. To this end, a novel multi-level feature aggregation structure with two types of feature aggregation nodes is employed. Experimental results on several benchmark datasets verify the effectiveness and superiorities of our proposed model over some state-of-the-art models.