Deep Feature Aggregation Research Articles

Deep Learning Features Can Improve Radiomics-Based Prostate Cancer Aggressiveness Prediction

PURPOSE Emerging evidence suggests that the use of artificial intelligence can assist in the timely detection and optimization of therapeutic approach in patients with prostate cancer. The conventional perspective on radiomics encompassing segmentation and the extraction of radiomic features considers it as an independent and sequential process. However, it is not necessary to adhere to this viewpoint. In this study, we show that besides generating masks from which radiomic features can be extracted, prostate segmentation and reconstruction models provide valuable information in their feature space, which can improve the quality of radiomic signatures models for disease aggressiveness classification. MATERIALS AND METHODS We perform 2,244 experiments with deep learning features extracted from 13 different models trained using different anatomic zones and characterize how modeling decisions, such as deep feature aggregation and dimensionality reduction, affect performance. RESULTS While models using deep features from full gland and radiomic features consistently lead to improved disease aggressiveness prediction performance, others are detrimental. Our results suggest that the use of deep features can be beneficial, but an appropriate and comprehensive assessment is necessary to ensure that their inclusion does not harm predictive performance. CONCLUSION The study findings reveal that incorporating deep features derived from autoencoder models trained to reconstruct the full prostate gland (both zonal models show worse performance than radiomics only models), combined with radiomic features, often lead to a statistically significant increase in model performance for disease aggressiveness classification. Additionally, the results also demonstrate that the choice of feature selection is key to achieving good performance, with principal component analysis (PCA) and PCA + relief being the best approaches and that there is no clear difference between the three proposed latent representation extraction techniques.

Multi-Branch Attention Fusion Network for Cloud and Cloud Shadow Segmentation

In remote sensing image processing, the segmentation of clouds and their shadows is a fundamental and vital task. For cloud images, traditional deep learning methods often have weak generalization capabilities and are prone to interference from ground objects and noise, which not only results in poor boundary segmentation but also causes false and missed detections of small targets. To address these issues, we proposed a multi-branch attention fusion network (MAFNet). In the encoder section, the dual branches of ResNet50 and the Swin transformer extract features together. A multi-branch attention fusion module (MAFM) uses positional encoding to add position information. Additionally, multi-branch aggregation attention (MAA) in the MAFM fully fuses the same level of deep features extracted by ResNet50 and the Swin transformer, which enhances the boundary segmentation ability and small target detection capability. To address the challenge of detecting small cloud and shadow targets, an information deep aggregation module (IDAM) was introduced to perform multi-scale deep feature aggregation, which supplements high semantic information, improving small target detection. For the problem of rough segmentation boundaries, a recovery guided module (RGM) was designed in the decoder section, which enables the model to effectively allocate attention to complex boundary information, enhancing the network’s focus on boundary information. Experimental results on the Cloud and Cloud Shadow dataset, HRC-WHU dataset, and SPARCS dataset indicate that MAFNet surpasses existing advanced semantic segmentation techniques.

Deep progressive feature aggregation network for multi-frame high dynamic range imaging

High dynamic range (HDR) imaging is an important task in image processing that aims to generate well-exposed images in scenes with varying illumination. Although existing multi-exposure fusion methods have achieved impressive results, generating high-quality HDR images in dynamic scenes remains difficult. The primary challenges are ghosting artifacts caused by object motion between low dynamic range images and distorted content in underexposure and overexposed regions. In this paper, we propose a deep progressive feature aggregation network for improving HDR imaging quality in dynamic scenes. To address the issues of object motion, our method implicitly samples high-correspondence features and aggregates them in a coarse-to-fine manner for alignment. In addition, our method adopts a densely connected network structure based on the discrete wavelet transform, which aims to decompose the input features into multiple frequency subbands and adaptively restore corrupted contents. Experiments show that our proposed method can achieve state-of-the-art performance under different scenes, compared to other promising HDR imaging methods. Specifically, the HDR images generated by our method contain cleaner and more detailed content, with fewer distortions, leading to better visual quality.

Understanding the Distributions of Aggregation Layers in Deep Neural Networks.

The process of aggregation is ubiquitous in almost all the deep nets' models. It functions as an important mechanism for consolidating deep features into a more compact representation while increasing the robustness to overfitting and providing spatial invariance in deep nets. In particular, the proximity of global aggregation layers to the output layers of DNNs means that aggregated features directly influence the performance of a deep net. A better understanding of this relationship can be obtained using information theoretic methods. However, this requires knowledge of the distributions of the activations of aggregation layers. To achieve this, we propose a novel mathematical formulation for analytically modeling the probability distributions of output values of layers involved with deep feature aggregation. An important outcome is our ability to analytically predict the Kullback-Leibler (KL)-divergence of output nodes in a DNN. We also experimentally verify our theoretical predictions against empirical observations across a broad range of different classification tasks and datasets.

FCPFNet: Feature Complementation Network with Pyramid Fusion for Semantic Segmentation

Traditional pyramid pooling modules have shown effective improvements in semantic segmentation tasks by capturing multi-scale feature information. However, their limitations arise from the shallow structure, which fails to fully extract contextual information, and the fused multi-scale feature information lacks distinctiveness, resulting in issues with the final segmentation discriminability. To address these issues, we proposes an effective solution called FCPFNet, which is based on global contextual prior for deep feature extraction of detailed information. Specifically, we introduce a novel deep feature aggregation module to extract semantic information from the output feature map of each layer through a deep aggregation of context information module, and expands the effective perception range. Additionally, we propose an Efficient Pyramid Pooling Module (EPPM) to capture distinctive features through communicating information between different sub-features and performs multi-scale fusion, which is integrated as a branch within the network to complement the information loss resulting from downsampling operations. Furthermore, in order to ensure the richness of image detail feature information and maintain a large receptive field to obtain more contextual information, EPPM concatenates the input feature map and the output feature map of the pyramid pooling module to acquire more comprehensive global contextual information. It has been demonstrated by experiment that the method described in this article achieves competitive performance on the challenging scene segmentation datasets Pascal VOC 2012, Cityscapes and Coco-Stuff, with MIOU of 81.0%, 78.8% and 40.1%, respectively.

An automatic histopathological image segmentation network based on global context-aware module and deep feature aggregation

Histopathological analysis is the gold standard for cancer diagnosis, but traditional manual identification methods are time-consuming, labor-intensive, and highly relied on the experience of pathologists. In recent years, deep learning methods can assist pathologists to achieve accurate and automated detection of lesion areas. However, lesion detection is a difficult task due to its various size, shape, and surrounding complex scenes. Existing methods ignore the global interaction between lesion area and normal area, so it is insufficient to extract important global features. In addition, most methods fail to effectively fuse deep and shallow features, resulting in the extracted features lacking discrimination, and these deficiencies lead to incomplete extraction and irregular boundaries of lesion. To address the above-mentioned issues, an end-to-end histopathological image segmentation network based on global context-aware module and deep feature aggregation (GCMDFA) is proposed for segmenting lesions in histopathological images (HIs). In the GCMDFA, a global context-aware module is applied to better explore the interactive relationship between any two pixels to directly capture global contextual information related to lesion tissue of HIs for achieving global features. To better decode the features of encoder, a deep feature aggregation structure is designed to effectively fuse deep and shallow layer features of HIs. Experimental results on three HI datasets (including public and external data) show that the proposed GCMDFA method is more competitive compared with some state-of-the-art (SOTA) methods. This research can assist pathologists to analyze HIs in clinical computer-aided diagnosis.

Emotion Recognition from Few-Channel EEG Signals by Integrating Deep Feature Aggregation and Transfer Learning

Emotion Recognition from Few-Channel EEG Signals by Integrating Deep Feature Aggregation and Transfer Learning

A new deep graph attention approach with influence and preference relationship reconstruction for rate prediction recommendation

Graph neural networks have been frequently applied in recommender systems due to their powerful representation abilities for irregular data. However, these methods still suffer from the difficulties such as the inflexible graph structure, sparse and highly imbalanced data, and relatively shallow networks, limiting rate prediction ability for recommendations. This paper presents a novel deep dynamic graph attention framework based on influence and preference relationship reconstruction (DGA-IPR) for recommender systems to learn optimal latent representations of users and items. The entire framework involves a user branch and an item branch. An influence-based dynamic graph attention (IDGA) module, a preference-based dynamic graph attention (PDGA) module, and an adaptive fine feature extraction (AFFE) module are respectively constructed for each branch. Concretely, the first two attention modules concentrate on reconstructing influence and preference relationship graphs, breaking imbalanced and fixed constraints of graph structures. Then a deep feature aggregation block and an adaptive feature fusion operation are built, improving the network depth and capturing potential high-order information expressions. Besides, AFFE is designed to acquire finer latent features for users and items. The DGA-IPR architecture is formed by integrating IDGA, PDGA, and AFFE for users and items, respectively. Experiments reveal the superiority of DGA-IPR over existing recommendation models.

Deep convolutional feature aggregation for fine-grained cultivar recognition

Fine-grained cultivar recognition has recently attracted considerable attention from researchers in pattern recognition and botany. However, this problem is highly challenging because the differences between cultivated species are so subtle that it is difficult to distinguish them effectively. This article presents a novel deep convolutional feature aggregation approach for fine-grained cultivar recognition. First, we propose a description method of the regional convolution covariance feature (RCCF), which describes the subtle changes in cultivated species by accumulating low-level convolution features and has a strong discriminating ability. Second, we also improve the regional maximum activation of convolutions (RMAC) and present a multiresolution RMAC high-level convolutional feature. Finally, we combine complementary RCCF with multiresolution RMAC features for fine-grained cultivar recognition, which significantly improves the recognition accuracy of fine-grained cultivated plants. We have carried out extensive tests on four benchmark cultivar plant datasets. The results show that our approach achieves state-of-the-art recognition performance on four benchmark cultivar plant datasets, surpassing other plant species identification methods.

Dual-branch hybrid network for lesion segmentation in gastric cancer images

The effective segmentation of the lesion region in gastric cancer images can assist physicians in diagnosing and reducing the probability of misdiagnosis. The U-Net has been proven to provide segmentation results comparable to specialists in medical image segmentation because of its ability to extract high-level semantic information. However, it has limitations in obtaining global contextual information. On the other hand, the Transformer excels at modeling explicit long-range relations but cannot capture low-level detail information. Hence, this paper proposes a Dual-Branch Hybrid Network based on the fusion Transformer and U-Net to overcome both limitations. We propose the Deep Feature Aggregation Decoder (DFA) by aggregating only the in-depth features to obtain salient lesion features for both branches and reduce the complexity of the model. Besides, we design a Feature Fusion (FF) module utilizing the multi-modal fusion mechanisms to interact with independent features of various modalities and the linear Hadamard product to fuse the feature information extracted from both branches. Finally, the Transformer loss, the U-Net loss, and the fused loss are compared to the ground truth label for joint training. Experimental results show that our proposed method has an IOU of 81.3%, a Dice coefficient of 89.5%, and an Accuracy of 94.0%. These metrics demonstrate that our model outperforms the existing models in obtaining high-quality segmentation results, which has excellent potential for clinical analysis and diagnosis. The code and implementation details are available at Github, https://github.com/ZYY01/DBH-Net/.

Simple and robust depth-wise cascaded network for polyp segmentation

The segmentation of the polyp region in colonoscopy images is considered difficult due to size, texture, and color variation. To segment polyps successfully, models based on convolutional neural networks (CNN), transformers, and their combinations have been developed. However, these methods are limited in that they can only model the local appearance of polyps or lack multi-level feature representation for spatial dependency in the decoding process. In this paper, we propose a simple, efficient yet powerful polyp segmentation framework that unifies the network with a multiscale cascaded path. The proposed MMS-Net utilizes multiscale and multipath convolutional operations in conjunction with multiple deep feature aggregation. The overall dense empowered features are sufficient for pixel-by-pixel detection of the polyp region. Extensive experiments on two popular benchmark datasets for polyp segmentation (Kvasir and CVC-Clinic DB) and two datasets of other medical applications (DRIVE and MC) are presented. The results show that our MMS-Net performs comparably to or better than other state-of-the-art methods despite having two or even three orders of magnitude fewer trainable parameters.

Artificial intelligence-based classification of pollen grains using attention-guided pollen features aggregation network

Visual classification of pollen grains is crucial for various agricultural applications, particularly for the protection, monitoring, and tracking of flora to preserve the biome and maintain the quality of honey-based products. Traditionally, pollen grain classification has been performed by trained palynologists using a light microscope. Despite their wide range of applications, still tiresome and time-consuming methods are used. Artificial intelligence (AI) can be used to automate the pollen grain classification process. Recently, numerous AI-based techniques for classifying pollen grains have been proposed. However, there is still possibility for performance enhancement including processing time, memory size, and accuracy. In this study, an attention-guided pollen feature aggregation network (APFA-Net) based on deep feature aggregation and channel-wise attention is proposed. Three publicly available datasets, POLLEN73S, POLLEN23E, and Cretan pollen, having a total of 7362 images from 116 distinct pollen types are used for experiments. The proposed method shows F-measure values of 97.37 %, 97.66 %, and 98.39 % with POLLEN73S, POLLEN23E, and Cretan Pollen datasets, respectively. We confirm that our method outperforms existing state-of-the-art methods.

Blind Image Quality Index for Authentic Distortions With Local and Global Deep Feature Aggregation

Blind image quality assessment (BIQA) for authentic distortions is still a great challenge, even in today’s deep learning era. It has been widely acknowledged that local and global features are both indispensable for IQA, which play complementary roles. While combining local and global features is straightforward in traditional handcrafted feature-based IQA metrics, it is not an easy task in the deep learning framework. This is mainly due to the fact that deep neural networks typically require input images with a fixed size. Current metrics either resize the image or use local patches as input, which are problematic in that they cannot integrate local and global aspects as well as their interactions to achieve comprehensive quality evaluation. Motivated by the above facts, this paper presents a new BIQA metric for authentic distortions by aggregating local and global deep features in a Vision-Transformer framework. In the proposed metric, selective local regions and global content are simultaneously input for complementary feature extraction, and the Vision-Transformer is employed to build the relationship between different local patches and image quality. Self-attention mechanism is further adopted to explore the interaction between local and global deep features, producing the final image quality score. Extensive experiments on five authentically distorted IQA databases demonstrate that the proposed metric outperforms the state-of-the-arts in terms of both prediction performance and generalization ability.

Deep Feature Aggregation Framework Driven by Graph Convolutional Network for Scene Classification in Remote Sensing.

Scene classification of high spatial resolution (HSR) images can provide data support for many practical applications, such as land planning and utilization, and it has been a crucial research topic in the remote sensing (RS) community. Recently, deep learning methods driven by massive data show the impressive ability of feature learning in the field of HSR scene classification, especially convolutional neural networks (CNNs). Although traditional CNNs achieve good classification results, it is difficult for them to effectively capture potential context relationships. The graphs have powerful capacity to represent the relevance of data, and graph-based deep learning methods can spontaneously learn intrinsic attributes contained in RS images. Inspired by the abovementioned facts, we develop a deep feature aggregation framework driven by graph convolutional network (DFAGCN) for the HSR scene classification. First, the off-the-shelf CNN pretrained on ImageNet is employed to obtain multilayer features. Second, a graph convolutional network-based model is introduced to effectively reveal patch-to-patch correlations of convolutional feature maps, and more refined features can be harvested. Finally, a weighted concatenation method is adopted to integrate multiple features (i.e., multilayer convolutional features and fully connected features) by introducing three weighting coefficients, and then a linear classifier is employed to predict semantic classes of query images. Experimental results performed on the UCM, AID, RSSCN7, and NWPU-RESISC45 data sets demonstrate that the proposed DFAGCN framework obtains more competitive performance than some state-of-the-art methods of scene classification in terms of OAs.

Auxiliary Segmentation Method of Osteosarcoma in MRI Images Based on Denoising and Local Enhancement

Osteosarcoma is a bone tumor which is malignant. There are many difficulties when doctors manually identify patients’ MRI images to complete the diagnosis. The osteosarcoma in MRI images is very complex, making its recognition and segmentation resource-consuming. Automatic osteosarcoma area segmentation can solve these problems to a certain extent. However, existing studies usually fail to balance segmentation accuracy and efficiency. They are either sensitive to noise with low accuracy or time-consuming. So we propose an auxiliary segmentation method based on denoising and local enhancement. The method first optimizes the osteosarcoma images, including removing noise using the Edge Enhancement based Transformer for Medical Image Denoising (Eformer) and using a non-parameter method to localize and enhance the tumor region in MRI images. Osteosarcoma was then segmented by Deep Feature Aggregation for Real-Time Semantic Segmentation (DFANet). Our method achieves impressive segmentation accuracy. Moreover, it is efficient in both time and space. It can provide information about the location and extent of the osteosarcoma as a basis for further diagnosis.

A Deep Feature Aggregation Network for Accurate Indoor Camera Localization

As scene coordinate regression (SCoRe) methods become prevailing in the area of visual camera localization, the issue of repetitive or sparse texture scenes continues to be a concern. Specifically, they will suffer from performance degeneration due to ambiguous patterns caused by visual similarity. In this work, we propose a novel network for camera localization through a single RGB image, with our key insight that taking only high-level feature maps as input can be difficult for the network to accurately model the regression problem due to ambiguous patterns and utilizing the rich spatial details in low-level feature maps can tackle this issue. The core components of the network are 1) Deep Feature Aggregation Module (DFAM), which eliminates the difference among the different level feature representations and fuses multi-level context information; 2) CoordConv Scheme, which further improves the discrimination of features in repetitive or sparse texture areas of the image; 3) Deep Supervision, which endows low-level feature maps with direct supervision from the ground truth to improve the accuracy of camera localization; 4) Uncertainty Modeling, which quantifies the prediction errors stemming from the intrinsic noise in the data. Moreover, to maximize the power of DFAM, we embed channel attention modules into it to prune redundant and noisy features, through which we can refine the different level feature maps. Our network is designed to be lightweight and efficient, and the proposed DFAM can be integrated into general SCoRe-based networks. Comprehensive experiments demonstrate the effectiveness of DFAM and the superiority of our network over the state-of-the-art methods on two benchmarks.

Image retrieval using contrastive weight aggregation histograms

Aggregating deep convolutional features for image retrieval has obtained excellent results in recent years; however, exploiting the several advantages of deep convolutional feature maps remains challenging. To address this problem, we propose a novel weighting method called the contrastive weight aggregation histogram to distinguish the slightly distinguishable and highly distinguishable features in deep convolutional features maps for image retrieval. The main highlights of this paper are as follows: (1) a contrastive weighting is proposed to represent the differential contribution of the slightly and highly distinguishable features. It can enhance the distinguishable information and further improve the representative power of deep convolutional features. (2) A novel method is introduced to generate contrastive weighting by comparing the slightly and highly distinguishable feature aggregation. It has the ability to exploit the several advantages of deep convolutional feature maps in terms of differentiating. Experiments demonstrated that the proposed contrastive weighting method outperforms methods that use the deep convolutional feature aggregation on five benchmark datasets.

Clustering Feature Constraint Multiscale Attention Network for Shadow Extraction From Remote Sensing Images

Shadow extraction is an important and challenging task in remote sensing image analysis because the presence of shadows not only reduces radiation information but also affects the interpretation of remote sensing images. In this article, a clustering feature constraint multiscale attention network for shadow extraction from remote sensing images is proposed. First, in addition to the pixel-level description of the traditional neural network, our method focuses on the clustering relationships between pixel pairs to obtain the pixel group features of shadows. The feature extraction capability of the network is improved with a reweighting mechanism at the pixel level and pixel group features. Second, we employ a feature fusion algorithm by considering contextual information to improve the network’s attention toward shadow areas and enhance the nonlinear expression ability during the encoding and decoding layers. Furthermore, considering the most prominent multiscale features of shadows in remote sensing images, a deep multiscale feature aggregation structure is established to better fit the multiscale feature expression of shadows. Finally, we construct a shadow extraction dataset to verify the proposed approach. We compare our method with the results of state-of-the-art deep learning models. The results show that the intersection over union (IOU) of our method is improved by 0.85%–9.51% and that the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F1$ </tex-math></inline-formula> -score is improved by 0.73–6.48. In addition, the test results for images with different resolutions prove that the proposed approach is more robust than the other methods.

Distortion Convolution Module for Semantic Segmentation of Panoramic Images Based on the Image-Forming Principle

Semantic segmentation of panoramic images plays a key role in many applications, such as security monitoring and autonomous driving. With the rapid development of deep learning, some deep networks are developed to segment panoramic images semantically. However, these networks don’t have the special modules to correct the image distortion of panoramic images according to the distortion principle, which makes feature extraction unreasonable during the convolution because of the image distortion. This article proposes a novel semantic segmentation network for panoramic images of outdoor scenes based on the distortion convolution. The network contains an encoder and a decoder. The encoder consists of a distortion convolutional module (DCM), a residual network, and an atrous spatial pyramid pooling (ASPP). The DCM is developed to correct the image distortion according to the image-forming principle. In the decoder, a deep feature aggregation network (DFAN) is designed to fully fuse low-level features with high-level features. The proposed network introduces the DCM and DFAN into the semantic segmentation of panoramic images, which improves the segmentation accuracy. The experiments demonstrate that the proposed network has good performance for different outdoor scenes and plays an important role in measurement applications.

Virtual Adversarial Training-Based Deep Feature Aggregation Network From Dynamic Effective Connectivity for MCI Identification.

Dynamic functional connectivity (dFC) network inferred from resting-state fMRI reveals macroscopic dynamic neural activity patterns for brain disease identification. However, dFC methods ignore the causal influence between the brain regions. Furthermore, due to the complex non-Euclidean structure of brain networks, advanced deep neural networks are difficult to be applied for learning high-dimensional representations from brain networks. In this paper, a group constrained Kalman filter (gKF) algorithm is proposed to construct dynamic effective connectivity (dEC), where the gKF provides a more comprehensive understanding of the directional interaction within the dynamic brain networks than the dFC methods. Then, a novel virtual adversarial training convolutional neural network (VAT-CNN) is employed to extract the local features of dEC. The VAT strategy improves the robustness of the model to adversarial perturbations, and therefore avoids the overfitting problem effectively. Finally, we propose the high-order connectivity weight-guided graph attention networks (cwGAT) to aggregate features of dEC. By injecting the weight information of high-order connectivity into the attention mechanism, the cwGAT provides more effective high-level feature representations than the conventional GAT. The high-level features generated from the cwGAT are applied for binary classification and multiclass classification tasks of mild cognitive impairment (MCI). Experimental results indicate that the proposed framework achieves the classification accuracy of 90.9%, 89.8%, and 82.7% for normal control (NC) vs. early MCI (EMCI), EMCI vs. late MCI (LMCI), and NC vs. EMCI vs. LMCI classification respectively, outperforming the state-of-the-art methods significantly.