Multi-level Feature Fusion Research Articles

Multilevel Feature Fusion for End-to-End Blind Image Quality Assessment

In this paper, a framework based on two feature extraction networks and a multilevel feature fusion (MFF) network is proposed. Multilevel degradation features can be obtained through this method, and combined with the human visual perception system, the local and global feature information contained in these features can be captured, which is conducive to the prediction of distorted images. First, a restored image approximating a reference image is generated by a restorative generative adversarial network (GAN). Furthermore, the multilevel degradation features of distorted images and the restored image features are extracted by EfficientNet. Second, the features extracted by EfficientNet are input into the MFF network and are fully expressed by the top-down, bottom-up and third edge joining methods. Moreover, the features provide more low-level details and high-level semantic features for the prediction of image quality scores. In addition, after the MFF stage, the framework calculates the score of each branch feature and obtains the average quality score. Experimental results show that our method achieves greatly improved prediction accuracy and performance on five standard databases.

RSLC-Deeplab: A Ground Object Classification Method for High-Resolution Remote Sensing Images

With the continuous advancement of remote sensing technology, the semantic segmentation of different ground objects in remote sensing images has become an active research topic. For complex and diverse remote sensing imagery, deep learning methods have the ability to automatically discern features from image data and capture intricate spatial dependencies, thus outperforming traditional image segmentation methods. To address the problems of low segmentation accuracy in remote sensing image semantic segmentation, this paper proposes a new remote sensing image semantic segmentation network, RSLC-Deeplab, based on DeeplabV3+. Firstly, ResNet-50 is used as the backbone feature extraction network, which can extract deep semantic information more effectively and improve the segmentation accuracy. Secondly, the coordinate attention (CA) mechanism is introduced into the model to improve the feature representation generated by the network by embedding position information into the channel attention mechanism, effectively capturing the relationship between position information and channels. Finally, a multi-level feature fusion (MFF) module based on asymmetric convolution is proposed, which captures and refines low-level spatial features using asymmetric convolution and then fuses them with high-level abstract features to mitigate the influence of background noise and restore the lost detailed information in deep features. The experimental results on the WHDLD dataset show that the mean intersection over union (mIoU) of RSLC-Deeplab reached 72.63%, the pixel accuracy (PA) reached 83.49%, and the mean pixel accuracy (mPA) reached 83.72%. Compared to the original DeeplabV3+, the proposed method achieved a 4.13% improvement in mIoU and outperformed the PSP-NET, U-NET, MACU-NET, and DeeplabV3+ networks.

DeepICSH: a complex deep learning framework for identifying cell-specific silencers and their strength from the human genome.

Silencers are noncoding DNA sequence fragments located on the genome that suppress gene expression. The variation of silencers in specific cells is closely related to gene expression and cancer development. Computational approaches that exclusively rely on DNA sequence information for silencer identification fail to account for the cell specificity of silencers, resulting in diminished accuracy. Despite the discovery of several transcription factors and epigenetic modifications associated with silencers on the genome, there is still no definitive biological signal or combination thereof to fully characterize silencers, posing challenges in selecting suitable biological signals for their identification. Therefore, we propose a sophisticated deep learning framework called DeepICSH, which is based on multiple biological data sources. Specifically, DeepICSH leverages a deep convolutional neural network to automatically capture biologically relevant signal combinations strongly associated with silencers, originating from a diverse array of biological signals. Furthermore, the utilization of attention mechanisms facilitates the scoring and visualization of these signal combinations, whereas the employment of skip connections facilitates the fusion of multilevel sequence features and signal combinations, thereby empowering the accurate identification of silencers within specific cells. Extensive experiments on HepG2 and K562 cell line data sets demonstrate that DeepICSH outperforms state-of-the-art methods in silencer identification. Notably, we introduce for the first time a deep learning framework based on multi-omics data for classifying strong and weak silencers, achieving favorable performance. In conclusion, DeepICSH shows great promise for advancing the study and analysis of silencers in complex diseases. The source code is available at https://github.com/lyli1013/DeepICSH.

Multilevel feature fusion of multi-domain vibration signals for bearing fault diagnosis

Multilevel feature fusion of multi-domain vibration signals for bearing fault diagnosis

MFCINet: multi-level feature and context information fusion network for RGB-D salient object detection

MFCINet: multi-level feature and context information fusion network for RGB-D salient object detection

DCMF-AFNet: An anchor-free photovoltaic hot-spot fault detection network based on deformable context transformer and bi-branch multi-level feature fusion

In the process of photovoltaic hot-spot detection by thermal infrared sensors, the fault features cannot be effectively represented due to low pixel ratios and complex environmental interference, which makes it difficult for the detection network to accurately detect hot-spot faults. Therefore, an anchor-free photovoltaic hot-spot fault detection algorithm based on deformable context Transformer and bi-branch multi-level feature fusion is proposed. First, to improve the feature extraction ability of the backbone network for small-scale hot-spot faults, a deformable context Transformer module is constructed. By building an offset network in a multi-headed self-attention mechanism, dynamic and static context information with small-scale fault features can be explored from shallow feature maps. Second, to solve the problem of low target saliency due to the complex background interference, a bi-branch multi-level feature fusion module is designed to aggregate global and local multi-level features in a parallel fusion manner, enabling the detection network to rapidly focus on the fault target region in complex environments. Then, an anchor-free mechanism is introduced, and a dynamic task alignment prediction module is proposed to avoid feature misalignment in the classification and localization tasks and further improve the algorithm detection accuracy. Finally, to verify the superiority of the proposed network, seven detection algorithms are selected for comparison experiments. The experimental results show that the DCMF-AFNet network can accurately detect multi-scale hot-spot targets under various harsh conditions, and the detection accuracy can reach 87.3%.

Research on Printmaking Image Classification and Creation Based on Convolutional Neural Network

As an important form of expression in modern civilization art, printmaking has a rich variety of types and a prominent sense of artistic hierarchy. Therefore, printmaking is highly favored around the world due to its unique artistic characteristics. Classifying print types through image feature elements will improve people’s understanding of print creation. Convolutional neural networks (CNNs) have good application effects in the field of image classification, so CNN is used for printmaking analysis. Considering that the classification effect of the traditional convolutional neural image classification model is easily affected by the activation function, the T-ReLU activation function is introduced. By utilizing adjustable parameters to enhance the soft saturation characteristics of the model and avoid gradient vanishing, a T-ReLU convolutional model is constructed. A better convolutional image classification model is proposed based on the T-ReLU convolutional model, taking into account the issue of subpar multi-level feature fusion in deep convolutional image classification models. Utilize normalization to analyze visual input, an eleven-layer convolutional network with residual units in the convolutional layer, and cascading thinking to fuse convolutional network defects. The performance test results showed that in the data test of different styles of artificial prints, the GT-ReLU model can obtain the best image classification accuracy, and the image classification accuracy rate is 0.978. The GT-ReLU model maintains a classification accuracy above 94.4% in the multi-dataset test classification performance test, which is higher than that of other image classification models. For the use of visual processing technology in the field of classifying prints, the research content provides good reference value.

Establishment and Evaluation of Intelligent Diagnostic Model for Ophthalmic Ultrasound Images Based on Deep Learning.

The goal of the work described here was to construct a deep learning-based intelligent diagnostic model for ophthalmic ultrasound images to provide auxiliary analysis for the intelligent clinical diagnosis of posterior ocular segment diseases. The InceptionV3-Xception fusion model was established by using two pre-trained network models-InceptionV3 and Xception-in series to achieve multilevel feature extraction and fusion, and a classifier more suitable for the multiclassification recognition task of ophthalmic ultrasound images was designed to classify 3402 ophthalmic ultrasound images. The accuracy, macro-average precision, macro-average sensitivity, macro-average F1 value, subject working feature curves and area under the curve were used as model evaluation metrics, and the credibility of the model was assessed by testing the decision basis of the model using a gradient-weighted class activation mapping method. The accuracy, precision, sensitivity and area under the subject working feature curve of the InceptionV3-Xception fusion model on the test set reached 0.9673, 0.9521, 0.9528 and 0.9988, respectively. The model decision basis was consistent with the clinical diagnosis basis of the ophthalmologist, which proves that the model has good reliability. The deep learning-based ophthalmic ultrasound image intelligent diagnosis model can accurately screen and identify five posterior ocular segment diseases, which is beneficial to the intelligent development of ophthalmic clinical diagnosis.

MLAGG-Net: Multi-level aggregation and global guidance network for pancreatic lesion segmentation in histopathological images

Pancreatic cancer is one of the most aggressive and lethal malignancies in the world. Automatic and accurate segmentation of lesion regions from histopathological images is essential for disease diagnosis and analysis. Nevertheless, it remains a challenging task due to the complicated pathological manifestations of the lesion, including the ambiguity of boundaries, the heterogeneity of textures, and the large variations in morphology, size and location. To address these problems, we propose a multi-level aggregation and global guidance network. Specifically, we utilize a multi-level feature fusion and distribution module that aims to alleviate the scale diversity and complex boundary problems in lesions by aggregating fine-grained information of low-level features and semantic information of high-level features. The cross-attention fusion module is proposed to perform the aggregation of hierarchical multi-level features, which can absorb effective information and suppress redundant information from them. This paper also introduces a global information fusion module to guide the network to locate the lesion areas with inconsistent distribution between the internal and boundary regions more accurately for more complete segmentation results. We conduct experiments on our PANC dataset and the public GlaS challenge dataset to verify the effectiveness of the proposed network. The experimental results show that the proposed method can achieve better segmentation performance with 90.02% Dice and 82.07% Jaccard compared to other state-of-the-art methods on our PANC dataset while achieving competitive results on the GlaS challenge dataset.

Wildfire Detection via a Dual-Channel CNN with Multi-Level Feature Fusion

Forest fires have devastating impacts on ecology, the economy, and human life. Therefore, the timely detection and extinguishing of fires are crucial to minimizing the losses caused by these disasters. A novel dual-channel CNN for forest fires is proposed in this paper based on multiple feature enhancement techniques. First, the features’ semantic information and richness are enhanced by repeatedly fusing deep and shallow features extracted from the basic network model and integrating the results of multiple types of pooling layers. Second, an attention mechanism, the convolutional block attention module, is used to focus on the key details of the fused features, making the network more efficient. Finally, two improved single-channel networks are merged to obtain a better-performing dual-channel network. In addition, transfer learning is used to address overfitting and reduce time costs. The experimental results show that the accuracy of the proposed model for fire recognition is 98.90%, with a better performance. The findings from this study can be applied to the early detection of forest fires, assisting forest ecosystem managers in developing timely and scientifically informed defense strategies to minimize the damage caused by fires.

For cervical cancer diagnosis: Tissue Raman spectroscopy and multi-level feature fusion with SENet attention mechanism

Cervical cancer ranks among the most prevalent forms of gynecological malignancies. Timely identification of cervical lesions and prompt intervention can effectively prevent the development of cervical cancer or enhance patients' chances of survival. In this study, we propose an innovative method based on Raman spectroscopy, i.e., a multi-level SENet attention mechanism feature fusion architecture (MAFA) for rapid diagnosis of cervical cancer and precancerous lesions. The convolution process of this architecture can extract features from shallow to deep layers, and the attention mechanism is added to achieve the fusion of features from different layers. The added attention mechanism can automatically determine the importance of each layer feature channel and assign weight values to that layer according to the importance of each layer to achieve the purpose of focusing the model on certain waveform features and improve the targeting of model learning. We collected Raman spectra of 212 cervical tissues containing cervical cancer and its precancerous lesions.The experimental results show that MAFA can effectively improve the diagnostic accuracy of VGGNet, GoogLeNet and ResNet models in the validation of Raman spectral data of cervical tissue. Among them, ResNet performed the best, with the highest average accuracy, precision, recall and F1-Score of 82.36%, 84.00%, 82.35% and 82.26%, respectively, when no feature fusion was performed. The evaluation metrics improved by 4.91%, 3.97%, 4.97%, and 5.06%, respectively, after using the MAFA; they also improved by 4.16%, 2.90%, 4.17%, and 4.32%, respectively, compared with the model that directly performs feature fusion without using the attention mechanism. Therefore, the MAFA proposed in this study is better than that of the neural network that directly fuses the features of each convolutional layer. The experimental results show that the performance of the MAFA proposed in this paper is significantly higher than that of traditional deep learning algorithms, indicating that the present architecture can effectively improve the diagnostic accuracy of deep learning networks for cervical cancer.

Multi-Level Difference Network for Change Detection from Very High-Resolution Remote Sensing Images: A Case Study in Open-Pit Mines

Automatic change detection based on remote sensing is playing an increasingly important role in the national economy construction. To address the problem of limited change detection accuracy in existing single-level difference networks, this study proposes the Multi-level Difference Network (MDNet) for automatic change detection of ground targets from very high-resolution (VHR) remote sensing images. An early-difference network and a late-difference network are combined by MDNet to extract multi-level change features. The early-difference network can focus on change information throughout to reduce the spurious changes in the change detection results, and the late-difference network can provide deep features of a single image for reducing rough boundaries and scattered holes in the change detection results, thus improving the accuracy. However, not all high-level features extracted by MDNet contribute to the recognition of image differences, and the multi-level change features suffer from cross-channel heterogeneity. Stacking them directly on channels does not make effective use of change information, thus limiting the performance of MDNet. Therefore, the Multi-level Change Features Fusion Module (MCFFM) is proposed in this study for the effective fusion of multi-level change features. In the experiments, the publicly available open-pit mine change detection (OMCD) dataset was used first to achieve a change detection of open-pit mines over a large area, with an F1-score of 89.2%, increasing by 1.3% to 5.9% compared to the benchmark methods. Then, a self-made OMCD dataset was used to achieve an F1-score of 92.8% for the localized and fine-scale change detection in open-pit mines, which is an improvement of 0.7% to 5.4% compared to the benchmark methods. Finally, the Season-varying Change Detection Dataset is used to verify that the MDNet proposed can detect changes in other scenarios very well. The experimental results show that the proposed MDNet has significantly improved the performance of change detection on the three datasets compared with six advanced deep learning models, which will contribute to the development of change detection with VHR remote sensing images.

Graphic image classification method based on an attention mechanism and fusion of multilevel and multiscale deep features

The paper explores the issue of privacy breaches caused by sharing and publishing images on social media platforms, which is becoming increasingly serious. While existing deep learning-based methods have achieved remarkable results in private image classification, they only consider single-level and single-scale features and neglect the problem of multilevel and multiscale features. To address this limitation, the paper proposes a novel graphic image classification method that fuses multilevel and multiscale deep features. The proposed method employs a multibranch convolutional neural network to extract multilevel features, which are then processed by a multilevel pooling layer to obtain multiscale features. Additionally, two feature fusion methods based on attention mechanism, Privacy-MSML (Privacy Multi-Scale Multi-Level) and Privacy-MLMS (Privacy Multi-Level Multi-Scale), are designed to fuse the multilevel and multiscale features for image classification. Both methods utilize Bi-LSTM and a self-attention mechanism to capture feature dependencies. The experimental results on public datasets demonstrate that the proposed methods effectively fuse multilevel and multiscale features, leading to a significant improvement in classification performance, which highlights the innovative contribution of the paper in addressing the issue of multilevel and multiscale features in private image classification.

DOMOPT: A Detection-Based Online Multi-Object Pedestrian Tracking Network for Videos

Due to the problem of low tracking accuracy and weak tracking stability of current multi-object pedestrian tracking algorithms in complex scenes for videos, a Detection-based Online Multi-Object Pedestrian Tracking (DOMOPT) network is proposed. First, a Multi-Level Feature Fusion (MLFF) pedestrian detection network is proposed based on the Center and Scale Prediction (CSP) algorithm. The pyramid convolutional neural network is used as the backbone to enhance the feature extraction capability for small objects. The shallow features and deep features at multiple levels are integrated to fully obtain the position and semantic information to further improve the detection performance for small objects. Then, on the basis of Joint Detection and Embedding (JDE) architecture, a Multi-Branch Pedestrian Appearance (MBPA) feature extraction network is proposed and added into the pedestrian detection network to extract the appearance feature vector corresponding to each pedestrian. The pedestrian appearance feature extraction is treated as a classification task jointly training with the pedestrian detection task, using the multi-task learning strategy. Experimental results show that the proposed network has better tracking accuracy and stability compared with state-of-the-art algorithms.

A dual-structure attention-based multi-level feature fusion network for automatic surface defect detection

A dual-structure attention-based multi-level feature fusion network for automatic surface defect detection

Deep Digging into the Generalization of Self-Supervised Monocular Depth Estimation

Self-supervised monocular depth estimation has been widely studied recently. Most of the work has focused on improving performance on benchmark datasets, such as KITTI, but has offered a few experiments on generalization performance. In this paper, we investigate the backbone networks (e.g., CNNs, Transformers, and CNN-Transformer hybrid models) toward the generalization of monocular depth estimation. We first evaluate state-of-the-art models on diverse public datasets, which have never been seen during the network training. Next, we investigate the effects of texture-biased and shape-biased representations using the various texture-shifted datasets that we generated. We observe that Transformers exhibit a strong shape bias and CNNs do a strong texture-bias. We also find that shape-biased models show better generalization performance for monocular depth estimation compared to texture-biased models. Based on these observations, we newly design a CNN-Transformer hybrid network with a multi-level adaptive feature fusion module, called MonoFormer. The design intuition behind MonoFormer is to increase shape bias by employing Transformers while compensating for the weak locality bias of Transformers by adaptively fusing multi-level representations. Extensive experiments show that the proposed method achieves state-of-the-art performance with various public datasets. Our method also shows the best generalization ability among the competitive methods.

Building Change Detection in Remote Sensing Imagery with Focal Self-Attention and Multi-Level Feature Fusion

Accurate and intelligent building change detection greatly contributes to effective urban development, optimized resource management, and informed decision-making in domains such as urban planning, land management, and environmental monitoring. Existing methodologies face challenges in effectively integrating local and global features for accurate building change detection. To address these challenges, we propose a novel method that uses focal self-attention to process the feature vector of input images, which uses a “focusing” mechanism to guide the calculation of the self-attention mechanism. By focusing more on critical areas when processing image features in different regions, focal self-attention can better handle both local and global information, and is more flexible and adaptive than other methods, improving detection accuracy. In addition, our multi-level feature fusion module groups the features and then constructs a hierarchical residual structure to fuse the grouped features. On the LEVIR-CD and WHU-CD datasets, our proposed method achieved F1-scores of 91.62% and 89.45%, respectively. Compared with existing methods, ours performed better on building change detection tasks. Our method therefore provides a framework for solving problems related to building change detection, with some reference value and guiding significance.

Transformer-Based Context Condensation for Boosting Feature Pyramids in Object Detection

Current object detectors typically have a feature pyramid (FP) module for multi-level feature fusion (MFF) which aims to mitigate the gap between features from different levels and form a comprehensive object representation to achieve better detection performance. However, they usually require heavy cross-level connections or iterative refinement to obtain better MFF result, making them complicated in structure and inefficient in computation. To address these issues, we propose a novel and efficient context modeling mechanism that can help existing FPs deliver better MFF results while reducing the computational costs effectively. In particular, we introduce a novel insight that comprehensive contexts can be decomposed and condensed into two types of representations for higher efficiency. The two representations include a locally concentrated representation and a globally summarized representation, where the former focuses on extracting context cues from nearby areas while the latter extracts general contextual representations of the whole image scene as global context cues. By collecting the condensed contexts, we employ a Transformer decoder to investigate the relations between them and each local feature from the FP and then refine the MFF results accordingly. As a result, we obtain a simple and light-weight Transformer-based Context Condensation (TCC) module, which can boost various FPs and lower their computational costs simultaneously. Extensive experimental results on the challenging MS COCO dataset show that TCC is compatible to four representative FPs and consistently improves their detection accuracy by up to 7.8% in terms of average precision and reduce their complexities by up to around 20% in terms of GFLOPs, helping them achieve state-of-the-art performance more efficiently. Code will be released at https://github.com/zhechen/TCC.

Edge-Guided Camouflaged Object Detection via Multi-Level Feature Integration.

Camouflaged object detection (COD) aims to segment those camouflaged objects that blend perfectly into their surroundings. Due to the low boundary contrast between camouflaged objects and their surroundings, their detection poses a significant challenge. Despite the numerous excellent camouflaged object detection methods developed in recent years, issues such as boundary refinement and multi-level feature extraction and fusion still need further exploration. In this paper, we propose a novel multi-level feature integration network (MFNet) for camouflaged object detection. Firstly, we design an edge guidance module (EGM) to improve the COD performance by providing additional boundary semantic information by combining high-level semantic information and low-level spatial details to model the edges of camouflaged objects. Additionally, we propose a multi-level feature integration module (MFIM), which leverages the fine local information of low-level features and the rich global information of high-level features in adjacent three-level features to provide a supplementary feature representation for the current-level features, effectively integrating the full context semantic information. Finally, we propose a context aggregation refinement module (CARM) to efficiently aggregate and refine the cross-level features to obtain clear prediction maps. Our extensive experiments on three benchmark datasets show that the MFNet model is an effective COD model and outperforms other state-of-the-art models in all four evaluation metrics (Sα, Eϕ, Fβw, and MAE).

SCNet: A spatial feature fused convolutional network for multi-channel EEG pathology detection

Deep learning models, especially convolutional neural networks (CNNs), have shown remarkable achievements in clinical electroencephalography (EEG) pathology detection, due to their prominent capability in automatic feature learning. However, most existing CNN-based models ignore the spatial correlations of the EEG signals and the latent complementary information provided by different convolutional layers, which is essentially an important clue for pathology detection. To this end, we propose SCNet, a Spatial Feature Fused Convolutional Network, for multi-channel EEG pathology detection. SCNet first designs a spatial information learning mechanism to capture the channel-wise spatial correlations via global pooling strategies. A multi-level feature fusion module, which combines feature maps learned by different convolutional layers, is then devised to fully leverage the complementarity of multi-level features. The efficacy of the proposed SCNet is experimentally evaluated on both datasets, and the results obtained outperform current representatives in the literature. Ablation studies on the spatial information learning and multi-level fusion modules further confirm their great potential in EEG-based diagnostic applications.