Multi-level Feature Fusion Research Articles

Context-Aware Attentive Multilevel Feature Fusion for Named Entity Recognition.

In the era of information explosion, named entity recognition (NER) has attracted widespread attention in the field of natural language processing, as it is fundamental to information extraction. Recently, methods of NER based on representation learning, e.g., character embedding and word embedding, have demonstrated promising recognition results. However, existing models only consider partial features derived from words or characters while failing to integrate semantic and syntactic information, e.g., capitalization, inter-word relations, keywords, and lexical phrases, from multilevel perspectives. Intuitively, multilevel features can be helpful when recognizing named entities from complex sentences. In this study, we propose a novel attentive multilevel feature fusion (AMFF) model for NER, which captures the multilevel features in the current context from various perspectives. It consists of four components to, respectively, capture the local character-level (CL), global character-level (CG), local word-level (WL), and global word-level (WG) features in the current context. In addition, we further define document-level features crafted from other sentences to enhance the representation learning of the current context. To this end, we introduce a novel context-aware attentive multilevel feature fusion (CAMFF) model based on AMFF, to fully leverage document-level features from all the previous inputs. The obtained multilevel features are then fused and fed into a bidirectional long short-term memory (BiLSTM)-conditional random field (CRF) network for the final sequence labeling. Extensive experiments on four benchmark datasets demonstrate that our proposed AMFF and CAMFF models outperform a set of state-of-the-art baseline methods and the features learned from multiple levels are complementary.

A Cross-Attention and Multilevel Feature Fusion Network for Breast Lesion Segmentation in Ultrasound Images

A Cross-Attention and Multilevel Feature Fusion Network for Breast Lesion Segmentation in Ultrasound Images

Position Fusing and Refining for Clear Salient Object Detection.

Multilevel feature fusion plays a pivotal role in salient object detection (SOD). High-level features present rich semantic information but lack object position information, whereas low-level features contain object position information but are mixed with noises such as backgrounds. Appropriately addressing the gap between low-and high-level features is important in SOD. We first propose a global position embedding attention (GPEA) module to minimize the discrepancy between multilevel features in this article. We extract the position information by utilizing the semantic information at high-level features to resist noises at low-level features. Object refine attention (ORA) module is introduced to refine features used to predict saliency maps further without any additional supervision and heighten discriminative regions near the salient object, such as boundaries. Moreover, we find that the saliency maps generated by the previous methods contain some blurry regions, and we design a pixel value (PV) loss to help the model generate saliency maps with improved clarity. Experimental results on five commonly used SOD datasets demonstrated that the proposed method is effective and outperforms the state-of-the-art approaches on multiple metrics.

MFFAGAN: Generative Adversarial Network With Multilevel Feature Fusion Attention Mechanism for Remote Sensing Image Super-Resolution

MFFAGAN: Generative Adversarial Network With Multilevel Feature Fusion Attention Mechanism for Remote Sensing Image Super-Resolution

KD-SCFNet: Towards more accurate and lightweight salient object detection via knowledge distillation

Most existing salient object detection (SOD) models are difficult to apply due to the complex and huge model structures. Although some lightweight models are proposed, the accuracy is barely satisfactory. In this paper, we design a novel semantics-guided contextual fusion network (SCFNet) that focuses on the interactive fusion of multi-level features for accurate and efficient salient object detection. Furthermore, we apply knowledge distillation to SOD task and provide a sizeable dataset KD-SOD80K. In detail, we transfer the rich knowledge from a seasoned teacher to the untrained SCFNet through unlabeled images, enabling SCFNet to learn a strong generalization ability to detect salient objects more accurately. The knowledge distillation based SCFNet (KD-SCFNet) achieves comparable accuracy to the state-of-the-art heavyweight methods with less than 1M parameters and 174 FPS real-time detection speed. Extensive experiments demonstrate the robustness and effectiveness of the proposed distillation method and SOD framework.

Multi-level feature fusion network combining attention mechanisms for polyp segmentation

Clinically, automated polyp segmentation techniques have the potential to significantly improve the efficiency and accuracy of medical diagnosis, thereby reducing the risk of colorectal cancer in patients. Unfortunately, existing methods suffer from two significant weaknesses that can impact the accuracy of segmentation. Firstly, features extracted by encoders are not adequately filtered and utilized. Secondly, semantic conflicts and information redundancy caused by feature fusion are not attended to. To overcome these limitations, we propose a novel approach for polyp segmentation, named MLFF-Net, which leverages multi-level feature fusion and attention mechanisms. Specifically, MLFF-Net comprises three modules: Multi-scale Attention Module (MAM), High-level Feature Enhancement Module (HFEM), and Global Attention Module (GAM). Among these, MAM is used to extract multi-scale information and polyp details from the shallow output of the encoder. In HFEM, the deep features of the encoders complement each other by aggregation. Meanwhile, the attention mechanism redistributes the weight of the aggregated features, weakening the conflicting redundant parts and highlighting the information useful to the task. GAM combines features from the encoder and decoder features, as well as computes global dependencies to prevent receptive field locality. Experimental results on five public datasets show that the proposed method not only can segment multiple types of polyps but also has advantages over current state-of-the-art methods in both accuracy and generalization ability.

DFN-PSAN: Multi-level deep information feature fusion extraction network for interpretable plant disease classification

Accurate identification of crop diseases is an effective way to promote the development of intelligent and modernized agricultural production, as well as to reduce the use of pesticides and improve crop yield and quality. Deep learning methods have achieved better performance in classifying input plant disease images. However, many plant disease datasets are often constructed from controlled scenarios, and these deep learning models may not perform well when tested in real-world agricultural environments, highlighting the challenges of transitioning to natural farm environments under the new demand paradigm of Agri 4.0. Based on the above reasons, this work proposes using a multi-level deep information feature fusion extraction network (DFN-PSAN) to achieve plant disease classification in natural field environments. DFN-PSAN adopts the YOLOv5 Backbone and Neck network as the base structure DFN and uses pyramidal squeezed attention (PSA) combined with multiple convolutional layers to design a novel classification network PSAN, which fuses and processes the multi-level depth information features output from DFN and highlights the critical regions of plant disease images with the help of pixel-level attention provided by PSA, thus realizing effective classification of multiple fine-grained plant diseases. The proposed DFN-PSAN was trained and tested on three plant disease datasets. The average accuracy and F1-score exceeded 95.27%. The PSA attention mechanism saved 26% of model parameters, achieving a competitive performance among existing related methods. In addition, this work effectively enhances the transparency of the features of the model attention to plant diseases through t-SNE with SHAP interpretable methods.

Attention combined pyramid vision transformer for polyp segmentation

Colorectal cancer (CRC) has become one of the most frequent cancers in the world. To prevent CRC, proper polyp localization in endoscopy images plays a vital role for detecting and removing colorectal polyps. Most polyp segmentation methods use convolutional neural networks (CNN) as their backbone, and have achieved promising results to effectively assist clinicians in their diagnosis. However, those CNN-based approaches have limitations in modeling accurate location and shape of polyps, due to the intrinsic locality property of convolutional operations. To address these limitations, this study proposes a novel network, namely Att-PVT, that combines CNN and Pyramid Vision Transformer (PVT) together for poly segmentation. The main challenge lies in maintaining long-range semantic information without sacrificing low-level features. Att-PVT applies multidimensional information extraction (MIE) to generate refined feature maps extracted from PVT for better feature representation. Cascaded context integration (CCI) is designed to adaptively aggregating the three highest layers of refined polyp features for learning semantic and location information. To accurately segment colorectal polyps, Att-PVT introduces multilevel feature fusion (MFF) module that explores the boundary information in the shallower layer based on the global map. The proposed workflow has undergone comparative experiments on three public datasets, namely Kvasir, ColonDB, and ETIS. The results show that the proposed approach achieves impressive mDice scores of 0.926, 0.817, and 0.794 for polyp segmentation tasks on these datasets, surpassing other state-of-the-art methods. This indicates the superior generalization and scalability of the proposed approach.

FACILE: A capsule network with fewer capsules and richer hierarchical information for malware image classification

The struggle between security researchers and malware perpetuates an endless arms race. Recent studies indicate that converting malware into grayscale images and using convolutional neural network or capsule network to classify and identify them is a promising approach. Nonetheless, training convolutional neural networks demands a significant amount of data and parameters, posing a challenge to obtaining enough training samples for malware detection tasks, particularly for new malware families. Additionally, a high number of parameters translates to low classification efficiency. Capsule networks offer a promising approach to achieve higher-level feature representations through dynamic routing between capsules, but research on capsule networks is still nascent, and there are two major challenges for malware classification tasks: the huge number of parameters, which makes model training difficult, and capsules shedding, which results in information loss during routing. To address these issues, we propose a capsule network called FACILE that uses fewer capsules and richer hierarchical information. In the initial feature extraction stage, we perform multi-level feature extraction and fusion using dynamic convolution. In the routing phase, we introduce balance coefficients to enhance the model's representational power and stabilize the training process. We conducted experiments on the VIRUS-MNIST, MalImg, and BIG2015 datasets and found that FACILE requires only 8.1% of the capsules and 1.8%-3.3% of the parameters compared to the original CapsNet, while reducing the error rate to 8.087%, 1.149%, and 2.797%, respectively. Furthermore, FACILE demonstrates competitive performance with a small number of parameters as the training samples decrease compared to convolutional neural network-based models such as ResNet, EfficientNet, and VGG.

Cross Modality Feature Fusion Framework for Diabetic Retinopathy Image Classification

Diabetic retinopathy (DR) is a prevalent ocular condition and a prominent contributor to vision impairment in individuals with diabetes. Consistent monitoring through fundus photography and prompt intervention represents the most efficient strategy for controlling this ailment. Given the substantial diabetic patient population and their extensive screening needs, there is a growing inclination towards harnessing computer-aided and entirely automated methods for diagnosing DR. Over the past years, deep neural networks have achieved remarkable progress across a wide range of applications. Consequently, automating the diagnosis of DR and delivering tailored recommendations to DR patients underscores the significance of accurate and intricate DR classification. In this work, we have present a cross modality feature fusion based framework for diabetic retinopathy (DR) image classification. Here, cross modality means RGB image and it’s green channel. Initially, we have present multi-scale multi- receptive feature extraction block to learn the local and global features from both the modalities. Further, the learned features at various scale are fused effectively with present multi-level feature fusion block for image classification task. We evaluated our present framework on MESSIDOR and IDRID database by comparing it to state-of-the-art (SOTA) deep learning frame- works for DR image classification. The result analysis clearly demonstrate that the present cross-modality feature fusion based classification framework outperforms existing SOTA frameworks in terms of various evaluation parameters.

A High-Resolution Network with Strip Attention for Retinal Vessel Segmentation

Accurate segmentation of retinal vessels is an essential prerequisite for the subsequent analysis of fundus images. Recently, a number of methods based on deep learning have been proposed and shown to demonstrate promising segmentation performance, especially U-Net and its variants. However, tiny vessels and low-contrast vessels are hard to detect due to the issues of a loss of spatial details caused by consecutive down-sample operations and inadequate fusion of multi-level features caused by vanilla skip connections. To address these issues and enhance the segmentation precision of retinal vessels, we propose a novel high-resolution network with strip attention. Instead of the U-Net-shaped architecture, the proposed network follows an HRNet-shaped architecture as the basic network, learning high-resolution representations throughout the training process. In addition, a strip attention module including a horizontal attention mechanism and a vertical attention mechanism is designed to obtain long-range dependencies in the horizontal and vertical directions by calculating the similarity between each pixel and all pixels in the same row and the same column, respectively. For effective multi-layer feature fusion, we incorporate the strip attention module into the basic network to dynamically guide adjacent hierarchical features. Experimental results on the DRIVE and STARE datasets show that the proposed method can extract more tiny vessels and low-contrast vessels compared with existing mainstream methods, achieving accuracies of 96.16% and 97.08% and sensitivities of 82.68% and 89.36%, respectively. The proposed method has the potential to aid in the analysis of fundus images.

Optimizing Road Safety: Advancements in Lightweight YOLOv8 Models and GhostC2f Design for Real-Time Distracted Driving Detection.

The rapid detection of distracted driving behaviors is crucial for enhancing road safety and preventing traffic accidents. Compared with the traditional methods of distracted-driving-behavior detection, the YOLOv8 model has been proven to possess powerful capabilities, enabling it to perceive global information more swiftly. Currently, the successful application of GhostConv in edge computing and embedded systems further validates the advantages of lightweight design in real-time detection using large models. Effectively integrating lightweight strategies into YOLOv8 models and reducing their impact on model performance has become a focal point in the field of real-time distracted driving detection based on deep learning. Inspired by GhostConv, this paper presents an innovative GhostC2f design, aiming to integrate the idea of linear transformation to generate more feature maps without additional computation into YOLOv8 for real-time distracted-driving-detection tasks. The goal is to reduce model parameters and computational load. Additionally, enhancements have been made to the path aggregation network (PAN) to amplify multi-level feature fusion and contextual information propagation. Furthermore, simple attention mechanisms (SimAMs) are introduced to perform self-normalization on each feature map, emphasizing feature maps with valuable information and suppressing redundant information interference in complex backgrounds. Lastly, the nine distinct distracted driving types in the publicly available SFDDD dataset were expanded to 14 categories, and nighttime scenarios were introduced. The results indicate a 5.1% improvement in model accuracy, with model weight size and computational load reduced by 36.7% and 34.6%, respectively. During 30 real vehicle tests, the distracted-driving-detection accuracy reached 91.9% during daylight and 90.3% at night, affirming the exceptional performance of the proposed model in assisting distracted driving detection when driving and contributing to accident-risk reduction.

A lightweight RGB superposition effect adjustment network for low-light image enhancement and denoising

In low-light conditions, under-exposure causes certain kinds of image quality degradations, including intensive noise, biased color and degraded contrast. And during the enhancement process, it is also easy to amplify noise or cause overexposure. To overcome these problems, in this paper, an unsupervised learning approach, which is a lightweight RGB superposition effect adjustment network for low light image enhancement (SEA-Net), is proposed. The detail feature module in the proposed network can predict the appropriate saturation and brightness of the color by adjusting the distance among the R, G and B values of the pixels, and then use image multi-level feature fusion to control the overall contrast of the image. The proposed refining extraction operation algorithm uses Hadamard product to amplify the features of image color information, which can effectively avoid color deviation in the enhanced image. The network removes noise by decomposing the image and reconstructing it. To prevent image overexposure, the network effectively suppresses pixel value overflow and avoid image information loss. Because only low-light images are required for training, the proposed loss functions guide network to generate more realistic images with better saturation and contrast. Despite its light weight, this method shows faster running time and more favorable performance than other competitors. In reality, it can be applied to improve the performance of advanced image tasks under low-light conditions, such as nighttime facial recognition.

Enhanced YOLOv5: An Efficient Road Object Detection Method.

Accurate identification of road objects is crucial for achieving intelligent traffic systems. However, developing efficient and accurate road object detection methods in complex traffic scenarios has always been a challenging task. The objective of this study was to improve the target detection algorithm for road object detection by enhancing the algorithm's capability to fuse features of different scales and levels, thereby improving the accurate identification of objects in complex road scenes. We propose an improved method called the Enhanced YOLOv5 algorithm for road object detection. By introducing the Bidirectional Feature Pyramid Network (BiFPN) into the YOLOv5 algorithm, we address the challenges of multi-scale and multi-level feature fusion and enhance the detection capability for objects of different sizes. Additionally, we integrate the Convolutional Block Attention Module (CBAM) into the existing YOLOv5 model to enhance its feature representation capability. Furthermore, we employ a new non-maximum suppression technique called Distance Intersection Over Union (DIOU) to effectively address issues such as misjudgment and duplicate detection when significant overlap occurs between bounding boxes. We use mean Average Precision (mAP) and Precision (P) as evaluation metrics. Finally, experimental results on the BDD100K dataset demonstrate that the improved YOLOv5 algorithm achieves a 1.6% increase in object detection mAP, while the P value increases by 5.3%, effectively improving the accuracy and robustness of road object recognition.

DMFF-Net: Double-stream multilevel feature fusion network for image forgery localization

With the rapid development of image processing technology, it has become increasingly easy to manipulate images, which poses a threat to the stability and security of people’s lives. Recent methods have proposed the fusion of RGB and noise features to uncover tampering traces. However, these approaches overlook the characteristics of features at different levels, leading to insufficient feature fusion. To address this problem, this paper proposes a double-stream multilevel feature fusion network (DMFF-Net). Unlike the traditional feature fusion approach, DMFF-Net adopts a graded feature fusion strategy. It classifies features into primary, intermediate, and advanced levels and introduces the Primary Feature Fusion Module (PFFM) and the Advanced Feature Fusion Module (AFFM) to achieve superior fusion results. Additionally, a multisupervision strategy is employed to decode the fused features into level-specific masks, including boundary, regular, and refined masks. The DMFF-Net is validated on publicly available datasets, including CASIA, Columbia, COVERAGE, and NIST16, as well as a real-life manipulated image dataset, IMD20, and achieves AUCs of 84.7%, 99.6%, 86.6%, 87.4% and 82.8%, respectively. Extensive experiments show that our DMFF-Net outperforms state-of-the-art methods in terms of image manipulation localization accuracy and exhibits improved robustness.

Point Cloud Deep Learning Network Based on Local Domain Multi-Level Feature

Point cloud deep learning networks have been widely applied in point cloud classification, part segmentation and semantic segmentation. However, current point cloud deep learning networks are insufficient in the local feature extraction of the point cloud, which affects the accuracy of point cloud classification and segmentation. To address this issue, this paper proposes a local domain multi-level feature fusion point cloud deep learning network. First, dynamic graph convolutional operation is utilized to obtain the local neighborhood feature of the point cloud. Then, relation-shape convolution is used to extract a deeper-level edge feature of the point cloud, and max pooling is adopted to aggregate the edge features. Finally, point cloud classification and segmentation are realized based on global features and local features. We use the ModelNet40 and ShapeNet datasets to conduct the comparison experiment, which is a large-scale 3D CAD model dataset and a richly annotated, large-scale dataset of 3D shapes. For ModelNet40, the overall accuracy (OA) of the proposed method is similar to DGCNN, RS-CNN, PointConv and GAPNet, all exceeding 92%. Compared to PointNet, PointNet++, SO-Net and MSHANet, the OA of the proposed method is improved by 5%, 2%, 3% and 2.6%, respectively. For the ShapeNet dataset, the mean Intersection over Union (mIoU) of the part segmentation achieved by the proposed method is 86.3%, which is 2.9%, 1.4%, 1.7%, 1.7%, 1.2%, 0.1% and 1.0% higher than PointNet, RS-Net, SCN, SPLATNet, DGCNN, RS-CNN and LRC-NET, respectively.

MLFFusion: Multi-level feature fusion network with region illumination retention for infrared and visible image fusion

Infrared image and visible light image fusion is widely used in night vision, surveillance, military and other fields. The focus of the fusion task is to integrate complementary information in visible and infrared light images and eliminate redundant information. In addition, most of the fusion tasks are performed in the harsh environment of low light, and it is worth studying how to maintain the lighting information of the fusion results. In order to solve the problems existing, firstly, we designs a multi-level feature module to fusion multi-source information. Different from the parallel layer fusion strategy of the traditional network, we proposed a fusion strategy combine parallel layers and depth layers. Secondly, we add attention computing to the feature extraction network to improve the performance of the feature extraction network. Thirdly, in order to make the fusion image have good illumination information, we design the area illumination retention module, improved the performance of the fusion algorithm in low-light environments. A large number of experiments show that the proposed method has excellent performance and will perform better in low light environment. In addition, the proposed algorithm also shows great potential in multi modal object detection.

SMFF-YOLO: A Scale-Adaptive YOLO Algorithm with Multi-Level Feature Fusion for Object Detection in UAV Scenes

Object detection in images captured by unmanned aerial vehicles (UAVs) holds great potential in various domains, including civilian applications, urban planning, and disaster response. However, it faces several challenges, such as multi-scale variations, dense scenes, complex backgrounds, and tiny-sized objects. In this paper, we present a novel scale-adaptive YOLO framework called SMFF-YOLO, which addresses these challenges through a multi-level feature fusion approach. To improve the detection accuracy of small objects, our framework incorporates the ELAN-SW object detection prediction head. This newly designed head effectively utilizes both global contextual information and local features, enhancing the detection accuracy of tiny objects. Additionally, the proposed bidirectional feature fusion pyramid (BFFP) module tackles the issue of scale variations in object sizes by aggregating multi-scale features. To handle complex backgrounds, we introduce the adaptive atrous spatial pyramid pooling (AASPP) module, which enables adaptive feature fusion and alleviates the negative impact of cluttered scenes. Moreover, we adopt the Wise-IoU(WIoU) bounding box regression loss to enhance the competitiveness of different quality anchor boxes, which offers the framework a more informed gradient allocation strategy. We validate the effectiveness of SMFF-YOLO using the VisDrone and UAVDT datasets. Experimental results demonstrate that our model achieves higher detection accuracy, with AP50 reaching 54.3% for VisDrone and 42.4% for UAVDT datasets. Visual comparative experiments with other YOLO-based methods further illustrate the robustness and adaptability of our approach.

Dynamic Message Propagation Network for RGB-D and Video Salient Object Detection

Exploiting long-range semantic contexts and geometric information is crucial to infer salient objects from RGB and depth features. However, existing methods mainly focus on excavating local features within fixed regions by continuously feeding forward networks. In this article, we introduce Dynamic Message Propagation (DMP) to dynamically learn context information within more flexible regions. We integrate DMP into a Siamese-based network to process the RGB image and depth map separately and design a multi-level feature fusion module to explore cross-level information between refined RGB and depth features. Extensive experiments show clear improvements of our method over 17 methods on six benchmark datasets for RGB-D salient object detection (SOD). Additionally, our method outperforms its competitors for the video SOD task. Code is available at https://github.com/chenbaian-cs/DMPNet .

SCA-Net: Multiscale Contextual Information Network for Building Extraction Based on High-Resolution Remote Sensing Images

Accurately extracting buildings is essential for urbanization rate statistics, urban planning, resource allocation, etc. The high-resolution remote sensing images contain rich building information, which provides an important data source for building extraction. However, the extreme abundance of building types with large differences in size, as well as the extreme complexity of the background environment, result in the accurate extraction of spatial details of multi-scale buildings, which remains a difficult problem worth studying. To this end, this study selects the representative Xinjiang Tumxuk urban area as the study area. A building extraction network (SCA-Net) with feature highlighting, multi-scale sensing, and multi-level feature fusion is proposed, which includes Selective kernel spatial Feature Extraction (SFE), Contextual Information Aggregation (CIA), and Attentional Feature Fusion (AFF) modules. First, Selective kernel spatial Feature Extraction modules are used for cascading composition, highlighting information representation of features, and improving the feature extraction capability. Adding a Contextual Information Aggregation module enables the acquisition of multi-scale contextual information. The Attentional Feature Fusion module bridges the semantic gap between high-level and low-level features to achieve effective fusion between cross-level features. The classical U-Net, Segnet, Deeplab v3+, and HRNet v2 semantic segmentation models are compared on the self-built Tmsk and WHU building datasets. The experimental results show that the algorithm proposed in this paper can effectively extract multi-scale buildings in complex backgrounds with IoUs of 85.98% and 89.90% on the two datasets, respectively. SCA-Net is a suitable method for building extraction from high-resolution remote sensing images with good usability and generalization.