Feature Fusion Module Research Articles

SGNN-T: Space graph neural network coupled transformer for molecular property prediction

Molecular properties play a crucial role in material discovery, protein interaction and drug development. The appearance of Graph Neural Network (GNN) significantly improved the performance of molecular property prediction. However, nodes in GNN only update the features of neighbor nodes, resulting in insufficient ability to encode global feature information. The self- attention mechanism in transformer can encode the global information except for local information of molecules, while its spatial information is insufficient. Since molecules are three-dimensional spatial structures, spatial geometry information is an important attribute for molecules properties. To consider these factors, a network model of Space Graph Neural Network coupled Transformer (SGNN-T) is proposed in this paper which can combine global and local molecule information with three-dimensional spatial structures for molecular properties prediction. In this model, Graph neural network Geometric Feature Fusion Module (GGFF) and Transformer Spatial Geometric Feature Enhancement Module (TSGFE) are included to enhance the spatial geometry learning ability of the network. The GGFF module constructs a parallel graph neural network by thinking over atoms, bonds and bond angles at the same time which effectively complements the spatial information of the network by leading into bond angles than normal GNN. The TSGFE module introduces the coordinates and centrality degree features coupled with the features by GGFF into transformer to further enhance the geometric expression ability of the module. Through these two parts, SGNN-T model can encode local and global information of molecules at the same time. Property prediction experiments are executed on the QM9, OMDB and MEGNet dataset. The results of MAE show the proposed model has the best performance than the popular models.

Data‐Driven Modeling and High‐Precision Tracking Control of a Soft Continuum Manipulator: Enabling Robotic Sorting of Multiwire Cables

As a new class of robots, soft continuum manipulators have attracted attention due to their flexibility and compliance. However, these characteristics create challenges for precise modeling and control. This study proposes a hybrid offline and online data‐driven scheme to achieve high‐precision tracking control of a soft continuum manipulator. First, a novel multiscale deep neural network learns the manipulator model offline. Specifically, the feature fusion module extracts highly discriminative features and captures long‐term dependencies from the temporal trajectory data. The self‐attention module strengthens the ability to represent fusion features and enhances the model prediction accuracy. Then, the learnt model is updated using multisensor data online, and the proposed controller further compensates for the updated model and enhances the tracking accuracy in the movement stage. Finally, the experimental results demonstrate a significant improvement in motion accuracy under different trajectory‐tracking scenarios (i.e., deviations of <1 mm in position and <0.8° in orientation). The example of the multiwire cable sorting proves the feasibility of the proposed scheme in high‐precision industrial applications.

HRTracker: Multi-Object Tracking in Satellite Video Enhanced by High-Resolution Feature Fusion and an Adaptive Data Association

Multi-object tracking in satellite videos (SV-MOT) is an important task with many applications, such as traffic monitoring and disaster response. However, the widely studied multi-object tracking (MOT) approaches for general images can rarely be directly introduced into remote sensing scenarios. The main reasons for this can be attributed to the following: (1) the existing MOT approaches would cause a significant rate of missed detection of the small targets in satellite videos; (2) it is difficult for the general MOT approaches to generate complete trajectories in complex satellite scenarios. To address these problems, a novel SV-MOT approach enhanced by high-resolution feature fusion and a two-step association method is proposed. In the high-resolution detection network, a high-resolution feature fusion module is designed to assist detection by maintaining small object features in forward propagation. By utilizing features of different resolutions, the performance of the detection of small targets in satellite videos is improved. Through high-quality detection and the use of an adaptive Kalman filter, the densely packed weak objects can be effectively tracked by associating almost every detection box instead of only the high-score ones. The comprehensive experimental results using the representative satellite video datasets (VISO) demonstrate that the proposed HRTracker with the state-of-the-art (SOTA) methods can achieve competitive performance in terms of the tracking accuracy and the frequency of ID conversion, obtaining a tracking accuracy score of 74.6% and an ID F1 score of 78.9%.

High‐order multilayer attention fusion network for 3D object detection

AbstractThree‐dimensional object detection based on the fusion of 2D image data and 3D point clouds has become a research hotspot in the field of 3D scene understanding. However, different sensor data have discrepancies in spatial position, scale, and alignment, which severely impact detection performance. Inappropriate fusion methods can lead to the loss and interference of valuable information. Therefore, we propose the High‐Order Multi‐Level Attention Fusion Network (HMAF‐Net), which takes camera images and voxelized point clouds as inputs for 3D object detection. To enhance the expressive power between different modality features, we introduce a high‐order feature fusion module that performs multi‐level convolution operations on the element‐wise summed features. By incorporating filtering and non‐linear activation, we extract deep semantic information from the fused multi‐modal features. To maximize the effectiveness of the fused salient feature information, we introduce an attention mechanism that dynamically evaluates the importance of pooled features at each level, enabling adaptive weighted fusion of significant and secondary features. To validate the effectiveness of HMAF‐Net, we conduct experiments on the KITTI dataset. In the “Car,” “Pedestrian,” and “Cyclist” categories, HMAF‐Net achieves mAP performances of 81.78%, 60.09%, and 63.91%, respectively, demonstrating more stable performance compared to other multi‐modal methods. Furthermore, we further evaluate the framework's effectiveness and generalization capability through the KITTI benchmark test, and compare its performance with other published detection methods on the 3D detection benchmark and BEV detection benchmark for the “Car” category, showing excellent results. The code and model will be made available on https://github.com/baowenzhang/High‐order‐Multilayer‐Attention‐Fusion‐Network‐for‐3D‐Object‐Detection.

CloudSwinNet: A hybrid CNN-transformer framework for ground-based cloud images fine-grained segmentation

Solar irradiance is the main factor affecting the output of a photovoltaic (PV) power station. The dominant role of ground-based clouds on the variation of direct solar radiation in the whole sky. Ground-based cloud segmentation plays a crucial role in photovoltaic power generation prediction. Despite advancements, current cloud segmentation methods fail to meet the requirements of this application. While convolutional neural networks demonstrate impressive segmentation capabilities in ground-based cloud segmentation tasks, their inherent limitations hinder further performance enhancement. Hence, this paper introduces CloudSwinNet, a hybrid CNN-Transformer framework for ground-based cloud images fine-grained segmentation. CloudSwinNet leverages concepts from convolutional neural networks, including stepwise downsampling, local convolution, and skip connections. To further explore the fine-grained features in the ground-based cloud images, we incorporates the Fine-grained Feature Fusion Module (Fg-FFM) in encoder structure, while the jump connection structure integrates the GloRe spatial graph inference module. These additions enable comprehensive learning of multi-scale and long-range dependencies. We conducted extensive experiments on the fine-grained ground-based cloud dataset, demonstrating that CloudSwinNet outperforms six semantic segmentation networks in both qualitative and quantitative evaluations. The results of ablation experiments prove the effectiveness of the module introduced in this paper.

GMS-YOLO: An Algorithm for Multi-Scale Object Detection in Complex Environments in Confined Compartments.

Many compartments are prone to pose safety hazards such as loose fasteners or object intrusion due to their confined space, making manual inspection challenging. To address the challenges of complex inspection environments, diverse target categories, and variable scales in confined compartments, this paper proposes a novel GMS-YOLO network, based on the improved YOLOv8 framework. In addition to the lightweight design, this network accurately detects targets by leveraging more precise high-level and low-level feature representations obtained from GhostHGNetv2, which enhances feature-extraction capabilities. To handle the issue of complex environments, the backbone employs GhostHGNetv2 to capture more accurate high-level and low-level feature representations, facilitating better distinction between background and targets. In addition, this network significantly reduces both network parameter size and computational complexity. To address the issue of varying target scales, the first layer of the feature fusion module introduces Multi-Scale Convolutional Attention (MSCA) to capture multi-scale contextual information and guide the feature fusion process. A new lightweight detection head, Shared Convolutional Detection Head (SCDH), is designed to enable the model to achieve higher accuracy while being lighter. To evaluate the performance of this algorithm, a dataset for object detection in this scenario was constructed. The experiment results indicate that compared to the original model, the parameter number of the improved model decreased by 37.8%, the GFLOPs decreased by 27.7%, and the average accuracy increased from 82.7% to 85.0%. This validates the accuracy and applicability of the proposed GMS-YOLO network.

Urban traffic tiny object detection via attention and multi-scale feature driven in UAV-vision

The unmanned aerial vehicle (UAV) city patrol is of great significance in ensuring the safety of residents’ lives and properties, as well as maintaining the normal operation of the city. However, the detection of UAV images faces challenges such as numerous small-scale objects, complex backgrounds, and high requirements for detection speed. In response to these issues, we introduce a Real-time Small Object Detection network in UAV-vision (RTS-Net), tailored for UAV patrols. Initially, we introduce a multiscale feature fusion module (MFFM) designed to augment the expressiveness of features across scales, thereby enhancing the detection of smaller objects. Subsequently, leveraging attention mechanisms, we present the coordinated attention detection module (CADM), which bolsters the detection model’s ability to accurately segregate objects from the background in expansive, complex scenarios. Lastly, a lightweight real-time feature extraction module (RFEM) is crafted to diminish model computational complexity and boost inference speed. On the UAV road patrol image dataset we constructed, our proposed method attains a detection accuracy of 89.9%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} mAP, breaking previous records. It surpasses all prevailing detection methods, particularly for small-scale objects. Simultaneously, it achieves an inference speed of 163.9 FPS. The experimental results show that RTS-Net can satisfy the accurate and efficient detection of ground objects by various different UAV platforms in different complex scenarios.

Radiogenomics based survival prediction of small-sample glioblastoma patients by multi-task DFFSP model.

In this paper, the multi-task dense-feature-fusion survival prediction (DFFSP) model is proposed to predict the three-year survival for glioblastoma (GBM) patients based on radiogenomics data. The contrast-enhanced T1-weighted (T1w) image, T2-weighted (T2w) image and copy number variation (CNV) is used as the input of the three branches of the DFFSP model. This model uses two image extraction modules consisting of residual blocks and one dense feature fusion module to make multi-scale fusion of T1w and T2w image features as backbone. Also, a gene feature extraction module is used to adaptively weight CNV fragments. Besides, a transfer learning module is introduced to solve the small sample problem and an image reconstruction module is adopted to make the model anatomy-aware under a multi-task framework. 256 sample pairs (T1w and corresponding T2w MRI slices) and 187 CNVs of 74 patients were used. The experimental results show that the proposed model can predict the three-year survival of GBM patients with the accuracy of 89.1 %, which is improved by 3.2 and 4.7 % compared with the model without genes and the model using last fusion strategy, respectively. This model could also classify the patients into high-risk and low-risk groups, which will effectively assist doctors in diagnosing GBM patients.

Frequency distribution-aware network based on discrete cosine transformation (DCT) for remote sensing image super resolution.

Single-image super-resolution technology based on deep learning is widely used in remote sensing. The non-local feature reflects the correlation information between different regions. Most neural networks extract various non-local information of images in the spatial domain but ignore the similarity characteristics of frequency distribution, which limits the performance of the algorithm. To solve this problem, we propose a frequency distribution aware network based on discrete cosine transformation for remote sensing image super-resolution. This network first proposes a frequency-aware module. This module can effectively extract the similarity characteristics of the frequency distribution between different regions by rearranging the frequency feature matrix of the image. A global frequency feature fusion module is also proposed. It can extract the non-local information of feature maps at different scales in the frequency domain with little computational cost. The experiments were on two commonly-used remote sensing datasets. The experimental results show that the proposed algorithm can effectively complete image reconstruction and performs better than some advanced super-resolution algorithms. The code is available at https://github.com/Liyszepc/FDANet.

EFS-Former: An Efficient Network for Fruit Tree Leaf Disease Segmentation and Severity Assessment

Fruit is a major source of vitamins, minerals, and dietary fiber in people’s daily lives. Leaf diseases caused by climate change and other factors have significantly reduced fruit production. Deep learning methods for segmenting leaf diseases can effectively mitigate this issue. However, challenges such as leaf folding, jaggedness, and light shading make edge feature extraction difficult, affecting segmentation accuracy. To address these problems, this paper proposes a method based on EFS-Former. The expanded local detail (ELD) module extends the model’s receptive field by expanding the convolution, better handling fine spots and effectively reducing information loss. H-attention reduces computational redundancy by superimposing multi-layer convolutions, significantly improving feature filtering. The parallel fusion architecture effectively utilizes the different feature extraction intervals of the convolutional neural network (CNN) and Transformer encoders, achieving comprehensive feature extraction and effectively fusing detailed and semantic information in the channel and spatial dimensions within the feature fusion module (FFM). Experiments show that, compared to DeepLabV3+, this method achieves 10.78%, 9.51%, 0.72%, and 8.00% higher scores for mean intersection over union (mIoU), mean pixel accuracy (mPA), accuracy (Acc), and F_score, respectively, while having 1.78 M fewer total parameters and 0.32 G lower floating point operations per second (FLOPS). Additionally, it effectively calculates the ratio of leaf area occupied by spots. This method is also effective in calculating the disease period by analyzing the ratio of leaf area occupied by diseased spots. The method’s overall performance is evaluated using mIoU, mPA, Acc, and F_score metrics, achieving 88.60%, 93.49%, 98.60%, and 95.90%, respectively. In summary, this study offers an efficient and accurate method for fruit tree leaf spot segmentation, providing a solid foundation for the precise analysis of fruit tree leaves and spots, and supporting smart agriculture for precision pesticide spraying.

LSR-Det: A Lightweight Detector for Ship Detection in SAR Images Based on Oriented Bounding Box

Convolutional neural networks (CNNs) have significantly advanced in recent years in detecting arbitrary-oriented ships in synthetic aperture radar (SAR) images. However, challenges remain with multi-scale target detection and deployment on satellite-based platforms due to the extensive model parameters and high computational complexity. To address these issues, we propose a lightweight method for arbitrary-oriented ship detection in SAR images, named LSR-Det. Specifically, we introduce a lightweight backbone network based on contour guidance, which reduces the number of parameters while maintaining excellent feature extraction capability. Additionally, a lightweight adaptive feature pyramid network is designed to enhance the fusion capability of the ship features across different layers with a low computational cost by incorporating adaptive ship feature fusion modules between the feature layers. To efficiently utilize the fused features, a lightweight rotating detection head is designed, incorporating the idea of sharing the convolutional parameters, thereby improving the network’s ability to detect multi-scale ship targets. The experiments conducted on the SAR ship detection dataset (SSDD) and the rotating ship detection dataset (RSDD-SAR) demonstrate that LSR-Det achieves an average precision (AP50) of 98.5% and 97.2% with 3.21 G floating point operations (FLOPs) and 0.98 M parameters, respectively, outperforming the current popular SAR arbitrary-direction ship target detection methods.

MAFDE-DN4: Improved Few-shot plant disease classification method based on Deep Nearest Neighbor Neural Network

Deep learning-based methods for accurately identifying plant diseases can be effective in improving crop yields. However, the effectiveness of these methods heavily relies on the availability of large-scale manually labeled datasets, which present technical and economic challenges. Few-shot learning methods can be generalized to new categories with a small number of samples, which is very promising in the field of plant disease recognition despite the limited sample size. However, due to the complexity of real-life scenarios, distribution of disease leaves, and significant intra-class variability and inter-class similarity resulting from crop species and disease species, existing methods perform poorly in the field of plant disease recognition. To address the above problems, this paper proposes an improved multi-scale attention fusion with discriminative enhancement deep nearest neighbor neural network MAFDE-DN4 based on DN4. Our approach makes three main contributions. First, we have designed a bidirectional weighted feature fusion module (BWFM) to enhance the aggregation of fine-grained features and enhance the network’s representation of complex disease images. Second, to tackle the issue of sparse feature descriptors being vulnerable to irrelevant noise in small sample conditions, a episodic attention module (EA) has been developed to produce scene category-relevant attention maps. This effectively mitigates the influence of irrelevant background information. Finally, we introduce additional spacing between category margins to enhance the original softmax loss function, amplify the inter-class differences to reduce the intra-class distances, and add L2 regularization constraint terms to stabilize the training process. To simulate different real-world scenarios, we set up different dataset settings. Under the 1-shot task and the 5-shot task, our method achieves 57.5% and 81.41% accuracy under the within-domain strategy and 36.54% and 51.23% accuracy under the cross-domain strategy. The experimental results show that our method outperforms existing related work in the field of plant disease recognition, whether it is a dataset with a single background or a field dataset with a complex background in a real scenario. MAFDE-DN4 based on Few-shot learning requires substantially less data on new categories of plant diseases.

A lightweight color and geometry feature extraction and fusion module for end-to-end 6D pose estimation

Although advancements in red–green–blue-depth (RGB-D)-based six degree-of-freedom (6D) pose estimation methods, severe occlusion remains challenging. Addressing this issue, we propose a novel feature fusion module that can efficiently leverage the color and geometry information in RGB-D images. Unlike prior fusion methods, our method employs a two-stage fusion process. Initially, we extract color features from RGB images and integrate them into a point cloud. Subsequently, an anisotropic separable set abstraction network-like network is utilized to process the fused point cloud, extracting both local and global features, which are then combined to generate the final fusion features. Furthermore, we introduce a lightweight color feature extraction network to reduce model complexity. Extensive experiments conducted on the LineMOD, Occlusion LineMOD, and YCB-Video datasets conclusively demonstrate that our method significantly enhances prediction accuracy, reduces training time, and exhibits robustness to occlusion. Further experiments show that our model is significantly smaller than the latest popular 6D pose estimation models, which indicates that our model is easier to deploy on mobile platforms.

SSFPN: Selective Spatial Feature Pyramid Network for Remote Sensing Object Detection

Abstract Object detection, particularly for oriented small objects, benefits greatly from the generation of multi-scale feature maps by the feature fusion module. Despite the success of series of Feature Pyramid Network models in general object detection, their application in remote sensing object detection has received limited attention. Small oriented objects require contextual information for detection, while various types of objects need different long-range context. In this paper, we propose an intuitive and simple fusion module to be added in Feature Pyramid Network called Selective Spatial Feature Pyramid Network (SSFPN) to address these challenges. SSFPN dynamically adjusts the spatial receptive field across multi-scale feature maps, enhancing the modeling of contextual variations among different objects in remote sensing scenarios. In extensive experiments, SSFPN has achieved competitive results, i.e., an improvement of 0.34% after adding SSFPN to the Oriented RCNN model. The codes are available at https://github.com/Atlantisming/SSFPN.

GLFuse: A Global and Local Four-Branch Feature Extraction Network for Infrared and Visible Image Fusion

Infrared and visible image fusion integrates complementary information from different modalities into a single image, providing sufficient imaging information for scene interpretation and downstream target recognition tasks. However, existing fusion methods often focus only on highlighting salient targets or preserving scene details, failing to effectively combine entire features from different modalities during the fusion process, resulting in underutilized features and poor overall fusion effects. To address these challenges, a global and local four-branch feature extraction image fusion network (GLFuse) is proposed. On one hand, the Super Token Transformer (STT) block, which is capable of rapidly sampling and predicting super tokens, is utilized to capture global features in the scene. On the other hand, a Detail Extraction Block (DEB) is developed to extract local features in the scene. Additionally, two feature fusion modules, namely the Attention-based Feature Selection Fusion Module (ASFM) and the Dual Attention Fusion Module (DAFM), are designed to facilitate selective fusion of features from different modalities. Of more importance, the various perceptual information of feature maps learned from different modality images at the different layers of a network is investigated to design a perceptual loss function to better restore scene detail information and highlight salient targets by treating the perceptual information separately. Extensive experiments confirm that GLFuse exhibits excellent performance in both subjective and objective evaluations. It deserves note that GLFuse effectively improves downstream target detection performance on a unified benchmark.

A Multi-Branch Feature Extraction Residual Network for Lightweight Image Super-Resolution

Single-image super-resolution (SISR) seeks to elucidate the mapping relationships between low-resolution and high-resolution images. However, high-performance network models often entail a significant number of parameters and computations, presenting limitations in practical applications. Therefore, prioritizing a light weight and efficiency becomes crucial when applying image super-resolution (SR) to real-world scenarios. We propose a straightforward and efficient method, the Multi-Branch Feature Extraction Residual Network (MFERN), to tackle lightweight image SR through the fusion of multi-information self-calibration and multi-attention information. Specifically, we have devised a Multi-Branch Residual Feature Fusion Module (MRFFM) that leverages a multi-branch residual structure to succinctly and effectively fuse multiple pieces of information. Within the MRFFM, we have designed the Multi-Scale Attention Feature Fusion Block (MAFFB) to adeptly extract features via convolution and self-calibration attention operations. Furthermore, we introduce a Dual Feature Calibration Block (DFCB) to dynamically fuse feature information using dynamic weight values derived from the upper and lower branches. Additionally, to overcome the limitation of convolution in solely extracting local information, we incorporate a Transformer module to effectively integrate global information. The experimental results demonstrate that our MFERN exhibits outstanding performance in terms of model parameters and overall performance.

Apple recognition in complex environments based on FC-DETR

This study aims to address the challenges of apple recognition in complex environments by proposing a solution based on the FC-DETR model. Apple cultivation is an important part of agriculture in Xinjiang, but the increasing shortage of labor has driven the demand for automated harvesting technologies. Therefore, research and development of apple recognition technology in complex environments have become crucial. While CNN architecture models can accurately identify apples in conventional settings, their inherent receptive field limitations prevent them from fully capturing global features in noisy and complex environments, making it difficult to achieve the accuracy and robustness required for practical applications. To solve this problem, this study proposes a real-time object detection model, FC-DETR. This model incorporates the innovative FEMA-BasicBlock residual module, the CAFM cross-scale adaptive feature fusion module, and the novel Inner-WIoU loss function, thereby enhancing feature processing, multi-scale feature selection and integration, and detection accuracy. Experimental results show that the FC-DETR model achieves an apple recognition accuracy of 87 % and a recall rate of 82 % in complex backgrounds while maintaining a lightweight design. This study not only makes significant advances in automated apple harvesting technology but also contributes to improving the efficiency and sustainability of the apple industry in Xinjiang and globally.

Three-Dimensional Gravity Inversion Based on Attention Feature Fusion.

Three-dimensional gravity inversion is a process of obtaining the location, shape, and physical property parameters of underground anomaly sources using gravity anomaly data observed on the surface. In recent years, with the rapid development of data-driven methods, the application of deep learning (DL) to 3D gravity inversion has also attracted wide attention and achieved certain results. In this paper, based on the U-Net network, a three-dimensional gravity inversion method using an attention feature fusion mechanism is proposed. Using U-Net as the basic framework, the coarse-grained semantic features and fine-grained semantic features in the encoder and decoder are connected by long hops, and the global and local semantic features are aggregated through the attention feature fusion module, which avoids feature loss in the network training process. Compared with the inversion results of the U-Net network, the proposed method has a higher vertical resolution and effectively alleviates the influence of the skin effect on three-dimensional gravity inversion. Ablation experiments show that the attention feature fusion module is the key to improving the vertical resolution and prediction accuracy of inversion results. Noise experiments show that the inversion network in this study has a strong anti-noise ability and good generalization performance. The experimental results of the inversion network used in the prediction of the SAN Nicolas deposit in Mexico show that the inversion network can clearly predict the basic location and general shape of the sulfur deposit, and the results are in good agreement with the known geological data.

VPA-Net: A visual perception assistance network for 3d lidar semantic segmentation

The semantic segmentation of 3D point clouds holds paramount importance in visual perception tasks of automatic driving, including obstacle avoidance, decision control, path planning, and map construction, etc. Multi-sensor fusion is a rational and pivotal technique for implementing LiDAR semantic segmentation. However, effectively fusing and utilizing multi-source data remains a challenging task. In this work, we introduce a novel visual perception-assisted point cloud segmentation network, termed VPA-Net. This network architecture employs a dual-branch design to effectively combine spatial information from point clouds and visual cues from images, thereby bolstering the performance of 3D LiDAR semantic segmentation. More specifically, the dual-branch network structure processes and fuses multi-modal data from both point clouds and RGB images. Subsequently, the intermediate features from the two branches are merged via the proposed attention-based feature fusion module. Furthermore, to address the challenge of precise boundary prediction in large-scale point cloud scene segmentation, we introduce a refinement module based on 3D sparse convolution to enhance the spatial information of the LiDAR point clouds. The effectiveness of our method is validated on SemanticKITTI and a more challenging 3D semantic segmentation dataset. Experimental results underscore significant improvement on SemanticKITTI, with our approach surpassing the state-of-the-art method, achieving a 7.1% higher mIoU than SalsaNext.

MLFF-Net: a multi-model late feature fusion network for skin disease classification

Early diagnosis is paramount to preventing skin diseases and reducing mortality, given their global prevalence. Visual detection by experts using dermoscopy images has become the gold standard for detecting skin cancer. However, a significant challenge in skin cancer detection and classification lies in the similarity of appearance among skin disease lesions and the complexity of dermoscopic images. In response, we developed multi-model late feature fusion network (MLFF-Net), a multi-model late feature fusion network tailored for skin disease detection. Our approach begins with image pre-processing techniques to enhance image quality. We then employ a two-stream network comprising an enhanced densely linked network (DenseNet-121) and a vision transformer (ViTb16). We leverage shallow and deep feature fusion, late fusion, and an attention module to enhance the model’s feature extraction efficiency. The subsequent feature fusion module constructs multi-receptive fields to capture disease information across various scales and uses generalized mean pooling (GeM) pooling to reduce the spatial dimensions of lesion characteristics. Finally, we implement and test our skin lesion categorization model, demonstrating its effectiveness. Despite the combination, convolutional neural network (CNN) outperforms ViT approaches, with our model enhancing the accuracy of the best model by 6.1%.