Deep Fusion Network Research Articles

Multi-Source Data Fusion for Vehicle Maintenance Project Prediction

Ensuring road safety is heavily reliant on the effective maintenance of vehicles. Accurate predictions of maintenance requirements can substantially reduce ownership costs for vehicle owners. Consequently, this field has attracted increasing attention from researchers in recent years. However, existing studies primarily focus on predicting a limited number of maintenance needs, predominantly based solely on vehicle mileage and driving time. This approach often falls short, as it does not comprehensively monitor the overall health condition of vehicles, thus posing potential safety risks. To address this issue, we propose a deep fusion network model that utilizes multi-source data, including vehicle maintenance record data and vehicle base information data, to provide comprehensive predictions for vehicle maintenance projects. To capture the relationships among various maintenance projects, we create a correlation representation using the maintenance project co-occurrence matrix. Furthermore, building on the correlation representation, we propose a deep fusion network that employs the attention mechanism to efficiently merge vehicle mileage and vehicle base information. Experiments conducted on real data demonstrate the superior performance of our proposed model relative to competitive baseline models in predicting vehicle maintenance projects.

Recognition of vehicle license plates in highway scenes with deep fusion network and connectionist temporal classification

AbstractLicense plate recognition is crucial in Intelligent Transportation Systems (ITS) for vehicle management, traffic monitoring, and security inspection. In highway scenarios, this task faces challenges such as diversity, blurriness, occlusion, and illumination variation of license plates. This article explores Recurrent Neural Networks based on Connectionist Temporal Classification (RNN‐CTC) for license plate recognition in challenging highway conditions. Four neural network models: ResNet50, ResNeXt, InceptionV3, and SENet, all combined with RNN‐CTC are comparatively evaluated. Furthermore, a novel architecture named ResNet50 Deep Fusion Network using Connectionist Temporal Classification (ResNet50‐DFN‐CTC) is proposed. Comparative and ablation experiments are conducted using the Highway License Plate Dataset of Southeast University (HLPD‐SU). Results demonstrate the superior performance of ResNet50‐DFN‐CTC in challenging highway conditions, achieving 93.158% accuracy with a processing time of 7.91 ms, outperforming other tested models. This research contributes to advancing license plate recognition technology for real‐world highway applications under adverse conditions.

Pyramid Deep Fusion Network for Two-Hand Reconstruction From RGB-D Images

Pyramid Deep Fusion Network for Two-Hand Reconstruction From RGB-D Images

SDRnet: A Deep Fusion Network for ISAR Ship Target Recognition Based on Feature Separation and Weighted Decision

Existing methods for inverse synthetic aperture radar (ISAR) target recognition typically rely on a single high-resolution radar signal type, such as ISAR images or high-resolution range profiles (HRRPs). However, ISAR images and HRRP data offer representations of targets across different aspects, each containing valuable information crucial for radar target recognition. Moreover, the process of generating ISAR images inherently facilitates the acquisition of HRRP data, ensuring timely data collection. Therefore, to fully leverage the different information from both HRRP data and ISAR images and enhance ISAR ship target recognition performance, we propose a novel deep fusion network named the Separation-Decision Recognition network (SDRnet). First, our approach employs a convolutional neural network (CNN) to extract initial feature vectors from ISAR images and HRRP data. Subsequently, a feature separation module is employed to derive a more robust target representation. Finally, we introduce a weighted decision module to enhance overall predictive performance. We validate our method using simulated and measured data containing ten categories of ship targets. The experimental results confirm the effectiveness of our approach in improving ISAR ship target recognition.

The application of Hyperpixel segmentation algorithm in Chinese painting

Chinese painting, as an important component of traditional Chinese art, not only expresses Chinese culture and history, but also contains rich artistic connotations. Therefore, extracting and analyzing the main objects of Chinese painting is of great significance. The research introduces a fusion deep network based super pixel segmentation algorithm for image segmentation, and utilizes user marked segmentation target objects and backgrounds. The maximum similarity region merging idea is adopted to extract the main subject objects in Chinese painting, thereby achieving a deep understanding of the work. The results show that the boundary recall and segmentation accuracy of the Chinese painting subject object segmentation algorithm based on super pixels on the BuptPainting dataset are as high as 94.38% and 98.06%, respectively, with an undersegmentation error rate of only 7.69%. Meanwhile, the average accuracy value, optimal data scale value, and optimal image scale value of this method on the BuptPainting dataset are as high as 0.657, 0.541, and 0.529, respectively. The method proposed by the research institute has significant advantages in segmenting the main objects of Chinese painting, providing new ideas and methods for image analysis and processing of modern Chinese painting, and helping to further explore the artistic charm and cultural connotations of Chinese painting.

NeXtFusion: Attention-Based Camera-Radar Fusion Network for Improved Three-Dimensional Object Detection and Tracking

Accurate perception is crucial for autonomous vehicles (AVs) to navigate safely, especially in adverse weather and lighting conditions where single-sensor networks (e.g., cameras or radar) struggle with reduced maneuverability and unrecognizable targets. Deep Camera-Radar fusion neural networks offer a promising solution for reliable AV perception under any weather and lighting conditions. Cameras provide rich semantic information, while radars act like an X-ray vision, piercing through fog and darkness. This work proposes a novel, efficient Camera-Radar fusion network called NeXtFusion for robust AV perception with an improvement in object detection accuracy and tracking. Our proposed approach of utilizing an attention module enhances crucial feature representation for object detection while minimizing information loss from multi-modal data. Extensive experiments on the challenging nuScenes dataset demonstrate NeXtFusion’s superior performance in detecting small and distant objects compared to other methods. Notably, NeXtFusion achieves the highest mAP score (0.473) on the nuScenes validation set, outperforming competitors like OFT (35.1% improvement) and MonoDIS (9.5% improvement). Additionally, NeXtFusion demonstrates strong performance in other metrics like mATE (0.449) and mAOE (0.534), highlighting its overall effectiveness in 3D object detection. Furthermore, visualizations of nuScenes data processed by NeXtFusion further demonstrate its capability to handle diverse real-world scenarios. These results suggest that NeXtFusion is a promising deep fusion network for improving AV perception and safety for autonomous driving.

Multi-feature Fusion Deep Network for Skin Disease Diagnosis

Multi-feature Fusion Deep Network for Skin Disease Diagnosis

Deep Learning Combined with Radiologist's Intervention Achieves Accurate Segmentation of Hepatocellular Carcinoma in Dual-Phase Magnetic Resonance Images.

Segmentation of hepatocellular carcinoma (HCC) is crucial; however, manual segmentation is subjective and time-consuming. Accurate and automatic lesion contouring for HCC is desirable in clinical practice. In response to this need, our study introduced a segmentation approach for HCC combining deep convolutional neural networks (DCNNs) and radiologist intervention in magnetic resonance imaging (MRI). We sought to design a segmentation method with a deep learning method that automatically segments using manual location information for moderately experienced radiologists. In addition, we verified the viability of this method to assist radiologists in accurate and fast lesion segmentation. In our study, we developed a semiautomatic approach for segmenting HCC using DCNN in conjunction with radiologist intervention in dual-phase gadolinium-ethoxybenzyl-diethylenetriamine penta-acetic acid- (Gd-EOB-DTPA-) enhanced MRI. We developed a DCNN and deep fusion network (DFN) trained on full-size images, namely, DCNN-F and DFN-F. Furthermore, DFN was applied to the image blocks containing tumor lesions that were roughly contoured by a radiologist with 10 years of experience in abdominal MRI, and this method was named DFN-R. Another radiologist with five years of experience (moderate experience) performed tumor lesion contouring for comparison with our proposed methods. The ground truth image was contoured by an experienced radiologist and reviewed by an independent experienced radiologist. The mean DSC of DCNN-F, DFN-F, and DFN-R was 0.69 ± 0.20 (median, 0.72), 0.74 ± 0.21 (median, 0.77), and 0.83 ± 0.13 (median, 0.88), respectively. The mean DSC of the segmentation by the radiologist with moderate experience was 0.79 ± 0.11 (median, 0.83), which was lower than the performance of DFN-R. Deep learning using dual-phase MRI shows great potential for HCC lesion segmentation. The radiologist-aided semiautomated method (DFN-R) achieved improved performance compared to manual contouring by the radiologist with moderate experience, although the difference was not statistically significant.

DFN: A deep fusion network for flexible single and multi-modal action recognition

Multi-modal action recognition methods can be generally classified into two categories: (1) fusing multi-modal features with simple concatenation or fusing the classification scores of individual modalities without considering the interaction among the multi-modalities; (2) using one of the modalities as privileged information in training to boost the recognition on the other modalities in inference. The former approach usually is not able to deal with the cases where one of the modalities is missing. In the latter, the trained classifier does not work on the privileged modality. To address these shortcomings, this paper presents a novel end-to-end trainable deep fusion network (DFN) that is able to improve the performance not only in the cases where all modalities are available and also in the cases where there is a missing modality. The DFN is simple yet effective with the capability of retrieving an estimation of one modality by using another modality through a Multilayer Perceptron (MLP). In order to better preserve structure information, the DFN first maps the individual modality features to a high dimensional Kronecker-product space and subsequently learns a low-dimensional discriminative space for classification. The effectiveness of the proposed DFN has been verified on three benchmark datasets: the large NTU RGB+D, UTD-MHAD, and SYSU-3D datasets and it has achieved state-of-the-art results.

Agile Frequency RCS-Based Deep Fusion Network for Ship and Corner Reflector Identification

Agile Frequency RCS-Based Deep Fusion Network for Ship and Corner Reflector Identification

A time series and deep fusion framework for rotating machinery fault diagnosis

For shafting rotation equipment, in the fault diagnosis based on vibration analysis, the sampling signal is a waveform of a short moment, which is quasi-static information at the corresponding moment. A single quasi-static piece of information does not necessarily contain obvious fault information, however, in a longer period much larger than the sampling period, the time series information composed of multiple quasi-static combinations may well-characterize the fault. Traditional deep learning algorithms often focus on fault features in quasi-static information, ignoring time series features, which results in low diagnostic accuracy. To solve the above problems, this paper firstly presents a fault diagnosis framework of time series and deep fusion network (TDFN). Then, a method to extract time series information and quasi-static information based on expert experience knowledge is proposed, and finally studies the method of using this framework to train the network and fuse the two types of features. Different from existing deep learning methods, TDFN includes time series blocks, depth blocks and fusion blocks. This paper uses laboratory data and industrial rotor data for verification, the results show that the overall accuracy of TDFN reached 93.8% and it does not experience a drop in accuracy after fusion. The diagnosis accuracy is higher than traditional networks, providing guiding suggestions for the management and operation of on-site industrial equipment.

An Intelligent Facial Expression Recognition System Using a Hybrid Deep Convolutional Neural Network for Multimedia Applications

Recognizing facial expressions plays a crucial role in various multimedia applications, such as human–computer interactions and the functioning of autonomous vehicles. This paper introduces a hybrid feature extraction network model to bolster the discriminative capacity of emotional features for multimedia applications. The proposed model comprises a convolutional neural network (CNN) and deep belief network (DBN) series. First, a spatial CNN network processed static facial images, followed by a temporal CNN network. The CNNs were fine-tuned based on facial expression recognition (FER) datasets. A deep belief network (DBN) model was then applied to integrate the segment-level spatial and temporal features. Deep fusion networks were jointly used to learn spatiotemporal features for discrimination purposes. Due to its generalization capabilities, we used a multi-class support vector machine classifier to classify the seven basic emotions in the proposed model. The proposed model exhibited 98.14% recognition performance for the JaFFE database, 95.29% for the KDEF database, and 98.86% for the RaFD database. It is shown that the proposed method is effective for all three databases, compared with the previous schemes for JAFFE, KDEF, and RaFD databases.

Skip and chain connected deep fusion network for lung cancer screening

Skip and chain connected deep fusion network for lung cancer screening

Graph-Aware Deep Fusion Networks for Online Spam Review Detection

Product reviews on e-commerce platforms play a critical role in shaping users’ purchasing decisions. Unfortunately, online reviews sometimes can be intentionally misleading to manipulate the ecosystem. To date, existing methods to automatically detect “spam reviews” either focus on sophisticated feature engineering with traditional classification models or rely on tuning neural networks with aggregated features. In this article, we develop a novel graph-based model, namely, graph-aware deep fusion networks (GDFNs) that use information from relevant metadata (review text, features of users, and items) and relational data (network) to capture the semantic information from their complex heterogeneous interactions via graph convolutional networks (GCNs). Besides, GDFN also uses a novel fusion technique to synthesize low-and high-order interactions with propagated information across multiple review-related subgraphs. Extensive experiments on publicly available datasets show that our proposed model is effective and outperforms several strong state-of-the-art baselines.

Multi-level spatial-temporal and attentional information deep fusion network for retinal vessel segmentation

Objective. Automatic segmentation of fundus vessels has the potential to enhance the judgment ability of intelligent disease diagnosis systems. Even though various methods have been proposed, it is still a demanding task to accurately segment the fundus vessels. The purpose of our study is to develop a robust and effective method to segment the vessels in human color retinal fundus images. Approach. We present a novel multi-level spatial-temporal and attentional information deep fusion network for the segmentation of retinal vessels, called MSAFNet, which enhances segmentation performance and robustness. Our method utilizes the multi-level spatial-temporal encoding module to obtain spatial-temporal information and the Self-Attention module to capture feature correlations in different levels of our network. Based on the encoder and decoder structure, we combine these features to get the final segmentation results. Main results. Through abundant experiments on four public datasets, our method achieves preferable performance compared with other SOTA retinal vessel segmentation methods. Our Accuracy and Area Under Curve achieve the highest scores of 96.96%, 96.57%, 96.48% and 98.78%, 98.54%, 98.27% on DRIVE, CHASE_DB1, and HRF datasets. Our Specificity achieves the highest score of 98.58% and 99.08% on DRIVE and STARE datasets. Significance. The experimental results demonstrate that our method has strong learning and representation capabilities and can accurately detect retinal blood vessels, thereby serving as a potential tool for assisting in diagnosis.

A dual track deep fusion network for citrus disease classification using group shuffle depthwise feature pyramid and Swin transformer

Citrus fruits are globally significant crops, valued for their unique taste, nutritional value, and versatility in culinary applications. However, these crops are vulnerable to diseases that can substantially reduce their yield and quality, leading to significant financial losses. It is crucial to efficiently identify and manage diseases affecting citrus fruits to ensure food security and support sustainable agriculture. Traditionally, identifying symptoms associated with these diseases requires expert scientific and observational skills. With the advent of deep learning methods, it has become possible to automatically identify disease patterns from images of affected plants. In this research, a novel dual branch deep-learning network was proposed for the classification of citrus plant diseases. The Group Shuffle Depthwise Feature Pyramid (GSDFP), which constitutes the first branch, utilizes convolution blocks to extract local features. The second branch leverages the Swin transformer to integrate global contextual awareness into the extracted features and facilitates the learning process through long-term dependencies. Subsequently, the features from the two branches are fused and passed through a shuffle attention module, effectively capturing contextual relationships between them. The effectiveness of the proposed approach was validated on two benchmark datasets, namely the Citrus Plant Dataset and the Citrus Disease Image Gallery Dataset. The proposed network obtained a classification accuracy of 98.19%.

SD2SDFNet: A novel deep fusion network based on statistical denoising and dual-dimension self-attention model for power transformer hybrid-space fault prognosis

Intelligent fault prognosis plays an essential role in ensuring the stability and safety of power systems. However, the prediction of power transformer faults encounters significant challenges, including ultra-high noise interference, imbalanced samples, and difficulties in extracting sensitive fault features. To address these problems, this paper proposes SD2SDFNet, a novel deep fusion network that utilizes statistical denoising and a dual-dimension self-attention network. The proposed approach incorporates a statistical denoising approach called successive variational mode decomposition and Cramer Von Misses (SVMD-CVM). This technique aims to decompose signals into intrinsic mode functions (IMFs) and eliminate noise from each corresponding mode. By employing SVMD-CVM, the scheme effectively mitigates the impact of noise on fault prediction. Furthermore, a dual-dimension self-attention model is developed to selectively capture salient features from different IMF and time step dimensions. This adaptive feature selection process enhances the network's ability to focus on relevant information. To integrate the extracted features from both dimensions, a deep fusion network is adopted. This fusion process combines the weighted features, resulting in a more comprehensive set of features that facilitates accurate fault prediction. Experimental evaluations conducted on hybrid-space datasets demonstrate that SD2SDFNet achieves outstanding performance in predicting inter-turn faults. The network exhibited a Mean Squared Error (MSE) of 3.25E-05 and a Relative Absolute Error (RAE) of 1.58% for sensor 1, surpassing state-of-the-art prediction methods.

Deep learning-based postoperative visual acuity prediction in idiopathic epiretinal membrane

BackgroundTo develop a deep learning (DL) model based on preoperative optical coherence tomography (OCT) training to automatically predict the 6-month postoperative visual outcomes in patients with idiopathic epiretinal membrane (iERM).MethodsIn this retrospective cohort study, a total of 442 eyes (5304 images in total) were enrolled for the development of the DL and multimodal deep fusion network (MDFN) models. All eyes were randomized into a training dataset with 265 eyes (60.0%), a validation dataset with 89 eyes (20.1%), and an internal testing dataset with the remaining 88 eyes (19.9%). The input variables for prediction consisted of macular OCT images and diverse clinical data. Inception-Resnet-v2 network was utilized to estimate the 6-month postoperative best-corrected visual acuity (BCVA). Concurrently, a regression model was developed using the clinical data and OCT parameters in the training data set for predicting postoperative BCVA. The reliability of the models was subsequently evaluated using the testing dataset.ResultsThe prediction DL algorithm exhibited a mean absolute error (MAE) of 0.070 logMAR and root mean square error (RMSE) of 0.11 logMAR in the testing dataset. The DL model demonstrated a robust promising performance with R2 = 0.80, notably superior to R2 = 0.49 of the regression model. The percentages of BCVA prediction errors within ± 0.20 logMAR amounted to 94.32% in the testing dataset.ConclusionsThe OCT-based DL model demonstrated sensitivity and accuracy in predicting postoperative BCVA in iERM patients. This innovative DL model exhibits substantial potential for integration into surgical planning protocols.

Research on equipment safety fault diagnosis method based on multi‐sensor fusion deep network in mechatronics equipment environment

AbstractThe safety performance and stability of mechatronics equipment play an important role in modern industrial production. However, using a single‐sensor signal cannot effectively ensure robustness in complex scenarios. Moreover, the efficient collection and real‐time transmission of information can improve the real‐time performance of fault detection. To this end, this article proposes a novel deep network based on multi‐sensor information fusion for mechatronics equipment fault detection. Firstly, three sensors are used to collect status information. We use the CC2420 model to transmit the collected signals to the server for storage and analysis. Secondly, we designed a multi‐sensor information fusion deep network. To better model local and global features, we introduced convolutional operations and the multi‐head attention mechanism to form the backbone of the network. The results on the self‐built dataset indicate that the proposed model fully utilizes the advantages of multimodal information and deep networks to achieve the optimal fault detection results.

Deep Fusion Network Based Sparse View CT Reconstructions for Clinical Diagnostic Scanners.

Sparse view CT scan has the advantage of reducing radiation exposure and scanning time in clinical diagnosis. However, the limited number of x-ray projections can make the reconstruction problem ill posed and result in image artifacts. To tackle the problem, we propose a novel model-based deep fusion network(DFN) satisfying the clinical set-up. It extracts fused features encoded from both the sinogram and the preliminary reconstructed image generated by filtered back projection (FBP) to improve the quality of reconstruction. The preliminary reconstructed image endows fused features with prior knowledge that facilitate the convergence of neural network to high-quality reconstruction images. We design a custom loss for training that enforces the network to learn both the pixel value and the integrity of the tissue structure. A synthetic sparse view breast CT dataset from American Association of Physicists in Medicine(AAPM) is used for training, validation and testing. The qualitative and quantitative evaluations show that the DFN reconstruction algorithm significantly improves in balancing between the image quality and reconstruction speed, hence enables fast and high quality CT reconstruction despite the sparse view limitations.