Deep Fusion Research Articles

Dynamic category-sensitive hypergraph inferring and homo-heterogeneous neighbor feature learning for drug-related microbe prediction.

The microbes in human body play a crucial role in influencing the functions of drugs, as they can regulate the activities and toxicities of drugs. Most recent methods for predicting drug-microbe associations are based on graph learning. However, the relationships among multiple drugs and microbes are complex, diverse, and heterogeneous. Existing methods often fail to fully model the relationships. In addition, the attributes of drug-microbe pairs exhibit long-distance spatial correlations, which previous methods have not integrated effectively. We propose a new prediction method named DHDMP which is designed to encode the relationships among multiple drugs and microbes and integrate the attributes of various neighbor nodes along with the pairwise long-distance correlations. First, we construct a hypergraph with dynamic topology, where each hyperedge represents a specific relationship among multiple drug nodes and microbe nodes. Considering the heterogeneity of node attributes across different categories, we developed a node category-sensitive hypergraph convolution network to encode these diverse relationships. Second, we construct homogeneous graphs for drugs and microbes respectively, as well as drug-microbe heterogeneous graph, facilitating the integration of features from both homogeneous and heterogeneous neighbors of each target node. Third, we introduce a graph convolutional network with cross-graph feature propagation ability to transfer node features from homogeneous to heterogeneous graphs for enhanced neighbor feature representation learning. The propagation strategy aids in the deep fusion of features from both types of neighbors. Finally, we design spatial cross-attention to encode the attributes of drug-microbe pairs, revealing long-distance correlations among multiple pairwise attribute patches. The comprehensive comparison experiments showed our method outperformed state-of-the-art methods for drug-microbe association prediction. The ablation studies demonstrated the effectiveness of node category-sensitive hypergraph convolution network, graph convolutional network with cross-graph feature propagation, and spatial cross-attention. Case studies on three drugs further showed DHDMP's potential application in discovering the reliable candidate microbes for the interested drugs. Source codes and supplementary materials are available at https://github.com/pingxuan-hlju/DHDMP.

Classification Method of Internal Defects in Metal Materials Based on Ultrasonic Point Cloud Data

Abstract The commonly used methods for analysing defects in metal materials include metallography and scanning electron microscopy(SEM). However, these methods require grinding and polishing of the material surface, which can only obtain the characteristics of material surface defects, making it difficult to effectively detect and characterize internal defects in metal materials. Therefore, a deep neural network model based on ultrasonic point cloud data was proposed in this paper, called the Global and Local Feature Fusion Network (GLFFN), to classify the internal defects in metal materials. Firstly, an ultrasonic microscope was used to perform X-layer scanning on the metal materials, obtaining ultrasonic images at different depths, from which ultrasonic point cloud data was extracted. Based on this, the GLFFN model enhances the classification accuracy of internal defects in metal materials by deep fusion of global and local features of the point cloud. With casting billets as the detection object, which contain three typical internal defects: inclusions, shrinkage cavities, and cracks. The accuracy of the new model for defect classification reached 88.24%, which is superior to the traditional methods, proving the effectiveness of the new model on the classification of internal defects in metal materials.

Reconstruction of natural images from human fMRI using a three-stage multi-level deep fusion model

BackgroundImage reconstruction is a critical task in brain decoding research, primarily utilizing functional magnetic resonance imaging (fMRI) data. However, due to challenges such as limited samples in fMRI data, the quality of reconstruction results often remains poor. New methodWe proposed a three-stage multi-level deep fusion model (TS-ML-DFM). The model employed a three-stage training process, encompassing components such as image encoders, generators, discriminators, and fMRI encoders. In this method, we incorporated distinct supplementary features derived separately from depth images and original images. Additionally, the method integrated several components, including a random shift module, dual attention module, and multi-level feature fusion module. ResultsIn both qualitative and quantitative comparisons on the Horikawa17 and VanGerven10 datasets, our method exhibited excellent performance.Comparison with existing methods: For example, on the primary Horikawa17 dataset, our method was compared with other leading methods based on metrics the average hash value, histogram similarity, mutual information, structural similarity accuracy, AlexNet(2), AlexNet(5), and pairwise human perceptual similarity accuracy. Compared to the second-ranked results in each metric, the proposed method achieved improvements of 0.99 %, 3.62 %, 3.73 %, 2.45 %, 3.51 %, 0.62 %, and 1.03 %, respectively. In terms of the SwAV top-level semantic metric, a substantial improvement of 10.53 % was achieved compared to the second-ranked result in the pixel-level reconstruction methods. ConclusionsThe TS-ML-DFM method proposed in this study, when applied to decoding brain visual patterns using fMRI data, has outperformed previous algorithms, thereby facilitating further advancements in research within this field.

Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models

As speech is the most natural way for humans to express emotions, studies on Speech Emotion Recognition (SER) have been conducted in various ways However, there are some areas for improvement in previous SER studies: (1) while some studies have performed multi-label classification, almost none have specifically utilized Korean speech data; (2) most studies have not utilized multiple features in combination for emotion recognition. Therefore, this study proposes deep fusion models for multi-label emotion classification using Korean speech data and follows four steps: (1) preprocessing speech data labeled with Sadness, Happiness, Neutral, Anger, and Disgust; (2) applying data augmentation to address the data imbalance and extracting speech features, including the Log-mel spectrogram, Mel-Frequency Cepstral Coefficients (MFCCs), and Voice Quality Features; (3) constructing models using deep fusion architectures; and (4) validating the performance of the constructed models. The experimental results demonstrated that the proposed model, which utilizes the Log-mel spectrogram and MFCCs with a fusion of Vision-Transformer and 1D Convolutional Neural Network–Long Short-Term Memory, achieved the highest average binary accuracy of 71.2% for multi-label classification, outperforming other baseline models. Consequently, this study anticipates that the proposed model will find application based on Korean speech, specifically mental healthcare and smart service systems.

A deep fusion model for stock market prediction with news headlines and time series data

A deep fusion model for stock market prediction with news headlines and time series data

Semantic Segmentation Network Based on Adaptive Attention and Deep Fusion Utilizing a Multi-Scale Dilated Convolutional Pyramid.

Deep learning has recently made significant progress in semantic segmentation. However, the current methods face critical challenges. The segmentation process often lacks sufficient contextual information and attention mechanisms, low-level features lack semantic richness, and high-level features suffer from poor resolution. These limitations reduce the model's ability to accurately understand and process scene details, particularly in complex scenarios, leading to segmentation outputs that may have inaccuracies in boundary delineation, misclassification of regions, and poor handling of small or overlapping objects. To address these challenges, this paper proposes a Semantic Segmentation Network Based on Adaptive Attention and Deep Fusion with the Multi-Scale Dilated Convolutional Pyramid (SDAMNet). Specifically, the Dilated Convolutional Atrous Spatial Pyramid Pooling (DCASPP) module is developed to enhance contextual information in semantic segmentation. Additionally, a Semantic Channel Space Details Module (SCSDM) is devised to improve the extraction of significant features through multi-scale feature fusion and adaptive feature selection, enhancing the model's perceptual capability for key regions and optimizing semantic understanding and segmentation performance. Furthermore, a Semantic Features Fusion Module (SFFM) is constructed to address the semantic deficiency in low-level features and the low resolution in high-level features. The effectiveness of SDAMNet is demonstrated on two datasets, revealing significant improvements in Mean Intersection over Union (MIOU) by 2.89% and 2.13%, respectively, compared to the Deeplabv3+ network.

Enhancing physically-based flood forecasts through fusion of long short-term memory neural network with unscented Kalman filter

Accurate flood forecasts are vital for reservoir operation and flood prevention. The unscented Kalman filter (UKF) excels in improving physically-based models flood forecasting but depends on precise noise estimation, posing challenges in complex uncertainty quantification. In this study, we propose a novel approach, the deep fusion of a long short-term memory (LSTM) neural network and unscented Kalman filter (LSTM-UKF), to enhance flood forecasts by adaptively updating hydrological model states. The LSTM efficiently learns noise-related information from the estimation of Kalman Gain, obviating the need for intricate uncertainty recognition and approximation. For comparative analysis, the LSTM-UKF and UKF filters are constructed to correct Xinanjiang (XAJ) hydrological model states. Both filters utilize streamflow observations to update the XAJ model states and facilitate its flood forecasting performance. Comprehensive evaluations were conducted using 3-hourly meteor-hydrological sequences from the Jianxi and Tianyi basins in China, focusing on the accuracy, stability, and reliability of multi-step-ahead flood forecasts. Results indicate that the LSTM-UKF performs superior to the UKF, with maximal advancements in Nash-Sutcliffe efficiency coefficient (NSE) by 9.1%, maximal reduction in root mean square error (RMSE) by 18.7%, and maximal decrease in mean absolute error (MAE) by 22.6%. Additionally, the LSTM-UKF exhibits better robustness and stability in non-stationary flood events. The proposed LSTM-UKF mitigates the over-reliance on noise estimates, reducing systematic biases and error accumulation in state estimates and enhancing hydrological model generalizability in operational flood prevention platforms and systems.

ITFuse: An interactive transformer for infrared and visible image fusion

Infrared and visible image fusion (IVIF) has attracted increasing attention from the community because of its pleasing results in downstream applications. However, most existing deep fusion models are either feature–level fusion or image–level fusion, leading to information loss. In this paper, we propose an interactive transformer for IVIF, termed ITFuse. In contrast to previous algorithms, ITFuse consists of feature interactive modules (FIMs) and a feature reconstruction module (FRM) to alternatively extract and integrate important features. Specifically, to adequately exploit the common properties of different source images, we design a residual attention block (RAB) for mutual feature representation. To aggregate the distinct characteristics that existed in the corresponding input images, we leverage interactive attention (ITA) to incorporate the complementary information for comprehensive feature preservation and interaction. In addition, cross-modal attention (CMA) and transformer block (TRB) are presented to fully merge the capitalized features and construct long-range relationships. Furthermore, we devise a pixel loss and a structural loss to train the proposed deep fusion model in an unsupervised manner for excess performance amelioration. Massive experiments on popular databases illustrate that our ITFuse performs better than other representative state-of-the-art methods in terms of both qualitative and quantitative assessments. The source code of the proposed method is available at https://github.com/tthinking/ITFuse.

DeepDate: A deep fusion model based on whale optimization and artificial neural network for Arabian date classification.

As agricultural technology continues to develop, the scale of planting and production of date fruit is increasing, which brings higher yields. However, the increasing yields also put a lot of pressure on the classification step afterward. Image recognition based on deep learning algorithms can help to identify and classify the date fruit species, even in natural light. In this paper, a deep fusion model based on whale optimization and an artificial neural network for Arabian date classification is proposed. The dataset used in this study includes five classes of date fruit images (Barhi, Khalas, Meneifi, Naboot Saif, Sullaj). The process of designing each model can be divided into three phases. The first phase is feature extraction. The second phase is feature selection. The third phase is the training and testing phase. Finally, the best-performing model was selected and compared with the currently established models (Alexnet, Squeezenet, Googlenet, Resnet50). The experimental results show that, after trying different combinations of optimization algorithms and classifiers, the highest test accuracy achieved by DeepDate was 95.9%. It takes less time to achieve a balance between classification accuracy and time consumption. In addition, the performance of DeepDate is better than that of many deep transfer learning models such as Alexnet, Squeezenet, Googlenet, VGG-19, NasNet, and Inception-V3. The proposed DeepDate improves the accuracy and efficiency of classifying date fruits and achieves better results in classification metrics such as accuracy and F1. DeepDate provides a promising classification solution for date fruit classification with higher accuracy. To further advance the industry, it is recommended that stakeholders invest in technology transfer programs to bring advanced image recognition and AI tools to smaller producers, enhancing sustainability and productivity across the sector. Collaborations between agricultural technologists and growers could also foster more tailored solutions that address specific regional challenges in date fruit production.

SOH estimation of lithium-ion batteries based on multi-feature deep fusion and XGBoost

State of Health (SOH) is a crucial metric for battery management systems, and accurate estimation of battery SOH is essential for the underlying management and maintenance of batteries. Traditional data-driven approaches lack in-depth analysis of health features and fail to exploit the deep-level health information among these features. Therefore, this paper extracts six health features from both measurement and calculation perspectives based on actual measurable charging curve data, and conducts aging analysis of health features from a mechanistic perspective and Spearman correlation analysis from a quantitative perspective. Additionally, a strategy combining deep learning with ensemble learning is proposed. This strategy utilizes one-dimensional convolutional neural networks to deeply integrate multiple original features, extracting fusion features with deep-level health information. Subsequently, XGBoost is employed to learn the fusion features for estimating battery SOH. Finally, the effectiveness and superiority of the proposed method are verified on battery aging datasets from the University of Maryland and our laboratory. Experimental results demonstrate that the RMSE and MAE of the proposed method are both below 1 % and outperform other methods.

Monocular Depth and Ego-motion Estimation with Scale Based on Superpixel and Normal Constraints

3D perception in intelligent VR/AR and autonomous vehicles (AV) applications is critical and attracting significant attention. The self-supervised monocular depth and ego-motion estimation serves as a more intelligent learning approach that provides the required scene depth and location for 3D perception. However, the existing self-supervised learning methods suffer from scale ambiguity, boundary blur and imbalanced depth distribution, limiting the practical applications of VR/AR and AV. In this paper, we propose a new self-supervised learning framework based on superpixel and normal constraints to address these problems. Specifically, we formulate a novel 3D edge structure consistency loss to alleviate the boundary blur of depth estimation. To address the scale ambiguity of estimated depth and ego-motion, we propose a novel surface normal network for efficient camera height estimation. The surface normal network is composed of a deep fusion module and a full-scale hierarchical feature aggregation module. Meanwhile, to realize the global smoothing and boundary discriminability of the predicted normal map, we introduce a novel fusion loss which is based on the consistency constraints of the normal in edge domains and superpixel regions. Experiments are conducted on several benchmarks, and the results illustrate that the proposed approach outperforms the state-of-the-art methods in depth, ego-motion and surface normal estimation.

Pyramid Deep Fusion Network for Two-Hand Reconstruction From RGB-D Images

Pyramid Deep Fusion Network for Two-Hand Reconstruction From RGB-D Images

LithoSegNet: Regional attention-based deep fusion of multi-scale and cross-stage features for real-time lithology segmentation

Lithology identification plays an important role in engineering construction, disaster prediction, reservoir evaluation, and other fields. However, with the development of mechanization and automation, traditional lithology recognition methods are gradually unable to meet existing demands in terms of recognition speed and efficiency. In order to achieve real-time, fast, and accurate identification of lithology, we designed a lightweight model backbone, proposed a multi-scale feature extraction module and a cross-stage feature fusion module based on regional attention, developed a step-wise fusion scheme for multi-stage feature, and developed a real-time lithology segmentation model. The experimental results show that the feature extraction module improves the mIoU of the model by 0.0021; The cross-stage feature fusion module improves the mIoU of the model by 0.0104; The step-wise fusion scheme can effectively fuse shallow spatial features and deep semantic features. The model mIoU using this scheme is 0.0246 and 0.0145 higher than that using SegFormer and SegNeXt fusion schemes, respectively. Our model LithoSegNet can achieve a mIoU of 0.9583 with a speed of 116.25 images per second, comprehensively surpassing many state-of-the-art models. The research results can provide technical support for automated geological sketching in field exploration and on-site construction, and lay the foundation for intelligent construction of underground engineering, which has important scientific and engineering application value.

Deep Fusion of Intrinsic Vibration Information and Grassmann Manifold-based Similarity for Fault Identification of Reciprocating Compressor

This paper introduces a new method based on deep belief networks (DBN) to integrate intrinsic vibration information and assess the similarity of subspaces established on the Grassmann manifold for intelligent fault diagnosis of a reciprocating compressor (RC). Initially, raw vibration signals undergo empirical mode decomposition (EMD) to break them down into multiple intrinsic mode functions (IMFs). This operation can reveal inherent vibration patterns of fault and other components hidden in the original signals. Subsequently, features are refined from all the IMFs and concatenated into a high-dimensional representative vector, offering localized and comprehensive insights into RC operation. Through DBN, the fault-sensitive information are further refined from the features to enhance their performance in fault identification. Finally, similarities among subspaces on the Grassmann manifold are computed to match fault types. The efficacy of the method is validated using field data. Comparative analysis with traditional approaches for feature dimension reduction, feature extraction, and Euclidean distance-based fault identification underscores the effectiveness and superiority of the proposed method in RC fault diagnosis. Conflict of Interest Statement The authors declare no conflicts of interest.

Global–Local Deep Fusion: Semantic Integration with Enhanced Transformer in Dual-Branch Networks for Ultra-High Resolution Image Segmentation

The fusion of global contextual information with local cropped block details is crucial for segmenting ultra-high resolution images. In this study, A novel fusion mechanism termed global–local deep fusion (GL-Deep Fusion) is introduced, based on an enhanced transformer architecture that efficiently integrates global contextual information and local details. Specifically, we propose the global–local synthesis networks (GLSNet), a dual-branch network where one branch processes the entire original image, while the other branch handles cropped local patches as input. The feature fusion of different branches in GLSNet is achieved through GL-Deep Fusion, significantly enhancing the accuracy of ultra-high resolution image segmentation. Identifying tiny overlapping items is a task where the model excels, demonstrating its particular effectiveness. To optimize GPU memory utilization, a dual-branch architecture was meticulously designed. This architecture proficiently leverages the features it extracts and seamlessly integrates them into the enhanced transformer framework of GL-Deep Fusion. Benchmarks on the DeepGlobe and Vaihingen datasets demonstrate the efficiency and accuracy of the proposed model. It significantly reduces GPU memory usage by 24.1% on the DeepGlobe dataset, enhancing segmentation accuracy by 0.8% over the baseline model. On the Vaihingen dataset, our model delivers a Mean F1 score of 90.2% and achieves a mIoU of 90.9%, highlighting its exceptional memory efficiency and segmentation precision.

Estimating high spatio-temporal resolution XCO2 using spatial features deep fusion model

The high temporal-spatial resolution estimation of XCO2 data is foundational for precision quantification of carbon dioxide sources and sinks at a regional scale. This study proposed an advanced XCO2 data estimation method, using spatial features deep fusion. Leveraging convolutional neural network (CNN) principles, SpatialFusionNet—a module was designed to amalgamate geographical features within a defined spatial range. This module captures and integrates the spatial characteristics of meteorological and surface environmental factors, enhancing its application to the XCO2 estimation model. Building upon machine learning methods including Extreme Gradient Boosting (XGBoost), Support Vector Machine (SVM), and Deep Neural Network (DNN), combined with the SpatialFusionNet module, the spatial features deep fusion models were constructed utilizing relationships among OCO-2 satellite XCO2 trajectory monitoring data in China, Copernicus Atmospheric Monitoring Service (CAMS) XCO2 reanalysis data, and meteorological factors, surface vegetation, and meteorological factors. Model performance improvements were significant, with SVM, DNN, and XGBoost showing respective RMSE reductions of 1.297 ppm, 0.480 ppm, and 0.200 ppm in ten-fold cross-validation based on OCO-2 trajectory samples. In data validation with TCCON Hefei station, the correlation between inversion data and ground-based data reached 0.85, affirming the method's high accuracy. Employing the spatial feature extraction module combined with DNN, the 2015 XCO2 annual spatial distribution of China, analyzing temporal-spatial distribution characteristics in China was generated. The DNN model, combining the SpatialFusionNet module, significantly contributes to the estimation of high temporal-spatial resolution XCO2 datasets, facilitating fine-scale quantification of regional carbon cycling.

Optimizing Emotional Insight through Unimodal and Multimodal Long Short-term Memory Models

The field of multimodal emotion recognition is increasingly gaining popularity as a research area. It involves analyzing human emotions across multiple modalities, such as acoustic, visual, and language. Emotion recognition is more effective as a multimodal learning task than relying on a single modality. In this paper, we present an unimodal and multimodal long short-term memory model with a class weight parameter technique for emotion recognition on the CMU-Multimodal Opinion Sentiment and Emotion Intensity dataset. In addition, a critical challenge lies in selecting the most effective fusion method for integrating multiple modalities. To address this, we applied four different fusion techniques: Early fusion, late fusion, deep fusion, and tensor fusion. These fusion methods improved the performance of multimodal emotion recognition compared to unimodal approaches. With the highly imbalanced number of samples per emotion class in the MOSEI dataset, adding a class weight parameter technique leads our model to outperform the state of the art on all three modalities — acoustic, visual, and language — as well as on all the fusion models. The challenges of class imbalance, which can lead to biased model performance, and using an effective fusion method for integrating multiple modalities often result in decreased accuracy in recognizing less frequent emotion classes. Our proposed model shows 2–3% performance improvement in the unimodal and 2% in the multimodal over the state-of-the-art achieved results.

MMFG: Multimodal-based Mutual Feature Gating 3D Object Detection

To address the problem that image and point cloud features are fused in a coarse fusion way and cannot achieve deep fusion, this paper proposes a multimodal 3D object detection architecture based on a mutual feature gating mechanism. First, since the feature aggregation approach based on the set abstraction layer cannot obtain fine-grained features, a point-based self-attention mechanism module is designed. This module is added to the extraction branch of point cloud features to achieve fine-grained feature aggregation while maintaining accurate location information. Second, a new gating mechanism is designed for the deep fusion of image and point cloud. Deep fusion is achieved by mutual feature weighting between the image and the point cloud. The newly fused features are then fed into a feature refinement network to extract classification confidence and 3D target bounding boxes. Finally, a multi-scale detection architecture is proposed to obtain a more complete object shape. The location-based encoding feature algorithm is also designed to focus the interest points in the region of interest adaptively. The whole architecture shows outstanding performance on the KITTI3D and nuSenece datasets, especially at the difficult level. It shows that the framework solves the problem of low detection rates in LiDAR mode due to the low number of surface points obtained from distant objects.

An Implicit Transformer-based Fusion Method for Hyperspectral and Multispectral Remote Sensing Image

There is an effective way to enhance the spatial resolution of hyperspectral remote sensing images by fusing them with multispectral remote sensing images. However, most of the existing deep fusion techniques adopt discretized explicit models to approximate the complex continuous nonlinear mapping in the fusion process, leading to limitations in enhancing the fidelity of spatial details. Additionally, existing algorithms commonly utilize discrete methods such as bilinear or bicubic interpolation during the hyperspectral upsampling process, leading to the loss of crucial spatial-spectral features. To this end, this study proposes a novel Implicit Transformer Fusion Generative Adversarial Network (ITF-GAN), which incorporates the continuity perception mechanism of implicit neural representation with the powerful self-attention mechanism of the Transformer architecture, which uses point-to-point implicit functions aiming to efficiently process information in both spatial and spectral dimensions. Besides, a guided implicit neural sampling module is introduced in the hyperspectral image up-sampling process to enhance the coordinated expression of features in the spatial and spectral domains, which improves the spatial resolution and spectral fidelity of the fused image during the upsampling process. A series of fusion experiments including 4x, 8x, and 16x scale factors have shown that ITF-GAN has significant advantages over current popular fusion algorithms in both objective evaluation indicators and subjective visual evaluation.

Graph feature fusion driven by deep autoencoder for advanced hyperspectral image unmixing

In this paper, we propose a pioneering approach for blind Hyperspectral Image (HSI) unmixing named Multi-Features Graph Deep Fusion Learning Networks for HSI Unmixing (MF-GDL). Our method leverages the power of multi-feature graph deep fusion learning networks. MF-GDL integrates spectral HSI graph features with complementary spatial graph features, overcoming the fixed window size limitation observed in prior deep learning approaches in capturing spatial relationships effectively. A key contribution lies in the introduction of multi-view graph learning, a novel aspect in HSI unmixing. By incorporating graph similarity-based spatial information, we address an important gap in the existing literature, marking the first attempt, to our knowledge, to tackle this challenge using a graph deep learning method. Furthermore, we introduce auxiliary information based on Extended Morphological Profiles (EMPs) to enrich spatial context beyond high-level spatial relationships between neighboring pixels. This multi-view fusion learning not only improves accuracy in material abundance estimation but also demonstrates the significance of considering multiple perspectives in HSI unmixing. We validate the efficacy of MF-GDL on four real datasets: Jasper Ridge, Samson, Urban, and Apex showcasing its robust performance and highlighting the substantial contribution of the multi-view aspect in advancing HSI unmixing techniques.