Fusion Framework Research Articles

A self-supervised fusion for infrared and visible images via multi-level contrastive auto-encoding

The objective of infrared and visible image fusion is to integrate the features of source images and acquire a fused image that highlights foreground objects and preserves background details. Nonetheless, due to ignoring the mutual information interaction between source images, prevailing image fusion methods always encounter challenges related to incomplete or redundant feature extraction. For this purpose, this study presents a self-supervised fusion framework,employing multi-level contrastive auto-encoding to realize the infrared and visible image fusion (IVIF) task, referred to as SS-MCAE. The SS-MCAE consists of a coarse-to-fine feature extraction network (CFEN) and a fusion network (FN), where CFEN contains a content-aware enhancement module (CEM) and a multi-level contrastive learning autoencoder (MCLA). Specifically, CFEN employs CEM to adaptively enhance brightness information and texture details from source images and the MCLA is introduced to refine and extract these features hierarchically. Within contrastive learning, a neoteric sample generator (SG) is introduced for constructing positive and negative samples, aiming to maximize feature extraction while diminishing redundant information. Furthermore, an innovative perceptual loss is developed to retain original features and guide the image reconstruction, establishing a more reliable relationship between extracted features and source images. Comprehensive experiments reveal that our proposed SS-MCAE is superior to current approaches in both visual effect and quantitative analysis.

Hybrid deep spatial and statistical feature fusion for accurate MRI brain tumor classification.

The classification of medical images is crucial in the biomedical field, and despite attempts to address the issue, significant challenges persist. To effectively categorize medical images, collecting and integrating statistical information that accurately describes the image is essential. This study proposes a unique method for feature extraction that combines deep spatial characteristics with handmade statistical features. The approach involves extracting statistical radiomics features using advanced techniques, followed by a novel handcrafted feature fusion method inspired by the ResNet deep learning model. A new feature fusion framework (FusionNet) is then used to reduce image dimensionality and simplify computation. The proposed approach is tested on MRI images of brain tumors from the BraTS dataset, and the results show that it outperforms existing methods regarding classification accuracy. The study presents three models, including a handcrafted-based model and two CNN models, which completed the binary classification task. The recommended hybrid approach achieved a high F1 score of 96.12 ± 0.41, precision of 97.77 ± 0.32, and accuracy of 97.53 ± 0.24, indicating that it has the potential to serve as a valuable tool for pathologists.

GRU-AE-wiener: A generative adversarial network assisted hybrid gated recurrent unit with Wiener model for bearing remaining useful life estimation

Bearings play a pivotal role in various mechanical systems, and their health directly impacts the reliability and safety of these systems. Consequently, extensive research has been dedicated to the estimation of Bearing Remaining Useful Life (RUL) through the lens of information fusion theory. The absence of comprehensive life-cycle degradation data for bearings, a common challenge within the information fusion domain, can hinder the accuracy and reliability of RUL prediction models. A novel hybrid data and model approach named GRU-AE-Wiener has been developed to address this limitation. This approach combines the power of Gated Recurrent Unit (GRU) and Wiener process models within the information fusion framework. Firstly, a Loop Generative Adversarial Network (Loop-GAN) is introduced to synthesize pseudo data to enhance the quality of synthetic data. Next, a bidirectional GRU model is structurally integrated with the Wiener process. In this design, the GRU model is configured in an Auto-Encoder-like structure, with the Wiener process serving as the hidden layer within this Auto-Encoder. Importantly, both the GRU and Wiener processes are jointly optimized with the assistance of Loop-GAN, emphasizing the collaborative nature of information fusion in this approach. The effectiveness of the proposed GRU-AE-Wiener is validated using the PHM 2012 dataset and XJTU-SY dataset. Experimental results underscore its superior RUL predictive performance compared to other deep learning models, highlighting the practical application of information fusion principles in bearing health assessment.

Global–Local Deep Fusion: Semantic Integration with Enhanced Transformer in Dual-Branch Networks for Ultra-High Resolution Image Segmentation

The fusion of global contextual information with local cropped block details is crucial for segmenting ultra-high resolution images. In this study, A novel fusion mechanism termed global–local deep fusion (GL-Deep Fusion) is introduced, based on an enhanced transformer architecture that efficiently integrates global contextual information and local details. Specifically, we propose the global–local synthesis networks (GLSNet), a dual-branch network where one branch processes the entire original image, while the other branch handles cropped local patches as input. The feature fusion of different branches in GLSNet is achieved through GL-Deep Fusion, significantly enhancing the accuracy of ultra-high resolution image segmentation. Identifying tiny overlapping items is a task where the model excels, demonstrating its particular effectiveness. To optimize GPU memory utilization, a dual-branch architecture was meticulously designed. This architecture proficiently leverages the features it extracts and seamlessly integrates them into the enhanced transformer framework of GL-Deep Fusion. Benchmarks on the DeepGlobe and Vaihingen datasets demonstrate the efficiency and accuracy of the proposed model. It significantly reduces GPU memory usage by 24.1% on the DeepGlobe dataset, enhancing segmentation accuracy by 0.8% over the baseline model. On the Vaihingen dataset, our model delivers a Mean F1 score of 90.2% and achieves a mIoU of 90.9%, highlighting its exceptional memory efficiency and segmentation precision.

FCAN : Speech emotion recognition network based on focused contrastive learning

Traditional speech emotion recognition (SER) models predominantly utilize cross-entropy loss for supervised optimization. Yet, given the limited size of SER datasets, relying solely on cross-entropy loss does not adequately capture the intricacies within the data. Moreover, conventional contrastive learning techniques, though improving performance, do not completely harness the inherent data structure across samples. To surmount these limitations, we introduce an innovative emotion recognition network founded on focused contrastive learning. Our model employs bidirectional gated recurrent unit (GRU) networks to seize critical contextual dependencies. By integrating an attention mechanism, we adeptly merge the contextual features of dialogues with individual utterance attributes, forming a comprehensive multi-stage feature fusion framework. This strategy extracts more elaborate emotional features from the constrained dataset. Additionally, by amalgamating the focal loss function into our training protocol, we mitigate imbalanced sample issues to a degree. We further refine our contrastive loss function by enacting a weighted strategy for intra-dialogue category assignments, where higher penalty weights are ascribed to harder-to-identify categories, thus incentivizing the model to concentrate on distinguishing subtler category differences. This refinement significantly bolsters the model’s proficiency in learning various emotional feature distributions, thereby enhancing its emotion recognition capabilities. Experimental validation on IEMOCAP and MELD benchmark datasets underpins our model’s superiority.

A multimodal framework for extraction and fusion of satellite images and public health data

In low- and middle-income countries, the substantial costs associated with traditional data collection pose an obstacle to facilitating decision-making in the field of public health. Satellite imagery offers a potential solution, but the image extraction and analysis can be costly and requires specialized expertise. We introduce SatelliteBench, a scalable framework for satellite image extraction and vector embeddings generation. We also propose a novel multimodal fusion pipeline that utilizes a series of satellite imagery and metadata. The framework was evaluated generating a dataset with a collection of 12,636 images and embeddings accompanied by comprehensive metadata, from 81 municipalities in Colombia between 2016 and 2018. The dataset was then evaluated in 3 tasks: including dengue case prediction, poverty assessment, and access to education. The performance showcases the versatility and practicality of SatelliteBench, offering a reproducible, accessible and open tool to enhance decision-making in public health.

Speeding up UAV-based crop variability assessment through a data fusion approach using spatial interpolation for site-specific management

Innovations in precision agriculture enhance complex tasks, reduce environmental impact, and increase food production and cost efficiency. One of the main challenges is ensuring rapid information availability for autonomous vehicles and standardizing processes across platforms to maximize interoperability. The lack of drone technology standardisation, communication barriers, high costs, and post-processing requirements sometimes hinder their widespread use in agriculture. This research introduces a standardized data fusion framework for creating real-time spatial variability maps using images from different Unmanned Aerial Vehicles (UAVs) for Site-Specific Crop Management (SSM). Two spatial interpolation methods were used (Inverse Distance Weight, IDW, and Triangulated Irregular Networks, TIN), selected for their computational efficiency and input flexibility. The proposed framework can use different UAV image sources and offers versatility, speed, and efficiency, consuming up to 98 % less time, energy, and computing requirements than standard photogrammetry techniques, providing rapid field information, allowing edge computing incorporation into the UAV data acquisition phase. Experiments conducted in Spain, Serbia, and Finland in 2022 under the H2020 FlexiGroBots project demonstrated a strong correlation between results from this method and those from standard photogrammetry techniques (up to r = 0.93). In addition, the correlation with Sentinel 2 satellite images was as strong as that obtained with photogrammetry-based orthomosaics (up to r = 0.8). The proposed approach could support irrigation leak detection, soil parameter estimation, weed management, and satellite integration for agriculture.

Modeling spatiotemporal temperature dynamics of large-format power batteries: A multi-source information fusion approach

Uneven temperature distributions can significantly impact battery performance and cycle life. Industrial applications, in particular, where unknown disturbances and limited sensors exist, pose significant challenges for accurately estimating the battery temperature field. This work proposes a multi-source information fusion framework for capturing the spatiotemporal temperature dynamics of pouch-type Lithium-ion batteries. First, we develop a system model of the battery thermal process based on physical insights, considering unknown parameter deviations and unmodeled dynamics. Next, we use the Galerkin-spectral method to expand the spatiotemporal variable and reduce the original infinite-dimensional system to a low-order model containing the most important system modes. The unknown component of the model is then entirely stripped and integrated into a nonlinear term. To close the reality gap of the physics-based model, we subsequently develop an error compensation model that learns the dynamic behavior from sparse observations. Ultimately, our framework design yields a fusion-driven model for reliable temperature field prediction. Simulations and experimental data validate the superior performance and generalization capability of the proposed method.

Exploration of deep learning-driven multimodal information fusion frameworks and their application in lower limb motion recognition

Research on Lower Limb Motion Recognition (LLMR) based on various wearable sensors has been widely applied in exoskeleton robots, exercise rehabilitation, etc. Typically, employing multimodal information tends to yield higher accuracy and stronger robustness compared to using unimodal information. Due to the inevitable reliance on feature engineering in shallow machine learning-based LLMR methods, this study leverages the powerful non-linear feature mapping capability of deep learning (DL) to construct several end-to-end LLMR frameworks, including: Convolutional Neural Networks (CNNs), CNN-Recurrent Neural Networks (RNNs) and CNN-Graph Neural Networks (GNNs). The effectiveness of the proposed frameworks is verified in distinct tasks, including the recognition of seven types of lower limb motions in healthy subjects and three types of motions in patients with stroke, as well as the phase recognition task during the sit-to-stand (SitTS) process in patients with stroke, achieving the highest mean accuracy of 95.198 %, 99.784 %, and 99.845 %, respectively. Further research and integration of two transfer learning techniques, adaptive Batch Normalization (BN) and model fine-tuning, significantly enhance the applicability of the proposed frameworks in inter-subject prediction. Additionally, systematic analyses are conducted to assess the strengths and weaknesses of different models in terms of recognition performance, complexity, and adaptability to variations in the number of modalities and sensor channels. Experimental results indicate that the proposed frameworks hold promise in providing potential support for the development of human-robot collaborative lower limb exoskeletons or rehabilitation robots.

Parallel Multilink Group Joint ICA: Fusion of 3D Structural and 4D Functional Data Across Multiple Resting fMRI Networks.

Multimodal neuroimaging research plays a pivotal role in understanding the complexities of the human brain and its disorders. Independent component analysis (ICA) has emerged as a widely used and powerful tool for disentangling mixed independent sources, particularly in the analysis of functional magnetic resonance imaging (fMRI) data. This paper extends the use of ICA as a unifying framework for multimodal fusion, introducing a novel approach termed parallel multilink group joint ICA (pmg-jICA). The method allows for the fusion of gray matter maps from structural MRI (sMRI) data to multiple fMRI intrinsic networks, addressing the limitations of previous models. The effectiveness of pmg-jICA is demonstrated through its application to an Alzheimer's dataset, yielding linked structure-function outputs for 53 brain networks. Our approach leverages the complementary information from various imaging modalities, providing a unique perspective on brain alterations in Alzheimer's disease. The pmg-jICA identifies several components with significant differences between HC and AD groups including thalamus, caudate, putamen with in the subcortical (SC) domain, insula, parahippocampal gyrus within the cognitive control (CC) domain, and the lingual gyrus within the visual (VS) domain, providing localized insights into the links between AD and specific brain regions. In addition, because we link across multiple brain networks, we can also compute functional network connectivity (FNC) from spatial maps and subject loadings, providing a detailed exploration of the relationships between different brain regions and allowing us to visualize spatial patterns and loading parameters in sMRI along with intrinsic networks and FNC from the fMRI data. In essence, developed approach combines concepts from joint ICA and group ICA to provide a rich set of output characterizing data-driven links between covarying gray matter networks, and a (potentially large number of) resting fMRI networks allowing further study in the context of structure/function links. We demonstrate the utility of the approach by highlighting key structure/function disruptions in Alzheimer's individuals.

A Fusion Approach for UAV Onboard Flight Trajectory Management and Decision Making Based on the Combination of Enhanced A* Algorithm and Quadratic Programming

The rapid advancement of unmanned aerial vehicle (UAV) technologies has led to an increasing demand for UAV operations in low-altitude, high-density, and complex airspace such as mountains or urban areas. In order to handle complex scenarios and ensure flight safety for UAVs with different flight missions beyond visual line of sight in such environments, a fusion framework of onboard autonomous flight trajectory management and decision-making system using global strategical path planning and local tactical trajectory optimization combination is proposed in this paper. The global strategical path planning is implemented by an enhanced A* algorithm under the multi-constraint of UAV positioning uncertainty and obstacle density to improve the safety and cost-effectiveness. The local tactical trajectory optimization is realized using quadratic programming to ensure smoothness, kinematic feasibility, and obstacle avoidance of the planned trajectory in dynamic environments. Receding-horizon control is used to ensure the flight path and trajectory planning efficiently and seamlessly. To assess the performance of the system, a terrain database and a navigation system are employed for environment and navigation performance simulation. The experimental results confirm that the fusion approach can realize better safety and cost-effectiveness through path planning with kino-dynamic feasible trajectory optimization.

DFG-HCEN: A distinctive-feature guided and hierarchical channel enhanced network-based infrared and visible image fusion

In this paper, we propose an unsupervised learning approach for the task of infrared and visible image fusion. This approach is called a distinctive-feature guided and hierarchical channel enhanced network-based infrared and visible image Fusion (DFG-HCEN). Instead of using complex fusion rules, DFG-HCEN uses multi-level fusion to achieve fusion results, effectively avoiding information loss during feature extraction. To improve the fusion effect, we designed a distinctive-feature guided module that strengthens the relationship between modules. Moreover, the proposed hierarchical channel enhanced and distinctive-Feature guided module aims to facilitate the fusion framework in efficiently integrating the multilevel complementary features of the source pictures. In addition, we incorporate a hybrid loss method for unsupervised training of the provided DFG-HCEN. The fidelity loss is used to constrain the pixel similarity between the fused result and source images. The application of luminance regularization loss has been shown to be an efficient method for addressing the problem of luminance degradation in fused images. We conducted extensive experiments, including visual examination and quantitative analysis, comparing DFG-HCEN with thirteen other state-of-the-art fusion techniques. The results demonstrate the superiority of DFG-HCEN. Moreover, the extended object detection experiments validate the ability of DFG-HCEN to fully support downstream tasks.

Two-stage fusion framework driven by domain knowledge for penetration prediction of laser welding

To solve the problem of industrial application difficulties caused by insufficient depth exploration and poor interpretability of the pure data drive neural network. A two-stage fusion framework driven by domain knowledge was proposed and focused on industrial applications: laser welding penetration monitoring. First, a multi-sensor acquisition system based on plasma and molten pool signals is developed, extracting multiple features from plasma and molten pool images. Two Back Propagation (BP) neural network information fusion models are established in the first fusion stage, leveraging the characteristics of the plasma plume and molten pool front. In the second fusion stage, the Dempster-Shafer (DS) evidence theory is employed to fuse the information from the two BP neural network models. The proposed method, achieving a high accuracy of 98.7 %, is validated by comparing its average accuracy and anti-interference capability with classical Convolutional Neural Network (CNN) and BP neural network models. The physical relationship between input and output is elucidated through an established physical model, enhancing the interpretability and anti-interference capability of the model, and thereby improving the stability and accuracy of the system.

Dynamic YOLO for small underwater object detection

The practical application of object detection inevitably encounters challenges posed by small objects. In underwater object detection, a crucial method for marine exploration, the presence of small objects in underwater environments significantly hampers the performance of detection. In this paper, a dynamic YOLO detector is proposed as a solution to alleviate this problem. Specifically, a light-weight backbone network is first constructed based on deformable convolution v3, with some specialized designs for small object detection. Secondly, a unified feature fusion framework based on channel-wise, scale-wise, and spatial-aware attention is proposed to fuse feature maps from different scales. This is particularly critical for detecting small objects since it allows us to fully exploit the enhanced capabilities offered by our proposed backbone network. Finally, a simple but effective detection head is designed to handle the conflict between classification and localization by disentangling and aligning the two tasks. Extensive experiments are conducted on benchmark datasets to demonstrate the effectiveness of the proposed model. Without bells and whistles, dynamic YOLO outperforms the recent state-of-the-art methods by a large margin of +0.8\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$+\\,0.8$$\\end{document} AP and +1.8\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$+\\,1.8$$\\end{document} APS\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ ext {AP}_{S}$$\\end{document} on the DUO dataset. Experimental results on Pascal VOC and MS COCO datasets also demonstrate the superiority of the proposed method. At last, ablation studies are conducted on DUO dataset to validate the effectiveness and efficiency of each design in dynamic YOLO. Source code will be available at https://github.com/chenjie04/Dynamic-YOLO.

Predicting stroke events with a proactive fusion system: a comprehensive study on imbalance class handling in computational biomechanics

Stroke, as a critical global health concern and the second leading cause of death, occurs when blood flow to the brain is interrupted. Although machine learning has advanced in medical safety, there is limited research on stroke prediction using information fusion systems. This study presents a fusion framework that combines multiple base classifiers and a Meta classifier to improve stroke prediction performance. The research utilizes Grid Search optimized models, such as Random Forest, Support Vector Machine, K Nearest Neighbors, AdaBoost, Gradient Boosting, Light Gradient Boosting, Categorical Boosting, and eXtreme Gradient Boosting for in-depth stroke analysis. Since stroke events are rare and unpredictable, classification outcomes can be deceptive due to imbalanced data. This article examines stroke probability by comparing three data balancing methods: over-sampling, under-sampling, and tomek-link synthetic minority over-sampling (SMOTE-TL) to enhance prediction accuracy. The findings reveal that AdaBoost as a meta-classifier attains the highest performance in the fusion framework, with a peak of 88.09% Recall and 83.66% F1 score. This innovative approach provides crucial insights into stroke prediction and can be a valuable resource for strengthening intervention efforts in advanced healthcare systems.

HVDFusion: an effective fusion framework based on Hilbert vibration decomposition for multi-focal and multi-sensor images

HVDFusion: an effective fusion framework based on Hilbert vibration decomposition for multi-focal and multi-sensor images

Self-confidence Fermatean fuzzy trust network-driven consensus fusion framework: A new energy vehicle power battery recycling service outlet selection

The selection of new energy vehicle power battery (NEVPB) recycling service outlet will promote the rapid development of new energy vehicle industry. In order to cope the green and low-carbon development path of NEVPB recycling under information uncertainty and interaction. This paper intends to construct a multidimensional consensus fusion framework of self-confidence Fermatean fuzzy trust network. Firstly, individuals express their opinions based on Fermatean fuzzy sets with self-confidence (FFS-SC). Subsequently, a novel trust propagation and aggregation is built to obtain complete trust relations, and weights of individuals are determined by trust network, self-confidence, and relative deviation. Meanwhile, the trust network- driven consensus feedback mechanism is developed to collect evaluation information. Furthermore, the correlation coefficient to eliminate the homogeneity influence of indicators. In addition, the decision results are derived by Multi-Attributive Border Approximation area Comparison (MABAC) method. Finally, an applied case with five NEVPB recycling service outlets selection is presented to verify the feasibility of the established framework, and the effectiveness is further demonstrated through sensitivity analysis and comparative analysis. The case results show that the indicator with demonstration of recycling service outlet and market supervision departments strengthen supervision indicators have great impact on NEVPB recycling service outlets selection. Furthermore, this paper has reference significance for standardizing the management policy and technology of power battery recycling and realizing the green and low-carbon environmental protection of the whole life cycle of new energy vehicles.

A Local-Global Attention Fusion Framework with Tensor Decomposition for Medical Diagnosis

A Local-Global Attention Fusion Framework with Tensor Decomposition for Medical Diagnosis

Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI.

The globally rising prevalence of mental disorders leads to shortfalls in timely diagnosis and therapy to reduce patients' suffering. Facing such an urgent public health problem, professional efforts based on symptom criteria are seriously overstretched. Recently, the successful applications of computer-aided diagnosis approaches have provided timely opportunities to relieve the tension in healthcare services. Particularly, multimodal representation learning gains increasing attention thanks to the high temporal and spatial resolution information extracted from neuroimaging fusion. In this work, we propose an efficient multimodality fusion framework to identify multiple mental disorders based on the combination of functional and structural magnetic resonance imaging. A multioutput conditional generative adversarial network (GAN) is developed to address the scarcity of multimodal data for augmentation. Based on the augmented training data, the multiheaded gating fusion model is proposed for classification by extracting the complementary features across different modalities. The experiments demonstrate that the proposed model can achieve robust accuracies of 75.1 ± 1.5 %, 72.9 ± 1.1 %, and 87.2 ± 1.5 % for autism spectrum disorder (ASD), attention deficit/hyperactivity disorder, and schizophrenia, respectively. In addition, the interpretability of our model is expected to enable the identification of remarkable neuropathology diagnostic biomarkers, leading to well-informed therapeutic decisions.

Benchmarking of industrial wastewater treatment processes using a complex probabilistic hesitant fuzzy soft Schweizer–Sklar prioritized-based framework

The objective of this research work is to effectively handle the intricate and uncertain nature of decision-making in the context of industrial wastewater treatment. The prioritization of wastewater treatment is of utmost importance in order to save the environment, promote human health, ensure adherence to legal regulations, conserve resources, and foster overall sustainability. The demand for efficient and sustainable wastewater treatment processes is increasing globally, but selecting appropriate treatment technologies for industrial effluents is challenging due to its complex nature. Our study aims to provide a systematic framework for evaluating and selecting treatments based on efficacy, affordability, and environmental impact. Aggregation operators are one of the fundamental ideas in the framework of information fusion for addressing real-life problems. Numerous scholars have made significant contributions to the development of aggregation operators specifically designed for multi-parameter decision-making (MPDM) whenever circumstances are characterized by uncertainty. Regrettably, the current operators can only be employed within strict limits and constraints. Therefore, this research formulates novel prioritized aggregation operators that alleviate the restrictive constraints associated with the current operators. After this, we present a new methodology that combines the Schweizer–Sklar prioritized aggregation operators with a complex probabilistic hesitant fuzzy soft framework. For this purpose, we develop the averaging aggregation operators and geometric aggregation operators. Following that, we proceed to examine the theorems and properties associated with them by providing rigorous proofs. Then, we develop a novel decision-making technique with a numerical example based on the wastewater treatment process. Through this novel technique, the best process is the activated sludge process. We perform the validity tests to evaluate the feasibility and effectiveness of MPDM techniques.