Fusion Problem Research Articles

A weighted gradient model with total variation regularization for multi-focus image fusion

In this paper, we consider the problem of the multi-focus image fusion and propose a novel weighted gradient model with the total variation regularization. The existence and uniqueness of the solution for the proposed model is established. We employ the semi-proximal alternating direction method of multipliers (sPADMM) algorithm to find the minimizer of the proposed model. We also show the convergence and convergence rate of the sPADMM algorithm under suitable conditions. Numerical results are given to demonstrate the effectiveness and competitiveness of the proposed method.

Experimental and Simulation Studies of Micro-Swing Arc Welding Process for X80M Pipeline

Pipe girth welds are prone to incomplete fusion problems in the automatic welding process of long-distance pipelines, which is often related to temperature inhomogeneity in the weld bead. The narrow gap and micro-swing welding technique was applied in pipeline construction to improve welding quality. The manuscript provides a detailed investigation of the micro-swing welding technique with a combination of welding experiments and numerical simulation. A swing welding strategy was proposed according to the actual welding condition in pipeline construction to study the formation mechanism of weld joints. The swing width grew to 1.25–1.35 times from the 3 to 6 o’clock position in the same filling layer. It also increased with filling layers, and filling layer 5 had the biggest swing width, almost two times that of filling layer 2. “Middle concave” morphology appeared at the 3 o’clock position, which could effectively avoid the occurrence of incomplete fusion, while “hump” morphology may appear at the 6 o’clock position, and incomplete fusion defects occurred if the next pass failed to eliminate the influence of the “hump”. The temperature field presented an obvious “sawtooth” shape at small swing frequencies, which could cause temperature inhomogeneity. It could be effectively eliminated when swing frequency reached over 5 Hz.

Calculations of Compression of a Spherical Layered System by Shock Waves Taking into Account the Transfer of Thermal Radiation in the Kinetic Model

At present, numerical simulation is the main, and often the only, tool for detailed description of some physical phenomena in the study of the processes of material compression by shock waves. The study of the behavior of shock and rarefaction waves on the simplest model tests helps in the analysis of more complex calculations, for example, problems of inertial thermonuclear fusion on laser facilities. The paper considers comparative calculations of a test problem that simulates compression of a spherical layered system consisting of two substances by shock waves.

Information Fusion over Network Dynamics with Unknown Correlations: An Overview

Survey/review study Information Fusion over Network Dynamics with Unknown Correlations: An Overview Wangyan Li 1, and Fuwen Yang 2,* 1 College of Science, University of Shanghai for Science and Technology, Shanghai 200093, China 2 Griffth School of Engineering, Griffth University, Gold Coast Campus, QLD 4222, Australia * Correspondence: fuwen.yang@griffth.edu.au Received: 24 October 2022 Accepted: 22 November 2022 Published: 23 June 2023 Abstract: Unknown correlations (UCs) generally exist in a wide spectrum of practical multi-source information fusion problems, and thereby, their corresponding fusion problems have become one of the most important topics in information fusion domain. During the past three decades, the research on this topic has been growing rapidly and extensively, and, as a result, various important advances have been reported. In this overview, we intend to summarize the culmination of years of development in the field of information fusion under UCs as a roadmap. First, the potential reasons leading to UCs are investigated. According to the unknown nature of correlations, we further divide UCs into two categories, i.e., fully UCs, and partially UCs. For each category, the corresponding fusion methods are reviewed. Next, this roadmap witnesses the recent development of information fusion under UCs in a distributed way thanks to the popularity of distributed sensing technology. In particular, the distributed fusion techniques based on consensus, diffusion, and multi-object tracking strategies for UCs are examined. Finally, some future perspectives on information fusion under UCs are pointed out.

Adaptive visual-tactile fusion recognition for robotic operation of multi-material system.

The use of robots in various industries is evolving from mechanization to intelligence and precision. These systems often comprise parts made of different materials and thus require accurate and comprehensive target identification. While humans perceive the world through a highly diverse perceptual system and can rapidly identify deformable objects through vision and touch to prevent slipping or excessive deformation during grasping, robot recognition technology mainly relies on visual sensors, which lack critical information such as object material, leading to incomplete cognition. Therefore, multimodal information fusion is believed to be key to the development of robot recognition. Firstly, a method of converting tactile sequences to images is proposed to deal with the obstacles of information exchange between different modalities for vision and touch, which overcomes the problems of the noise and instability of tactile data. Subsequently, a visual-tactile fusion network framework based on an adaptive dropout algorithm is constructed, together with an optimal joint mechanism between visual information and tactile information established, to solve the problem of mutual exclusion or unbalanced fusion in traditional fusion methods. Finally, experiments show that the proposed method effectively improves robot recognition ability, and the classification accuracy is as high as 99.3%.

An Automatic Digital Camouflage Pattern Generation Method based on Texture Structure

This article investigates the problem of scattered burst signal detection based on multiple sensors to obtain overall decisions. In the explosion detection system studied in the article, sensors independently transmit their decisions on measuring explosion information to the data fusion processing terminal, which provides overall decisions based on fusion rules. The researchers focus on the data fusion theory of the distributed parallel detection burst point data fusion system based on the Bayesian rule. This paper has obtained the data fusion rule and sensor decision criteria that make the overall system optimal, and proposed a nonlinear Gauss Seidel mathematical variable algorithm that optimizes the data fusion rule and multi-sensor decision criteria The data fusion problem when detecting burst point signals with two different and three identical types of sensors. The data fusion algorithm proposed in this article is validated and simulated through computer experiments on the detection of three types of sensors. The relevant experimental data show that the performance of a data fusion system based on Bayesian detection is significantly improved compared with the sensor acquisition of burst point information. In the experiment, the risk of Bayesian missing detection of burst point signal coefficient of the data fusion system using three sensors with the same performance is reduced by 32.7%.

EFPN: Effective medical image detection using feature pyramid fusion enhancement

Feature pyramid networks (FPNs) are widely used in the existing deep detection models to help them utilize multi-scale features. However, there exist two multi-scale feature fusion problems for the FPN-based deep detection models in medical image detection tasks: insufficient multi-scale feature fusion and the same importance for multi-scale features. Therefore, in this work, we propose a new enhanced backbone model, EFPNs, to overcome these problems and help the existing FPN-based detection models to achieve much better medical image detection performances. We first introduce an additional top-down pyramid to help the detection networks fuse deeper multi-scale information; then, a scale enhancement module is developed to use different sizes of kernels to generate more diverse multi-scale features. Finally, we propose a feature fusion attention module to estimate and assign different importance weights to features with different depths and scales. Extensive experiments are conducted on two public lesion detection datasets for different medical image modalities (X-ray and MRI). On the mAP and mR evaluation metrics, EFPN-based Faster R-CNNs improved 1.55% and 4.3% on the PenD (X-ray) dataset, and 2.74% and 3.1% on the BraTs (MRI) dataset, respectively. EFPN-based Faster R-CNNs achieve much better performances than the state-of-the-art baselines in medical image detection tasks. The proposed three improvements are all essential and effective for EFPNs to achieve superior performances; and besides Faster R-CNNs, EFPNs can be easily applied to other deep models to significantly enhance their performances in medical image detection tasks.

Soil carbon content prediction using multi-source data feature fusion of deep learning based on spectral and hyperspectral images

Visible near-infrared reflectance spectroscopy (VNIR) and hyperspectral images (HSI) have their respective advantages in soil carbon content prediction, and the effective fusion of VNIR and HSI is of great significance for improving the prediction accuracy. But the contribution difference analysis of multiple features in the multi-source data is inadequate, and there is a lack of in-depth research on the contribution difference analysis of artificial feature and deep learning feature. In order to solve the problem, soil carbon content prediction methods based on VNIR and HSI multi-source data feature fusion are proposed. The multi-source data fusion network under the attention mechanism and the multi-source data fusion network with artificial feature are designed. For the multi-source data fusion network based on the attention mechanism, the information are fused through the attention mechanism according to the contribution difference of each feature. For the other network, artificial feature are introduced to fuse multi-source data. The results show that multi-source data fusion network based on the attention mechanism can improve the prediction accuracy of soil carbon content, and multi-source data fusion network combined with artificial feature has better prediction effect. Compared with two single-source data from the VNIR and HSI, the relative percent deviation of Neilu, Aoshan Bay and Jiaozhou Bay based on multi-source data fusion network combined with artificial feature are increased by 56.81% and 149.18%, 24.28% and 43.96%, 31.16% and 28.73% respectively. This study can effectively solve the problem of the deep fusion of multiple features in the soil carbon content prediction by VNIR and HSI, so as to improve the accuracy and stability of soil carbon content prediction, promote the application and development of soil carbon content prediction in spectral and hyperspectral image, and provide technical support for the study of carbon cycle and the carbon sink.

Multi-Modality Adaptive Feature Fusion Graph Convolutional Network for Skeleton-Based Action Recognition

Graph convolutional networks are widely used in skeleton-based action recognition because of their good fitting ability to non-Euclidean data. While conventional multi-scale temporal convolution uses several fixed-size convolution kernels or dilation rates at each layer of the network, we argue that different layers and datasets require different receptive fields. We use multi-scale adaptive convolution kernels and dilation rates to optimize traditional multi-scale temporal convolution with a simple and effective self attention mechanism, allowing different network layers to adaptively select convolution kernels of different sizes and dilation rates instead of being fixed and unchanged. Besides, the effective receptive field of the simple residual connection is not large, and there is a great deal of redundancy in the deep residual network, which will lead to the loss of context when aggregating spatio-temporal information. This article introduces a feature fusion mechanism that replaces the residual connection between initial features and temporal module outputs, effectively solving the problems of context aggregation and initial feature fusion. We propose a multi-modality adaptive feature fusion framework (MMAFF) to simultaneously increase the receptive field in both spatial and temporal dimensions. Concretely, we input the features extracted by the spatial module into the adaptive temporal fusion module to simultaneously extract multi-scale skeleton features in both spatial and temporal parts. In addition, based on the current multi-stream approach, we use the limb stream to uniformly process correlated data from multiple modalities. Extensive experiments show that our model obtains competitive results with state-of-the-art methods on the NTU-RGB+D 60 and NTU-RGB+D 120 datasets.

CBFM: Contrast Balance Infrared and Visible Image Fusion Based on Contrast-Preserving Guided Filter

Infrared (IR) and visible image fusion is an important data fusion and image processing technique that can accurately and comprehensively integrate the thermal radiation and texture details of source images. However, existing methods neglect the high-contrast fusion problem, leading to suboptimal fusion performance when thermal radiation target information in IR images is replaced by high-contrast information in visible images. To address this limitation, we propose a contrast-balanced framework for IR and visible image fusion. Specifically, a novel contrast balance strategy is proposed to process visible images and reduce energy while allowing for detailed compensation of overexposed areas. Moreover, a contrast-preserving guided filter is proposed to decompose the image into energy-detail layers to reduce high contrast and filter information. To effectively extract the active information in the detail layer and the brightness information in the energy layer, we proposed a new weighted energy-of-Laplacian operator and a Gaussian distribution of the image entropy scheme to fuse the detail and energy layers, respectively. The fused result was obtained by adding the detail and energy layers. Extensive experimental results demonstrate that the proposed method can effectively reduce the high contrast and highlighted target information in an image while simultaneously preserving details. In addition, the proposed method exhibited superior performance compared to the state-of-the-art methods in both qualitative and quantitative assessments.

Improved Generalized IHS Based on Total Variation for Pansharpening

Pansharpening refers to the fusion of a panchromatic (PAN) and a multispectral (MS) image aimed at generating a high-quality outcome over the same area. This particular image fusion problem has been widely studied, but until recently, it has been challenging to balance the spatial and spectral fidelity in fused images. The spectral distortion is widespread in the component substitution-based approaches due to the variation in the intensity distribution of spatial components. We lightened the idea using the total variation optimization to improve upon a novel GIHS-TV framework for pansharpening. The framework drew the high spatial fidelity from the GIHS scheme and implemented it with a simpler variational expression. An improved L1-TV constraint to the new spatial–spectral information was introduced to the GIHS-TV framework, along with its fast implementation. The objective function was solved by the Iteratively Reweighted Norm (IRN) method. The experimental results on the “PAirMax” dataset clearly indicated that GIHS-TV could effectively reduce the spectral distortion in the process of component substitution. Our method has achieved excellent results in visual effects and evaluation metrics.

Practice and research on non-foreign heritage culture inheritance education in universities based on information fusion technology

Abstract It has gradually become a trend to infiltrate the knowledge of non-heritage culture and inherit the spirit of non-heritage culture and ethnic crafts in art education in colleges and universities. The purpose of this paper is to discuss the significance of ethnic crafts in non-heritage culture to art education in colleges and universities, the problems faced by the integration of ethnic crafts in non-heritage culture into art education in colleges and universities, and the transmission path of ethnic crafts in non-heritage culture in art education in colleges and universities. We hope to enhance deep connection between college art education and non-heritage culture by increasing research efforts, so that students will pay attention to the preservation of non-heritage culture and pass on the ethnic crafts of non-heritage culture in college studies. This paper develops research education and the cultivation of intangible cultural heritage through information fusion-based technology. The construction objectives of the multi-source information integration platform are proposed, and the overall architecture, functions, interfaces, and hardware architecture of the platform are designed. Analyze the connection between information fusion and information integration, and clarify the implementation process of multi-source information fusion technology. The two-level information fusion model of diagnosis constructed in this paper can avoid the error of diagnosis results caused by relying on neural network fault diagnosis alone, and the accuracy of fault diagnosis is optimized, which can be applied to the construction of working face. It solves the fusion problem that fault diagnosis results from different information sources exist with each other. It can provide scientific guidance for operation and management units to make decisions, and help to improve the level of education of NRM culture transmission.

On Computing Evidential Centroid Through Conjunctive Combination: An Impossibility Theorem

The theory of belief functions (TBF) is now a widespread framework to deal and reason with uncertain and imprecise information, in particular to solve information fusion and clustering problems. Combination functions (rules) and distances are essential tools common to both the clustering and information fusion problems in the context of TBF, which have generated considerable literature. Distances and combination between evidence corpus of TBF are indeed often used within various clustering and classification algorithms, however their interplay and connections have seldom been investigated, which is the topic of this paper. More precisely, we focus on the problem of aggregating evidence corpus to obtain a representative one, and we show through an impossibility theorem that in this case, there is a fundamental contradiction between the use of conjunctive combination rules on the one hand, and the use of distances on the other hand. Rather than adding new methodologies, such results are instrumental in guiding the user among the many methodologies that already exist. To illustrate the interest of our results, we discuss different cases where they are at play.

Conflict Evidence Fusion Algorithm Based on Cosine Distance and Information Entropy

<p>Dealing with high conflict evidence, traditional evidence theory sometimes has certain limitations, and results in fusion results contrary to common sense. In order to solve the problem of high conflict evidence fusion, this paper analyzes traditional evidence theory and proposes an evidence fusion method that combines cosine distance and information entropy. Cosine distance can measure the directionality between two vectors. The better the directionality, the more similar the two vectors are. Therefore, this article uses cosine distance to determine the similarity between evidences, and then calculates the credibility of each piece of evidence. Information entropy can calculate the amount of information for each evidence. The greater the information entropy, the greater the uncertainty of the evidence. Therefore, this article uses information entropy to measure the uncertainty of the evidence. Then, the credibility and uncertainty of the evidence are fused to calculate the weight of the evidence. Then we use d-s evidence theory for evidence fusion. The numerical example shows that the method is feasible and effective in dealing with conflict evidence. </p> <p> </p>

Trust–Region Nonlinear Optimization Algorithm for Orientation Estimator and Visual Measurement of Inertial–Magnetic Sensor

This paper proposes a novel robust orientation estimator to enhance the accuracy and robustness of orientation estimation for inertial–magnetic sensors of the small consumer–grade drones. The proposed estimator utilizes a trust–region strategy within a nonlinear optimization framework, transforming the orientation fusion problem into a nonlinear optimization problem based on the maximum likelihood principle. The proposed estimator employs a trust–region Dogleg gradient descent strategy to optimize orientation precision and incorporates a Huber robust kernel to minimize interference caused by acceleration during the maneuvering process of the drone. In addition, a novel method for evaluating the performance of orientation estimators is also presented based on visuals. The proposed method consists of two parts: offline calibration of the basic cube using Augmented Reality University of Cordoba (ArUco) markers and online orientation measurement of the sensor carrier using a nonlinear optimization solver. The proposed measurement method’s accuracy and the proposed estimator’s performance are evaluated under low–dynamic (rotation) and high–dynamic (shake) conditions in the experiment. The experimental findings indicate that the proposed measurement method obtains an average re–projection error of less than 0.1 pixels. The proposed estimator has the lowest average orientation error compared to conventional orientation estimation algorithms. Despite the time–consuming nature of the proposed estimator, it exhibits greater robustness and precision, particularly in highly dynamic environments.

Region-aware RGB and near-infrared image fusion

This paper proposes a region-aware fusion method, called RaIF, for RGB and near-infrared (NIR) outdoor scenery image fusion. The method is motivated by the observation that current fusion approaches produce gray appearance in overexposed sky regions and distortion in vegetation regions. RaIF generates the region probability maps by exploiting their specific characteristics in the visible and NIR spectra. It recovers the overexposed sky regions by employing the intrinsic channel correlation between RGB and NIR images, and enhances the vegetation regions in an adjustable manner. RaIF formulates image fusion problem as a gradient-domain optimization problem with luminance and chromaticity regularizations. Experimental results validate the superiority of RaIF that produces fused images with improved appearance in the sky and vegetation regions, and achieves the state-of-the-art performance quantitatively and qualitatively. Furthermore, RaIF can act as a refinement module that improves the fusion results of current deep learning based approaches. It is also capable of recovering specular highlight regions other than sky overexposure.

Fractional-Order Variational Image Fusion and Denoising Based on Data-Driven Tight Frame

Multi-modal image fusion can provide more image information, which improves the image quality for subsequent image processing tasks. Because the images acquired using photon counting devices always suffer from Poisson noise, this paper proposes a new three-step method based on the fractional-order variational method and data-driven tight frame to solve the problem of multi-modal image fusion for images corrupted by Poisson noise. Thus, this article obtains fused high-quality images while removing Poisson noise. The proposed image fusion model can be solved by the split Bregman algorithm which has significant stability and fast convergence. The numerical results on various modal images show the excellent performance of the proposed three-step method in terms of numerical evaluation metrics and visual quality. Extensive experiments demonstrate that our method outperforms state-of-the-art methods on image fusion with Poisson noise.

Evidential transformer for pavement distress segmentation

AbstractDistress segmentation assigns each pixel of a pavement image to one distress class or background, which provides a simplified representation for distress detection and measurement. Even though remarkably benefiting from deep learning, distress segmentation still faces the problems of poor calibration and multimodel fusion. This study has proposed a deep neural network by combining the Dempster–Shafer theory (DST) and a transformer network for pavement distress segmentation. The network, called the evidential segmentation transformer, uses its transformer backbone to obtain pixel‐wise features from input images. The features are then converted into pixel‐wise mass functions by a DST‐based evidence layer. The pixel‐wise masses are utilized for performing distress segmentation based on the pignistic criterion. The proposed network is iteratively trained by a new learning strategy, which represents uncertain information of ambiguous pixels by mass functions. In addition, an evidential fusion strategy is proposed to fuse heterogeneous transformers with different distress classes. Experiments using three public data sets (Pavementscape, Crack500, and CrackDataset) show that the proposed networks achieve state‐of‐the‐art accuracy and calibration on distress segmentation, which allows for measuring the distress shapes more accurately and stably. The proposed fusion strategy combines heterogeneous transformers while remaining a performance not less than those of the individual networks on their respective data sets. Thus, the fusion strategy makes it possible to use the existing networks to build a more general and accurate one for distress segmentation.

Multimodal Medical Image Fusion: The Perspective of Deep Learning

Multimodal medical image fusion involves the integration of medical images originating from distinct modalities and captured by various sensors, with the aim to enhance image quality, minimize redundant information, and preserve specific features, ultimately leading to increased efficiency and accuracy in clinical diagnoses. In recent years, the emergence of deep learning techniques has propelled significant advancements in image fusion, addressing the limitations of conventional methods that necessitate manual design of activity level measurement and fusion rules. This paper initially presents a systematic description of the multimodal medical image fusion problem, delineating the interrelationships between different fusion modalities while summarizing their characteristics and functions. Subsequently, it reviews the theories and enhancement approaches associated with deep learning in the medical image fusion domain, striving for a comprehensive overview of the state-of-the-art developments in this field from a deep learning perspective. These developments encompass multimodal feature extraction methods based on convolutional techniques, adversarial learning-based methods, convolutional sparse representation and stacked autoencoder-based signal processing methods, and unified models. Lastly, the paper summarizes the enhancement techniques for multimodal medical image fusion methods, highlighting the pressing issues and challenges encountered by deep learning approaches in this domain.

UEFSD: Unsupervised medical images fusion based on exclusive features and saliency detection for SPECT-MRI images of glioma

Background and ObjectiveTo address the key feature extraction problem in glioma image fusion due to blurred edge and texture features, with certain exclusive features entangled with noise, we propose an unsupervised fusion network based on exclusive feature unentanglement and saliency detection (UEFSD). Single-photon emission computed tomography (SPECT) and magnetic resonance imaging (MRI) are taken as examples. MethodsFirst, the SPECT image in RGB space is converted to YCbCr space. Then, Y-channel SPECT and MRI images are fed into the UEFSD-based network and the attribute and object encoders are respectively used to separate the attribute vector of exclusive information for each modality and object feature mapping based on this attribute vector. Subsequently, the attribute vector and object feature maps are fused through the fusion decoder to obtain the fused image. In the training phase, first, the image is reconstructed using the attribute vector and object feature maps of the other modalities. The feature displaying the largest difference between the source and reconstructed images is identified as the exclusive feature that cannot be obtained via reconstruction. We trained the encoders and decoders in an unsupervised manner using exclusive feature loss as a high-level constraint and introducing a saliency mask as a low-level constraint. The trained attribute encoders and object encoders feed information to the fusion decoder to generate fused images. ResultUEFSD-based network was trained on Harvard Medical School's public dataset and tested by fusing MRI-T2 glioma images with their SPECT-T1 and SPECT-Tc counterparts. The experimental results demonstrate that UEFSD performs optimally in both subjective and objective indices when compared with nine other existing networks. ConclusionThe proposed UEFSD-based network can obtain clear edge and texture glioma features, which can effectively improve the rich and natural colour information of medical images and has the potential for computer-aided diagnosis system application.