Final Fusion Result Research Articles

Infrared and Visible Image Fusion in a Multilevel Low-Rank Decomposition Framework Based on Guided Filtering and Feature Extraction ∗

A novel infrared and visible image fusion method in a multilevel low-rank decomposition framework based on guided filtering and feature extraction is proposed to address the lack of edge information and blurred details in fused images. Based on multilevel low-rank decomposition, the fusion strategy of base part and detail contents has been improved. Firstly, the source infrared and visible images are decomposed to the base part coefficients and n -level detail content coefficients by multilevel low-rank decomposition. Secondly, the base part coefficients are learned by the VGG-19 network to get the weight map, and then, the improved weight map is obtained by guided filtering, and the coefficients of the base part are fused to acquire the fused base part coefficients. The n -level detail content coefficients are fused using the rule of dynamic level measurement with maximum value and then reconstructed to obtain the final fused detail content coefficients. Finally, the fused base part and detailed content information are superimposed to get the final fusion result. The results show that the fusion algorithm can effectively preserve the edge and detail features of the source image. Compared with other state-of-the-art fusion methods, the proposed method performs better in objective assessment and visual quality. The average value of evaluation metrics E N and M I have been improved by 0.5337 and 1.0673 on the six pair images.

Objectivity meets subjectivity: A subjective and objective feature fused neural network for emotion recognition

Using multimodal fusion method to deal with emotion recognition task has become a trend. The fusion vector can more comprehensively reflect the subject’s emotional change state, so as to obtain a more accurate emotion recognition effect. However, different fusion input or feature fusion methods have different effects on the final fusion results. In this paper, we propose a subjective and objective feature fused neural network model (SOFNN) for emotion recognition, which can effectively learn spatial–temporal information from EEG signals and dynamically integrate EEG signals with eye movement signals. Specifically, we extract more abundant spatial and temporal information from the original EEG signal through a series of 1-D convolution kernels of different sizes and we verify the effectiveness of the extracted features through experiments. The size of the 1-D convolution kernels is determined by the characteristics (such as sampling rate and number of channels) of the original EEG signal. Then, we design a subjective and objective feature fusion framework to adjust the proportion of the two features through the dynamic learning of the weight vector, so as to fully exploit their respective advantages. We evaluate the performance of our model on the SEED-IV dataset, which is a common dataset. For the recognition task of four emotions (happy, sad, fear and neutral), our model achieves 86.27% accuracy and 10.16% standard deviation, which are better than the existing methods. In addition, we design a variety of ablation experiments to verify the effectiveness of each module in our model. The experiment results show that our model can make better use of the complementary relationship between subjective and objective features, which can achieve better emotion recognition effect.

Infrared and visible image fusion based on visibility enhancement and hybrid multiscale decomposition

Infrared and visible image fusion technology aims to integrate the heat source information of infrared image into the visible image to generate a more informative image. Many fusion methods proposed in recent years have problems such as loss of detailed information and low contrast. In this paper, we propose a novel infrared and visible image fusion method based on visibility enhancement and hybrid multiscale decomposition. Firstly, we propose an effective image pre-processing method to increase the details and improve the quality of the source image. Then, the pre-processed images are decomposed by ℓ1−ℓ0 decomposition model to obtain the base and detail layers. Thirdly, for the base layer fusion part, we propose a meaningful method based on the weight of the visual saliency illumination map (VSIM), which not only preserves the contrast information but also does it guarantee the overall structure of the fusion result. For the detail layer fusion part, we take the advantage of convolutional neural network (CNN) to obtain the decision map of the fused detail layer. Next, we employ Laplacian and Gaussian pyramids to decompose the detail layers and decision map, respectively and then fuse them by a synthetic detail fusion strategy. Finally, we reconstruct the base and detail layers to generate the final fusion result. Experiment results demonstrate that our results are more in line with the human visual system (HVS) and outperform some state-of-the-art methods on quantitative metrics.

Learning Spectral Cues for Multispectral and Panchromatic Image Fusion.

Recently, deep learning based multispectral (MS) and panchromatic (PAN) image fusion methods have been proposed, which extracted features automatically and hierarchically by a series of non-linear transformations to model the complicated imaging discrepancy. But they always pay more attention to the extraction and compensation of spatial details and use the mean squared error or mean absolute error as a loss function, regardless of the preservation of spectral information contained in multispectral images. For the sake of the improvements in both spatial and spectral resolution, this paper presents a novel fusion model that takes the spectral preservation into consideration, and learns the spectral cues from the process of generating a spectrally refined multispectral image, which is constrained by a spectral loss between the generated image and the reference image. Then these spectral cues are used to modulate the PAN features to obtain final fusion result. Experimental results on reduced-resolution and full-resolution datasets demonstrate that the proposed method can obtain a better fusion result in terms of visual inspection and evaluation indices when compared with current state-of-the-art methods.

Infrared and visible image fusion based on two-scale decomposition and improved saliency detection

Aiming at the problems of poor target saliency, loss of background information and time-consuming in image fusion, a fast image fusion algorithm combining two-scale decomposition and improved saliency detection is proposed. Mean filtering is used to decompose the source image into a base layer and a detail layer. The maximum symmetric surround (MSS) saliency detection algorithm is improved to obtain the dim suppressed MSS algorithm. dim suppressed MSS saliency detection and guided filtering is used to generate fusion rules for each layer. The inverse transformation of two-scale decomposition is used for the fusion sub-image of the base layer and the detail layer to obtain the final fusion result. Experimental results show that the algorithm consumes less time and has better fusion quality, which reflects the feasibility of the proposed algorithm.

Decision fusion method for fault diagnosis based on closeness and Dempster-Shafer theory

Decision fusion is an effective way to resolve the conflict of diagnosis results. Aiming at the problem that Dempster-Shafer (DS) theory deals with the high conflict of evidence and produces wrong results, a decision fusion algorithm for fault diagnosis based on closeness and DS theory is proposed. Firstly, the relevant concepts of DS theory are introduced, and the normal distribution membership function is used as the evidence closeness. Secondly, the harmonic average is introduced, and the weight of each evidence is established according to the product of closeness of each evidence and its harmonic average. Thirdly, the weight of conflicting evidence is regularized, and the final decision fusion result is obtained by using the Dempster’s rule. Lastly, the simulation and application examples are designed. Simulation and application results show that the method can effectively reduce the impact of diagnostic information conflicts and improve the accuracy of decision fusion. What’s more, the method considers the overall average distribution of evidence in the identification framework, it can reduce evidence conflicts while preserving important diagnostic information.

A New Method to Handle Conflict when Combining Evidences Using Entropy Function and Evidence Angle with an Effective Application in Fault Diagnosis

Addressing the problem of fusing highly conflicting evidences in Dempster–Shafer theory is one of the most necessary, important, and difficult research directions all the time, and so far we have published two papers related to it. In this paper, another novel method to handle conflict when combining evidences is proposed, where evidence distance, evidence angle, and improved entropy function, three key tools, are used for constructing the final weight of each body of evidence. This newly proposed approach mainly consists of three steps: firstly, both evidence distance and evidence angle determine the initial weight together; secondly, making use of the improved entropy modifies the initial weight to get the final weight; lastly, the classical D-S combination rule will be applied to obtain final fusion results. Still a classical numeric example and a real fault diagnosis application both demonstrate its effectiveness and efficiency, and compared with other current popular methods including two of our previous works, this new approach can converge fast and reduce most uncertainty of decision-making when fusing highly conflicting evidences.

Optimization Algorithm of Bearings Only Tracking based on Cubature Kalman Filter

In view of the shortcomings of low precision and poor performance in the process of bearings only tracking (BOT) in wireless sensor networks(WSNs), an optimization algorithm of BOT based on the cubature Kalman filter(CKF) in WSNs is proposed. The sensors with spatial distribution are divided into several clusters, the number of sensors in each cluster is the same, and each sensor detects data to form a local estimation. Then uploads it to the cluster head (CH) node data fusion, so as to get the accurate information of the target. In order to improve the positioning accuracy, the CKF is used to process the data of each sensor, and then it is uploaded to the CH node to ensure the tracking ability of the filter to the changes of structural parameters. The CH node uses weighted data fusion algorithm to get the final fusion result. The simulation results verify the effectiveness of the method.

Infrared and visible image fusion via global variable consensus

In this paper, we propose an infrared and visible image fusion framework based on the consensus problem. Most current infrared and visible image fusion models aim to transfer only one characteristic of each source domain to the final fusion result. This mechanism limits the performances of fusion algorithms under different conditions. We present a general fusion framework based to solve the global variable consensus optimization problem through altering direction method of multipliers (ADMM). We identified that combination of the local operators allows smooth transfer of superficial characteristics of the source domain into the fusion result. Our modification of ADMM enables us to expand the fusion algorithm's compatibility by tackling various setting including dimensionality, data types and style. The qualitative and quantitative experiment results demonstrate that, compared with other state-of-the-art algorithms, the proposed method can provide competitive performance in transferring features, structures, and information from source images to fusion results.

Color-compensated multi-scale exposure fusion based on physical features

The dynamic range captured by ordinary digital cameras is much smaller than that in natural scenes, so a single image cannot fully capture the real scene details when shooting high dynamic scenes. Traditional multi-exposure image fusion algorithms often cause inconsistencies in local brightness and corresponding scenes, which easily leads to problems such as color distortion and loss of details. Based on this, a novel multi-scale exposure fusion approach based on physical features is presented in this paper. Specifically, we first employ a novel Retinex model to get the illumination maps of original images and adopt the obtained illuminations map to construct the exposure maps. Then, a feature descriptor named patterns of oriented edge magnitudes (POEM) is introduced to extract local contrast from source images. Moreover, the extracted feature is combined to construct a weight map, and the weight term is integrated with the Laplacian pyramid algorithm to obtain an initial fusion image. Finally, simple detail enhancement and color compensation operations are performed to get the final fusion result. The numerous experiments have revealed that the proposed approach yields comparative and even better results in comparison with some state-of-the-art techniques both in subjective and objective evaluation.

U-GAN Model for Infrared and Visible Images Fusion

Infrared and visible image fusion is an effective method to solve the lack of single sensor imaging. The purpose is that the fusion images are suitable for human eyes and conducive to the next application and processing. In order to solve the problems of incomplete feature extraction, loss of details, and less samples of common data sets, it is not conducive to training, an end-to-end network architecture for image fusion is proposed. U-net is introduced into image fusion, and the final fusion result is obtained by using the generative adversarial network. Through its special convolution structure, the important feature information is extracted to the maximum extent, and the sample does not need to be cut to avoid the problem of reducing the fusion accuracy, but also to improve the training speed. Then the U-net extracted feature is confronted with the discriminator containing infrared image, and the generator model is obtained. The experimental results show that the present algorithm can obtain the fusion image with clear outline, prominent texture and obvious target. SD, SF, SSIM, AG and other indicators are obviously improved.

A new patch-based two-scale decomposition for infrared and visible image fusion

Typically, in existing two-scale fusion method, source images are first processed by a filter to obtain two layers, including a base layer and a detail layer. Second, a specific fusion strategy is applied to fuse these layers. Herein, the fusion of the base layer influences the contrast and clearness of final fusion result because the base layer may contain large scale variations in intensity. However, due to the inherent limitation of the base layer based on the filtering decomposition, the contrast of fusion result is easily reduced, or even the detail of the source images, in the fusion result, may be blurred. In this paper, a new two-scale fusion approach is proposed. In the process of two-scale decomposition, a new patch-based low-rank representation is proposed to obtain the two layers of the source images. In the proposed decomposition approach, the expression of the base layer is converted from a single filtering decomposition to a transform domain. It consists of a dictionary and corresponding coefficients. In the process of fusion, we design a global statistical fusion rule based on the coefficient value to achieve the base layer fusion, which improves the contrast of the final fusion result. For detail layer fusion, the L1-norm strategy is chosen to integrate small scale features contained in the detail layer into the final fusion result, which can contain clear detail. Experimental results illustrate that the proposed framework outperforms traditional approaches on the public database.

An intelligent fault diagnosis approach based on Dempster-Shafer theory for hydraulic valves

Detecting faults in hydraulic valves are of great significance to improve the reliability and security of the whole hydraulic system. However, it is difficult to detect multiple faults in hydraulic valves using existing approaches due to closed structural components and complex hydraulic system itself. Therefore, an intelligent fault diagnosis approach based on Dempster-Shafer (DS) theory is proposed specifically for detecting several faults occurred in hydraulic valves. Actually, it is classified in the ensemble learning in terms of the information fusion theory. In this approach, signal segments containing fault information are selected to structure sample sets firstly. Then sample sets are simultaneously fed into the single classifier including long short-term memory networks (LSTM), convolutional neural network (CNN) and random forests (RF). Through learning spontaneously in these intelligent classification approaches, fault features are concluded and the probability of each type fault is respectively revealed. All probabilities are constructed as basic probability assignment (BPA) functions, which are further calculated in the information fusion process in terms of DS theory. Finally, the fault types are identified by the final fusion results. Experimental investigations are performed to validate performance of the present approach (taken a solenoid controlled pilot operated directional valve as an example). It is shown that the average accuracy ratio of proposed intelligent fault diagnosis approach is 98.5% for six fault types detection. The study does provide an effective access to detect faults in hydraulic valves.

Rolling Guidance Filtering-Orientated Saliency Region Extraction Method for Visible and Infrared Images Fusion

Different image types can be obtained by different sensors, but all the useful information cannot be extracted from a single image. Infrared images can capture the heat source information of scene targets in low light or severe weather conditions. Visible images provide more detail information about the scene. To obtain the rich image information, we propose a visible and infrared images fusion method based on rolling guidance filtering and saliency region extraction in this paper. A multi-scale image decomposition framework is built by using the edge preserving-smoothing algorithm. The image is decomposed into one base layer with different scales and several detail layers. Meanwhile, the saliency region extraction is implemented on each decomposition layer by combining with the rolling guidance filtering. Weight reconstruction is adopted to obtain the final fusion result. The results show that the proposed algorithm has good subjective and objective evaluation results, better fusion performance and robustness compared to other state-of-the-art fusion methods.

Weighted Conflict Evidence Combination Method Based on Hellinger Distance and the Belief Entropy

In the Dempster-Shafer evidence theory, how to effectively measure the degree of conflict between two bodies of evidence is still an open question. To solve this problem, we propose a weighted conflict evidence combination method based on Hellinger distance and the belief entropy. This method uses the probability transformation function to deal with the multi-subset focal elements firstly. Next, the Hellinger distance is introduced to measure the degree of conflict among the evidence. Moreover, improved belief entropy is also employed to quantify the uncertainty of the basic belief assignments. Further, Hellinger distance and the improved belief entropy are combined to construct the weight coefficient concerning evidence, and finally, the Dempster combination rule is used for fusion. The final fusion results of proposed method on fault diagnosis experiment and target recognition experiment are 0.9018 and 0.9895 respectively, apparently higher than that of other methods, revealing the advantages of the proposed method.

Multiple GNSS inter-system biases in precise time transfer

Inter-system bias (ISB) is important for the fusion of multiple global navigation satellite system data. They should be estimated or corrected for precise position, navigation, and timing solutions. In this study, ISB estimation models are developed for the global positioning system, GLONASS, BeiDou Navigation Satellite System, and Galileo systems. Six stations, with three different types of receivers, are used to analyze the ISB characteristics of different systems. A pair of time-link stations, which have a short baseline and are connected to a common external time frequency from the laboratory, are used to analyze the performance of three different estimation strategies (epoch-wise variable, 30 min piece-wise constant, and daily constant). These stations are also used to evaluate the influence of ISB constrains on the precise estimation of time transfer, coordinates, and the troposphere. The results illustrate: (a) ISBs variation exists between different systems, and the same types of receivers also exhibit large differences between the different stations due to the receiver hardware delay; (b) the precision of the different measurements and their related errors, the estimated ISBs that exhibit large differences between pseudorange and carrier phase observations, and the different ISB estimation strategies can also cause differences in the final fusion results; and (c) using the estimated ISB as constraints can improve the precise estimation of time transfer, coordinates, and the troposphere in the real-time solution mode.

A New Method to Improve Fault Location Accuracy in Transmission Line Based on Fuzzy Multi-Sensor Data Fusion

Fast and accurate fault location in transmission lines has great significance for power system stability and power supply reliability. The location results of the various algorithms are diverse. The conflicting results influence the operator’s analysis on fault-based quick restoration of power systems. This paper proposes a new method to improve fault location accuracy in transmission lines, relying on the multi-sources fault location results from distance relays of two ends. The location results from two ends and a fault position are related to each other. According to these relationships, a multi-sensor data fusion-based transmission line fault location model is built. In the model, fuzzy inference systems are formed and serve as the initial fusion step to identify fault scenarios. Weighted covariance fusion coefficients are calculated and used to determine the final fusion result. The case study is carried out using PSCAD/EMTDC and field recording data from substations. The performance of the method is tested for different fault types, fault positions, fault resistances, fault inception angles, load flow, line parameters and two-terminal unsynchronized angles.

Multi‐sensor medical image fusion using pyramid‐based DWT: a multi‐resolution approach

Generally, image fusions are carried out on two-dimensional (2D) images, but the operations with the 2D images increase the computational complexity than vector data. This study presents an algorithm with Laplacian pyramid built on 1D discrete wavelet transform (DWT) called modified multi-resolution DWT (MMDWT) for multi-sensor medical image fusion which was found to be efficient for n -level decomposition and can work with all mother wavelets with less computational complexity. The MMDWT methodology is compared with the discrete cosine transform, DWT, stationary WT, curvelet transform, principal component analysis, fuzzy and neurofuzzy technique and the performance measure is analysed. The performance evaluation of the MMDWT technique is illustrated using several sets of medical images provided by Health Care Global Enterprises Ltd. (HCG) Hospital Bangalore, based on the subjective and objective analyses. The result is validated by radiologists from HCG for subjective evaluation. The outcome of the MMDWT methodology is analysed with existing fusion algorithms and reveals the supremacy of the final fusion results.

Time Series Data Fusion Based on Evidence Theory and OWA Operator.

Time series data fusion is important in real applications such as target recognition based on sensors’ information. The existing credibility decay model (CDM) is not efficient in the situation when the time interval between data from sensors is too long. To address this issue, a new method based on the ordered weighted aggregation operator (OWA) is presented in this paper. With the improvement to use the Q function in the OWA, the effect of time interval on the final fusion result is decreased. The application in target recognition based on time series data fusion illustrates the efficiency of the new method. The proposed method has promising aspects in time series data fusion.

Novel Data Fusion Algorithm Based on Event-Driven and Dempster–Shafer Evidence Theory

When a new event occurs, the nodes in the neighborhood of the event sense and then send many packets to the sink node. Such circumstances need their networks to be simultaneously reliable and event-driven. Moreover, it should be remove redundant packets in order to lower the average energy consumption. A data fusion algorithm based on event-driven and Dempster–Shafer evidence theory is proposed in this paper to reduce data packet quantities and reserve energy for wireless sensor networks upon detecting abnormal data. Sampling data is compared against the set threshold, and the nodes enter the relevant state only when there are abnormal datum; at this point, cluster formation begins. All cluster members incorporate a local forwarding history to decide whether to forward or to drop recent sampling data. Dempster–Shafer evidence theory is exploited to process the data. The basic belief assignment function, with which the output of each cluster member is characterized as a weighted-evidence, is constructed. Then, the synthetic rule is subsequently applied to each cluster head to fuse the evidences gathered from cluster member nodes to obtain the final fusion result. Simulation results demonstrate that the proposed algorithm can effectively ensure fusion result accuracy while saving energy.