Fusion Metric Research Articles

Rethinking the Effectiveness of Objective Evaluation Metrics in Multi-Focus Image Fusion: A Statistic-Based Approach.

As an effective technique to extend the depth-of-field (DOF) of optical lenses, multi-focus image fusion has recently become an active topic in image processing community. However, a major problem remaining unsolved in this field is the lack of universal criteria in selecting objective evaluation metrics. Consequently, the metrics utilized in different studies often vary significantly, leading to high difficulties in achieving unbiased evaluation. To address this problem, this paper proposes a statistic-based approach for verifying the effectiveness of objective metrics in multi-focus image fusion. The core idea is to adopt statistical correlation measures to evaluate the performance consistency between a certain fusion metric and some popular full-reference image quality assessment models. In addition, a convolutional neural network (CNN)-based fusion metric is presented to measure the similarity between the source images and the fused image based on the semantic features at multiple abstraction levels. A comparative study is conducted to evaluate 20 existing fusion metrics using the proposed statistic-based approach on a large-scale, realistic and with-ground-truth multi-focus image fusion dataset recently released. Experimental results demonstrate the feasibility of the proposed approach in evaluating the effectiveness of objective metrics and the advantage of our CNN-based metric.

ADQE: Obtain Better Deep Learning Models by Evaluating the Augmented Data Quality Using Information Entropy

Data augmentation, as a common technique in deep learning training, is primarily used to mitigate overfitting problems, especially with small-scale datasets. However, it is difficult for us to evaluate whether the augmented dataset truly benefits the performance of the model. If the training model is relied upon in each case to validate the quality of the data augmentation and the dataset, it will take a lot of time and resources. This article proposes a simple and practical approach to evaluate the quality of data augmentation for image classification tasks, enriching the theoretical research on data augmentation quality evaluation. Based on the information entropy, multiple dimensional metrics for data quality augmentation are established, including diversity, class balance, and task relevance. Additionally, a comprehensive data augmentation quality fusion metric is proposed. Experimental results on the CIFAR-10 and CUB-200 datasets show that our method maintains optimal performance in a variety of scenarios. The cosine similarity between the score of our method and the precision of model is up to 99.9%. A rigorous evaluation of data augmentation quality is necessary to guide the improvement of DL model performance. The quality standards and evaluation defined in this article can be utilized by researchers to train high-performance DL models in situations where data are limited.

MCA-Based Multi-Channel Fusion Attacks Against Cryptographic Implementations

Multi-channel fusion attacks (MCFAs) are a more effective kind of side-channel analysis approaches for evaluating the side-channel vulnerability of a cryptographic device as compared to side-channel attacks exploiting only mono-channel leakage. However, research for MCFAs is insufficient to explore their greatest potential. Considering that multi-channel leakages are generated by the same crypto device and collected individually, the paper holds the opinion that there exists a linear combination of the multi-channel leakages or leakage features or attack results to launch a fusion attack with the highest performance. Correspondingly, this paper develops three optimal strategies of MCFAs on the basis of multiple correlation analysis (MCA). In addition, a new MCA-based leakage fusion metric applied for measuring whether or not to combine a channel with other channels for fusion attacks is constructed as well. The proposed metric can select the channels with similar noise levels for combination and reduce the possibility of negative influences on leakage fusion caused by high-noise channels. Effectiveness of the proposed optimal fusion attacks and the new fusion metric are verified by practical experiments in typical scenarios, respectively.

Local linear embedding algorithm of mutual neighborhood based on multi-information fusion metric

Local linear embedding (LLE) algorithm is an effective tool, which mines low-dimensional features in high-dimensional space. However, the local region and inner structure directly affect the performance of the LLE algorithm. To address this problem, the LLE algorithm of mutual neighborhood by employing multi-information fusion metric (MIFM-MNLLE) is proposed. First, the Euclidean distance and cosine similarity method are combined to evaluate the similary among samples, by which the accuracy of selected neighbors can be improved. Subsequently, the idea of mutual neighbor structure is utilized to construct the neighbor and the mutual neighbor graph of the sample to describe the internal structure of the data set. Finally, the coefficient of LLE is rectified according to the mutual neighborhood relationship between the sample and its neighbors, so as to extract significant features effectively. Extensive experimental results show that the proposed method has better performance compared with the existing methods on bearing datasets.

Statistical Goodness Factor ‘ᴦ’ for Image Fusion Algorithm Based on UGGD Parameters

In this paper we propose a novel pyramid decomposition based Image fusion metric, Gamma Factor or Goodness of Fit ‘ᴦ’ which describes the statistically amount of information fused by the image fusion algorithm. We first apply steerable pyramid decomposition and then a fitting model for Univariate Generalised Gaussian Distribution (UGGD) parameter estimation. From the UGGD; P and S fitting model coefficients are computed. To estimate the optimum weights for computation a huge data set of complimentary images are used. Using these weights, amount of information contributed by each image to form a fused image can be estimated. Experimental results show the tremendous matching with the quantise information

A Novel De-Ghosting Image Fusion Technique for Multi-Exposure, Multi-Focus Images Using Guided Image Filtering

In this paper, a novel de-ghosting image fusion technique is presented, which enhances the quality of low dynamic range images using multi-level exposures taken from the ordinary camera and also removes the ghosting artifact. In the proposed algorithm, first, the source images, taken under different exposure settings, are decomposed into base and detail layers using two-scale decomposition. The base and detail layers contain small and large-scale variation details of the source images, respectively. The Laplacian-of-Gaussian filter is applied to the source images to get the edge information. Afterward, the saliency map of the edges is computed. To remove the ghosting artifacts, a weight matrix is calculated by applying the median filter on the histogram equalized source images. The weight matrix is combined with the saliency map to generate more accurate weights. The separate weights for the base and detail layers are calculated using guided image filters. Finally, the base and detail layers’ weights are fused with the source images to generate a vivid and enhanced image without any artifacts. The proposed technique is evaluated both qualitatively and quantitatively. The comparison of our technique in terms of Yang’s Metric ( $Q_{Y}$ ), Quality Mutual Information ( $Q_{MI}$ ), Gradient-based Fusion Metric ( $Q_{G}$ ) and Chen Blum’s Metric ( $Q_{CB}$ ) with other state-of-the-art techniques proves that the proposed technique outperforms existing techniques.

Analysis of the Emerging AOMedia AV1 Video Coding Format for OTT Use-Cases

To achieve improved quality of service and efficiencies with higher resolution, high dynamic range, and wide color gamut in online media delivery, there is a need for advanced video compression standards. Alliance for Open Media (AOM), a joint development foundation, is targeting AOMedia Video 1 (AV1) as a royalty-free video coding format that is likely to be finalized before the end of 2017. Built on top of Google’s VP9 codec, AV1 has brought in new coding tools from other open source royalty-free codecs. Multiple new royalty-free coding tools have been contributed by the members of AOM. In this paper, we provide an analysis of the coding gains offered by the default and experimental tools in AV1 and the corresponding decoding complexity increase for the over-the-top adaptive bit-rate streaming delivery use-cases. Also, the coding gains are compared against the compression performance of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$ </tex-math></inline-formula> 264, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$ </tex-math></inline-formula> 265, and libVP9 open-source codecs to highlight the potential bit savings possible with AV1 using multiple video quality metrics such as peak signal-to-noise ratio, structure similarity, and Netflix’s Video Multi-Assessment Fusion metric. These results indicate that AV1 offers competitive compression performance over H.265 without significantly increasing the decoding complexity.

A Novel Image Fusion Metric for Intelligent Manufacturing Information System

An image fusion metric is commonly used to evaluate a fusion scheme because subjective evaluation cannot work in an intelligent manufacturing information system. In this study, an objective image fusion metric based on a log-Gabor filter (LGIMF) is presented. This metric can be calculated in five steps: (1) filtering the source and fused images into sub-bands, (2) constructing an ideal synthesis image by applying the maximization principle from the sub-band of the source images, (3) capturing the variation information between the real fused image and the ideal synthesis image in each sub-band, (4) measuring the sub-band fusion visual information by using the signalto- noise ratio model, and (5) weighting the sub-band fusion visual information to determine the overall quality. In our experiment, the proposed fusion metric is compared with other well-known metrics by using a subjective test dataset. We found that the LGIMF was more consistent in subjective perception compared with the other metrics.

Multi-Channel Fusion Attacks

Side channel attacks (SCAs) are a kind of the most powerful means to evaluate the physical security of a crypto device. Multi-channel fusion attacks (MCFAs) belong in SCAs and utilize multi-channel leakages simultaneously. As compared with the known mono-channel attacks, MCFAs have higher potential leakage utilization. However, previous research about MCFAs is scarce. MCFAs have not been studied systematically and classified explicitly. It is hard to select MCFAs strategies properly and perform MCFAs efficiently according to the existing research. In light of this, we classify MCFAs into three groups from the view of fusion for the first time, including data-level, feature-level, and decision-level fusion attacks. Accordingly, we construct six MCFAs and verify their effectiveness in typical scenarios. The applicable scopes and the different performances of MCFAs with different fusion activities are also first discussed. To the best of our knowledge, this paper is the first to systematically investigate MCFAs. We hope that this paper will be helpful to understand MCFAs and offer advice on MCFAs against the different implementations of a crypto algorithm. Besides, a fusion metric that determines which channels are suitable for combination is also proposed and verified by practical experiments.

Weber-aware weighted mutual information evaluation for infrared–visible image fusion

A performance metric for infrared and visible image fusion is proposed based on Weber’s law. To indicate the stimulus of source images, two Weber components are provided. One is differential excitation to reflect the spectral signal of visible and infrared images, and the other is orientation to capture the scene structure feature. By comparing the corresponding Weber component in infrared and visible images, the source pixels can be marked with different dominant properties in intensity or structure. If the pixels have the same dominant property label, the pixels are grouped to calculate the mutual information (MI) on the corresponding Weber components between dominant source and fused images. Then, the final fusion metric is obtained via weighting the group-wise MI values according to the number of pixels in different groups. Experimental results demonstrate that the proposed metric performs well on popular image fusion cases and outperforms other image fusion metrics.

Multimodal gray image fusion metric based on complex wavelet structural similarity

A novel multimodal gray image fusion metric is proposed based on complex wavelet structural similarity (CW-SSIM). The idea behind is that consistent phase changes with the wavelet coefficients corresponding to certain distortions in the image. Comparing with other common objective fusion metrics, the proposed metric correspond better to the subjective evaluation in the experiments.

Spatial Quality Assessment of Pan-Sharpened High Resolution Satellite Imagery Based on an Automatically Estimated Edge Based Metric

Most of the existing pan-sharpening quality assessment methods consider only the spectral quality and there are just few investigations, which concentrate on spatial characteristics. Spatial quality of pan-sharpened images is vital in elaborating the capability of object extraction, identification, or reconstruction, especially regarding man-made objects and their application for large scale mapping in urban areas. This paper presents an Edge based image Fusion Metric (EFM) for spatial quality evaluation of pan-sharpening in high resolution satellite imagery. Considering Modulation Transfer Function (MTF) as a precise measurement of edge response, MTFs of pan-sharpened images are assessed and compared to those obtained from the original multispectral or panchromatic images. Spatial quality assessment of pan-sharpening is done by comparison of MTF curves of the pan-sharpened and reference images. The capability of the proposed method is evaluated by quality assessment of two different residential and industrial urban areas of WorldView-2 pan-sharpened images. Obtained results clearly show the wide spatial discrepancy in quality of Pan-sharpened images, resulting from different fusion methods, and confirm the need for spatial quality assessment of fused products. The results also prove the capability of the proposed EFM as a powerful tool for evaluation and comparison of different image fusion techniques and products.

Image Fusion Metric Based on Visual Saliency Weighted Structural Similarity

In this paper, a novel non-reference objective quality metric for night vision image fusion is presented. The metric extracts visual salient regions of the source images based on the mechanism of human visual attention, calculates the visual saliency weights with consideration of the physical characteristics of the input images, and evaluates the quality of the fused image with a visual saliency weighted structural similarity. Experimental results show that the proposed approach is consistent with human visual inspection better than recent state of the art image fusion evaluation methods, and can be used to assess the fusion schemes that are not performed at the same level.

Novel Fused Image Quality Measures Based on Structural Similarity

Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we present new approaches using structural similarity (SSIM) index for assessing grayscale fused image quality and color fusion method quality in image fusion. The advantages of grayscale fused image quality measure using SSIM are that the metric does not require a reference image and can be easily computed. The advantages of color fusion metric using SSIM are that it correlates strongly with human perception and can also be easily computed. Numerous simulations demonstrate that our measures conform to subjective evaluations and can be able to assess different image fusion methods.

Video fusion performance evaluation based on structural similarity and human visual perception

In order to evaluate different video fusion algorithms in temporal stability and consistency as well as in spatial information transfer, a novel objective video fusion quality metric is proposed with the structural similarity (SSIM) index and the perception characteristics of human visual system (HVS) in this paper. Firstly, for each frame, two sub-indices, i.e., the spatial fusion quality index and the temporal fusion quality index, are defined by the weighted local SSIM indices. Secondly, for the current frame, an individual-frame fusion quality measure is obtained by integrating the above two sub-indices. Lastly, the proposed global video fusion metric is constructed as the weighted average of all the individual-frame fusion quality measures. In addition, according to the perception characteristics of HVS, some local and global spatial–temporal information, such as local variance, pixel movement, global contrast, background motion and so on, is employed to define the weights in the proposed metric. Several sets of experimental results demonstrate that the proposed metric can evaluate different video fusion algorithms accurately, and the evaluation results coincide with the subjective results well.

A new image fusion performance metric based on visual information fidelity

Because subjective evaluation is not adequate for assessing work in an automatic system, using an objective image fusion performance metric is a common approach to evaluate the quality of different fusion schemes. In this paper, a multi-resolution image fusion metric using visual information fidelity (VIF) is presented to assess fusion performance objectively. This method has four stages: (1) Source and fused images are filtered and divided into blocks. (2) Visual information is evaluated with and without distortion information in each block. (3) The visual information fidelity for fusion (VIFF) of each sub-band is calculated. (4) The overall quality measure is determined by weighting the VIFF of each sub-band. In our experiment, the proposed fusion assessment method is compared with several existing fusion metrics using the subjective test dataset provided by Petrovic. We found that VIFF performs better in terms of both human perception matching and computational complexity.

FUSION OF SURVEILLANCE IMAGES IN INFRARED AND VISIBLE BAND USING CURVELET, WAVELET AND WAVELET PACKET TRANSFORM

This paper presents two methods for fusion of infrared (IR) and visible surveillance images. The first method combines Curvelet Transform (CT) with Discrete Wavelet Transform (DWT). As wavelets do not represent long edges well while curvelets are challenged with small features, our objective is to combine both to achieve better performance. The second approach uses Discrete Wavelet Packet Transform (DWPT), which provides multiresolution in high frequency band as well and hence helps in handling edges better. The performance of the proposed methods have been extensively tested for a number of multimodal surveillance images and compared with various existing transform domain fusion methods. Experimental results show that evaluation based on entropy, gradient, contrast etc., the criteria normally used, are not enough, as in some cases, these criteria are not consistent with the visual quality. It also demonstrates that the Petrovic and Xydeas image fusion metric is a more appropriate criterion for fusion of IR and visible images, as in all the tested fused images, visual quality agrees with the Petrovic and Xydeas metric evaluation. The analysis shows that there is significant increase in the quality of fused image, both visually and quantitatively. The major achievement of the proposed fusion methods is its reduced artifacts, one of the most desired feature for fusion used in surveillance applications.

A Nonreference Image Fusion Metric Based on the Regional Importance Measure

In this paper, we present a novel metric for evaluation of image fusion algorithms, based on evaluation of similarity of regions in images to be fused with the corresponding regions in the fused image. The metric uses several factors to quantify the importance of regions in each of the input images, such as contrast, size, and shape of region. The similarity of the corresponding regions in an input image and the fused image is measured using a wavelet-based mutual information measure. Experimental results show that the proposed metric's ranking of different image fusion methods is more consistent with the subjective quality of the fused image than the state-of-the-art image fusion metrics.

Adaptive Fusion of Multimodal Surveillance Image Sequences in Visual Sensor Networks

In this paper we present a novel method of fusing of the sequences of images obtained from multimodal surveillance cameras and subject to distortions typical for visual sensor networks environment. The proposed fusion method uses the structural similarity measure (SSIM) to measure a level of noise in regions of a received image in order to optimize the selection of regions in the fused image. The region-based image fusion algorithm using the dual-tree complex wavelet transform (DT-CWT) is used to fuse the selected regions. The performance of the proposed method was extensively tested for a number of multimodal surveillance image sequences and proposed method outperformed the state-of-the-art algorithms, increasing significantly the quality of the fused image, both visually and in terms of the Petrovic image fusion metric.

Region-Based Multimodal Image Fusion Using ICA Bases

In this paper, we present a novel multimodal image fusion algorithm in the independent component analysis (ICA) domain. Region-based fusion of ICA coefficients is implemented, where segmentation is performed in the spatial domain and ICA coefficients from separate regions are fused separately. The ICA coefficients from given regions are consequently weighted using the Piella fusion metric in order to maximize the quality of the fused image. The proposed method exhibits significantly higher performance than the basic ICA algorithm and also shows improvement over other state-of-the-art algorithms