Multi-exposure Image Fusion Research Articles

An Improved Multi-Exposure Image Fusion Method for Intelligent Transportation System

In this paper, an improved multi-exposure image fusion method for intelligent transportation systems (ITS) is proposed. Further, a new multi-exposure image dataset for traffic signs, TrafficSign, is presented to verify the method. In the intelligent transportation system, as a type of important road information, traffic signs are fused by this method to obtain a fused image with moderate brightness and intact information. By estimating the degree of retention of different features in the source image, the fusion results have adaptive characteristics similar to that of the source image. Considering the weather factor and environmental noise, the source image is preprocessed by bilateral filtering and dehazing algorithm. Further, this paper uses adaptive optimization to improve the quality of the output image of the fusion model. The qualitative and quantitative experiments on the new dataset show that the multi-exposure image fusion algorithm proposed in this paper is effective and practical in the ITS.

Suspect Multifocus Image Fusion Based on Sparse Denoising Autoencoder Neural Network for Police Multimodal Big Data Analysis

In recent years, the success rate of solving major criminal cases through big data has been greatly improved. The analysis of multimodal big data plays a key role in the detection of suspects. However, the traditional multiexposure image fusion methods have low efficiency and are largely time-consuming due to the artifact effect in the image edge and other sensitive factors. Therefore, this paper focuses on the suspect multiexposure image fusion. The self-coding neural network based on deep learning has become a hotspot in the research of data dimension reduction, which can effectively eliminate the irrelevant and redundant learning data. In the case of limited field depth, due to the limited focusing depth of the camera, the focusing plane cannot obtain the global clear image of the target in the depth scene, which is prone to defocusing and blurring phenomena. Therefore, this paper proposes a multifocus image fusion based on a sparse denoising autoencoder neural network. To realize an unsupervised end-to-end fusion network, the sparse denoising autoencoder neural network is adopted to extract features and learn fusion rules and reconstruction rules simultaneously. The initial decision graph of the multifocus image is taken as a prior input to learn the rich detailed information of the image. The local strategy is added to the loss function to ensure that the image is restored accurately. The results show that this method is superior to the state-of-the-art fusion methods.

Strong ghost removal in multi-exposure image fusion using hole-filling with exposure congruency

It is the most crucial problem to remove ghost in the multi-exposure image fusion of dynamic scene. The traditional fusion methods have good effects to remove weak ghosts. However, they cannot effectively remove strong ghosts. This paper proposes a new strong ghost removal method in multi-exposure image fusion using hole-filling with exposure congruency. First, analyzing the characteristics of strong ghosts, a detection scheme for strong ghost regions is designed by combining histogram matching and exposure difference detection. Subsequently, to effectively extract image local features, a multi-scale fusion network for non-strong ghost regions is designed to obtain a pre-fused image. Further, based on the distribution characteristics of strong ghosts, a hole-filling model with exposure congruency is designed to remove the strong ghosts. Experimental results show that compared with the state-of-the-art methods, the proposed method can obtain better performance in both of subjective and objective evaluation, particularly in terms of effectively removing strong ghosts.

Superpixel-Based Quality Assessment of Multi-Exposure Image Fusion for Both Static and Dynamic Scenes.

Multi-exposure image fusion (MEF) algorithms have been used to merge a stack of low dynamic range images with various exposure levels into a well-perceived image. However, little work has been dedicated to predicting the visual quality of fused images. In this work, we propose a novel and efficient objective image quality assessment (IQA) model for MEF images of both static and dynamic scenes based on superpixels and an information theory adaptive pooling strategy. First, with the help of superpixels, we divide fused images into large- and small-changed regions using the structural inconsistency map between each exposure and fused images. Then, we compute the quality maps based on the Laplacian pyramid for large- and small-changed regions separately. Finally, an information theory induced adaptive pooling strategy is proposed to compute the perceptual quality of the fused image. Experimental results on three public databases of MEF images demonstrate the proposed model achieves promising performance and yields a relatively low computational complexity. Additionally, we also demonstrate the potential application for parameter tuning of MEF algorithms.

Large Dynamic Range Scatter Signal Acquisition Based on Multi-exposure Image Fusion

Large Dynamic Range Scatter Signal Acquisition Based on Multi-exposure Image Fusion

Deep Coupled Feedback Network for Joint Exposure Fusion and Image Super-Resolution.

Nowadays, people are getting used to taking photos to record their daily life, however, the photos are actually not consistent with the real natural scenes. The two main differences are that the photos tend to have low dynamic range (LDR) and low resolution (LR), due to the inherent imaging limitations of cameras. The multi-exposure image fusion (MEF) and image super-resolution (SR) are two widely-used techniques to address these two issues. However, they are usually treated as independent researches. In this paper, we propose a deep Coupled Feedback Network (CF-Net) to achieve MEF and SR simultaneously. Given a pair of extremely over-exposed and under-exposed LDR images with low-resolution, our CF-Net is able to generate an image with both high dynamic range (HDR) and high-resolution. Specifically, the CF-Net is composed of two coupled recursive sub-networks, with LR over-exposed and under-exposed images as inputs, respectively. Each sub-network consists of one feature extraction block (FEB), one super-resolution block (SRB) and several coupled feedback blocks (CFB). The FEB and SRB are to extract high-level features from the input LDR image, which are required to be helpful for resolution enhancement. The CFB is arranged after SRB, and its role is to absorb the learned features from the SRBs of the two sub-networks, so that it can produce a high-resolution HDR image. We have a series of CFBs in order to progressively refine the fused high-resolution HDR image. Extensive experimental results show that our CF-Net drastically outperforms other state-of-the-art methods in terms of both SR accuracy and fusion performance. The software code is available here https://github.com/ytZhang99/CF-Net.

UNIFusion: A Lightweight Unified Image Fusion Network

Advanced image fusion frameworks have focused on constructing a unified system to deliver general fusion solutions for all the potential subtasks, including visible and infrared image fusion, multi-focus image fusion, and multi-exposure image fusion. However, the fusion performance of these unified solutions cannot be guaranteed as the involved subtasks exhibit essential diversity. Besides, the huge volume of these models also sacrifices their flexibility in practical applications. To this end, we propose a lightweight unified fusion network to balance the multi-level information featured across different channels and different layers. In particular, We construct a novel network architecture based on the Ghost module endowed with a new loss function. The depth of the designed network is increased, while the number of parameters is reduced by an order of magnitude compared to existing learning-based fusion approaches. In principle, our network belongs to the autoencoder paradigm, comprising three parts: encoder, fusion layer, and decoder, respectively. Considering that the efficacy of existing autoencoder-based approaches is impeded by the single fusion strategy, we use the guided filtering approach to decompose the source image into a base layer and a detail layer to extend the diversity of the inputs. Drawing on this, we can design different fusion strategies in these two layers for adapting different image fusion tasks. In the fusion layer, besides the three basic fusion strategies, i.e., average, max, and spatial attention, an additional gradient perception strategy is proposed to handle the multi-focus image fusion problem, improving the corresponding fusion performance. Qualitative and quantitative experiments show that our UNIFusion achieves promising results in these image fusion tasks, compared to the state of the art.

Deep Gradual Multi-Exposure Fusion Via Recurrent Convolutional Network

The performance of multi-exposure image fusion (MEF) has been recently improved with deep learning techniques but there are still a couple of problems to be overcome. In this paper, we propose a novel MEF network based on recurrent neural network (RNN). Multi-exposure images have different useful information depending on their exposure levels, and in order to fuse them complementarily, we first extract the local detail and global context features of input source images, and both features are separately combined. A weight map is learned from the local features for effectively fusing according to the importance of each source image. Adopting RNN as a backbone network enables gradual fusion, where more inputs result in further improvement of the fusion gradually. Also, information can be transferred to the deeper level of the network. Experimental results show that the proposed method achieves the reduction of fusion artifacts and improves detail restoration performance, compared to conventional methods.

Multi-exposure image fusion based on image decomposition and color prior

Multi-exposure image fusion based on image decomposition and color prior

Ghost-Free Deep High-Dynamic-Range Imaging Using Focus Pixels for Complex Motion Scenes.

Multi-exposure image fusion inevitably causes ghost artifacts owing to inaccurate image registration. In this study, we propose a deep learning technique for the seamless fusion of multi-exposed low dynamic range (LDR) images using a focus-pixel sensor. For auto-focusing in mobile cameras, a focus-pixel sensor originally provides left (L) and right (R) luminance images simultaneously with a full-resolution RGB image. These L/R images are less saturated than the RGB images because they are summed up to be a normal pixel value in the RGB image of the focus pixel sensor. These two features of the focus pixel image, namely, relatively short exposure and perfect alignment are utilized in this study to provide fusion cues for high dynamic range (HDR) imaging. To minimize fusion artifacts, luminance and chrominance fusions are performed separately in two sub-nets. In a luminance recovery network, two heterogeneous images, the focus pixel image and the corresponding overexposed LDR image, are first fused by joint learning to produce an HDR luminance image. Subsequently, a chrominance network fuses the color components of the misaligned underexposed LDR input to obtain a 3-channel HDR image. Existing deep-neural-network-based HDR fusion methods fuse misaligned multi-exposed inputs directly. They suffer from visual artifacts that are observed mostly in saturated regions because pixel values are clipped out. Meanwhile, the proposed method reconstructs missing luminance with aligned unsaturated focus pixel image first, and thus, the luma-recovered image provides the cues for accurate color fusion. The experimental results show that the proposed method not only accurately restores fine details in saturated areas, but also produce ghost-free high-quality HDR images without pre-alignment.

Details-preserving multi-exposure image fusion based on dual-pyramid using improved exposure evaluation

Due to a limited dynamic range of widely used image recorders, it is difficult to record complete information on real scenes using a single image, and a restricted range of contrast, brightness and chromaticity can only be recorded. To mitigate this matter, a set of images of the identical scenery could be firstly captured at different exposure situations, and next be merged into an informative image via image fusion. In the paper, we present a well details-preserving image fusion technique via improved exposure evaluation and dual-pyramid model. The proposed method owning to its advantages of cost-effective, computation-stable and adaptive image processing can achieve a high dynamic range image from a set of multiple exposure sequences even for extremely complicated scenes. Experimental results demonstrated that the proposed method has richer details and better visual effects compared with the other commonly used techniques in most cases. Therefore, it could provide useful help and inspiration for image processing and enhancement field such as digital photography, remote sensing and medical imaging.

FuseVis: Interpreting Neural Networks for Image Fusion Using Per-Pixel Saliency Visualization

Image fusion helps in merging two or more images to construct a more informative single fused image. Recently, unsupervised learning-based convolutional neural networks (CNN) have been used for different types of image-fusion tasks such as medical image fusion, infrared-visible image fusion for autonomous driving as well as multi-focus and multi-exposure image fusion for satellite imagery. However, it is challenging to analyze the reliability of these CNNs for the image-fusion tasks since no groundtruth is available. This led to the use of a wide variety of model architectures and optimization functions yielding quite different fusion results. Additionally, due to the highly opaque nature of such neural networks, it is difficult to explain the internal mechanics behind its fusion results. To overcome these challenges, we present a novel real-time visualization tool, named FuseVis, with which the end-user can compute per-pixel saliency maps that examine the influence of the input image pixels on each pixel of the fused image. We trained several image fusion-based CNNs on medical image pairs and then using our FuseVis tool we performed case studies on a specific clinical application by interpreting the saliency maps from each of the fusion methods. We specifically visualized the relative influence of each input image on the predictions of the fused image and showed that some of the evaluated image-fusion methods are better suited for the specific clinical application. To the best of our knowledge, currently, there is no approach for visual analysis of neural networks for image fusion. Therefore, this work opens a new research direction to improve the interpretability of deep fusion networks. The FuseVis tool can also be adapted in other deep neural network-based image processing applications to make them interpretable.

Multi-exposure image fusion based on recursive filter

Multi-exposure image fusion technology is to fuse images of the same scene with different exposures into a high-quality image that contains more scene information. This paper proposes a multi-exposure image fusion method based on a recursive filter. The method first calculates three characteristic indicators of the multi-exposure image sequence: image contrast, entropy, and exposure information. And use three characteristic indicators to estimate the initial image fusion. In order to obtain a more accurate weight map for image fusion, a recursive filter is used to refine the weight map. Finally, the weighted sum method is used to construct the fused image. The experimental results show that the fusion results obtained by the algorithm in this paper have high image clarity, better subjective visual fusion effects and objective performance indicators, and maintain a high computational efficiency.

Hue-Correction Scheme Considering Non-Linear Camera Response for Multi-Exposure Image Fusion

We propose a novel hue-correction scheme for multi-exposure image fusion (MEF). Various MEF methods have so far been studied to generate higher-quality images. However, there are few MEF methods considering hue distortion unlike other fields of image processing, due to a lack of a reference image that has correct hue. In the proposed scheme, we generate an HDR image as a reference for hue correction, from input multi-exposure images. After that, hue distortion in images fused by an MEF method is removed by using hue information of the HDR one, on the basis of the constant-hue plane in the RGB color space. In simulations, the proposed scheme is demonstrated to be effective to correct hue-distortion caused by conventional MEF methods. Experimental results also show that the proposed scheme can generate high-quality images, regardless of exposure conditions of input multi-exposure images.

A Novel Fast Single Image Dehazing Algorithm Based on Artificial Multiexposure Image Fusion

Poor weather conditions, such as fog, haze, and mist, cause visibility degradation in captured images. Existing imaging devices lack the ability to effectively and efficiently mitigate the visibility degradation caused by poor weather conditions in real time. Image depth information is used to eliminate hazy effects by using existing physical model-based approaches. However, the imprecise depth information always affects dehazing performance. This article proposes an image fusion-based algorithm to enhance the performance and robustness of image dehazing. Based on a set of gamma-corrected underexposed images, pixelwise weight maps are constructed by analyzing both global and local exposedness to guide the fusion process. The spatial-dependence of luminance of the fused image is reduced, and its color saturation is balanced in the dehazing process. The performance of the proposed solution is confirmed in both theoretical analysis and comparative experiments.

GANFuse: a novel multi-exposure image fusion method based on generative adversarial networks

In this paper, a novel multi-exposure image fusion method based on generative adversarial networks (termed as GANFuse) is presented. Conventional multi-exposure image fusion methods improve their fusion performance by designing sophisticated activity-level measurement and fusion rules. However, these methods have a limited success in complex fusion tasks. Inspired by the recent FusionGAN which firstly utilizes generative adversarial networks (GAN) to fuse infrared and visible images and achieves promising performance, we improve its architecture and customize it in the task of extreme exposure image fusion. To be specific, in order to keep content of extreme exposure image pairs in the fused image, we increase the number of discriminators differentiating between fused image and extreme exposure image pairs. While, a generator network is trained to generate fused images. Through the adversarial relationship between generator and discriminators, the fused image will contain more information from extreme exposure image pairs. Thus, this relationship can realize better performance of fusion. In addition, the method we proposed is an end-to-end and unsupervised learning model, which can avoid designing hand-crafted features and does not require a number of ground truth images for training. We conduct qualitative and quantitative experiments on a public dataset, and the experimental result shows that the proposed model demonstrates better fusion ability than existing multi-exposure image fusion methods in both visual effect and evaluation metrics.

Multi‐exposure image fusion via a pyramidal integration of the phase congruency of input images with the intensity‐based maps

The most important part of the common algorithms for multi-exposure image fusion (MEF) is the selection of features and metrics that are appropriate for weight map extraction. This study presents a structure-based multi-exposure image fusion by employing the phase congruency (PC) of the input image. The main idea behind PC-based analysis is that the locations of image key attributes are at points where frequency components are maximally in phase. PC detects the details of an image invariant to its contrast and also emphasises on the texture- or structure-based features. In this work, alongside intensity-based maps, the extracted PC-based map is utilised for MEF in a pyramidal manner. Several experiments conducted on the benchmark dataset including a variety of natural multi-exposed image sequences to evaluate the proposed algorithm. Quantitative evaluations in terms of MEF structural similarity index and visual quality assessments show that the proposed method achieves better performance and produces comparable fused images in comparison to other approaches.

An improved algorithm of multi-exposure image fusion by detail enhancement

Multi-exposure image fusion is an effective method for depicting high dynamic range of the target scene in a single image. However, there are still some problems remaining: the preserving of global contrast, the preserving of the local details in saturated regions, and the existence of halo artifacts. To solve these problems, this paper proposes a new multi-exposure image fusion algorithm with detail enhancement. Firstly, the well-exposedness evaluation function, the chromatic information evaluation function and the local detail preserved function are used to measure weight maps. Then, an improved multi-exposure fusion framework based on pyramid decomposition is proposed to further enhance the details. The experimental results demonstrate that the proposed algorithm can preserve more details than the state-of-the-art. In the view of appreciation, our approach could produce a more realistic brightness distribution of target scene as well as avoid halo artifacts.

Multi-exposure image fusion based on linear embeddings and watershed masking

High dynamic range imaging (HDRI) is a challenging technology but yet demanding for modern imaging applications. Low-cost image sensors have limited dynamic range, and it is not always possible to capture and display natural scenes with high contrast and information loss in any exposure is inevitable. Three solutions for HDRI are using expensive high dynamic range (HDR) cameras with HDR-compatible displays, tone mapping operators for low dynamic range (LDR) screens, and capturing and fusing multiple exposures of the same LDR scene via image fusion algorithms. Companies that produce user grade devices prefer multi-exposure fusion (MEF) approaches to obtain HDR-like images for LDR screens due to its low cost. Hence, merging a stack of images containing different exposures of the same scene into a single informative image is an attractive research field. In this study, a novel, simple yet effective method is proposed for static image exposure fusion. The developed technique is based on weight map extraction via linear embeddings and watershed masking. The main advantage lies in watershed masking-based adjustment for obtaining accurate weights for image fusion. The comprehensive experimental comparisons demonstrate very strong visual and statistical results, and this approach should facilitate future MEF studies.

Color-compensated multi-scale exposure fusion based on physical features

The dynamic range captured by ordinary digital cameras is much smaller than that in natural scenes, so a single image cannot fully capture the real scene details when shooting high dynamic scenes. Traditional multi-exposure image fusion algorithms often cause inconsistencies in local brightness and corresponding scenes, which easily leads to problems such as color distortion and loss of details. Based on this, a novel multi-scale exposure fusion approach based on physical features is presented in this paper. Specifically, we first employ a novel Retinex model to get the illumination maps of original images and adopt the obtained illuminations map to construct the exposure maps. Then, a feature descriptor named patterns of oriented edge magnitudes (POEM) is introduced to extract local contrast from source images. Moreover, the extracted feature is combined to construct a weight map, and the weight term is integrated with the Laplacian pyramid algorithm to obtain an initial fusion image. Finally, simple detail enhancement and color compensation operations are performed to get the final fusion result. The numerous experiments have revealed that the proposed approach yields comparative and even better results in comparison with some state-of-the-art techniques both in subjective and objective evaluation.