Multi-exposure Image Fusion Methods Research Articles

A dual domain multi-exposure image fusion network based on spatial-frequency integration

Multi-exposure image fusion aims to generate a single high-dynamic image by integrating images with different exposures. Existing deep learning-based multi-exposure image fusion methods primarily focus on spatial domain fusion, neglecting the global modeling ability of the frequency domain. To effectively leverage the global illumination modeling ability of the frequency domain, we propose a novelty perspective on multi-exposure image fusion via the Spatial-Frequency Integration Framework, named MEF-SFI. Initially, we revisit the properties of the Fourier transform on the 2D image, and verify the feasibility of multi-exposure image fusion in the frequency domain where the amplitude and phase component is able to guide the integration of the illumination information. Subsequently, we present the deep Fourier-based multi-exposure image fusion framework, which consists of a spatial path and frequency path for local and global modeling separately. Specifically, we introduce a Spatial-Frequency Fusion Module to facilitate efficient interaction between dual domains and capture complementary information from input images with different exposures. Finally, we combine a dual domain loss function to ensure the retention of complementary information in both the spatial and frequency domains. Extensive experiments on the PQA-MEF dataset demonstrate that our method achieves visual-appealing fusion results against state-of-the-art multi-exposure image fusion approaches. Our code is available at https://github.com/SSyangguang/MEF-freq.

Robust HDR reconstruction using 3D patch based on two-scale decomposition

In this paper, we propose an effective 3D patch-based match and fusion method by taking account of dynamic or multi-view scenes in a multi-exposure image sequence using two-scale decomposition. As opposed to most multi-exposure image fusion methods, the proposed method does not require a pre-alignment step to reduce ghosting artifacts. Considering that pixel values are affected by multi-view or multi-exposure scenes, we use a uniform matching approach to match and find similar patches in different exposure images and then fuse them at each scale. By searching all similar patches (instead of the optimal patch) in the searching window to form the 3D patch when facing limited image information, we can fully use the complementary information of the multi-exposure images to preserve information about moving objects and scene details. The experimental results show that the proposed method not only performs well on dynamic scenes but also consistently generates high-quality fused images in multi-view scenes.

EMEF: Ensemble Multi-Exposure Image Fusion

Although remarkable progress has been made in recent years, current multi-exposure image fusion (MEF) research is still bounded by the lack of real ground truth, objective evaluation function, and robust fusion strategy. In this paper, we study the MEF problem from a new perspective. We don’t utilize any synthesized ground truth, design any loss function, or develop any fusion strategy. Our proposed method EMEF takes advantage of the wisdom of multiple imperfect MEF contributors including both conventional and deep learning-based methods. Specifically, EMEF consists of two main stages: pre-train an imitator network and tune the imitator in the runtime. In the first stage, we make a unified network imitate different MEF targets in a style modulation way. In the second stage, we tune the imitator network by optimizing the style code, in order to find an optimal fusion result for each input pair. In the experiment, we construct EMEF from four state-of-the-art MEF methods and then make comparisons with the individuals and several other competitive methods on the latest released MEF benchmark dataset. The promising experimental results demonstrate that our ensemble framework can “get the best of all worlds”. The code is available at https://github.com/medalwill/EMEF.

Multi-exposure image fusion via perception enhanced structural patch decomposition

Multi-exposure image fusion (MEF) is an affordable and convenient option for high-dynamic-range imaging. Current MEF methods are prone to visually unrealistic results since they take no account of perceptual factors. To address this problem, a multi-exposure image fusion method is proposed based on perception enhanced structural patch decomposition, namely PESPD-MEF. An image patch is first decomposed into four components: perceptual gain, signal strength, signal structure, and mean intensity. Then, the enhancement rule is designed for perceptual gain, and the latter three elements are fused independently in different ways. Finally, the fused components are aggregated to generate informative and perception-realistic results. Moreover, the multi-scale framework is adopted to boost the fused performance. The proposed method is also extended to address single low-light image enhancement issue. Experimental results demonstrate that the proposed approach outperforms the state-of-the-art methods by a large margin in terms of perceptual realism. The source code is available at https://github.com/Junchao2018/PESPD-MEF.

AIM-MEF: Multi-exposure image fusion based on adaptive information mining in both spatial and frequency domains

Multi-Exposure Image Fusion (MEF) is a simple and effective solution to obtain high dynamic range images by integrating important information from a series of low dynamic range source images with different exposure levels. Despite promising results have been achieved, two issues remain unresolved in existing deep learning-based MEF methods. One is that the information in the frequency domain is underutilized, and the other is that the useful information of hard-to-learn pixels is not fully exploited. To address the above issues, we propose AIM-MEF: a multi-exposure image fusion framework based on adaptive information mining in both spatial and frequency domains. Concretely, we propose an image fusion loss in the frequency domain to mine the frequency domain information for the first time in the MEF field. In order to adaptively exploit the hard-to-learn but useful information in the source images, we propose a hard information mining strategy in both the spatial and the frequency domains, and perform information mining in pixel and frequency levels of the source images. In addition, we propose an adaptive fusion strategy based on information abundance to perform information mining at the image level. We compared AIM-MEF to 10 traditional and deep learning-based methods on two public datasets with eight objective metrics, and AIM-MEF outperformed these baselines in both subjective and objective evaluations. Codes will be publicly available.

Multi-exposure image fusion of dynamic scenes using consistency detection and weighted least squares filtering

The high dynamic range image obtained by multi-exposure image fusion (MEF) contains more detailed information, which has broad application prospects and crucial practical significance in many fields. However, due to the introduction of the ghosting artifacts caused by object movement and camera shake, MEF in the dynamic scenes has always been a challenge. To address this problem, we designed a deghosting method for MEF. The over- and under-exposed images are corrected by intensity mapping and consistency detection to obtain the aligned latent images. Then the high- and low-frequency components of the latent images are generated using weighted least squares filtering. The blending weights are calculated based on the image luminance and the exposedness function. These frequency components are integrated into the final deghosted image with more texture details and vivid color. A comprehensive evaluation experiment is carried out, proving that the proposed method has a better visual effect and stable performance than the state-of-the-art deghosting MEF methods.

Multi-exposure electric power monitoring image fusion method without ghosting based on exposure fusion framework and color dissimilarity feature.

To solve the ghosting artifacts problem in dynamic scene multi-scale exposure fusion, an improved multi-exposure fusion method has been proposed without ghosting based on the exposure fusion framework and the color dissimilarity feature of this study. This fusion method can be further applied to power system monitoring and unmanned aerial vehicle monitoring. In this study, first, an improved exposure fusion framework based on the camera response model was applied to preprocess the input image sequence. Second, the initial weight map was estimated by multiplying four weight items. In removing the ghosting weight term, an improved color dissimilarity feature was used to detect the object motion features in dynamic scenes. Finally, the improved pyramid model as adopted to retain detailed information about the poor exposure areas. Experimental results indicated that the proposed method improves the performance of images in terms of sharpness, detail processing, and ghosting artifacts removal and is superior to the five existing multi-exposure image fusion (MEF) methods in quality evaluation.

Multi-Exposure Image Fusion via Deformable Self-Attention.

Most multi-exposure image fusion (MEF) methods perform unidirectional alignment within limited and local regions, which ignore the effects of augmented locations and preserve deficient global features. In this work, we propose a multi-scale bidirectional alignment network via deformable self-attention to perform adaptive image fusion. The proposed network exploits differently exposed images and aligns them to the normal exposure in varying degrees. Specifically, we design a novel deformable self-attention module that considers variant long-distance attention and interaction and implements the bidirectional alignment for image fusion. To realize adaptive feature alignment, we employ a learnable weighted summation of different inputs and predict the offsets in the deformable self-attention module, which facilitates that the model generalizes well in various scenes. In addition, the multi-scale feature extraction strategy makes the features across different scales complementary and provides fine details and contextual features. Extensive experiments demonstrate that our proposed algorithm performs favorably against state-of-the-art MEF methods.

Review of Multi-Exposure Image Fusion Methods

Review of Multi-Exposure Image Fusion Methods

Multi-Exposure Image Fusion via Multi-Scale and Context-Aware Feature Learning

In this letter, a deep learning (DL)-based multi-exposure image fusion (MEF) method via multi-scale and context-aware feature learning is proposed, aiming to overcome the defects of existing traditional and DL-based methods. The proposed network is based on an auto-encoder architecture. First, an encoder that combines the convolutional network and Transformer is designed to extract multi-scale features and capture the global contextual information. Then, a multi-scale feature interaction (MSFI) module is devised to enrich the scale diversity of extracted features using cross-scale fusion and Atrous spatial pyramid pooling (ASPP). Finally, a decoder with a nest connection architecture is introduced to reconstruct the fused image. Experimental results show that the proposed method outperforms several representative traditional and DL-based MEF methods in terms of both visual quality and objective assessment. The source code and trained model are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/yuliu316316/MSCA-MEF</uri> .

ACE-MEF: Adaptive Clarity Evaluation-Guided Network With Illumination Correction for Multi-Exposure Image Fusion

For a natural scene with nonuniform environment light, the captured visible images are always under- or over-exposed because of the limited dynamic range of digital imaging devices. Multi-exposure image fusion (MEF) is a mainstream and effective solution. For a local region that has friendly visual effect in one exposure setting but extremely bad-exposed in another, most existing MEF methods have the ability to transfer the scene detail information to the fused images. However, they will be affected by the over-high or -low light inevitably thus resulting in local visibility reduction. To address this issue, we propose an adaptive clarity evaluation-guided network with illumination correction for MEF in a coarse-to-fine manner, which is termed as ACE-MEF. To be specific, our ACE-MEF is mainly composed of two modules: clarity preservation network (CPN) and illumination adjustment network (IAN). Based on the adaptive clarity evaluation, CPN could be trained to coarsely preserve the environment light and texture details of the clearer regions in source images. Therefore, the need for labeled reference images that are time-consuming to obtain could be mitigated. By measuring the parameter maps of gamma function, IAN is able to refine and correct the local bad-exposed regions so that more details could be further revealed. Extensive experiments demonstrate that our method outperforms multiple state-of-the-art algorithms qualitatively and quantitatively.

Deep HDR Deghosting by Motion-Attention Fusion Network.

Multi-exposure image fusion (MEF) methods for high dynamic range (HDR) imaging suffer from ghosting artifacts when dealing with moving objects in dynamic scenes. The state-of-the-art methods use optical flow to align low dynamic range (LDR) images before merging, introducing distortion into the aligned LDR images from inaccurate motion estimation due to large motion and occlusion. In place of pre-alignment, attention-based methods calculate the correlation between the reference LDR image and non-reference LDR images, thus excluding misaligned regions in LDR images. Nevertheless, they also exclude the saturated details at the same time. Taking advantage of both the alignment and attention-based methods, we propose an efficient Deep HDR Deghosting Fusion Network (DDFNet) guided by optical flow and image correlation attentions. Specifically, the DDFNet estimates the optical flow of the LDR images by a motion estimation module and encodes that optical flow as a flow feature. Additionally, it extracts correlation features between the reference LDR and other non-reference LDR images. The optical flow and correlation features are employed to adaptably combine information from LDR inputs in an attention-based fusion module. Following the merging of features, a decoder composed of Dense Networks reconstructs the HDR image without ghosting. Experimental results indicate that the proposed DDFNet achieves state-of-the-art image fusion performance on different public datasets.

Multi-Exposure Image Fusion Algorithm Based on Improved Weight Function.

High-dynamic-range (HDR) image has a wide range of applications, but its access is limited. Multi-exposure image fusion techniques have been widely concerned because they can obtain images similar to HDR images. In order to solve the detail loss of multi-exposure image fusion (MEF) in image reconstruction process, exposure moderate evaluation and relative brightness are used as joint weight functions. On the basis of the existing Laplacian pyramid fusion algorithm, the improved weight function can capture the more accurate image details, thereby making the fused image more detailed. In 20 sets of multi-exposure image sequences, six multi-exposure image fusion methods are compared in both subjective and objective aspects. Both qualitative and quantitative performance analysis of experimental results confirm that the proposed multi-scale decomposition image fusion method can produce high-quality HDR images.

Multi-Exposure Image Fusion Techniques: A Comprehensive Review

Multi-exposure image fusion (MEF) is emerging as a research hotspot in the fields of image processing and computer vision, which can integrate images with multiple exposure levels into a full exposure image of high quality. It is an economical and effective way to improve the dynamic range of the imaging system and has broad application prospects. In recent years, with the further development of image representation theories such as multi-scale analysis and deep learning, significant progress has been achieved in this field. This paper comprehensively investigates the current research status of MEF methods. The relevant theories and key technologies for constructing MEF models are analyzed and categorized. The representative MEF methods in each category are introduced and summarized. Then, based on the multi-exposure image sequences in static and dynamic scenes, we present a comparative study for 18 representative MEF approaches using nine commonly used objective fusion metrics. Finally, the key issues of current MEF research are discussed, and a development trend for future research is put forward.

Multi‐exposure image fusion based on feature evaluation with adaptive factor

The authors present a new multi-exposure images fusion method based on feature evaluation with adaptive factor. It is noticed the existing multi-exposure fusion algorithm is not well adapted to the input images, which are overall bright or dark, the fused image quality is not pretty good, and the details are not preserved completely. So an adaptive factor to adapt the intensity of input images is presented here. First, the exposure assessment weight, texture change weight, and colour intensity weight are calculated by a sliding window. Finally, the images are fused by using a pyramid to avoid the seams. Twenty exposure input images of different scenes are selected, the subjective and objective aspects are analysed and compared with several existing multi-exposure image fusion methods. The experimental results show that the proposed method can retain more details and obtain satisfactory visual effects on static scenes.

Benchmarking and comparing multi-exposure image fusion algorithms

Multi-exposure image fusion (MEF) is an important area in computer vision and has attracted increasing interests in recent years. Apart from conventional algorithms, deep learning techniques have also been applied to MEF. However, although many efforts have been made on developing MEF algorithms, the lack of benchmarking studies makes it difficult to perform fair and comprehensive performance comparison among MEF algorithms, thus hindering the development of this field significantly. In this paper, we fill this gap by proposing a benchmark of multi-exposure image fusion (MEFB), which consists of a test set of 100 image pairs, a code library of 21 algorithms, 20 evaluation metrics, 2100 fused images, and a software toolkit. To the best of our knowledge, this is the first benchmarking study in the field of MEF. This paper also gives a literature review on MEF methods with a focus on deep learning-based algorithms. Extensive experiments have been conducted using MEFB for comprehensive performance evaluation and for identifying effective algorithms. We expect that MEFB will serve as an effective platform for researchers to compare the performance of MEF algorithms.

Suspect Multifocus Image Fusion Based on Sparse Denoising Autoencoder Neural Network for Police Multimodal Big Data Analysis

In recent years, the success rate of solving major criminal cases through big data has been greatly improved. The analysis of multimodal big data plays a key role in the detection of suspects. However, the traditional multiexposure image fusion methods have low efficiency and are largely time-consuming due to the artifact effect in the image edge and other sensitive factors. Therefore, this paper focuses on the suspect multiexposure image fusion. The self-coding neural network based on deep learning has become a hotspot in the research of data dimension reduction, which can effectively eliminate the irrelevant and redundant learning data. In the case of limited field depth, due to the limited focusing depth of the camera, the focusing plane cannot obtain the global clear image of the target in the depth scene, which is prone to defocusing and blurring phenomena. Therefore, this paper proposes a multifocus image fusion based on a sparse denoising autoencoder neural network. To realize an unsupervised end-to-end fusion network, the sparse denoising autoencoder neural network is adopted to extract features and learn fusion rules and reconstruction rules simultaneously. The initial decision graph of the multifocus image is taken as a prior input to learn the rich detailed information of the image. The local strategy is added to the loss function to ensure that the image is restored accurately. The results show that this method is superior to the state-of-the-art fusion methods.

Hue-Correction Scheme Considering Non-Linear Camera Response for Multi-Exposure Image Fusion

We propose a novel hue-correction scheme for multi-exposure image fusion (MEF). Various MEF methods have so far been studied to generate higher-quality images. However, there are few MEF methods considering hue distortion unlike other fields of image processing, due to a lack of a reference image that has correct hue. In the proposed scheme, we generate an HDR image as a reference for hue correction, from input multi-exposure images. After that, hue distortion in images fused by an MEF method is removed by using hue information of the HDR one, on the basis of the constant-hue plane in the RGB color space. In simulations, the proposed scheme is demonstrated to be effective to correct hue-distortion caused by conventional MEF methods. Experimental results also show that the proposed scheme can generate high-quality images, regardless of exposure conditions of input multi-exposure images.

GANFuse: a novel multi-exposure image fusion method based on generative adversarial networks

In this paper, a novel multi-exposure image fusion method based on generative adversarial networks (termed as GANFuse) is presented. Conventional multi-exposure image fusion methods improve their fusion performance by designing sophisticated activity-level measurement and fusion rules. However, these methods have a limited success in complex fusion tasks. Inspired by the recent FusionGAN which firstly utilizes generative adversarial networks (GAN) to fuse infrared and visible images and achieves promising performance, we improve its architecture and customize it in the task of extreme exposure image fusion. To be specific, in order to keep content of extreme exposure image pairs in the fused image, we increase the number of discriminators differentiating between fused image and extreme exposure image pairs. While, a generator network is trained to generate fused images. Through the adversarial relationship between generator and discriminators, the fused image will contain more information from extreme exposure image pairs. Thus, this relationship can realize better performance of fusion. In addition, the method we proposed is an end-to-end and unsupervised learning model, which can avoid designing hand-crafted features and does not require a number of ground truth images for training. We conduct qualitative and quantitative experiments on a public dataset, and the experimental result shows that the proposed model demonstrates better fusion ability than existing multi-exposure image fusion methods in both visual effect and evaluation metrics.

A Precise Multi-Exposure Image Fusion Method Based on Low-level Features.

Multi exposure image fusion (MEF) provides a concise way to generate high-dynamic-range (HDR) images. Although the precise fusion can be achieved by existing MEF methods in different static scenes, the corresponding performance of ghost removal varies in different dynamic scenes. This paper proposes a precise MEF method based on feature patches (FPM) to improve the robustness of ghost removal in a dynamic scene. A reference image is selected by a priori exposure quality first and then used in the structure consistency test to solve the image ghosting issues existing in the dynamic scene MEF. Source images are decomposed into spatial-domain structures by a guided filter. Both the base and detail layer of the decomposed images are fused to achieve the MEF. The structure decomposition of the image patch and the appropriate exposure evaluation are integrated into the proposed solution. Both global and local exposures are optimized to improve the fusion performance. Compared with six existing MEF methods, the proposed FPM not only improves the robustness of ghost removal in a dynamic scene, but also performs well in color saturation, image sharpness, and local detail processing.