Multi-exposure Image Fusion Research Articles

GALFusion: Multi-Exposure Image Fusion via a Global–Local Aggregation Learning Network

The goal of multi-exposure image fusion is to generate synthetic results with abundant details and balanced exposure from low dynamic range(LDR) images. The existing multi-exposure fusion methods often use convolution operations to extract features. However, these methods only consider the pixel values in local view field and ignore the long-range dependencies between pixels. To solve the aforementioned problem, we propose a global-local aggregation network for fusing extreme exposure images in an unsupervised way. Firstly, we design a collaborative aggregation module, composed of two sub-modules covering a non-local attention inference module and a local adaptive learning module, to mine the relevant features from source images. So that we successfully formulate a feature extraction mechanism with aggregating global and local information. Secondly, we provide a special fusion module to reconstruct fused images, which effectively avoids artifacts and suppresses information decay. Moreover, we further fine-tune the fusion results by a recursive refinement module to capture more textural details from source images. The results of both comparative and ablation analyses on two datasets demonstrate that GALFusion achieves the best marks in terms of MEF-SSIM and PSNR, outperforming the existing 12 state-of-the-art fusion methods.

DMEF: Multi-Exposure Image Fusion Based on a Novel Deep Decomposition Method

In this paper, we propose a novel deep decomposition approach based on Retinex theory for multi-exposure image fusion, termed as DMEF. According to the assumption of Retinex theory, we firstly decompose the source images into illumination and reflection maps by the data-driven decomposition network, among which we introduce the pathwise interaction block that reactivates the deep features lost in one path and embeds them into another path. Therefore, loss of illumination and reflection features during decomposition can be effectively suppressed. And then the high dynamic range illumination map could be obtained by fusing the separated illumination maps in the fusion network. Thus, the reconstructed details in under-exposed and over-exposed regions will be clearer with the help of the fused reflection map which contains complete high-frequency scene information. Finally, the fused illumination and reflection maps are multiplied pixel-by-pixel to obtain the final fused image. Moreover, to retain the discontinuity in the illumination map where gradient of reflection map changes steeply, we introduce the structure-preservation smoothness loss function to retain the structure information and eliminate visual artifacts in these regions. The superiority of our proposed network is demonstrated by applying extensive experiments compared with other state-of-the-art fusion methods subjectively and objectively.

Review of Multi-Exposure Image Fusion Methods

Review of Multi-Exposure Image Fusion Methods

Rethinking multi-exposure image fusion with extreme and diverse exposure levels: A robust framework based on Fourier transform and contrastive learning

Multi-exposure image fusion (MEF) is an important technique for generating high dynamic range images. However, most existing MEF studies focus on fusing a moderately over-exposed image and a moderately under-exposed image, and they are not robust in fusing images with extreme and diverse exposure levels. In this paper, we propose a robust MEF framework based on Fourier transform and contrastive learning. Specifically, we develop a Fourier transform-based pixel intensity transfer strategy to synthesize images with diverse exposure levels from normally exposed natural images and train an encoder–decoder network to reconstruct the original natural image. In this way, the encoder and decoder can learn to extract features from images with diverse exposure levels and generate fused images with normal exposure. We propose a contrastive regularization loss to further enhance the capability of the network in recovering normal exposure levels. In addition, we construct an extreme MEF benchmark dataset and a random MEF benchmark dataset for a more comprehensive evaluation of MEF algorithms. We extensively compare our method with fifteen competitive traditional and deep learning-based MEF algorithms on three benchmark datasets, and our method outperforms the other methods in both subjective visual effects and objective evaluation metrics. Our code, datasets and all fused images will be released.

Three-stage fusion framework for single image dehazing

Fog generally causes loss or blurring of image information, significantly affecting work and life. This study proposes a three-stage fusion method comprising multiscale dark channel image fusion based on the dark channel prior, multiexposure image fusion using a gamma-correction technique, and a final fusion combining the restoration-based and enhancement-based methods to ensure good saturation and contrast in the dehazing result. The proposed method is experimentally evaluated in detail, and qualitative as well as quantitative analyses are performed. The obtained results indicate that our method can effectively improve the haze removal performance and is advantageous for dealing with long-distance hazy images.

MUFusion: A general unsupervised image fusion network based on memory unit

Existing image fusion approaches are committed to using a single deep network to solve different image fusion problems, achieving promising performance in recent years. However, devoid of the ground-truth output, in these methods, only the appearance from source images can be exploited during the training process to generate the fused images, resulting in suboptimal solutions. To this end, we advocate a self-evolutionary training formula by introducing a novel memory unit architecture (MUFusion). In this unit, specifically, we utilize the intermediate fusion results obtained during the training process to further collaboratively supervise the fused image. In this way, our fusion results can not only learn from the original input images, but also benefit from the intermediate output of the network itself. Furthermore, an adaptive unified loss function is designed based on this memory unit, which is composed of two loss items, i.e., content loss and memory loss. In particular, the content loss is calculated based on the activity level maps of source images, which can constrain the output image to contain specific information. On the other hand, the memory loss is obtained based on the previous output of our model, which is utilized to force the network to yield fusion results with higher quality. Considering the handcrafted activity level maps cannot consistently reflect the accurate salience judgement, we put two adaptive weight items between them to prevent this degradation phenomenon. In general, our MUFusion can effectively handle a series of image fusion tasks, including infrared and visible image fusion, multi-focus image fusion, multi-exposure image fusion, and medical image fusion. Particularly, the source images are concatenated in the channel dimension. After that, a densely connected feature extraction network with two scales is used to extract the deep features of the source images. Following this, the fusion result is obtained by two feature reconstruction blocks with skip connections from the feature extraction network. Qualitative and quantitative experiments on 4 image fusion subtasks demonstrate the superiority of our MUFusion, compared to the state-of-the-art methods.

GeFuNet: A knowledge-guided deep network for the infrared and visible image fusion

Deep learning has achieved great success in computer vision and image processing applications. Several pioneering works also succeeded in image fusion. However, due to the lack of ground truth, deep networks for image fusion are challenging to be well trained, restricting their fusion performance. In this paper, we design a knowledge-guided deep network, Generative-Fusion Network (GeFuNet), which takes natural images as its training data and ground truth. GeFuNet consists of two sub-networks: data generation subnet and image fusion subnet. In training, guided by the knowledge that real infrared and visible images respectively mainly contain contour and texture information, the data generation subnet is trained to generate pseudo infrared and pseudo visible images. The image fusion subnet is then trained using these pseudo images and further supervised by the natural images. The image fusion subnet adopts a hierarchical block to effectively extract and fuse multi-level information from the source images, which is learned by detail enhancement training. During the inference, the trained image fusion subnet is used to fuse real infrared and visible images. The experimental results show that GeFuNet can extract a complete contour and more details from the infrared and visible images, which are fused into the final image. The results also prove the applicability of GeFuNet on multi-scale image fusion, multi-focus image fusion, medical image fusion, and multi-exposure image fusion.

Deep HDR Deghosting by Motion-Attention Fusion Network.

Multi-exposure image fusion (MEF) methods for high dynamic range (HDR) imaging suffer from ghosting artifacts when dealing with moving objects in dynamic scenes. The state-of-the-art methods use optical flow to align low dynamic range (LDR) images before merging, introducing distortion into the aligned LDR images from inaccurate motion estimation due to large motion and occlusion. In place of pre-alignment, attention-based methods calculate the correlation between the reference LDR image and non-reference LDR images, thus excluding misaligned regions in LDR images. Nevertheless, they also exclude the saturated details at the same time. Taking advantage of both the alignment and attention-based methods, we propose an efficient Deep HDR Deghosting Fusion Network (DDFNet) guided by optical flow and image correlation attentions. Specifically, the DDFNet estimates the optical flow of the LDR images by a motion estimation module and encodes that optical flow as a flow feature. Additionally, it extracts correlation features between the reference LDR and other non-reference LDR images. The optical flow and correlation features are employed to adaptably combine information from LDR inputs in an attention-based fusion module. Following the merging of features, a decoder composed of Dense Networks reconstructs the HDR image without ghosting. Experimental results indicate that the proposed DDFNet achieves state-of-the-art image fusion performance on different public datasets.

SGFusion: A saliency guided deep-learning framework for pixel-level image fusion

Pixel-level image fusion, which merges different modal images into an informative image, has attracted more and more attention. Despite many methods that have been proposed for pixel-level image fusion, there is a lack of effective image fusion methods that can simultaneously deal with different tasks. To address this problem, we propose a saliency guided deep-learning framework for pixel-level image fusion called SGFusion, which is an end-to-end fusion network and can be applied to a variety of fusion tasks by training one model. In specific, the proposed network uses the dual-guided encoding, image reconstruction decoding, and the saliency detection decoding processes to simultaneously extract the feature maps and saliency maps in different scales from the image. The saliency detection decoding is used as fusion weights to merge the features of image reconstruction decoding for generating the fusion image, which can effectively extract meaningful information from the source images and make the fusion image more in line with visual perception. Experiments indicate that the proposed fusion method achieves state-of-the-art performance in infrared and visible image fusion, multi-exposure image fusion, and medical image fusion on various public datasets.

Ghost-Free Multi-Exposure Image Fusion Technology Based on the Multi-Scale Block LBP Operator

This paper proposes a ghost-free multi-exposure image fusion technique based on the multi-scale block LBP (local binary pattern) operator. The method mainly includes two steps: first, the texture variation, brightness, and spatial consistency weight maps of the image are computed, and then these three image features are used to construct the initial weight map. Finally, the multi-resolution method is used to fuse the images to obtain the resulting image. The main advantage of this technique lies in the step of extracting the details of the source image based on the multi-scale block LBP operator, which is used to preserve the details of the brightest and darkest areas in high dynamic range scenes and preserve the texture features of the source image. Another advantage is that a new LBP operator-based motion detection method is proposed for fusing multi-exposure images in dynamic scenes containing moving objects. In addition, this paper also studies two spatially consistent weight distribution methods and compares and discusses the effects of these two methods on the results of dynamic image fusion. Through a large number of experimental comparisons, the superiority and feasibility of this method are proved.

Recognition of characters on curved metal workpiece surfaces based on multi-exposure image fusion and deep neural networks

Accurate industry online scene text recognition techniques for character on curved metal-workpieces are investigated. To increase the contrast between the characters and their backgrounds, and to reduce the interactions of corrosion et al., image pyramid fusion method is proposed to synthesize high-quality detail-enhanced images from image sequences with multi-exposure times. With this method, the three most important reflection-related image quality properties: contrast, saturation, and exposure, can be enhanced simultaneously. A Resnet-50 network is then used as a feature extraction network to fuse features from different scales of feature maps, the masks and the envelope contour coordinates of character text lines can then be obtained by combining global contextual information through semantic segmentation. Finally, characters included in the irregular text lines are recognized with an attention mechanism-based text rectification recognition algorithm. The advantages and effectiveness of the proposed method are statistically analyzed with the data from different production lines. Code will be available in https://github.com/HuaxiongWu-ZJLG/CSCM-Recognition.git.

Ghosting-free multi-exposure image fusion for static and dynamic scenes

The visual system enables humans to perceive all details of the real-world with vivid colors, while high dynamic range (HDR) technology aims at capturing natural scenes in a closer way to human perception through a large dynamic range of color gamut. Especially for traditional –low dynamic range (LDR)– devices, HDR-like image generation is an attractive research topic. Blending a stack of input LDR exposures is called multi-exposure image fusion (MEF). MEF is indeed a very challenging problem and it is highly prone to halo effects or ghosting and motion blur in the cases when there are spatial discontinuities in between input exposures. To overcome these artifacts, MEF keeps the “best” quality regions of each exposure via a weight characterization scheme. This paper proposes an effective weight map extraction framework which relies on principal component analysis, adaptive well-exposedness and saliency maps. The characterized maps are later refined by a guided filter and a blended output image is obtained via pyramidal decomposition. Comprehensive experiments and comparisons demonstrate that the developed algorithm generates very strong statistical and visual results for both static and dynamic scenes. In addition, the designed method is successfully applied to the visible-infrared image fusion problem without any further optimization.

Attention-Guided Global-Local Adversarial Learning for Detail-Preserving Multi-Exposure Image Fusion

Deep learning networks have recently demonstrated yielded impressive progress for multi-exposure image fusion. However, how to restore realistic texture details while correcting color distortion is still a challenging problem to be solved. To alleviate the aforementioned issues, in this paper, we propose an attention-guided global-local adversarial learning network for fusing extreme exposure images in a coarse-to-fine manner. Firstly, the coarse fusion result is generated under the guidance of attention weight maps, which acquires the essential region of interest from both sides. Secondly, we formulate an edge loss function, along with a spatial feature transform layer, for refining the fusion process. So that it can take full use of the edge information to deal with blurry edges. Moreover, by incorporating global-local learning, our method can balance pixel intensity distribution and correct the color distortion on spatially varying source images from both image/patch perspectives. Such a global-local discriminator ensures all the local patches of the fused images align with realistic normal-exposure ones. Extensive experimental results on two publicly available datasets show that our method drastically outperforms state-of-the-art methods in visual inspection and objective analysis. Furthermore, sufficient ablation experiments prove that our method has significant advantages in generating high-quality fused results with appealing details, clear targets, and faithful color. Source code will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/JinyuanLiu-CV/AGAL</uri> .

Unified gradient- and intensity-discriminator generative adversarial network for image fusion

This study proposes a unified gradient- and intensity-discriminator generative adversarial network for various image fusion tasks, including infrared and visible image fusion, medical image fusion, multi-focus image fusion, and multi-exposure image fusion. On the one hand, we unify all fusion tasks into discriminating a fused image’s gradient and intensity distributions based on a generative adversarial network. The generator adopts a dual-encoder–single-decoder framework to extract source image features by using different encoder paths. A dual-discriminator is employed to distinguish the gradient and intensity, ensuring that the generated image contains the desired geometric structure and conspicuous information. The dual adversarial game can tackle the generative adversarial network’s mode collapse problem. On the other hand, we define a loss function based on the gradient and intensity that can be adapted to various fusion tasks by using varying relevant parameters with the source images. Qualitative and quantitative experiments on publicly available datasets demonstrate our method’s superiority over state-of-the-art methods.

TransMEF: A Transformer-Based Multi-Exposure Image Fusion Framework Using Self-Supervised Multi-Task Learning

In this paper, we propose TransMEF, a transformer-based multi-exposure image fusion framework that uses self-supervised multi-task learning. The framework is based on an encoder-decoder network, which can be trained on large natural image datasets and does not require ground truth fusion images. We design three self-supervised reconstruction tasks according to the characteristics of multi-exposure images and conduct these tasks simultaneously using multi-task learning; through this process, the network can learn the characteristics of multi-exposure images and extract more generalized features. In addition, to compensate for the defect in establishing long-range dependencies in CNN-based architectures, we design an encoder that combines a CNN module with a transformer module. This combination enables the network to focus on both local and global information. We evaluated our method and compared it to 11 competitive traditional and deep learning-based methods on the latest released multi-exposure image fusion benchmark dataset, and our method achieved the best performance in both subjective and objective evaluations. Code will be available at https://github.com/miccaiif/TransMEF.

Multi-Exposure Image Fusion Based on Weighted Average Adaptive Factor and Local Detail Enhancement

In order to adapt to the local brightness and contrast of input image sequences, we propose a new weighted average adaptive factor well-exposure weight evaluation scheme. The exposure weights of brighter and darker pixels are determined according to the local average brightness and expected brightness. We find that in the traditional multi-exposure image fusion scheme, the brighter and darker regions of the scene lose many details. To solve this problem, we first propose a standard to determine the brighter and darker regions and then use a fast local Laplacian filter to enhance the image in the region. This paper selects 16 multi-exposure images of different scenes for subjective and objective analysis and compares them with eight existing multi-exposure fusion schemes. The experimental results show that the proposed method can enhance the details appropriately while preserving the details in static scenes and adapting to the input image brightness.

Blind quality assessment of tone-mapped images using multi-exposure sequences

The tone mapping operator (TMO) enables high dynamic range (HDR) images to be presented on low dynamic range (LDR) consumer electronic devices. However, the results obtained by this method are not always ideal due to the reduced number of bits. In comparison, the multi-exposure image fusion (MEF) bypasses the intermediate HDR image composition and directly produces an image presented on standard devices. Inspired by this, this paper proposes a quality assessment method for tone-mapped image (TMI) based on generating multi-exposure sequences. Specifically, the method uses a generative adversarial network (GAN) to generate a set of sequences with different exposure levels based on the TMIs. Then a two-branch convolutional neural network (CNN) is used to extract features from the tone-mapped images and the multi-exposure reference sequences, respectively. Finally, the transformer is used to mine the intrinsic connections between TMIs and multi-exposure sequences and learn the mapping relationships from feature space to quality space. We conducted extensive experiments on the ESPL-LIVE HDR database. The applicability and effectiveness of the proposed method are verified by comparing and analyzing relevant features and model configurations with existing mainstream evaluation algorithms.

Multi-exposure image fusion for dynamic scenes with ghosting removal

Multi-exposure image fusion algorithms in dynamic scenes usually have ghosts. Most of the existing ghost-free methods have complex calculations and the details will be lost. To solve these problems, we present a multi-exposure fusion method, which can remove ghosts and preserve more details of the image. The method detects ghost images by the improved frame difference method and calculates the early weight maps, combining the ghost detection binary image and initial weight maps to redefine weight maps. Then with the improved Laplacian pyramid algorithm, a high-quality image with more details is fused quickly. This is a significant improvement over existing techniques by objective and subjective comparisons.

Detail Enhancement Multi-Exposure Image Fusion Based on Homomorphic Filtering

Due to the large dynamic range of real scenes, it is difficult for images taken by ordinary devices to represent high-quality real scenes. To obtain high-quality images, the exposure fusion of multiple exposure images of the same scene is required. The fusion of multiple images results in the loss of edge detail in areas with large exposure differences. Aiming at this problem, this paper proposes a new method for the fusion of multi-exposure images with detail enhancement based on homomorphic filtering. First, a fusion weight map is constructed using exposure and local contrast. The exposure weight map is calculated by threshold segmentation and an adaptively adjustable Gaussian curve. The algorithm can assign appropriate exposure weights to well-exposed areas so that the fused image retains more details. Then, the weight map is denoised using fast-guided filtering. Finally, a fusion method for the detail enhancement of Laplacian pyramids with homomorphic filtering is proposed to enhance the edge information lost by Laplacian pyramid fusion. The experimental results show that the method can generate high-quality images with clear edges and details as well as similar color appearance to real scenes and can outperform existing algorithms in both subjective and objective evaluations.

Smoke removal and image enhancement of laparoscopic images by an artificial multi-exposure image fusion method

In laparoscopic surgery, image quality is often degraded by surgical smoke or by side effects of the illumination system, such as reflections, specularities, and non-uniform illumination. The degraded images complicate the work of the surgeons and may lead to errors in image-guided surgery. Existing enhancement algorithms mainly focus on enhancing global image contrast, overlooking local contrast. Here, we propose a new Patch Adaptive Structure Decomposition utilizing the Multi-Exposure Fusion technique to enhance the local contrast of laparoscopic images for better visualization. The set of under-exposure level images is obtained from a single input blurred image by using gamma correction. Spatial linear saturation is applied to enhance image contrast and to adjust the image saturation. The Multi-Exposure Fusion (MEF) is used on a series of multi-exposure images to obtain a single clear and smoke-free fused image. MEF is applied by using adaptive structure decomposition on all image patches. Image entropy based on the texture energy is used to calculate image energy strength. The texture entropy energy determined the patch size that is useful in the decomposition of image structure. The proposed method effectively eliminate smoke and enhance the degraded laparoscopic images. The qualitative results showed that the visual quality of the resultant images is improved and smoke-free. Furthermore, the quantitative scores computed of the metrics: FADE, Blur, JNBM, and Edge Intensity are significantly improved as compared to other existing methods.