Multi-focus Image Research Articles

YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer

Infrared and visible image fusion is aims to generate a composite image that can simultaneously describe the salient target in the infrared image and texture details in the visible image of the same scene. Since deep learning (DL) exhibits great feature extraction ability in computer vision tasks, it has also been widely employed in handling infrared and visible image fusion issue. However, the existing DL-based methods generally extract complementary information from source images through convolutional operations, which results in limited preservation of global features. To this end, we propose a novel infrared and visible image fusion method, i.e., the Y-shape dynamic Trans- former (YDTR). Specifically, a dynamic Transformer module (DTRM) is designed to acquire not only the local features but also the significant context information. Furthermore, the proposed network is devised in a Y-shape to comprehensively maintain the thermal radiation information from the infrared image and scene details from the visible image. Considering the specific information provided by the source images, we design a loss function that consists of two terms to improve fusion quality: a structural similarity (SSIM) term and a spatial frequency (SF) term. Extensive experiments on mainstream datasets illustrate that the proposed method outperforms both classical and state- of-the-art approaches in both qualitative and quantitative assess- ments. We further extend the YDTR to address other infrared and RGB-visible images and multi-focus images without fine- tuning, and the satisfactory fusion results demonstrate that the proposed method has good generalization capability. Our code is available at <uri>https://github.com/tthinking/YDTR</uri>.

Trainable Self-Guided Filter for Multi-Focus Image Fusion

Trainable Self-Guided Filter for Multi-Focus Image Fusion

Multi-focus image fusion based on HOSVD and parameter adaptive PCNN in fast local Laplacian filtering domain

To augment edge preservation capabilities and detail enhancement in multi-focus image fusion, a new Multi-focus image fusion algorithm is proposed in this paper. Firstly, we select fast local laplacian filtering (FLLF) as the multi-scale decomposition tool to decompose the source image into an approximate image and multiple detail images. Secondly, higher-order singular value decomposition (HOSVD) is applied to fuse the approximate image. Thirdly, the parameter-adaptive pulse coupled-neural network (PA-PCNN) model is used to fuse the detailed images. Finally, exploiting the combination of the fusion results of the approximate image and the detail images, the final fusion image is obtained using the FLLF inverse transform. The experimental results show that the proposed method achieves excellent performance in both subjective visual perception and objective index evaluation.

IFSepR: A General Framework for Image Fusion Based on Separate Representation Learning

This paper proposes an image fusion framework based on separate representation learning, called IFSepR. We believe that both the co-modal image and the multi-modal image have common and private features based on prior knowledge, exploiting this disentangled representation can help to image fusion, especially to fusion rule design. Inspired by the autoencoder network and contrastive learning, a multi-branch encoder with contrastive constraints is built to learn the common and private features of paired images. In the fusion stage, based on the disentangled features, a general fusion rule is designed to integrate the private features, then combining the fused private features and the common feature are fed into the decoder, reconstructing the fused image. We perform a series of evaluations on three typical image fusion tasks, including multi-focus image fusion, infrared and visible image fusion, medical image fusion. Quantitative and qualitative comparison with five state-of-art image fusion methods demonstrates the advantages of our proposed model.

Multi-focus image fusion algorithm based on SML and difference image

Multi-focus image fusion algorithm based on SML and difference image

Multi-Scale Dilated Convolutional Neural Network Based Multi-Focus Image Fusion Algorithm

Multi-Scale Dilated Convolutional Neural Network Based Multi-Focus Image Fusion Algorithm

SIFT feature extraction method for the defocused blurred area of multi-focus images

SIFT feature extraction method for the defocused blurred area of multi-focus images

The review of multi-focus image fusion methods based on deep learning

The review of multi-focus image fusion methods based on deep learning

Depth-Distilled Multi-Focus Image Fusion

Depth-Distilled Multi-Focus Image Fusion

A Convolutional Neural Network-Based Conditional Random Field Model for Structured Multi-Focus Image Fusion Robust to Noise

The limited depth of field of optical lenses, makes multi-focus image fusion (MFIF) algorithms of vital importance. Lately, Convolutional Neural Networks (CNN) have been widely adopted in MFIF methods, however their predictions mostly lack structure and are limited by the size of the receptive field. Moreover, since images have noise due to various sources, the development of MFIF methods robust to image noise is required. A novel robust to noise Convolutional Neural Network-based Conditional Random Field (mf-CNNCRF) model is introduced. The model takes advantage of the powerful mapping between input and output of CNN networks and the long range interactions of the CRF models in order to reach structured inference. Rich priors for both unary and smoothness terms are learned by training CNN networks. The α -expansion graph-cut algorithm is used to reach structured inference for MFIF. A new dataset, which includes clean and noisy image pairs, is introduced and is used to train the networks of both CRF terms. A low-light MFIF dataset is also developed to demonstrate real-life noise introduced by the camera sensor. Qualitative and quantitative evaluation prove that mf-CNNCRF outperforms state-of-the-art MFIF methods for clean and noisy input images, while being more robust to different noise types without requiring prior knowledge of noise.

MLNet: A Multi-Domain Lightweight Network for Multi-Focus Image Fusion

Existing multi-focus image fusion (MFIF) methods are difficult to achieve satisfactory results in both fusion performance and rate simultaneously. The spatial domain methods are hard to determine the focus/defocus boundary (FDB), and the transform domain methods are likely to damage the content information of the source images. Moreover, the deep learning-based MFIF methods are usually confronted with low rate due to complex models and enormous learnable parameters. To address these issues, we propose a multi-domain lightweight network (MLNet) for MFIF, which can achieve competitive results in both performance and rate. The proposed MLNet mainly includes three modules, namely focus extraction (FE), focus measure (FM) and image fusion (IF). In the interpretable FE module, the image features extracted by discrete cosine transform-based convolution (DCTConv) and local binary pattern-based convolution (LBPConv) are concatenated and fed into the FM module. DCTConv based on transform domain takes DCT coefficients to construct a fixed convolution kernel without parameter learning, which can effectively capture the high/low frequency content of the image. LBPConv based on spatial domain can achieve structure features and gradient information from source images. In the FM module, a 3-layer 1&#x00D7;1 convolution with a few learnable parameters is employed to generate the initial decision map, which has the properties of flexible input. The fused image is obtained by the IF module according to the final decision map. In terms of quantitative and qualitative evaluations, extensive experiments validate that the proposed method outperforms existing state-of-the-art methods on three public datasets. In addition, the proposed MLNet contains only 0.01M parameters, which is 0.2&#x0025; of the first CNN-based MFIF method [25].

Deep fusion prior for plenoptic super-resolution all-in-focus imaging

Multi-focus image fusion (MFIF) and super-resolution (SR) are the inverse problem of imaging model, purposes of MFIF and SR are obtaining all-in-focus and high-resolution 2D mapping of targets. Though various MFIF and SR methods have been designed; almost all the them deal with MFIF and SR separately. This paper unifies MFIF and SR problems in the physical perspective as the multi-focus image super resolution fusion (MFISRF), and we propose a novel unified dataset-free unsupervised framework named deep fusion prior (DFP) based-on deep image prior (DIP) to address such MFISRF with single model. Experiments have proved that our proposed DFP approaches or even outperforms those state-of-art MFIF and SR method combinations. To our best knowledge, our proposed work is a dataset-free unsupervised method to simultaneously implement the multi-focus fusion and super-resolution task for the first time. Additionally, DFP is a general framework, thus its networks and focus measurement tactics can be continuously updated to further improve the MFISRF performance. DFP codes are open source available at http://github.com/GuYuanjie/DeepFusionPrior.

DGS-Fuse: unsupervised image fusion network combining global information

To better preserve the global semantic information and edge information of the original image during the multi-focus image fusion process, we propose an encoder–decoder network, which combines pixel loss function, multiscale structural similarity loss function and total variation loss function to further reduce the detail loss in the image reconstruction process. The model introduces dynamic convolution and global context network to improve its feature expression ability and global context modeling ability. In the fusion stage, the fusion strategy based on the edge feature map is used to fuse the two feature maps output by the encoder to obtain the final decision map. Combining the gradient information of the deep feature map and the edge feature map, the fusion strategy can more effectively extract the flat parts of the focused and defocused regions and enhance the edge features of the image. Finally, the proposed algorithm is compared with seven advanced fusion algorithms, and the results are superior to other fusion algorithms in subjective and objective evaluation indexes.

High‐resolution 2.5D multifocusing imaging of a crooked seismic profile in a crystalline rock environment: Results from the Larder Lake area, Ontario, Canada

ABSTRACTA high‐resolution seismic reflection transect was acquired over a hard‐rock geological setting along an existing roadway in the Larder Lake area of the Superior Craton of Canada for the Metal Earth project in 2017. This profile, as well as other Metal earth transects, primarily aims to enhance the knowledge and to better understand the subsurface geology of the Abitibi Greenstone Belt within the Canadian Shield. The complex geological settings of the study area as well as the tribulations caused by the survey geometry have made the imaging and velocity field estimation more challenging. A recently introduced 2.5D multifocusing stacking method is one potential solution for processing crooked‐line seismic data with a poor signal‐to‐noise ratio. The 2.5D multifocusing approach offers more realistic modelling of the zero‐offset wavefield by explicitly accounting for the midpoint dispersion and cross‐dip effects. The main practical problem of the 2.5D multifocusing implementation is the simultaneous determination of the optimal wavefield parameters for each image point and time location. We address this optimization problem using a multidimensional constrained differential evolution global optimization algorithm, as this improves the efficiency and accuracy of the estimation. We have also designed an efficient processing sequence for multifocusing seismic imaging. The performance of the 2.5D multifocusing procedure has been examined on a synthetic model, generated using the same real acquisition geometry. Numerical tests demonstrate that the 2.5D multifocusing technique can produce a more focused stack with the primary reflections appearing at their poststack correct locations, and the procedure can also provide reliable estimates of interface dips. Due to the importance and difficulty of imaging the data, several conventional and advanced processing strategies have been attempted on the transect, specifically: 2D phase‐shift time migration of a dip moveout corrected stack; 2D prestack Kirchhoff time migration; swath 3D poststack migration; and our 2.5D multifocusing imaging algorithm. We found that applying the 2.5D multifocusing stacking algorithm followed by a poststack time migration approach improved the resolution of the image significantly compared to all the conventional and advanced methods and identified new reflections. The 2.5D multifocusing method also focused the steeply dipping reflections more coherently, which resolved ambiguities in geological architecture by understanding the location and continuity of structures. The method also accurately extracts 3D structural information and results in an improved signal‐to‐noise ratio.

Multi-focus image fusion: Transformer and shallow feature attention matters

The depth of field (DOF) of camera equipment is generally limited, so it is very difficult to get a fully focused image with all the objects clear after taking only one shot. A way to obtain a fully focused image is to use a multi-focus image fusion method, which fuses multiple images with different focusing depths into one image. However, most of the existing methods focus too much on the fusion accuracy of a single pixel, ignoring the integrity of the target and the importance of shallow features, resulting in internal errors and boundary artifacts, which need to be repaired after a long time of post-processing. In order to solve these problems, we propose a cascade network based on Transformer and attention mechanism, which can directly obtain the decision map and fusion result of focusing/defocusing region through end-to-end processing of source image, avoiding complex post-processing. For improving the fusion accuracy, this paper introduces the joint loss function, which can optimize the network parameters from three aspects. Furthermore, In order to enrich the shallow features of the network, a global attention module with shallow features is designed. Extensive experiments were conducted, including a large number of ablation experiments, 6 objective measures and a variety of subjective visual comparisons. Compared with 9 state-of-the-art methods, the results show that the proposed network structure can improve the quality of multi-focus fusion images and the performance is optimal.

Preference Learning to Multifocus Image Fusion via Generative Adversarial Network

Multifocus image fusion (MFIF) is to produce an all-in-focus fused image by integrating a pair of multifocus images with the same scene. In this article, an effective focus detection method is proposed for MFIF, by a generative adversarial network (GAN) with preference learning (PL). Benefitting from more obvious focus characteristics from the luminance channel in HSV, we take this luminance as the input of our GAN to carry out the easier focus detection, instead of the grayscale images in existing methods. On the other hand, to train our GAN more effectively, we utilize the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${\ell }_{{2},{1}}$ </tex-math></inline-formula> norm to construct a focus fidelity loss with structural group sparseness, to regularize the generator loss, pledging a more accurate focus confidence map. More importantly, a novel learning strategy, termed PL, is further developed to enhance model training. Functionally, it assigns a larger learning weight to a sample more difficult to be learned. Extensive experiments demonstrate that our proposed method is superior to other state-of-the-art methods.

ZMFF: Zero-shot multi-focus image fusion

Multi-focus image fusion (MFF) is an effective way to eliminate the out-of-focus blur generated in the imaging process. The difficulties in distinguishing different blur levels and the lack of real supervised data make multi-focus image fusion remain a challenging task after decades of research. According to deep image prior (DIP) (Ulyanov et al., 2018), a neural network itself can capture the low-level statistics of a single image and is successfully used as a prior for solving many inverse problems without the need for handmade priors or priors learned from large-scale datasets. Motivated by this idea, we propose a novel multi-focus image fusion framework named ZMFF comprised of a deep image prior network to model the deep prior of the fused image and a deep mask prior network to model the deep prior of the focus map corresponding to each source image. Without the labor-intensive training pair collection, our method achieves zero-shot learning and avoids the domain shifting problem due to the inconsistency between the manually degraded multi-focus images and the real ones. As far as we know, it is the first unsupervised and untrained deep model for the MFF task. Extensive experiments on both synthetic and real-world datasets demonstrate the promising performance, generalization and flexibility of our approach. Source code is available at https://github.com/junjun-jiang/ZMFF.

Robust extension of light fields with probable 3D distribution based on iterative scene estimation from multi-focus images

In this study, we propose an iterative approach to perform pseudo-scene reconstruction based on multi-focus images for estimating the 4D light fields of the scene robustly. An analogous scene-independent linear filtering approach has been proposed to efficiently achieve free viewpoint image reconstruction and scene refocusing with virtual bokeh by solely extracting 3D frequency components. However, it only succeeded in synthesizing limited regions of 4D light fields from multi-focus images, which can lead to awful degradation of reconstructed free viewpoint images outside the range due to the lack of the corresponding frequency components. In this paper, we directly utilize a linear relationship of 3D convolution combining multi-focus images and 3D scene by 3D blurring filters to estimate the 3D distribution itself of the scene instead of deriving the scene-independent linear filters from the relationship. In other words, we perform the scene reconstruction as a simplified convex optimization problem by modeling multi-focus images in multi-layered blurring composed of the scene and a bokeh filter. Algebraic reconstruction techniques of computed tomography, which uses the classical steepest descent method combined with projection onto convex sets (POCS) to ensure the convexity of the problem constraints, are introduced into such 3D scene reconstruction. The new method can be highly anticipated to assist us in extensively estimating robust 4D light fields. Experimental results illustrate that this approach enables good quality 3D image reconstruction with a wider range of 4D light fields. We also discuss the reduction of high computational costs and enormous memory requirements imposed by iterative algebraic reconstruction techniques. The quality of the obtained 4D light fields is comprehensively evaluated using both synthetic and real multi-focus images of practicable image resolution and depth range, additionally to an analysis of the faster processing times when comparing our iterative approach to the performance of the prior linear filters.

Infrared and visible image fusion using a guiding network to leverage perceptual similarity

In infrared (IR) and visible image fusion, visual appearance of fused images produced by an end-to-end fusion network relies on a loss function that defines the desired properties of the fusion results. The previous approaches mainly focus on measuring the similarity of an output image with respective IR and visible images using pixel-wise loss functions such as mean square error or structural similarity. This can lead, however, to preserve only local information in the fused images, and it may lose semantic information in the scene. In this work, we propose a new joint loss function for the image fusion, which considers visual quality and subsequent applications simultaneously. Our joint loss function guides the fusion network by transferring the knowledge of a pre-trained network to leverage perceptual similarity. The contrast loss function serves to prevent the reduction of global and local image contrasts in the fused image. The conventional data fidelity is also considered in the proposed function. Then, the joint loss function is used to train our U-Net like fusion network, a hybrid of convolutional and transformer blocks. Experimental results show that the proposed method preserves detailed scene structure from the both of source images and demonstrates the better quantitative results. Furthermore, the proposed fusion schemes can be used for combining multi-exposure and multi-focus image pairs. To show the further effectiveness of IR and visible image fusion, we apply the results of fusion methods for downstream tasks so that the proposed method improves the cases of only using original modalities.

MUFusion: A general unsupervised image fusion network based on memory unit

Existing image fusion approaches are committed to using a single deep network to solve different image fusion problems, achieving promising performance in recent years. However, devoid of the ground-truth output, in these methods, only the appearance from source images can be exploited during the training process to generate the fused images, resulting in suboptimal solutions. To this end, we advocate a self-evolutionary training formula by introducing a novel memory unit architecture (MUFusion). In this unit, specifically, we utilize the intermediate fusion results obtained during the training process to further collaboratively supervise the fused image. In this way, our fusion results can not only learn from the original input images, but also benefit from the intermediate output of the network itself. Furthermore, an adaptive unified loss function is designed based on this memory unit, which is composed of two loss items, i.e., content loss and memory loss. In particular, the content loss is calculated based on the activity level maps of source images, which can constrain the output image to contain specific information. On the other hand, the memory loss is obtained based on the previous output of our model, which is utilized to force the network to yield fusion results with higher quality. Considering the handcrafted activity level maps cannot consistently reflect the accurate salience judgement, we put two adaptive weight items between them to prevent this degradation phenomenon. In general, our MUFusion can effectively handle a series of image fusion tasks, including infrared and visible image fusion, multi-focus image fusion, multi-exposure image fusion, and medical image fusion. Particularly, the source images are concatenated in the channel dimension. After that, a densely connected feature extraction network with two scales is used to extract the deep features of the source images. Following this, the fusion result is obtained by two feature reconstruction blocks with skip connections from the feature extraction network. Qualitative and quantitative experiments on 4 image fusion subtasks demonstrate the superiority of our MUFusion, compared to the state-of-the-art methods.