Multi-focus Fusion Research Articles

A Lightweight Pixel-Level Unified Image Fusion Network.

In recent years, deep-learning-based pixel-level unified image fusion methods have received more and more attention due to their practicality and robustness. However, they usually require a complex network to achieve more effective fusion, leading to high computational cost. To achieve more efficient and accurate image fusion, a lightweight pixel-level unified image fusion (L-PUIF) network is proposed. Specifically, the information refinement and measurement process are used to extract the gradient and intensity information and enhance the feature extraction capability of the network. In addition, these information are converted into weights to guide the loss function adaptively. Thus, more effective image fusion can be achieved while ensuring the lightweight of the network. Extensive experiments have been conducted on four public image fusion datasets across multimodal fusion, multifocus fusion, and multiexposure fusion. Experimental results show that L-PUIF can achieve better fusion efficiency and has a greater visual effect compared with state-of-the-art methods. In addition, the practicability of L-PUIF in high-level computer vision tasks, i.e., object detection and image segmentation, has been verified.

Trans2Fuse: Empowering image fusion through self-supervised learning and multi-modal transformations via transformer networks

Image fusion enhances a single image by integrating information from multiple sources with complementary data. Present end-to-end fusion methods often face overfitting or intricate parameter tuning due to inadequate task-specific training data. To address this, two-stage approaches utilize encoder–decoder networks trained on extensive natural image datasets, yet suffer from limited performance due to domain disparities. In this work, we devise a novel encoder–decoder fusion framework and introduce a self-supervised scheme based on destruction–reconstruction. This approach facilitates task-specific feature learning by proposing three auxiliary tasks: pixel intensity non-linear transformation for multi-modal fusion, brightness transformation for multi-exposure fusion, and noise transformation for multi-focus fusion. By randomly selecting one task during model training, we mutually reinforce different fusion tasks, enhancing network generalizability. We innovate an encoder combining Convolutional Neural Network (CNN) and Transformer to extract both local and global features. Rigorous evaluations against 11 traditional and deep learning-based methods span four benchmark datasets: infrared-visible fusion, medical fusion, multi-exposure fusion, and multi-focus fusion. Comprehensive assessments, encompassing nine metrics from diverse viewpoints, consistently demonstrate the superior performance of our approach in all scenarios. We will make our code, datasets, and fused images publicly available.

An end-to-end anti-shaking multi-focus image fusion approach

In this paper, an end-to-end multi-focus fusion network is proposed to solve the mis-registration problem of the captured images. This paper takes the shaking and camera breathing effects that usually exist when acquiring multiple partially focused images into account. The new network consists of a homography pre-correction module, a deformable convolutional fine-correction module, an attention based fusion module and a reconstruction module. The homography pre-correction module is proposed to achieve a coarse correction for multi-focus images. The deformable convolution fine-correction module enables the fusion algorithm to achieve an effective fine alignment for small shaking in multi-focus images. The attention based fusion module and the image reconstruction module are proposed to achieve high definition fusion results. This paper produces a dedicated shaking dataset for training and testing. The algorithm’s superior performance in depth-of-field extension and generalisation to different scenes can be seen through sufficient experiments on test datasets, real scenes and image sequences. The ablation experiments also demonstrate the necessity of each module in the algorithm.

Multi-Focus Image Fusion via Distance-Weighted Regional Energy and Structure Tensor in NSCT Domain.

In this paper, a multi-focus image fusion algorithm via the distance-weighted regional energy and structure tensor in non-subsampled contourlet transform domain is introduced. The distance-weighted regional energy-based fusion rule was used to deal with low-frequency components, and the structure tensor-based fusion rule was used to process high-frequency components; fused sub-bands were integrated with the inverse non-subsampled contourlet transform, and a fused multi-focus image was generated. We conducted a series of simulations and experiments on the multi-focus image public dataset Lytro; the experimental results of 20 sets of data show that our algorithm has significant advantages compared to advanced algorithms and that it can produce clearer and more informative multi-focus fusion images.

Effective lung cancer diagnosis using multi-focus fusion of CT and PET images with deep learning strategies

ABSTRACT Lung cancer is among the most prevalent forms of the disease. This study describes multi-modal image fusion for lung cancer diagnosis using a combination of PET and CT images. The proposed approach involves breaking down the images into constituent parts, amalgamating the low and high-frequency bands, and using a Siamese Inception V3 approach to form a weight map that merges the pixel motion data from the images. The fused images are acquired using the inverse DTCWT of the fused coefficient, and deep EfficientNet is used to extract features from the fused image. The embedded whale optimization with mean grey wolf optimization methods are used to choose features. The proposed approach is tested on CT and PET images of normal and lung cancer, and a deep learning technique, CNN/BiLSTM, is used to classify lung cancer. The proposed classifier has an AUC of 92.34 percent, which is superior to other proposed categorizers.

Deep fusion prior for plenoptic super-resolution all-in-focus imaging

Multi-focus image fusion (MFIF) and super-resolution (SR) are the inverse problem of imaging model, purposes of MFIF and SR are obtaining all-in-focus and high-resolution 2D mapping of targets. Though various MFIF and SR methods have been designed; almost all the them deal with MFIF and SR separately. This paper unifies MFIF and SR problems in the physical perspective as the multi-focus image super resolution fusion (MFISRF), and we propose a novel unified dataset-free unsupervised framework named deep fusion prior (DFP) based-on deep image prior (DIP) to address such MFISRF with single model. Experiments have proved that our proposed DFP approaches or even outperforms those state-of-art MFIF and SR method combinations. To our best knowledge, our proposed work is a dataset-free unsupervised method to simultaneously implement the multi-focus fusion and super-resolution task for the first time. Additionally, DFP is a general framework, thus its networks and focus measurement tactics can be continuously updated to further improve the MFISRF performance. DFP codes are open source available at http://github.com/GuYuanjie/DeepFusionPrior.

Multi-focus image fusion: Transformer and shallow feature attention matters

The depth of field (DOF) of camera equipment is generally limited, so it is very difficult to get a fully focused image with all the objects clear after taking only one shot. A way to obtain a fully focused image is to use a multi-focus image fusion method, which fuses multiple images with different focusing depths into one image. However, most of the existing methods focus too much on the fusion accuracy of a single pixel, ignoring the integrity of the target and the importance of shallow features, resulting in internal errors and boundary artifacts, which need to be repaired after a long time of post-processing. In order to solve these problems, we propose a cascade network based on Transformer and attention mechanism, which can directly obtain the decision map and fusion result of focusing/defocusing region through end-to-end processing of source image, avoiding complex post-processing. For improving the fusion accuracy, this paper introduces the joint loss function, which can optimize the network parameters from three aspects. Furthermore, In order to enrich the shallow features of the network, a global attention module with shallow features is designed. Extensive experiments were conducted, including a large number of ablation experiments, 6 objective measures and a variety of subjective visual comparisons. Compared with 9 state-of-the-art methods, the results show that the proposed network structure can improve the quality of multi-focus fusion images and the performance is optimal.

CRITICAL REFLECTION ON QUANTITATIVE ASSESSMENT OF IMAGE FUSION QUALITY

Abstract. Image fusion technique has been extended its development from multi-sensor fusion, multi-model fusion to multi-focus fusion. More and more advanced techniques such as deep learning have been integrated into the development of image fusion algorithms. However, as an important aspect, fusion quality assessment has been received less attention. This paper intends to reflect on the commonly used indices for quantitative assessment and investigate how they can represent the fusion quality regarding spectral preservation and spatial improvement. We found that image dissimilarities are unavoidable due to the spectral coverage of different image sensors. Image fusion should integrate these dissimilarities when they are representing spatial improvement. Such integration will naturally change the pixel values. However, as the quality indices for the assessment of spectral preservation are measuring image dissimilarities, the integration of spatial information will lead to a low fusion quality assessment. For the evaluation of spatial improvement, the quality indices only work if the spatial details have been lost; however, in the case of spatial details gain, these indices do not reflect them as spatial improvements. Moreover, this paper raises attention to image processing procedures involved in image fusion, including image geo-registration, image clipping and image resampling, which will change image statistics and thereby influence the quality assessment when statistical indices are used.

De-Fencing and Multi-Focus Fusion Using Markov Random Field and Image Inpainting

Multi-focus image fusion aims at combining source information from differently focused images. Fusion of multi-focus images has great applications in machine vision. The paper focuses on removal of fence occlusions in multi-focus images. The proposed model extracts fence occlusion map using salient image features, and refined by morphological operators. Binary operators, and inpainting methods are used for fence removal and restoration. The proposed model estimates the fence area using statistical characteristics in focused regions. Similarly, binary filtering is used to perform thinning of enlarged areas for optimised restoration. The proposed model employs guided filtering for consistency verification. Fusion and restoration results are compared using several (reference and no-reference based) image quality metrics. Simulations show that proposed scheme achieves better results (visually and quantitatively) as compared to existing state-of-the-art techniques.

High-quality image multi-focus fusion to address ringing and blurring artifacts without loss of information

High-quality image multi-focus fusion to address ringing and blurring artifacts without loss of information

Multi-Focus Network to Decode Imaging Phenotype for Overall Survival Prediction of Gastric Cancer Patients.

Gastric cancer (GC) is the third leading cause of cancer-associated deaths globally. Accurate risk prediction of the overall survival (OS) for GC patients shows significant prognostic value, which helps identify and classify patients into different risk groups to benefit from personalized treatment. Many methods based on machine learning algorithms have been widely explored to predict the risk of OS. However, the accuracy of risk prediction has been limited and remains a challenge with existing methods. Few studies have proposed a framework and pay attention to the low-level and high-level features separately for the risk prediction of OS based on computed tomography images of GC patients. To achieve high accuracy, we propose a multi-focus fusion convolutional neural network. The network focuses on low-level and high-level features, where a subnet to focus on lower-level features and the other enhanced subnet with lateral connection to focus on higher-level semantic features. Three independent datasets of 640 GC patients are used to assess our method. Our proposed network is evaluated by metrics of the concordance index and hazard ratio. Our network outperforms state-of-the-art methods with the highest concordance index and hazard ratio in independent validation and test sets. Our results prove that our architecture can unify the separate low-level and high-level features into a single framework, and can be a powerful method for accurate risk prediction of OS.

The Fusion of Multi-Focus Images Based on the Complex Shearlet Features-Motivated Generative Adversarial Network

The traditional methods for multi-focus image fusion, such as the typical multi-scale geometric analysis theory-based methods, are usually restricted by sparse representation ability and the transferring efficiency of the fusion rules for the captured features. Aiming to integrate the partially focused images into the fully focused image with high quality, the complex shearlet features-motivated generative adversarial network is constructed for multi-focus image fusion in this paper. Different from the popularly used wavelet, contourlet, and shearlet, the complex shearlet provides more flexible multiple scales, anisotropy, and directional sub-bands with the approximate shift invariance. Therefore, the features in complex shearlet domain are more effective. With of help of the generative adversarial network, the whole procedure of multi-focus fusion is modeled to be the process of adversarial learning. Finally, several experiments are implemented and the results prove that the proposed method outperforms the popularly used fusion algorithms in terms of four typical objective metrics and the comparison of visual appearance.

A multi-focus image fusion framework based on multi-scale sparse representation in gradient domain

Existing multi-focus fusion methods cannot identify focus regions of multiple source images at a pixel level well, which limits the fusion performance. To solve this problem, we propose a novel multi-focus image fusion framework that performs focus region detection by a pixel-level focus measure. First, we propose the multi-scale sparse representation in gradient domain (MSSRGD) that combines the advantages of the sparse representation and the multi-scale approach to yield a data-driven and block-level focus measure. Second, the measure values are extended to the pixel level with the adaptive recursive filter (ARF) which can make full use of the original spatial information. Finally, the fused image is generated according to an accurate decision map estimated by a pixel-wise comparison. The experimental results demonstrate that the proposed method outperforms the conventional and state-of-the-art methods in terms of both visual perceptions and objective evaluations.

Multi-focus Fusion Method Based on Multi-Scale Local Weighted Variance

Multi-focus Fusion Method Based on Multi-Scale Local Weighted Variance

基于多聚焦图像序列融合的筒状类工件内壁形貌重构方法

基于多聚焦图像序列融合的筒状类工件内壁形貌重构方法

基于减影图像与NSML的多焦点融合图像增强

基于减影图像与NSML的多焦点融合图像增强

RASF and DEDR-POCS Image Fusion through Multi-layer Perceptron in SAR Imagery Systems

This paper presents a novel multi-layer perceptron (MLP) based image fusion technique, which fuses two synthetic aperture radar (SAR) images, obtained from the same spatial reflectivity map, acquired with a conventional low-cost fractional synthetic aperture radar (Fr-SAR) system, enhanced via two different methodologies. The first image is enhanced using the traditional descriptive experiment design regularization (DEDR) framework through the projection onto convex solution sets (POCS) method; the second image is enhanced with the DEDR framework by incorporating the robust adaptive spatial filtering (RASF) solution operator. This work describes a MLP based technique applied to the pixel level multi-focus fusion problem characterized by the use of image windows with the idea of reducing noise and determining which pixel is clearer between the two images. Experimental results show that the proposed novel method outperforms the discrete wavelet transform based most competing approach.

Parameter-free Gaussian PSF Model for Extended Depth of Field in Brightfield Microscopy.

Due to their limited depth of field, conventional brightfield microscopes cannot image thick specimens entirely in focus. A common way to obtain an all-in-focus image is to acquire a z-stack of images by optically sectioning the specimen and then apply a multi-focus fusion method. Unfortunately, for undersampled image stacks, fusion methods cannot remove the blur in regions where the in-focus position is between two optical sections. In this work, we propose a parameter-free Gaussian PSF model in which the all-in-focus image together with both the depth map and sampling distances in image plane are estimated from the image sequence automatically, without knowledge on the z-stack acquisition. In a maximum a posteriori framework, an iteratively reweighted least squares method is used to estimate the image and an adaptive scaled gradient descent method is utilized to estimate the depth map and sampling distances efficiently. Experiments on synthetic and real data demonstrate that the proposed method outperforms the current state-of-the-art, mitigating fusion artifacts and recovering sharper edges.

Surface profiling of object using varifocal lens with image contrast

The measurement of automated and fast focusing on the improvement of speed and accuracy is an important issue in industrial inspection and biomedical microscopy. In this study, we developed a novel optical imaging system in which varifocal lenses are used to characterize the 3D surface topography of samples. The performance of the proposed system was evaluated using twelve focusing algorithms. Experimental results demonstrate the feasibility and effectiveness of the proposed system in the near real-time construction of multi-focus fusion images with large depth of field from which to derive 3D surface profiles.

A novel multi-focus image fusion method using multiscale shearing non-local guided averaging filter

The multiscale transform based image fusion method cannot effectively preserve detail information and easily produce artifacts. Faced with these problems, we present a novel multi-focus fusion method based on multiscale shearing non-local guided averaging filter (MSNLGA). First, we construct a new multiscale geometrical analysis tool called MSNLGA, which combines the non-local guided averaging filter with the shearing filter bank. The MSNLGA can represent the intrinsic geometric structure of image sparsely because of its good property in multiscale, multi-direction and shift-invariance. Then, the MSNLGA is used to decompose source images to obtain approximate subbands and directional detail subbands. For the approximate subbands, we extract the anti-noise spatial frequency feature from the source images to guide its fusion. For the directional detail subbands, we introduce the convolutional sparse representation, which is a model that can achieve sparse representation of an entire subband, to represent each subband so as to obtain the activity level measurement to fuse directional detail subbands. Finally, the fused image can be obtained by the inverse MSNLGA of the fused subbands. The experimental results show that the proposed method can be competitive with or even outperform the state-of-the-art fusion methods in terms of both visual and quantitative evaluations.