Multi-focus Image Fusion Research Articles

Multi-focus image fusion with convolutional neural network based on Dempster-Shafer theory

Convolutional neural networks (CNN) have been applied to many fields including image classification. Multi-focus image fusion can be regarded as the classification of focused areas and unfocused areas. Therefore, CNN has been widely used in multi-focus image fusion. However, most methods only use information from the last convolutional layer to complete the fusion task, which leads to a suboptimal fusion result. Aiming to solve this problem, we propose a novel convolutional neural network based on the Dempster-Shafer theory (DST) for multi-focus image fusion. Firstly, as a theoretical method for dealing with uncertain issues, the DST is introduced to fuse the results from different branch layers, thus increasing the reliability of the results. Also, a gradient residual block is designed to boost the utilization of edge information by the network while reducing the dimension from feature maps in the branch layers, thereby improving the performance of the network and reducing the number of training parameters. Compared with other state-of-the-art fusion methods, the decision map of the proposed method is more precise. And objectively, the average metrics of our proposed method for the 20 images from the ``Lytro" and ``Nature" datasets perform best in terms of information entropy, mutual information, structural similarity metric, and visual perception metric.

GeFuNet: A knowledge-guided deep network for the infrared and visible image fusion

Deep learning has achieved great success in computer vision and image processing applications. Several pioneering works also succeeded in image fusion. However, due to the lack of ground truth, deep networks for image fusion are challenging to be well trained, restricting their fusion performance. In this paper, we design a knowledge-guided deep network, Generative-Fusion Network (GeFuNet), which takes natural images as its training data and ground truth. GeFuNet consists of two sub-networks: data generation subnet and image fusion subnet. In training, guided by the knowledge that real infrared and visible images respectively mainly contain contour and texture information, the data generation subnet is trained to generate pseudo infrared and pseudo visible images. The image fusion subnet is then trained using these pseudo images and further supervised by the natural images. The image fusion subnet adopts a hierarchical block to effectively extract and fuse multi-level information from the source images, which is learned by detail enhancement training. During the inference, the trained image fusion subnet is used to fuse real infrared and visible images. The experimental results show that GeFuNet can extract a complete contour and more details from the infrared and visible images, which are fused into the final image. The results also prove the applicability of GeFuNet on multi-scale image fusion, multi-focus image fusion, medical image fusion, and multi-exposure image fusion.

An unsupervised multi‐focus image fusion method based on Transformer and U‐Net

This work presents a multi-focus image fusion method based on Transformer and U-Net with an unsupervised training fashion. In this work, the authors introduce Transformer into image fusion because it has great ability to capture the global dependencies and low-frequency features. In image processing, convolutional neural network (CNN) has good performance of detailed feature extraction but a weakness for global feature extraction, and Transformer has limited power in local or detailed information extraction but a strong capacity for global feature extraction. Thus, this work combines the advantages of CNN and Transformer to propose an unsupervised decision map making model for image fusion joint U-Net. The authors construct a model including feature extraction and feature reconstruction modules which correspond to the encoder and decoder network of U-Net, respectively. In addition, perceptual loss is introduced on the basis of structural similarity loss because the combination of these two loss functions can achieve better performance with lower training cost. Experiments show that the proposed image fusion method performs better fusion performance compared with the existing methods.

Multi-Focus Image Fusion Based on Hessian Matrix Decomposition and Salient Difference Focus Detection.

Multi-focus image fusion integrates images from multiple focus regions of the same scene in focus to produce a fully focused image. However, the accurate retention of the focused pixels to the fusion result remains a major challenge. This study proposes a multi-focus image fusion algorithm based on Hessian matrix decomposition and salient difference focus detection, which can effectively retain the sharp pixels in the focus region of a source image. First, the source image was decomposed using a Hessian matrix to obtain the feature map containing the structural information. A focus difference analysis scheme based on the improved sum of a modified Laplacian was designed to effectively determine the focusing information at the corresponding positions of the structural feature map and source image. In the process of the decision-map optimization, considering the variability of image size, an adaptive multiscale consistency verification algorithm was designed, which helped the final fused image to effectively retain the focusing information of the source image. Experimental results showed that our method performed better than some state-of-the-art methods in both subjective and quantitative evaluation.

Multi-focus image fusion dataset and algorithm test in real environment.

DATA REPORT article Front. Neurorobot., 18 October 2022 Volume 16 - 2022 | https://doi.org/10.3389/fnbot.2022.1024742

Multi-scale siamese networks for multi-focus image fusion

Multi-scale siamese networks for multi-focus image fusion

An improved algorithm for three-dimensional reconstruction of nonwoven microstructures by using multi-focus images

The filtration performance of nonwovens, such as virus protection and air purification, depends largely on their microstructure, that is, their internal structure. With the development of manufacturing techniques, nonwovens can be produced with complex microstructures to deliver low air resistance and high filtration efficiency. Therefore, it seems important to reconstruct an accurate three-dimensional (3D) model to describe the various microstructures for performance evaluation and product development. Multi-focus image fusion is one of the most effective ways used for the 3D nonwoven reconstruction. However, the overlapping and crossing of fibers in multi-focus images taken on an optional microscope make 3D fiber reconstruction incomplete and discontinuous. In this study, we developed a novel fiber segmentation and 3D nonwoven reconstruction algorithm to restore fiber continuity. The fiber web image was firstly thinned into the fiber mid-axis or skeleton, and fiber fragments were detected by searching the intersections. The features, such as the angle and depth, of each fiber fragment were obtained using Harris corner detection. A cluster technique was used to group the head-points of the fiber fragments that have similar features into one cluster. All fiber fragments in the same cluster are fitted with a polynomial curve to form a complete fiber in the 3D space. The experimental results show that this new method is suitable for the reconstruction of nonwovens with complex structures, such as fiber bonding or crossing, and can automatically detect and connect fiber fragments in 3D space. It provides an effective way for the subsequent simulation/evaluation of filtration characteristics.

Reference bioimaging to assess the phenotypic trait diversity of bryophytes within the family Scapaniaceae

Macro- and microscopic images of organisms are pivotal in biodiversity research. Despite that bioimages have manifold applications such as assessing the diversity of form and function, FAIR bioimaging data in the context of biodiversity are still very scarce, especially for difficult taxonomic groups such as bryophytes. Here, we present a high-quality reference dataset containing macroscopic and bright-field microscopic images documenting various phenotypic characters of the species belonging to the liverwort family of Scapaniaceae occurring in Europe. To encourage data reuse in biodiversity and adjacent research areas, we annotated the imaging data with machine-actionable metadata using community-accepted semantics. Furthermore, raw imaging data are retained and any contextual image processing like multi-focus image fusion and stitching were documented to foster good scientific practices through source tracking and provenance. The information contained in the raw images are also of particular interest for machine learning and image segmentation used in bioinformatics and computational ecology. We expect that this richly annotated reference dataset will encourage future studies to follow our principles.

Multi-level receptive field feature reuse for multi-focus image fusion

Multi-level receptive field feature reuse for multi-focus image fusion

Improved Procedure for Multi-Focus Images Using Image Fusion with qshiftN DTCWT and MPCA in Laplacian Pyramid Domain

Multi-focus image fusion (MIF) uses fusion rules to combine two or more images of the same scene with various focus values into a fully focused image. An all-in-focus image refers to a fully focused image that is more informative and useful for visual perception. A fused image with high quality is essential for maintaining shift-invariant and directional selectivity characteristics of the image. Traditional wavelet-based fusion methods, in turn, create ringing distortions in the fused image due to a lack of directional selectivity and shift-invariance. In this paper, a classical MIF system based on quarter shift dual-tree complex wavelet transform (qshiftN DTCWT) and modified principal component analysis (MPCA) in the laplacian pyramid (LP) domain is proposed to extract the focused image from multiple source images. In the proposed fusion approach, the LP first decomposes the multi-focus source images into low-frequency (LF) components and high-frequency (HF) components. Then, qshiftN DTCWT is used to fuse low and high-frequency components to produce a fused image. Finally, to improve the effectiveness of the qshiftN DTCWT and LP-based method, the MPCA algorithm is utilized to generate an all-in-focus image. Due to its directionality, and its shift-invariance, this transform can provide high-quality information in a fused image. Experimental results demonstrate that the proposed method outperforms many state-of-the-art techniques in terms of visual and quantitative evaluations.

(Retracted) Application of image fusion algorithm based on digital media technology in modeling design

The product modeling design is the concrete manifestation of realizing the unified identification goal of the corporate image. It can shape and disseminate the corporate image, create individual brands, and win in the fierce market competition. The product modeling design needs to serve the overall image design of the enterprise, with product design as the core, around people’s needs for products, and to the greatest extent possible to meet the needs of individuals and society to obtain a general sense of recognition. It changes people’s way of life and improves the quality and level of life. Different pieces of information need to be combined when designing a product. To achieve the best effect, we aim to study the application of image fusion algorithm based on digital media technology in modeling design. Digital media technology is a technology that combines abstract numbers, physical media, and computer technology. It is a generation of digital media technology for digital audio, digital video, digital film, computer animation, virtual reality, and so on. A compressed sensing multifocus image fusion algorithm based on non-subsampled shearlet transform is proposed. It improves the efficiency of the fusion algorithm without affecting the visual effect. We analyze the shape design of wearable smart products, summarizes the characteristics of product shape design, as well as the aesthetic value is shown, and summarizes the entry point of product shape design. The experimental results of this paper show that the proportion of choosing the classic style is 55.3%, which shows that people are more comfortable with the classic style, so they are more likely to buy. Health monitoring is the mainstream function of wearable products, and its proportion is as high as 71.3%, indicating that people are more concerned about health issues.

Conditional Random Field-Guided Multi-Focus Image Fusion.

Multi-Focus image fusion is of great importance in order to cope with the limited Depth-of-Field of optical lenses. Since input images contain noise, multi-focus image fusion methods that support denoising are important. Transform-domain methods have been applied to image fusion, however, they are likely to produce artifacts. In order to cope with these issues, we introduce the Conditional Random Field (CRF) CRF-Guided fusion method. A novel Edge Aware Centering method is proposed and employed to extract the low and high frequencies of the input images. The Independent Component Analysis—ICA transform is applied to high-frequency components and a Conditional Random Field (CRF) model is created from the low frequency and the transform coefficients. The CRF model is solved efficiently with the -expansion method. The estimated labels are used to guide the fusion of the low-frequency components and the transform coefficients. Inverse ICA is then applied to the fused transform coefficients. Finally, the fused image is the addition of the fused low frequency and the fused high frequency. CRF-Guided fusion does not introduce artifacts during fusion and supports image denoising during fusion by applying transform domain coefficient shrinkage. Quantitative and qualitative evaluation demonstrate the superior performance of CRF-Guided fusion compared to state-of-the-art multi-focus image fusion methods.

Multi-focus image fusion based on unsupervised learning

Multi-focus image fusion based on unsupervised learning

Unified gradient- and intensity-discriminator generative adversarial network for image fusion

This study proposes a unified gradient- and intensity-discriminator generative adversarial network for various image fusion tasks, including infrared and visible image fusion, medical image fusion, multi-focus image fusion, and multi-exposure image fusion. On the one hand, we unify all fusion tasks into discriminating a fused image’s gradient and intensity distributions based on a generative adversarial network. The generator adopts a dual-encoder–single-decoder framework to extract source image features by using different encoder paths. A dual-discriminator is employed to distinguish the gradient and intensity, ensuring that the generated image contains the desired geometric structure and conspicuous information. The dual adversarial game can tackle the generative adversarial network’s mode collapse problem. On the other hand, we define a loss function based on the gradient and intensity that can be adapted to various fusion tasks by using varying relevant parameters with the source images. Qualitative and quantitative experiments on publicly available datasets demonstrate our method’s superiority over state-of-the-art methods.

Improved Image Fusion Method Based on Sparse Decomposition

In the principle of lens imaging, when we project a three-dimensional object onto a photosensitive element through a convex lens, the point intersecting the focal plane can show a clear image of the photosensitive element, and the object point far away from the focal plane presents a fuzzy image point. The imaging position is considered to be clear within the limited size of the front and back of the focal plane. Otherwise, the image is considered to be fuzzy. In microscopic scenes, an electron microscope is usually used as the shooting equipment, which can basically eliminate the factors of defocus between the lens and the object. Most of the blur is caused by the shallow depth of field of the microscope, which makes the image defocused. Based on this, this paper analyzes the causes of defocusing in a video microscope and finds out that the shallow depth of field is the main reason, so we choose the corresponding deblurring method: the multi-focus image fusion method. We proposed a new multi-focus image fusion method based on sparse representation (DWT-SR). The operation burden is reduced by decomposing multiple frequency bands, and multi-channel operation is carried out by GPU parallel operation. The running time of the algorithm is further reduced. The results indicate that the DWT-SR algorithm introduced in this paper is higher in contrast and has much more details. It also solves the problem that dictionary training sparse approximation takes a long time.

UIFGAN: An unsupervised continual-learning generative adversarial network for unified image fusion

In this paper, we propose a novel unsupervised continual-learning generative adversarial network for unified image fusion, termed as UIFGAN. In our model, for multiple image fusion tasks, a generative adversarial network for training a single model with memory in a continual-learning manner is proposed, rather than training an individual model for each fusion task or jointly training multiple tasks. We use elastic weight consolidation to avoid forgetting what has been learned from previous tasks when training multiple tasks sequentially. In each task, the generation of the fused image comes from the adversarial learning between a generator and a discriminator. Meanwhile, a max-gradient loss function is adopted for forcing the fused image to obtain richer texture details of the corresponding regions in two source images, which applies to most typical image fusion tasks. Extensive experiments on multi-exposure, multi-modal and multi-focus image fusion tasks demonstrate the advantages of our method over the state-of-the-art approaches.

Multi-focus image fusion with enhancement filtering for robust vascular quantification using photoacoustic microscopy.

Accurate identification and quantification of microvascular patterns are important for clinical diagnosis and therapeutic monitoring using optical-resolution photoacoustic microscopy (OR-PAM). Due to its limited depth of field, conventional OR-PAM may not fully reveal microvascular patterns with enough details in depth range, which affects the segmentation and quantification. Here, we propose a robust vascular quantification approach via combining multi-focus image fusion with enhancement filtering (MIFEF). The multi-focus image fusion is constructed based on multi-scale gradients and image matting to improve image fusion quality by considerably achieving accurate focus measurement for initial segmentation as well as decision map refinement. The enhancement filtering identifies the vessels and handles noise without deforming microvasculature. The performance of the MIFEF were evaluated employing a leaf phantom, mouse livers and brains. The proposed method for OR-PAM can significantly facilitate the clinical provision of optical biopsy of vascular-related diseases.

Reliable and robust low rank representation based noisy images multi-focus image fusion

Reliable and robust low rank representation based noisy images multi-focus image fusion

Fusion of Multi-Focus Images using Jellyfish Search Optimizer

When obtaining an image of a scene, the lens focuses on objects at a certain distance, and objects at other distances are blurred. This is called the limited depth of field problem. An approach for solving this problem is multi-focus image fusion. A clearer view of the entire scene is obtained by using the multi-focus image fusion method. For this method, at least two images captured at different focuses are combined. Various algorithms have been developed for multi-focus image fusion methods. For multi-focus image fusion, pixel-level block-based methods are commonly used. The block size is a factor that significantly affects the fusion performance. As a result, the block size parameter must be improved. The Jellyfish search optimization algorithm (JSA) is used to propose a block-based multi-focus image fusion approach based on the optimal selection of clearer image blocks from source images. The results of DWTPCA, DCHWT, APCA, PCA, SWTDWT and SWT methods, which are traditional image fusion methods, and ABC (artificial bee colony) and JSA optimization algorithms, which are metaheuristic methods, are compared. In addition, it has been determined that the JSA method has better performance than other traditional methods when compared both visually and quantitatively.

Efficient Multi-focus Image Fusion Using Parameter Adaptive Pulse Coupled Neural Network Based Consistency Verification

Efficient Multi-focus Image Fusion Using Parameter Adaptive Pulse Coupled Neural Network Based Consistency Verification