Multi-focus Image Fusion Research Articles

Multi-Focus Microscopy Image Fusion Based on Swin Transformer Architecture

In this study, we introduce the U-Swin fusion model, an effective and efficient transformer-based architecture designed for the fusion of multi-focus microscope images. We utilized the Swin-Transformer with shifted window and path merging as the encoder for extracted hierarchical context features. Additionally, a Swin-Transformer-based decoder with patch expansion was designed to perform the un-sampling operation, generating the fully focused image. To enhance the performance of the feature decoder, the skip connections were applied to concatenate the hierarchical features from the encoder with the decoder up-sample features, like U-net. To facilitate comprehensive model training, we created a substantial dataset of multi-focus images, primarily derived from texture datasets. Our modulators demonstrated superior capability for multi-focus image fusion to achieve comparable or even better fusion images than the existing state-of-the-art image fusion algorithms and demonstrated adequate generalization ability for multi-focus microscope image fusion. Remarkably, for multi-focus microscope image fusion, the pure transformer-based U-Swin fusion model incorporating channel mix fusion rules delivers optimal performance compared with most existing end-to-end fusion models.

Optimized MobileNetV3: a deep learning-based Parkinson’s disease classification using fused images

Background and Objective Parkinson’s disease (PD) is a progressive neurological condition that manifests motor and non-motor symptoms. Early in the course of the disease, PD patients frequently experience vocal difficulties. In the beginning, preprocessing procedures were used with multi-focus image fusion to enhance the quality of input images. It is essential to diagnose and treat PD early to ensure that patients live healthy and productive lives. Methods Tremors, rigidity in the muscles, slow movement, difficulty balance, and other psychological symptoms are some of the disease’s symptoms. One of the critical mechanisms supporting PD identification and assessment is the dynamics of handwritten records. Several machine-learning techniques have been researched for the early detection of this disease. Yet the main problem with most of these manual feature extraction methods is their poor performance and accuracy. Results This cannot be acceptable when discovering such a chronic condition. For this purpose, a powerful deep learning model is suggested to help with the early diagnosis of Parkinson’s disease. Therefore, we proposed MobileNetV3-based classification. To enhance the classification performances even more, the MobileNetV3-based approach was optimized by the Improved Dwarf Mongoose Optimization algorithm (IDMO). Conclusion The Pyramid channel-based feature attention network (PCFAN) chooses the critical features. The efficiency of the approaches is tested using the PPMI and NTUA datasets. Our proposed approach obtains 99.34% accuracy, 98.53% sensitivity, 97.78% specificity, and 99.12% F-score compared to previous methods.

A defocus and similarity attention-based cascaded network for multi-focus and misaligned image fusion

Multi-focus image fusion uses multiple images focused on different depths to generate a clear image covering the whole scene. The existing multi-focus image fusion methods do not consider the defocus successive variation in close-range photography and the camera shaking in sequence image shooting, and most methods cannot process multiple images simultaneously. We proposed an end-to-end deep learning network to generate the all-in-focus image from multi-focus and misaligned images to solve these problems. Specifically, taking multiple multi-focus and misaligned source images as input, our Defocus and Similarity Attention Fusion Network (DSAF-Net) first generates the corresponding defocus map through the Defocus-Net, and warps the source images and defocus maps to a unified camera view according to the optical flow estimated by the OpticalFlow-Net. Finally, our model applies a coarse-to-fine correspondence matching scheme to obtain the similarity weight maps, combined with the warped source images and defocus maps to fuse into a clear image. For training and testing our DSAF-Net, a multi-focus and misaligned cultural heritage photography dataset (WHU-MFM) is constructed using Blender Cycles renderer. Experiments results demonstrate that our method outperforms the state-of-the-art methods both qualitatively and quantitatively. DSAF-Net is available at https://github.com/PeimingCHEN/DSAF-Net. And WHU-MFM is available at https://github.com/PeimingCHEN/WHU-MFM-Dataset.

All-in-focus large-FOV GRIN lens imaging by multi-focus image fusion

Gradient refractive index (GRIN) lenses are useful for miniaturized and in-vivo imaging. However, the intrinsic field-dependent aberrations of these lenses can deteriorate imaging resolution and limit the effective field of view. To address these aberrations, adaptive optics (AO) has been applied which inevitably requires the incorporation of additional hardware. Here we focus on field curvature aberration and propose a computational correction scheme which fuses a z-stack of images into a single in-focus image over the entire field of view (FOV), with no AO required. We validate our method by all-in-focus wide-field imaging of a printed letter sample and fluorescently labeled mouse brain slices. The method can also provide what we believe to be a new and valuable option for imaging enhancement in the scanning-modality use of GRIN lens microscopy.

Novel hybrid CNN with Bi-LSTM multi-focus image fusion method based on modified tetrolet transform in MRI and CT images

The process of fusing different images from various imaging modalities into a single, fused image that contains a wealth of information and improves the usability of medical images in real-world applications is known as medical image fusion. The most useful features from data can be automatically extracted by deep learning models. In the recent past, the field of image fusion has been preparing to introduce a deep learning model. In this work we can achieve the multi-Focus medical image fusion by hybrid deep learning models. Here the relevant health care data are collected from database (CT & MRI brain images). Following the input images are pre-processed using sliding window and the abnormal data is eliminated using distribution map method. Further the proposed work comprises 3 steps, 1) the proposed method is used to extract the features from the input image using the modified Tetrolet transform (MMT), which uses a brain image as an input image. This model is capable of identifying anomalous trends in time series data and automatically deriving from the input data characteristics that characterise the system state.2) Propose a novel hybrid model based on CNN with Bi-LSTM (Bi-directional Short Term Memory) multi-focus image fusion method to overcome the difficulty faced by the existing fusion methods. 3) This hybrid model are used to predict the brain tumor present in the fused image. Finally, experimental results are evaluated using a variety of performance measures. From the results, we can see that our suggested model contributes to an increase in predictive performance while also lowering the complexity in terms of storage and processing time.

An autoencoder deep residual network model for multi focus image fusion

An autoencoder deep residual network model for multi focus image fusion

Multi-focus image fusion via adaptive fractional differential and guided filtering

Multi-focus image fusion via adaptive fractional differential and guided filtering

FusionCPP: Cooperative fusion of infrared and visible light images based on PCNN and PID control systems

The imaging results of infrared and visible light sensors are highly complementary. Thus, combining the two images can compensate for the shortcomings of a single sensor. However, previous fusion methods have not fully utilized the collaborative guidance of infrared and visible images in the fusion process, leading to a poor correlation between extracted features and cannot guarantee feature fusion accuracy. In order to address these challenges, this paper proposes a collaborative fusion method for infrared and visible images based on pulse-coupled neural networks (PCNN) and proportional-integral-derivative (PID) control systems, called FusionCPP. First, a coupled neural network with a dual pulse structure is designed for the feature extraction section. Unlike traditional PCNN, this network includes two pulse generators, one of which is utilized to extract the salient features of the image, while the other is employed to extract the common pulse layer of the source image and multiply the pulse layer by the coupling iteration value to obtain the base layer. Second, a detail layer is obtained by subtracting the base layer from the source image. This method accurately extracts salient and detailed features based on the coupling between source images. Finally, a closed-loop PID control system is utilized as the fusion strategy in the image fusion task, which judges the difference between the fusion result and the source image in real-time through the feedback effect of the system. The controller adaptively adjusts the fusion weight based on the difference to fuse image features into the new image accurately. Experimental results indicate that the proposed FusionCPP can maintain significant contrast and rich textures. Besides, the proposed FusionCPP has extended to multi-focus image fusion and target detection tasks to verify its effectiveness.

FusionDiff: Multi-focus image fusion using denoising diffusion probabilistic models

Multi-focus image fusion (MFIF) is an image enhancement technology with broad application prospects that can effectively extend the depth-of-field of optical lenses. This paper presents a novel MFIF method, named FusionDiff, based on denoising diffusion probabilistic models (DDPM). FusionDiff uses DDPM to fuse two source images by iteratively performing multiple denoising operations. To predict noise accurately and train the model efficiently, a lightweight U-Net architecture is designed as the conditional noise predictor. FusionDiff does not depend on any specific activity level measurement method, fusion rule or complex feature extraction network. It overcomes many algorithm design and training difficulties in existing image fusion methods. FusionDiff is noise-resistant and can still produce outstanding fusion results from source images with noise interference. In addition, FusionDiff is a few-shot learning method, which makes it suitable for image fusion tasks where training samples are relatively scarce. Experiments show that FusionDiff outperforms representative state-of-the-art methods in both visual perception and quantitative metrics. The code is available at https://github.com/lmn-ning/ImageFusionhttps://github.com/lmn-ning/ImageFusion

Multi-image transformer for multi-focus image fusion

Multi-Focus Image Fusion (MFIF) is an image enhancement task that fuses images in which different regions are in focus to achieve an all-in-focus image. In recent years, Generative Adversarial Networks (GANs)-based approaches have significantly improved the MFIF on Convolutional Neural Network (CNN) architectures. However, despite vision transformers (ViTs) achieving more successful results than CNNs in many high and low-level vision problems due to their ability to provide global connectivity, they have not been employed for MFIF until this study. We develop a Multi-image Transformer (MiT) for MFIF by being inspired by a Spatial-Temporal Transformer Network (STTN) so that global connection can be modeled along multiple input images. We call the proposed transformers-based MFIF model MiT-MFIF as it uses the developed MiT as a core component. We have made various modifications to the baseline transformer to be able to utilize vision transformers in MFIF tasks. Comprehensive experiments on standard MFIF datasets demonstrate the effectiveness of the proposed MiT-MFIF. Moreover, proposed method does not require any post-processing step like in competitor GAN-based methods while outperforming the state-of-the-art MFIF methods.

AFCANet: An adaptive feature concatenate attention network for multi-focus image fusion

For multi-focus image fusion, the existing deep learning based methods cannot effectively learn the texture features and semantic information of the source image to generate high-quality fused images. Thus, we develop a new adaptive feature concatenate attention network named AFCANet, which adaptively learns cross-layer features and retains the texture features and semantic information of images to generate visually appealing fully focused images. In AFCANet, the encoder-decoder network is used as the backbone network. In the unsupervised training stage, an adaptive cross-layer skip connection mode is designed, and a cross-layer adaptive coordinate attention module is built to acquire meaningful information from the image along with ignoring unimportant information to obtain a better image fusion effect. In addition, in the middle of the encoder-decoder network, we also introduce an effective channel attention module to fully learn the output of the encoder, and accelerate network convergence. In the inference stage, we apply the pixel-based spatial frequency fusion rules to fuse the adaptive features learned by the encoder, which can successfully combine the texture and semantic information of the image and produce a more precise decision map. Extensive experiments on public datasets and the HBU-CVMDSP dataset show that our AFCANet can effectively improve the accuracy of the decision map in the focus and defocus regions, as well as improve the ability to retain the abundant details and edge features of the source image.

Multi-focus image fusion by using swarm and physics based metaheuristic algorithms: a comparative study with archimedes, atomic orbital search, equilibrium, particle swarm, artificial bee colony and jellyfish search optimizers

Multi-focus image fusion by using swarm and physics based metaheuristic algorithms: a comparative study with archimedes, atomic orbital search, equilibrium, particle swarm, artificial bee colony and jellyfish search optimizers

Multi-focus image fusion using structure-guided flow

Existing deep learning based methods have shown their advantages in multi-focus image fusion task. However, most methods still suffer from inaccurate focus region detection. In this paper, we employ the property of part-whole relationships embedded by the Capsule Network (CapsNet) to solve the problem. Specifically, we introduce CapsNet in multi-focus image fusion task, and design a structure-guided flow module, which fully utilizes structure information to help locate focus regions. CapsNet is introduced to extract structure features by supervising gradient information of the image. Compared with traditional convolutional neural networks (CNNs), CapsNet takes into account the correlation of features from different positions, such that it encodes more compact features. Once structure features are obtained, a flow alignment module is introduced to learn flow field between structure and image features, and propagate effectively structure features to image features to make confident focus region detection. Experimental results show the proposed method achieves robust fusion performance on three publicly available multi-focus datasets, and outperforms or is comparable to the state-of-the-art methods.

Multi-Focus Image Fusion via PAPCNN and Fractal Dimension in NSST Domain

Multi-focus image fusion is a popular technique for generating a full-focus image, where all objects in the scene are clear. In order to achieve a clearer and fully focused fusion effect, in this paper, the multi-focus image fusion method based on the parameter-adaptive pulse-coupled neural network and fractal dimension in the nonsubsampled shearlet transform domain was developed. The parameter-adaptive pulse coupled neural network-based fusion rule was used to merge the low-frequency sub-bands, and the fractal dimension-based fusion rule via the multi-scale morphological gradient was used to merge the high-frequency sub-bands. The inverse nonsubsampled shearlet transform was used to reconstruct the fused coefficients, and the final fused multi-focus image was generated. We conducted comprehensive evaluations of our algorithm using the public Lytro dataset. The proposed method was compared with state-of-the-art fusion algorithms, including traditional and deep-learning-based approaches. The quantitative and qualitative evaluations demonstrated that our method outperformed other fusion algorithms, as evidenced by the metrics data such as QAB/F, QE, QFMI, QG, QNCIE, QP, QMI, QNMI, QY, QAG, QPSNR, and QMSE. These results highlight the clear advantages of our proposed technique in multi-focus image fusion, providing a significant contribution to the field.

Robust multi-focus image fusion using focus property detection and deep image matting

Multi-focus image fusion techniques are designed to integrate the focused regions of multiple source images to produce an all-in-focus image. Unfortunately, the fusion results generated by existing methods are prone to suffer from chromatic aberrations and blurred boundary regions. In this paper, a multi-focus image fusion framework combining focus property detection and deep image matting is proposed to achieve high-quality fusion results. Specifically, we introduce spatial frequency detection blocks, which integrate the spatial frequency information of source images at different levels to achieve effective focus property detection for the initial decision. Then, the trimap acquired from the initial decision is optimized using a deep matting model so as to further detect the focused/defocused boundaries and obtain a more precise decision map. Meanwhile, a new hybrid loss function is introduced, and a large-scale dataset is created to improve the performance of the fusion framework. Finally, the fusion result is obtained by integrating the focused regions via the decision map. Compared to 18 other advanced fusion methods, the proposed method outperforms all of them in terms of the seven objective metrics evaluated on two publicly available datasets. These experimental results provide strong evidence that the proposed fusion framework excels not only in subjective vision but also in quantitative analysis. The code of the proposed method will be available at https://github.com/govenda/DIMF.

Adversarial attacks on multi-focus image fusion models

Multi-focus image fusion aims to create an all-in-focus clear image by fusing a set of partially focused images. In recent years, various multi-focus image fusion methods based on deep learning have been proposed, but there is no thorough study evaluating their robustness for adversarial attacks. In this paper, we investigate the robustness of deep learning based multi-focus image fusion models to adversarial attacks. First, we generated adversarial examples which can significantly reduce the fusion quality of image fusion models. Then, a metric defocus attack intensity (DAI) was proposed to quantitatively evaluate the robustness of different models for adversarial attacks. At last, we analyzed the factors affecting model robustness, including model size and post-processing steps. Besides, we successfully attacked recent image fusion models in the black-box scene by utilizing the transferability of adversarial examples. Experimental results show that state-of-the-art image fusion models are also vulnerable to adversarial attacks, and some observations in image classifier robustness studies are not transferable to image fusion task.

Two-Stage Focus Measurement Network with Joint Boundary Refinement for Multifocus Image Fusion

Focus measurement, one of the key tasks in multifocus image fusion (MFIF) frameworks, identifies the clearer parts of multifocus images pairs. Most of the existing methods aim to achieve disposable pixel-level focus measurement. However, the lack of sufficient accuracy often gives rise to misjudgments in the results. To this end, a novel two-stage focus measurement with joint boundary refinement network is proposed for MFIF. In this work, we adopt a coarse-to-fine strategy to gradually achieve block-level and pixel-level focus measurement for producing more fine-grained focus probability maps, instead of directly predicting at the pixel level. In addition, the joint boundary refinement optimizes the performance on the focused/defocused boundary component (FDB) during the focus measurement. To improve feature extraction capability, both CNN and transformer are employed to, respectively, encode local patterns and capture long-range dependencies. Then, the features from two input branches are legitimately aggregated by modeling the spatial complementary relationship in each pair of multifocus images. Extensive experiments demonstrate that the proposed model achieves state-of-the-art performance in both subjective perception and objective assessment.

Combining transformers with CNN for multi-focus image fusion

Recently, deep convolutional neural network (CNN) based methods for multi-focus image fusion have achieved adequate performance. However, most of them cannot obtain spatially continuous results, especially in smooth regions and edges between focused and defocused regions. In this paper, we propose a novel end-to-end method, which merits both Transformers and CNNs, as a strong alternative for multi-focus image fusion task. Transformer has advantages over a CNN in that it can extract global features. It is able to make the fusion results to be spatially consistent. The proposed architecture consists of CNN and transformer branches, where transformer branches take feature map patches as inputs and leverages the transformer to propagate global contexts among patches. Moreover, in order to improve feature representation, we introduce online knowledge distillation learning strategy (KDL). The strategy achieves better interactions between global features and local features. Specifically, we design hard target and soft target by simply yet effectively ensembling outputs of two branches, which are used to supervise CNN and transformer branches. The experiments demonstrate the superiority of our proposed architecture and achieve competitive results with state-of-the-art methods.

An end-to-end anti-shaking multi-focus image fusion approach

In this paper, an end-to-end multi-focus fusion network is proposed to solve the mis-registration problem of the captured images. This paper takes the shaking and camera breathing effects that usually exist when acquiring multiple partially focused images into account. The new network consists of a homography pre-correction module, a deformable convolutional fine-correction module, an attention based fusion module and a reconstruction module. The homography pre-correction module is proposed to achieve a coarse correction for multi-focus images. The deformable convolution fine-correction module enables the fusion algorithm to achieve an effective fine alignment for small shaking in multi-focus images. The attention based fusion module and the image reconstruction module are proposed to achieve high definition fusion results. This paper produces a dedicated shaking dataset for training and testing. The algorithm’s superior performance in depth-of-field extension and generalisation to different scenes can be seen through sufficient experiments on test datasets, real scenes and image sequences. The ablation experiments also demonstrate the necessity of each module in the algorithm.

Miniature Probe for Optomechanical Focus-Adjustable Optical-Resolution Photoacoustic Endoscopy

Photoacoustic microscopy (PAM) is a promising imaging modality because it is able to reveal optical absorption contrast in high resolution on the order of a micrometer. It can be applied in an endoscopic approach by implementing PAM into a miniature probe, termed photoacoustic endoscopy (PAE). Here we develop a miniature focus-adjustable PAE (FA-PAE) probe characterized by both high resolution (in micrometers) and large depth of focus (DOF) via a novel optomechanical design for focus adjustment. To realize high resolution and large DOF in a miniature probe, a 2-mm plano-convex lens is specially adopted, and the mechanical translation of a single-mode fiber is meticulously designed to allow the use of multi-focus image fusion (MIF) for extended DOF. Compared with existing PAE probes, our FA-PAE probe achieves high resolution of 3-5 μm within unprecedentedly large DOF of >3.2 mm, more than 27 times the DOF of the probe without performing focus adjustment for MIF. The superior performance is first demonstrated by imaging both phantoms and animals including mice and zebrafish in vivo by linear scanning. Further, in vivo endoscopic imaging of a rat's rectum by rotary scanning of the probe is conducted to showcase the capability of adjustable focus. Our work opens new perspectives for PAE biomedical applications.