Focus-aware and deep restoration network with transformer for multi-focus image fusion

Abstract

Multi-focus image fusion (MFIF) aims to break the limitations of physical devices and integrate different focused regions from multiple source images into a single fully focused image. However, existing fusion methods suffer from two issues when designing the fusion framework: over-reliance on post-processing operations or underestimation of their impact, leading to unsatisfactory visual results. To this end, we propose a fusion model that combines focus-aware and deep restoration. Specifically, we first employ a designed focus-aware module to integrate spatial frequency information from various levels of depth features in the source images. This integration produces an initial decision map that reflects the focus information. Then, we treat the optimization of the decision map as an image restoration task, rather than relying on commonly used post-processing operations in the MFIF field. In the restoration task, the initial decision map is passed as input along with the gradient information of the source images to the deep restoration network combined with the transformer for obtaining the final decision map. To facilitate the training of the reconstruction network, we also propose a large dataset with labels and a new set of loss functions. Extensive experimental results conclusively demonstrate that the proposed method produces fusion results of superior visual quality. This is substantiated by an assessment across eight objective metrics, which encompass normalized mutual information (1.193), nonlinear correlation information entropy (0.846), gradient-based similarity measurement (0.727), phase congruency-based measurements (0.844), structural similarity-based assessments (0.902), human perception-inspired evaluations (0.808), peak signal-to-noise ratio (74.55), and correlation coefficients (0.971). These metrics collectively affirm the outstanding fusion performance of our method. The code will be able to be found at https://github.com/govenda/FARfusion.