Image Fusion Research Articles

Underwater image enhancement via color correction and multi-feature image fusion

Abstract The light attenuation underwater causes the actual underwater images to suffer from color cast, low contrast, and weak illumination. To address these issues, an effective fusion-based method is proposed, which realizes color correction, brightness adjustment, contrast, and detail enhancement of underwater images. Concretely, we first design an adaptive color correction method via dominant color channel judgment and lower color channel compensation. Then, we detect the brightness of each input image and propose a gamma correction function based on the gradient of the cumulative histogram to adjust the brightness of the low-light images. Subsequently, global histogram stretching and adaptive fractional differentiation techniques are employed to process the brightness-adjusted image, and then the global contrast-enhanced version and detail-enhanced version are generated respectively. To integrate the advantages of both versions, a channel fusion method based on the Lab color space is used to fuse the luminance and color of the two versions separately. The experimental results demonstrate the effectiveness of the proposed method in improving the color and illumination of underwater images, as well as enhancing the clarity of images. Moreover, the testing results on multiple datasets validate the excellent stability of this method.

Height estimation of sugarcane tip cutting position based on multimodal alignment and depth image fusion

Sugarcane tip cutting is essential to reducing the rate of impurities in the harvest. To achieve adaptive regulation of the tip-cutting position by a sugarcane harvester, we propose a method for estimating the height of the tip-cutting position of sugarcane. The RGB and Binocular depth cameras are aligned to process the sugarcane tip region image. This involves threshold segmentation, morphological operations, and contour detection to identify the tip-cutting position and upper boundary contours. The depth image is segmented using contour pixel information and merged to form a colour depth image of the sugarcane's tip. This image is then transformed using depth data and triangular parallax principles to determine the height of the sugarcane tip-cutting position. The proposed method was evaluated in various sugarcane plantation environments. Comparative analysis between the proposed method and manual measurements of actual cutting position heights revealed that the RMSE ranged from 1.22 cm to 1.78 cm, and R2 varied between 0.79 and 0.86. These results demonstrate the effectiveness of the proposed method in accurately extracting the height information of sugarcane tip-cutting positions, which has a specific application value for the adaptive adjustment of the tip-cutting device of the sugarcane harvester.

Bayesian multi-exposure image fusion for robust high dynamic range ptychography

Bayesian multi-exposure image fusion for robust high dynamic range ptychography

UAV vision-based crack quantification and visualization of bridges: system design and engineering application

Accurately measuring visible cracks in bridges is crucial for their structural health diagnosis, damage detection, performance evaluation, and maintenance planning. The primary means of visual crack detection still relies heavily on manual visual inspection, an inefficient process that can pose significant safety risks. This article develops a unmanned aerial vehicle (UAV) vision-based surface crack measurement methodology and visualization scheme for the bridges that can detect and measure cracks automatically with improved efficiency. The surface crack measurement methodology is achieved by designing a three-stage crack sensing system including the You Only Look Once-based crack recognition, U-shaped network-based crack segmentation, and deep-vision-based crack width calculation. This workflow is integrated into a comprehensive UAV inspection system, which is intended for operation at the field. The surface crack visualization scheme is accomplished by taking advantage of time-series image fusion, GPS information migration, and three-dimensional (3D) point cloud technique to reconstruct the 3D geometrical model of the tested bridge, which is convenient for unveiling the crack information in the bridge. The proposed methodology was successfully validated by a case study on an arch bridge. The achievement of this article promotes the UAV vision-based bridge’s surface crack inspection technology to a new status that no preparation for pasting calibration marker is needed, and crack identification, segmentation, and width calculation are realized promptly during the UAV flying on-site, as well as damage evaluation for bridges is visually fulfilled based on the reconstructed digital-graphical 3D model. The working environments and influencing factors to the developed system are sufficiently discussed. Certain limitations in the current application are pointed out for future improvements.

DDFNet-A: Attention-Based Dual-Branch Feature Decomposition Fusion Network for Infrared and Visible Image Fusion

The fusion of infrared and visible images aims to leverage the strengths of both modalities, thereby generating fused images with enhanced visible perception and discrimination capabilities. However, current image fusion methods frequently treat common features between modalities (modality-commonality) and unique features from each modality (modality-distinctiveness) equally during processing, neglecting their distinct characteristics. Therefore, we propose a DDFNet-A for infrared and visible image fusion. DDFNet-A addresses this limitation by decomposing infrared and visible input images into low-frequency features depicting modality-commonality and high-frequency features representing modality-distinctiveness. The extracted low and high features were then fused using distinct methods. In particular, we propose a hybrid attention block (HAB) to improve high-frequency feature extraction ability and a base feature fusion (BFF) module to enhance low-frequency feature fusion ability. Experiments were conducted on public infrared and visible image fusion datasets MSRS, TNO, and VIFB to validate the performance of the proposed network. DDFNet-A achieved competitive results on three datasets, with EN, MI, VIFF, QAB/F, FMI, and Qs metrics reaching the best performance on the TNO dataset, achieving 7.1217, 2.1620, 0.7739, 0.5426, 0.8129, and 0.9079, respectively. These values are 2.06%, 11.95%, 21.04%, 21.52%, 1.04%, and 0.09% higher than those of the second-best methods, respectively. The experimental results confirm that our DDFNet-A achieves better fusion performance than state-of-the-art (SOTA) methods.

HyperMRI: hyperspectral and magnetic resonance fusion methodology for neurosurgery applications.

Magnetic resonance imaging (MRI) is a common technique in image-guided neurosurgery (IGN). Recent research explores the integration of methods like ultrasound and tomography, among others, with hyperspectral (HS) imaging gaining attention due to its non-invasive real-time tissue classification capabilities. The main challenge is the registration process, often requiring manual intervention. This work introduces an automatic, markerless method for aligning HS images with MRI. This work presents a multimodal system that combines RGB-Depth (RGBD) and HS cameras. The RGBD camera captures the patient's facial geometry, which is used for registration with the preoperative MR through ICP. Once MR-depth registration is complete, the integration of HS data is achieved using a calibrated homography transformation. The incorporation of external tracking with a novel calibration method allows camera mobility from the registration position to the craniotomy area. This methodology streamlines the fusion of RGBD, HS and MR images within the craniotomy area. Using the described system and an anthropomorphic phantom head, the system has been characterised by registering the patient's face in 25 positions and 5 positions resulted in a fiducial registration error of 1.88 ± 0.19 mm and a target registration error of 4.07 ± 1.28 mm, respectively. This work proposes a new methodology to automatically register MR and HS information with a sufficient accuracy. It can support the neurosurgeons to guide the diagnosis using multimodal data over an augmented reality representation. However, in its preliminary prototype stage, this system exhibits significant promise, driven by its cost-effectiveness and user-friendly design.

Constrained Optimization Guided Approach for Multispectral and Panchromatic Image Fusion

Constrained Optimization Guided Approach for Multispectral and Panchromatic Image Fusion

Study on flame quenching characteristics inside the Tesla valve structure flame arrester unit

Flame arrester is a kind of safety device designed to prevent the spread of flames. In order to achieve a more efficient flame arrester unit structure, this paper proposes a flame arrester unit based on the Tesla valve structure. A comparative study of the quenching characteristics of deflagration flames of propane-air premixtures in traditional crimped ribbon and Tesla valve structure flame arrester units is conducted through numerical simulations and experiments. The results indicate that the flame arrester unit based on the Tesla valve structure exhibits more efficient flame quenching performance compared to the crimped ribbon flame arrester unit. The flame quenching process fusion image shows that the flame quenching length of Tesla valve flame arrester unit with the same cross-sectional area is only 61% of the crimped ribbon flame arrester unit. In addition to heat transfer and wall effect, the collision of airflow in the branched pipe and main pipe within the Tesla valve structure flame arrester unit also plays a significant role in hindering the further advancement of flames. The quenching process of the Tesla valve structure flame arrester unit can be divided into four stages: expansion, diversion, convergence, and contraction stages. Among the five structural parameters that make up the Tesla valve structure flame arrester unit, the angle is the most critical factor affecting quenching performance, with the angle and entrance structure having a significant impact on flow performance. Furthermore, experimental evidence confirms the typical morphological changes the flame front structure undergoes during flame propagation: spherical (hemispherical),finger-shaped and tulip-shaped. However, tulip-shaped flames do not always propagate continuously to the end of the pipeline.

Image fusion-based low-dose CBCT enhancement method for visualizing miniscrew insertion in the infrazygomatic crest

Digital dental technology covers oral cone-beam computed tomography (CBCT) image processing and low-dose CBCT dental applications. A low-dose CBCT image enhancement method based on image fusion is proposed to address the need for subzygomatic small screw insertion. Specifically, firstly, a sharpening correction module is proposed, where the CBCT image is sharpened to compensate for the loss of details in the underexposed/over-exposed region. Secondly, a visibility restoration module based on type II fuzzy sets is designed, and a contrast enhancement module using curve transformation is designed. In addition to this, we propose a perceptual fusion module that fuses visibility and contrast of oral CBCT images. As a result, the problems of overexposure/underexposure, low visibility, and low contrast that occur in oral CBCT images can be effectively addressed with consistent interpretability. The proposed algorithm was analyzed in comparison experiments with a variety of algorithms, as well as ablation experiments. After analysis, compared with advanced enhancement algorithms, this algorithm achieved excellent results in low-dose CBCT enhancement and effective observation of subzygomatic small screw implantation. Compared with the best performing method, the evaluation metric is 0.07–2 higher on both datasets. The project can be found at: https://github.com/sunpeipei2024/low-dose-CBCT.

TRAFMEL: Multimodal Entity Linking Based on Transformer Reranking and Multimodal Co-Attention Fusion

Multimodal entity linking aims to link mentions to target entities in the multimodal knowledge graph. The current multimodal entity linking mainly focuses on the global fusion of text and image, seldom fully exploring the correlation between modalities. In order to improve the fusion effect of multimodal feature, we propose a multimodal entity linking model based on a Multimodal Co-Attention Fusion strategy. This strategy is designed to enable text and image to guide each other for extracting features, thus making full exploration of the correlation between modalities to improve the fine-grained feature fusion effect. Furthermore, we also design a candidate entity generation strategy based on Transformer, which combines multiple candidate entity sets and adjusts the candidate entity ranking to obtain high-quality candidate entity sets. We perform experiments on domain datasets and public datasets, and the experimental results demonstrate that our model has a good performance in candidate entity generation and multimodal feature fusion, outperforming the state-of-the-art baseline models.

MACCNet: Multiscale Attention and Cross- Convolutional Network for Infrared and Visible Image Fusion

MACCNet: Multiscale Attention and Cross- Convolutional Network for Infrared and Visible Image Fusion

Advanced Imaging Guidance for Ductus Venosus Stenting in Obstructed Total Anomalous Pulmonary Venous Return.

Obstructed infracardiac total anomalous pulmonary venous return is nearly always a surgical emergency in which infants present in severe cardiopulmonary distress. Ductal venosus stenting can provide a temporizing option for premature, low birth weight infants with high risk for surgical complications. In challenging anatomic cases, virtual reality, 3D-printed models, and fusion image guidance can aid in procedural planning and provide support for successful intervention.

Exploring Ear Recognition: Image Fusion and Deep Learning with Thermal and Visible Images

Exploring Ear Recognition: Image Fusion and Deep Learning with Thermal and Visible Images

Parallel Hierarchical Multi-View Feature Fusion Based on Canonical Correlation Analysis for Mammogram Retrieval

Due to the diversity of image sources, content-based multi-source image fusion and retrieval have shown promising capabilities in computer vision tasks, and especially when applied in Computer-Aided Diagnosis (CAD) to automate and improve the accuracy of medical image analysis. The combination of computer vision and CAD systems has the potential to revolutionize healthcare by augmenting the expertise of clinicians, improving overall diagnostic accuracy and helping experts in the clinical decision-making process by classifying and retrieving similar annotated clinical images to a given query. In the context of multi-view mammography interpretation, the concept of multi-view feature fusion has recently been studied to improve retrieval performance while effectively guaranteeing the complementarity of both MLO and CC views. However, conventional multi-view feature fusion makes descriptors long and lacks to take into consideration the relationship between descriptors. To deal with this issue, we propose two hierarchical multi-view feature fusion methods, for multi-view mammogram retrieval, based on the Canonical Correlation Analysis (CCA), which is the most commonly used multivariate parametric test. In fact, we have adapted CCA to determine the relationship between two descriptors by processing latent correlation factors. Moreover, after extracting descriptors for each view, a comparative study of texture and shape fusion descriptors is proposed in order to identify the more discriminative features for multi-view mammogram retrieval. Then, a query-dependent distance metric preserving both visual resemblance and semantic similarity is carried out to dynamically determine the more appropriate distance measure for each query image. Extensive experiments on the challenging Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM) have demonstrated the effectiveness of the proposed hierarchical multi-view feature fusion for mammogram retrieval, which outperforms the performance achieved either by conventional fused information or by single view information. To improve the transparency of our paper, the source code of the proposed method and the related dataset (including readme files) are publicly accessible through the following GitHub link: https://github.com/ABDERRAHIMMAR/Multi-View-Feature-Fusion-for-Mammogram-Retrieval . This open-access resource empowers researchers and practitioners to delve deeper into our methodology, fostering collaboration and advancements in the field of computer-aided diagnosis and medical image analysis.

Multi-exposure image fusion using structural weights and visual saliency map

Multi-exposure image fusion using structural weights and visual saliency map

G2NPAN: GAN-guided nuance perceptual attention network for multimodal medical fusion image quality assessment

Multimodal medical fusion images (MMFI) are formed by fusing medical images of two or more modalities with the aim of displaying as much valuable information as possible in a single image. However, due to the different strategies of various fusion algorithms, the quality of the generated fused images is uneven. Thus, an effective blind image quality assessment (BIQA) method is urgently required. The challenge of MMFI quality assessment is to enable the network to perceive the nuances between fused images of different qualities, and the key point for the success of BIQA is the availability of valid reference information. To this end, this work proposes a generative adversarial network (GAN) -guided nuance perceptual attention network (G2NPAN) to implement BIQA for MMFI. Specifically, we achieve the blind evaluation style via the design of a GAN and develop a Unique Feature Warehouse module to learn the effective features of fused images from the pixel level. The redesigned loss function guides the network to perceive the image quality. In the end, the class activation mapping supervised quality assessment network is employed to obtain the MMFI quality score. Extensive experiments and validation have been conducted in a database of medical fusion images, and the proposed method is superior to the state-of-the-art BIQA method.

Image Fusion Method Based on Snake Visual Imaging Mechanism and PCNN

The process of image fusion is the process of enriching an image and improving the image’s quality, so as to facilitate the subsequent image processing and analysis. With the increasing importance of image fusion technology, the fusion of infrared and visible images has received extensive attention. In today’s deep learning environment, deep learning is widely used in the field of image fusion. However, in some applications, it is not possible to obtain a large amount of training data. Because some special organs of snakes can receive and process infrared information and visible information, the fusion method of infrared and visible light to simulate the visual mechanism of snakes came into being. Therefore, this paper takes into account the perspective of visual bionics to achieve image fusion; such methods do not need to obtain a significant amount of training data. However, most of the fusion methods for simulating snakes face the problem of unclear details, so this paper combines this method with a pulse coupled neural network (PCNN). By studying two receptive field models of retinal nerve cells, six dual-mode cell imaging mechanisms of rattlesnakes and their mathematical models and the PCNN model, an improved fusion method of infrared and visible images was proposed. For the proposed fusion method, eleven groups of source images were used, and three non-reference image quality evaluation indexes were compared with seven other fusion methods. The experimental results show that the improved algorithm proposed in this paper is better overall than the comparison method for the three evaluation indexes.

Light-weight Fusion Network for UAV Visible-light and Infrared Images based on Real-time Brain-like Intelligent Computing

Abstract. Multiple sensors equipped on the Unmanned Aerial Vehicle (UAV) enables the acquisition of multi-modal and multi-source remote sensing data. UAV remote sensing usually faces with real-time or near-real-time tasks in complex and highly dynamic environments, such as disaster monitoring, traffic management, border patrol and so on. Under these conditions, the image fusion algorithm needs to be high efficiency, precision and reliability. In this paper, we proposed an intelligent real-time fusion network for UAV multi-source remote sensing data based on AI brain-like chips, and deployed the algorithm on the UAV platform to achieve online high-efficiency computing. Firstly, we have developed a novel image fusion algorithm named SFNet for infrared and visible image fusion based on ShuffleNetv2. Then, we use ZCA and l1-norm to process the remodeled deep feature. The weight maps are generated by bi-cubic interpolation and soft-max operation. Finally, the fused image is reconstructed by weighted-average operation. The proposed SFNet is deployed on the Lynxi KA200 brain computing chip, and a comprehensive inference test is carried out with UAV remote sensing data. Several State-Of-The-Art (SOTA) data fusion algorithms are deployed on the same chip for experimental comparison. The proposed SFNet is proved to have faster inference speed and better feature extraction results on brain-like chips. It is more suitable for real-time UAV remote sensing image fusion tasks.

Keratoconus Progression Determined at the First Visit: A Deep Learning Approach With Fusion of Imaging and Numerical Clinical Data.

Multiple clinical visits are necessary to determine progression of keratoconus before offering corneal cross-linking. The purpose of this study was to develop a neural network that can potentially predict progression during the initial visit using tomography images and other clinical risk factors. The neural network's development depended on data from 570 keratoconus eyes. During the initial visit, numerical risk factors and posterior elevation maps from Scheimpflug imaging were collected. Increase of steepest keratometry of 1 diopter during follow-up was used as the progression criterion. The data were partitioned into training, validation, and test sets. The first two were used for training, and the latter for performance statistics. The impact of individual risk factors and images was assessed using ablation studies and class activation maps. The most accurate prediction of progression during the initial visit was obtained by using a combination of MobileNet and a multilayer perceptron with an accuracy of 0.83. Using numerical risk factors alone resulted in an accuracy of 0.82. The use of only images had an accuracy of 0.77. The most influential risk factors in the ablation study were age and posterior elevation. The greatest activation in the class activation maps was seen at the highest posterior elevation where there was significant deviation from the best fit sphere. The neural network has exhibited good performance in predicting potential future progression during the initial visit. The developed neural network could be of clinical significance for keratoconus patients by identifying individuals at risk of progression.

Fusion of full-field optical angiography images via gradient feature detection

Full-field optical angiography (FFOA)—a real-time non-invasive imaging technique for extracting biological blood microcirculation information—contributes to an in-depth understanding of the functional and pathological changes of biological tissues. However, owing to the limitation of the depth-of-field (DOF) of optical lenses, existing FFOA imaging methods cannot capture an image containing every blood-flow information. To address this problem, this study develops a long-DOF full-field optical angiography imaging system and proposes a novel multi-focus image fusion scheme to expand the DOF. First, FFOA images with different focal lengths are acquired by the absorption intensity fluctuation modulation effect. Second, an image fusion scheme based on gradient feature detection in a nonsubsampled contourlet transform domain is developed to capture focus features from FFOA images and synthesize an all-focused image. Specifically, FFOA images are decomposed by NSCT into coefficients and low-frequency difference images; thereafter, two gradient feature detection-based fusion rules are used to select the pre-fused coefficients. The experimental results of both phantom and animal cases show that the proposed fusion method can effectively extend the DOF and address practical FFOA image defocusing problems. The fused FFOA image can provide a more comprehensive description of blood information than a single FFOA image.