Image Fusion Research Articles

LEAFusion: An Infrared and Visible Light Image Fusion Network Resilient to Harsh Light Environment Interference Based on Harsh Light Environment Aware

LEAFusion: An Infrared and Visible Light Image Fusion Network Resilient to Harsh Light Environment Interference Based on Harsh Light Environment Aware

Aggressive bone tumours: What a radiologist can offer to the surgeon?

The complexity of aggressive bone tumours necessitates a comprehensive approach that combines radiological assessments with clinical and pathological findings. Radiologists offer valuable insights to surgeons throughout the diagnostic and therapeutic journey, enhancing the precision and efficacy of surgical interventions. Radiologists contribute to definitive diagnosis in the majority of benign and certain aggressive tumours, although the complexity of aggressive bone tumours often requires histopathological confirmation. They assist in characterizing tumours and evaluating their extension, providing critical information for treatment planning. Radiologists guide biopsy procedures, ensuring representative tissue samples while minimizing morbidity and the risk of tumour spread. In preoperative planning, radiologists construct detailed 3D reconstructions of tumours, aiding surgeons in strategizing surgical approaches and anticipating challenges. During surgery, radiologists offer intraoperative guidance through techniques like image fusion and intraoperative MRI, enhancing surgical precision. Post-surgical surveillance for tumour recurrence heavily relies on radiological imaging, with functional MR sequences providing valuable insights. Radiologists also play a significant role in image-guided therapeutic interventions for aggressive bone tumours, offering procedures like osteoplasty and ablation techniques for pain relief and tumour control. In conclusion, radiologists are indispensable members of the multidisciplinary team and offer expertise in diagnosis, biopsy guidance, preoperative planning, intraoperative guidance, post-surgical surveillance, and interventional therapy. Their collaborative efforts significantly optimize patient's outcome and quality of life.

Image Dehazing Enhancement Strategy Based on Polarization Detection of Space Targets

In view of the fact that the technology of polarization detection performs better at identifying targets through clouds and fog, the recognition ability of the space target detection system under haze conditions will be improved by applying the technology. However, due to the low ambient brightness and limited target radiation information during space target detection, the polarization information of space target is seriously lost, and the advantages of polarization detection technology in identifying targets through clouds and fog cannot be effectively exerted under the condition of haze detection. In order to solve the above problem, a dehazing enhancement strategy specifically applied to polarization images of space targets is proposed. Firstly, a hybrid multi-channel interpolation method based on regional correlation analysis is proposed to improve the calculation accuracy of polarization information during preprocessing. Secondly, an image processing method based on full polarization information inversion is proposed to obtain the degree of polarization of the image after inversion and the intensity of the image after dehazing. Finally, the image fusion method based on discrete cosine transform is used to obtain the dehazing polarization fusion enhancement image. The effectiveness of the proposed image processing strategy is verified by carrying out simulated and real space target detection experiments. Compared with other methods, by using the proposed image processing strategy, the quality of the polarization images of space targets obtained under the haze condition is significantly improved. Our research results have important practical implications for promoting the wide application of polarization detection technology in the field of space target detection.

Contrast-enhanced ultrasound can differentiate the level of glioma infiltration and correlate it with biological behavior: a study based on local pathology.

The objective of this study is to assess the diagnostic efficacy of contrast-enhanced ultrasound (CEUS) in determining the level of glioma infiltration and to investigate its correlation with pathological markers. A prospective study involving 16 adult glioma patients was conducted. Preoperative US-(Magnetic Resonance)MR fusion imaging was utilized for tumor infiltration localization, while CEUS was employed to assess hemodynamic alterations. Parameters such as peak intensity (PI), rise time (RT), time to peak (TTP), and area under the curve (AUC) were measured. Utilizing contralateral normal brain tissue as the reference standard. The Kruskal-Wallis H-test was conducted to compare CEUS and pathological parameters (significance level, p < 0.05; bonferroni correction) among tumor margins, infiltration zones, and normal tissues, as well as between low-grade glioma (LGG) and high-grade glioma (HGG) within the infiltration zone, based on whole slide pathological images analysis. Spearman correlation analysis was employed to determine the correlation coefficient between hemodynamics and pathology. Receiver operating characteristic (ROC) curves were drawn to evaluate the performance of CEUS in tumor classification. From tumor margin to normal tissue, PI, AUC, Ki67, EGFR, and 1p/19q showed a significant decreasing trend, while TTP, IDH-1, and MGMT gradually increased. RT was lower at the tumor margin but did not show statistically significant differences. In the infiltration zones, there was a significant increase in parameters such as PI, normalized PI (Nor_PI), AUC, and Ki67 from LGG to HGG, while RT, Nor_RT, TTP, Nor_TTP, IDH-1, and MGMT significantly decreased. Nor_AUC and EGFR increased but were not significant, and 1p/19q decreased but was not significant. RT and Nor_TTP were independent risk factors for distinguishing between LGG and HGG in the infiltration zone, with a combined diagnostic efficacy ROC of 0.891. The sensitivity reached 96.64% and the specificity reached 82.35%. There was a significant correlation between hemodynamic indicators and pathological indicators. CEUS can effectively differentiate levels of infiltration zones, which correlates with their biological behavior, with RT + Nor_TTP showing particularly highest diagnostic efficacy. These findings contribute to improving the accuracy of diagnosing infiltration zones and provide essential biological insights for subsequent treatments.

Enhanced Preoperative Planning for Intracranial Aneurysms Through Multimodal Image Fusion of Silent/Time-of-Magnetic Resonance Angiography and Computed Tomography Using 3DSlicer: A Comparative Efficacy Analysis With Computed Tomography Angiography.

This study evaluates the effectiveness of multimodal image fusion (MIF) using silent and time-of-flight (TOF) magnetic resonance angiography (MRA) and computed tomography (CT) for preoperative planning in patients with intracranial aneurysms who have contraindications to contrast media. A retrospective study included 40 patients with intracranial aneurysms, diagnosed using three-dimensional computed tomography angiography (CTA). These patients underwent both Silent and TOF MRA scans, followed by a CTA scan. The multi-image fusion (MIF) technique, applied using 3DSlicer software, integrated the silent/TOF-MRA with CT images for preoperative assessment. This study compared the image quality, aneurysm detection sensitivity, and anatomic accuracy of the MIF images with those of three-dimensional CTA. Silent-MRA-CT fusion images demonstrated higher sensitivity (95.5%) and lower false negative rates (4.5%) compared with TOF-MRA-CT. Furthermore, silent-MRA-CT fusion images outperformed TOF-MRA-CT in terms of signal homogeneity, venous signal interference suppression, and aneurysm visibility (all P < 0.05). The interclass correlation coefficient and kappa values for aneurysm morphology and shape indicated superior measurement consistency and shape concordance of silent-MRA-CT with CTA compared with TOF-MRA-CT (all P < 0.01). This study supports the use of silent/TOF-MRA-CT fusion imaging as a reliable alternative to CTA, noting that silent-MRA-CT closely mirrors CTA. Contrast-free MRA-CT fusion images have the potential to be used for preoperative planning in patients with intracranial aneurysms who have contraindications to contrast.

A novel infrared and visible image fusion network based on cross-modality reinforcement and multi-attention fusion strategy

A novel infrared and visible image fusion network based on cross-modality reinforcement and multi-attention fusion strategy

Multi-Focus Image Fusion Algorithm Based on Multi-Task Learning and PS-ViT

Multi-Focus Image Fusion Algorithm Based on Multi-Task Learning and PS-ViT

Estimation of Canopy Water Content by Integrating Hyperspectral and Thermal Imagery in Winter Wheat Fields

Rapid and accurate estimation of canopy water content (CWC) is important for agricultural water management and food security. Due to the complexity of dynamic changes in water transport during plant growth, estimation of CWC using a single sensor often leads to high uncertainty in the results. Multi-sensor data fusion is one of the solutions to this problem, but suitable spectral preprocessing methods and data fusion methods still need further research. The objectives of this study were to characterize the performance of two varieties at different growth stages under five water stress conditions and screen hyperspectral sensitive spectral bands by using continuous wavelet transform (CWT) and a successive projection algorithm (SPA). Ultimately, the CWC prediction model of winter wheat hyperspectral characteristic bands and thermal imaging information fusion was created using the GRA algorithm. The results showed that canopy temperature parameters and spectral parameters responded significantly to water deficits in winter wheat. Using the CWT-SPA method, a total of 285 hyperspectral feature bands with wavelet decomposition scales ranging from one to eight were selected. The sensitive bands were mainly distributed in the following ranges: 545–561, 746–1348, 1561–1810, and 2122–2430 nm. The GRA algorithm has good multi-source data model fusion capability, and its constructed prediction model based on hyperspectral and thermal image fusion has high accuracy on the canopy water content in winter wheat (R2 = 0.930, RMSE = 5.44%, nRMSE = 7.94%). Compared to the single-feature spectral model (R2 = 0.864, RMSE = 5.92%, nRMSE = 8.63%) and thermal image CWC prediction model (R2 = 0.813, RMSE = 7.22%, nRMSE = 10.49%), the model prediction accuracy based on the GRA algorithm is increased by 7.64% and 13.69%, respectively.

ACFNet: An adaptive cross-fusion network for infrared and visible image fusion

ACFNet: An adaptive cross-fusion network for infrared and visible image fusion

Prognostic nomogram model based on quantitative metrics of subregions surrounding residual cavity in glioblastoma patients

BackgroundThe hyperintensity area surrounding the residual cavity on postoperative fluid-attenuated inversion recovery (FLAIR) image is a potential site for glioblastoma (GBM) recurrence. This study aimed to develop a nomogram using quantitative metrics from subregions of this area, prior to chemoradiotherapy (CRT), to predict early GBM recurrence.MethodsAdult patients with GBM diagnosed between October 2018 and October 2022 were retrospectively analyzed. Quantitative metrics, including the mean, maximum, minimum, median values, and standard deviation of FLAIR signal intensity (SI) (measured using 3D-Slicer software), were extracted from the following subregions surrounding the residual cavity on post-contrast T1-weighted (CE-T1WI)-FLAIR fusion images: the enhancing region (ER), non-enhancing region (NER), and combined ER + NER. Independent prognostic factors were identified using Cox regression and least absolute shrinkage and selection operator (LASSO) analyses and were incorporated into the prediction nomogram model. The model’s performance was evaluated using the C-index, calibration curves, and decision curves.ResultsA total of 129 adult GBM patients were enrolled and randomly assigned to a training (n = 90) and a validation cohorts (n = 39) in a 7:3 ratio. Sixty-nine patients experienced postoperative recurrence. Cox regression analysis identified subventricular zone involvement, the median FLAIR intensity in the ER, the rFLAIR (relative FLAIR intensity compared to the contralateral normal region) of ER + NER, and corpus callosum involvement as independent prognostic factors. For predicting recurrence within 1 year after surgery, the nomogram model had a C-index of 0.733 in the training cohort and 0.746 in the validation cohort. Based on the nomogram score, post-operative GBM patients could be stratified into high- and low-risk for recurrence.ConclusionsNomogram models which based on quantitative metrics from FLAIR hyperintensity subregions may serve as potential markers for assessing GBM recurrence risk. This approach could enhance clinical decision-making and provide an alternative method for recurrence estimation in GBM patients.

Random-Coupled Neural Network

Improving the efficiency of current neural networks and modeling them on biological neural systems have become prominent research directions in recent years. The pulse-coupled neural network (PCNN) is widely used to mimic the computational characteristics of the human brain in computer vision and neural network fields. However, PCNN faces limitations such as limited neural connections, high computational costs, and a lack of stochastic properties. This study proposes a random-coupled neural network (RCNN) to address these limitations. RCNN employs a stochastic inactivation process, selectively inactivating neural connections using a random inactivation weight matrix. This method reduces the computational burden and allows for extensive neural connections. RCNN encodes constant stimuli as periodic spike trains and periodic stimuli as chaotic spike trains, reflecting the information encoding characteristics of biological neural systems. Our experiments applied RCNN to image segmentation and fusion tasks, demonstrating its robustness, efficiency, and high noise resistance. Results indicate that RCNN surpasses traditional methods in performance across these applications.

Parallel-way: Multi-modality-based brain tumor segmentation using parallel capsule network

ABSTRACT Brain tumors present a formidable diagnostic challenge due to their aberrant cell growth. Accurate determination of tumor location and size is paramount for effective diagnosis. Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET) are pivotal tools in clinical diagnosis, yet tumor segmentation within their images remains challenging, particularly at boundary pixels, owing to limited sensitivity. Recent endeavors have introduced fusion-based strategies to refine segmentation accuracy, yet these methods often prove inadequate. In response, we introduce the Parallel-Way framework to surmount these obstacles. Our approach integrates MRI and PET data for a holistic analysis. Initially, we enhance image quality by employing noise reduction, bias field correction, and adaptive thresholding, leveraging Improved Kalman Filter (IKF), Expectation Maximization (EM), and Improved Vibe Algorithm (IVib), respectively. Subsequently, we conduct multi-modality image fusion through the Dual-Tree Complex Wavelet Transform (DTWCT) to amalgamate data from both modalities. Following fusion, we extract pertinent features using the Advanced Capsule Network (ACN) and reduce feature dimensionality via Multi-objective Diverse Evolution-based selection. Tumor segmentation is then executed utilizing the Twin Vision Transformer with dual attention mechanism. Implemented our Parallel-Way framework which exhibits heightened model performance. Evaluation across multiple metrics, including accuracy, sensitivity, specificity, F1-Score, and AUC, underscores its superiority over existing methodologies.

Adaptive Transformer-Based Multi-Modal Image Fusion for Real-Time Medical Diagnosis and Object Detection

In recent years, medical diagnosis and object detection have been significantly enhanced by the integration of multi-modal image fusion techniques. This study proposes an Adaptive Transformer-Based Multi-Modal Image Fusion (AT-MMIF) framework designed for real-time medical diagnosis and object detection. The framework employs a Transformer architecture to capture both global and local feature correlations across multiple imaging modalities, including MRI, CT, PET, and X-ray, for more accurate diagnostic results and faster object detection in medical imagery. The fusion process incorporates spatial and frequency-domain information to improve the clarity and detail of the output images, enhancing diagnostic accuracy. The adaptive attention mechanism within the Transformer dynamically adjusts to the relevant features of different image types, optimizing fusion in real time. This leads to an improved sensitivity (98.5%) and specificity (96.7%) in medical diagnosis. Additionally, the model significantly reduces false positives and negatives, with an F1 score of 97.2% in object detection tasks. The AT-MMIF framework is further optimized for real-time processing with an average inference time of 120 ms per image and a model size reduction of 35% compared to existing multi-modal fusion models. By leveraging the strengths of Transformer architectures and adaptive learning, the proposed framework offers a highly efficient and scalable solution for real-time medical diagnosis and object detection in various clinical settings, including radiology, oncology, and pathology.

An Unsupervised Image Fusion Approach Using Convolution Neural Network for Feature Enhancement of Medical Brain Images

Image fusion is the process of amalgamation of two or more images into a single image that contains composite, enriched with diverse details from the original images. In the medical field, image fusion serves as an indispensable tool for elevating the precision of medical imagery and facilitating diagnostic processes. With the advent of deep learning, there has been a significant increase in the accuracy and effectiveness of image fusion techniques. This paper presents a deeplearningbased approach for medical image fusion that combines the advantages of deep-learning techniques with traditional image fusion methods. The proposed method is evaluated on medical data from different modalities, and the experimental results show that the proposed method outperforms existing state-of-the-art image fusion techniques.

Mixed reality combined with surgical navigation technology assisted in parapharyngeal space tumors surgery: a clinical study

Objective: To assess the feasibility and application of mixed reality combined with surgical navigation technology in parapharyngeal space tumor surgery, and to provide a reference for the development and promotion of this technology. Methods: In this study, retrospective data collection was conducted on 16 patients with parapharyngeal space tumors who were treated at the Department of Oral and Maxillofacial Surgery, Peking University School and Hospital of Stomatology from June 2020 to June 2023. The patient's age was (39.6±17.8) years, with 4 males and 12 females. Mixed reality combined with surgical navigation technology was utilized to assist physicians in the treatment of these patients. Mixed reality combined with surgical navigation technology was used to assist physicians in treatment of these patients. The application steps included acquisition of image data, processing of image data [three-dimensional (3D) reconstruction, image fusion, and virtual surgical design], development of surgical navigation plan, connection of mixed reality and navigation system, automatic registration and intraoperative guidance and validation. In the preoperative plan, landmark points were placed on the virtual tumor and surrounding important structures reconstructed using digital software, serving to guide the localization of crucial anatomical structures. Intraoperative positioning deviation, surgical time, intraoperative blood loss, and postoperative complications were recorded and analyzed to evaluate the clinical application effectiveness of mixed reality combined with surgical navigation technology. Results: With the assistance of mixed reality combined with surgical navigation technology, 16 patients successfully underwent tumor resection. All patients were accurately diagnosed preoperatively by 3D reconstruction and image fusion technology, and a comprehensive preoperative plan was formulated; intraoperatively, mixed reality combined with surgical navigation technology was utilized for the localization of important structures. The average localization deviation of 38 landmark points during the operation were (4.43±1.96) mm, with 62% (26/42) of the points having a deviation of ≥0 and<5 mm. The average duration of the operation was (149.6±53.9) min and the blood loss was 70 (45, 150) ml. The average postoperative follow-up was 16 months, and five patients experienced postoperative complications involving facial paralysis, hoarseness, and choking. Conclusions: Mixed reality combined with surgical navigation technology can achieve the three-dimensional visualization of oral and maxillofacial anatomical structures to achieve precise preoperative diagnosis. During surgery, the technology can real-time display the relationship between soft tissue tumors and the surrounding important anatomical structures, guide surgical operation, and enhance the safety of surgery.

Recording Reflection Volume Holographic Gratings with Enhanced Performance by Two‐Stage Photopolymers Containing Bifunctional Monomers

AbstractA holographic optical waveguide (HOW) is an optimal solution for augmented reality (AR) displays, in which reflection volume holographic gratings are core optical elements. In this study, five high‐refractive‐index (≈1.6) bifunctional acrylate monomers (TPBSAs) are designed and synthesized. A series of two‐stage photopolymers are prepared using TPBSAs as writing monomers. Among them, one containing 9 wt.% TPBSA‐3 shows the best holographic recording performance with a sensitivity as high as 26.59 cm mJ−1. Using it as the recording medium, a reflection VHG can be obtained with high spatial frequency (5634 lines per mm), diffraction efficiency (97.14%), refractive index modulation (0.029), and good transparency within 400−800 nm after photobleaching. Furthermore, a HOW capable of achieving virtual and real image fusion is fabricated.

Hierarchical Fusion of Infrared and Visible Images Based on Channel Attention Mechanism and Generative Adversarial Networks

In order to solve the problem that existing visible and infrared image fusion methods rely only on the original local or global information representation, which has the problem of edge blurring and non-protrusion of salient targets, this paper proposes a layered fusion method based on channel attention mechanism and improved Generative Adversarial Network (HFCA_GAN). Firstly, the infrared image and visible image are decomposed into a base layer and fine layer, respectively, by a guiding filter. Secondly, the visible light base layer is fused with the infrared image base layer by histogram mapping enhancement to improve the contour effect. Thirdly, the improved GAN algorithm is used to fuse the infrared and visible image refinement layer, and the depth transferable module and guided fusion network are added to enrich the detailed information of the fused image. Finally, the multilayer convolutional fusion network with channel attention mechanism is used to correlate the local information of the layered fusion image, and the final fusion image containing contour gradient information and useful details is obtained. TNO and RoadSence datasets are selected for training and testing. The results show that the proposed algorithm retains the global structure features of multilayer images and has obvious advantages in fusion performance, model generalization and computational efficiency.

Correlation of prosthesis position on mid-term pacing burden in patients with new permanent pacemaker after TAVI

Abstract Background Prior studies suggest a strong correlation of the final prosthesis position on the occurrence of new conduction disturbances resulting in permanent pacemaker (PPM) implantation after transcatheter aortic valve implantation (TAVI). A high burden right ventricular pacing (PPM stimulation rate of &amp;gt; 20 %) is associated with poor long-term outcomes. So far, data about predictors influencing the PPM stimulation rate after TAVI are scarce. Purpose Thus, the purpose of this study was to evaluate the influence of the three-dimensional transcatheter heart valve (THV) position, as well as device landing zone characteristics or peri-/post-procedural factors on the stimulation rate in patients with new implanted PPM as TAVI complication. Materials: We evaluated the final THV position of 387 patients (106 with self-expanding Evolut R THV, 281 with balloon-expandable Sapien 3 THV) using fusion imaging of pre- and post-TAVI computed tomography angiography (CTA) to obtain a three-dimensional reconstruction of the THV within the native annulus region. The length of the THV above and below the native annulus was measured within the fused images to assess the implantation depth. PPM stimulation rates were examined pre-discharge and after 3 months. Patient’s clinical long-term outcome was evaluated over 5 years via standardized questionnaire as well as contacting general practitioners. Results Out of the 387 patients 112 (29%) received a new PPM pre-discharge after TAVI between 2014 and 2022. They were grouped into patients with stimulations rate below and above 20%. A stimulation rate &amp;gt; 20% in the PPM control before discharge and after three months was detected in 69 % and 67 % of them, respectively. Using uni- and multivariate logistic regression analysis, only new occurred third degree atrioventricular blocks (p=0.007) was identified as predictor for high burden right ventricular pacing after PPM-implantation as well as after 3 months. The implantation depth of the THV showed no influence on the PPM stimulation rate – neither the mean value nor the exact implantation depth adjacent the left, right and non-coronary cusp (P=0.455, P=0.108 and P=0.932, respectively). There were no significant differences regarding MACE between both groups during long-term follow-up; only the degree of dyspnea 5 years after TAVI was significantly higher in patients with a stimulation rate &amp;gt; 20% (p=0.01). Conclusion We detected no correlation of the exact three-dimensional THV position on mid-term pacing burden in 112 patients implanted with a new PPM after TAVI using CTA fusion imaging. Only a new occurred third degree atrioventricular block was a predictor for a high burden right ventricular pacing (stimulation rate &amp;gt;20%). These results provide important insights into which patients actually require PPM implantation after TAVI.

A Deep Unfolding Network for Multispectral and Hyperspectral Image Fusion

Multispectral and hyperspectral image fusion (MS/HS fusion) aims to generate a high-resolution hyperspectral (HRHS) image by fusing a high-resolution multispectral (HRMS) and a low-resolution hyperspectral (LRHS) images. The deep unfolding-based MS/HS fusion method is a representative deep learning paradigm due to its excellent performance and sufficient interpretability. However, existing deep unfolding-based MS/HS fusion methods only rely on a fixed linear degradation model, which focuses on modeling the relationships between HRHS and HRMS, as well as HRHS and LRHS. In this paper, we break free from this observation model framework and propose a new observation model. Firstly, the proposed observation model is built based on the convolutional sparse coding (CSC) technique, and then a proximal gradient algorithm is designed to solve this model. Secondly, we unfold the iterative algorithm into a deep network, dubbed as MHF-CSCNet, where the proximal operators are learned using convolutional neural networks. Finally, all trainable parameters can be automatically learned end-to-end from the training pairs. Experimental evaluations conducted on various benchmark datasets demonstrate the superiority of our method both quantitatively and qualitatively compared to other state-of-the-art methods.

Advances in the diagnosis of prostate cancer based on image fusion

Image fusion currently plays an important role in the diagnosis of prostate cancer (PCa). Selecting and developing a good image fusion algorithm is the core task of achieving image fusion, which determines whether the fusion image obtained is of good quality and can meet the actual needs of clinical application. In recent years, it has become one of the research hotspots of medical image fusion. In order to make a comprehensive study on the methods of medical image fusion, this paper reviewed the relevant literature published at home and abroad in recent years. Image fusion technologies were classified, and image fusion algorithms were divided into traditional fusion algorithms and deep learning (DL) fusion algorithms. The principles and workflow of some algorithms were analyzed and compared, their advantages and disadvantages were summarized, and relevant medical image data sets were introduced. Finally, the future development trend of medical image fusion algorithm was prospected, and the development direction of medical image fusion technology for the diagnosis of prostate cancer and other major diseases was pointed out.