Multimodal Image Fusion Research Articles

Transarterial embolization for anterior cranial fossa dural arteriovenous fistula based on multi-modal three-dimensional imaging

Background: Dural arteriovenous fistula (DAVF) in the anterior cranial fossa (ACF) is known to show a high risk of intracranial hemorrhage. Recently, multi-modal fusion imaging with computed tomography angiography, computed tomography venography, and three-dimensional (3D) rotation angiography have been used preoperatively to ensure anatomical safety. We report on endovascular treatment as a first-line approach for ACFDAVF based on the understanding of vascular anatomy obtained from multi-modal fusion imaging. Methods: All patients with ACF-DAVF treated endovascularly as a first-line approach were included in this study. Analyses took into account complications (particularly visual function), immediate angiographic outcomes, and follow-up findings in consecutive patients with ACF-DAVF treated with interventional treatment based on multi-modal fusion imaging. Results: Five patients with ACF-DAVF underwent six sessions of transarterial embolization (TAE) in our institution. The five male patients (mean age, 74.5 years; range, 60–84 years) were treated with liquid embolic agents (Onyx, four procedures; n-butyl 2-cyanoacrylate, two procedures). No difference was seen between preoperative image evaluation and image evaluation during the endovascular procedure, and in all cases, a microcatheter was navigated into a target artery assumed from preoperative multi-modal imaging, allowing treatment completion in a single procedure. In all cases, the shunt disappeared completely and visual function after procedure was maintained. At the last follow-up, all patients showed a modified Rankin scale score of 0 or 1 with no recurrences. Conclusion: Multi-modal fusion imaging facilitates a 3D understanding of the vascular anatomy, allowing TAE as the first-line treatment for ACF-DAVF.

An ensemble deep learning model for medical image fusion with Siamese neural networks and VGG-19.

Multimodal medical image fusion methods, which combine complementary information from many multi-modality medical images, are among the most important and practical approaches in numerous clinical applications. Various conventional image fusion techniques have been developed for multimodality image fusion. Complex procedures for weight map computing, fixed fusion strategy and lack of contextual understanding remain difficult in conventional and machine learning approaches, usually resulting in artefacts that degrade the image quality. This work proposes an efficient hybrid learning model for medical image fusion using pre-trained and non-pre-trained networks i.e. VGG-19 and SNN with stacking ensemble method. The model leveraging the unique capabilities of each architecture, can effectively preserve the detailed information with high visual quality, for numerous combinations of image modalities in image fusion challenges, notably improved contrast, increased resolution, and lower artefacts. Additionally, this ensemble model can be more robust in the fusion of various combinations of source images that are publicly available from Havard-Medical-Image-Fusion Datasets, GitHub. and Kaggle. Our proposed model performance is superior in terms of visual quality and performance metrics to that of the existing fusion methods in literature like PCA+DTCWT, NSCT, DWT, DTCWT+NSCT, GADCT, CNN and VGG-19.

Current Applications of the Three-Dimensional Printing Technology in Neurosurgery: A Review.

In the recent years, three-dimensional (3D) printing technology has emerged as a transformative tool, particularly in health care, offering unprecedented possibilities in neurosurgery. This review explores the diverse applications of 3D printing in neurosurgery, assessing its impact on precision, customization, surgical planning, and education. A literature review was conducted using PubMed, Web of Science, Embase, and Scopus, identifying 84 relevant articles. These were categorized into spine applications, neurovascular applications, neuro-oncology applications, neuroendoscopy applications, cranioplasty applications, and modulation/stimulation applications. 3D printing applications in spine surgery showcased advancements in guide devices, prosthetics, and neurosurgical planning, with patient-specific models enhancing precision and minimizing complications. Neurovascular applications demonstrated the utility of 3D-printed guide devices in intracranial hemorrhage and enhanced surgical planning for cerebrovascular diseases. Neuro-oncology applications highlighted the role of 3D printing in guide devices for tumor surgery and improved surgical planning through realistic models. Neuroendoscopy applications emphasized the benefits of 3D-printed guide devices, anatomical models, and educational tools. Cranioplasty applications showed promising outcomes in patient-specific implants, addressing biomechanical considerations. The integration of 3D printing into neurosurgery has significantly advanced precision, customization, and surgical planning. Challenges include standardization, material considerations, and ethical issues. Future directions involve integrating artificial intelligence, multimodal imaging fusion, biofabrication, and global collaboration. 3D printing has revolutionized neurosurgery, offering tailored solutions, enhanced surgical planning, and invaluable educational tools. Addressing challenges and exploring future innovations will further solidify the transformative impact of 3D printing in neurosurgical care. This review serves as a comprehensive guide for researchers, clinicians, and policymakers navigating the dynamic landscape of 3D printing in neurosurgery.

A novel 3D multimodal fusion imaging surgical guidance in microvascular decompression for primary trigeminal neuralgia and hemifacial spasm

BackgroundNeurovascular compression (NVC) is a primary etiology of trigeminal neuralgia (TN) and hemifacial spasm (HFS). Despite Magnetic Resonance Tomographic Angiography (MRTA) being a useful tool for 3D multimodal fusion imaging (MFI) in microvascular decompression (MVD) surgery planning, it may not visualize smaller arterial vessels and veins effectively. We validate a novel computed tomography angiography and venography (CTA/V) - diffusion tensor tractography (DTT) -3D-MFI to enhance the MVD surgical guidance.MethodsIn this prospective study, 80 patients with unilateral primary TN or HFS who underwent MVD surgery were included. Imaging was conducted using CTA/V-DTT-3D-MFI compared with CT-MRTA-3D-MFI in predicting the responsible vessel and assessing the severity of NVC. Surgical outcomes were subsequently analyzed. Neurosurgery residents were provided with questionnaires to evaluate and compare the two approaches.ResultsCTA/V-DTT-3D-MFI significantly improved accuracy in identifying the responsible vessel (kappa = 0.954) and NVC (kappa = 0.969) compared to CT-MRTA-3D-MFI, aligning well with surgical findings. CTA/V-DTT-3D-MFI also exhibited higher sensitivity in identifying responsible vessels (98.0%) and NVC (98.7%) than CT-MRTA-3D-MFI. Additionally, CTA/V-DTT-3D-MFI showed fewer complications, shorter operation times, and lower recurrence after one year (all p < 0.05). Resident neurosurgeons emphasized that CTA/V-DTT-3D-MFI greatly assisted in formulating precise surgical strategies for more accurate identification and protection of responsible vessels and nerves (all p < 0.001).ConclusionCTA/V-DTT-3D-MFI enhances MVD surgery guidance, improving accuracy in identifying responsible vessels and NVC for better outcomes. This advanced imaging plays a crucial role in safer and more effective MVD surgery, as well as in training neurosurgeons.

Adaptive infrared patterns for microscopic surface reconstructions.

Multi-zoom microscopic surface reconstructions of operating sites, especially in ENT surgeries, would allow multimodal image fusion for determining the amount of resected tissue, for recognizing critical structures, and novel tools for intraoperative quality assurance. State-of-the-art three-dimensional model creation of the surgical scene is challenged by the surgical environment, illumination, and the homogeneous structures of skin, muscle, bones, etc., that lack invariant features for stereo reconstruction. An adaptive near-infrared pattern projector illuminates the surgical scene with optimized patterns to yield accurate dense multi-zoom stereoscopic surface reconstructions. The approach does not impact the clinical workflow. The new method is compared to state-of-the-art approaches and is validated by determining its reconstruction errors relative to a high-resolution 3D-reconstruction of CT data. 200 surface reconstructions were generated for 5 zoom levels with 10 reconstructions for each object illumination method (standard operating room light, microscope light, random pattern and adaptive NIR pattern). For the adaptive pattern, the surface reconstruction errors ranged from 0.5 to 0.7 mm, as compared to 1-1.9 mm for the other approaches. The local reconstruction differences are visualized in heat maps. Adaptive near-infrared (NIR) pattern projection in microscopic surgery allows dense and accurate microscopic surface reconstructions for variable zoom levels of small and homogeneous surfaces. This could potentially aid in microscopic interventions at the lateral skull base and potentially open up new possibilities for combining quantitative intraoperative surface reconstructions with preoperative radiologic imagery.

A review of deep learning approaches for multimodal image segmentation of liver cancer.

This review examines the recent developments in deep learning (DL) techniques applied to multimodal fusion image segmentation for liver cancer. Hepatocellular carcinoma is a highly dangerous malignant tumor that requires accurate image segmentation for effective treatment and disease monitoring. Multimodal image fusion has the potential to offer more comprehensive information and more precise segmentation, and DL techniques have achieved remarkable progress in this domain. This paper starts with an introduction to liver cancer, then explains the preprocessing and fusion methods for multimodal images, then explores the application of DL methods in this area. Various DL architectures such as convolutional neural networks (CNN) and U-Net are discussed and their benefits in multimodal image fusion segmentation. Furthermore, various evaluation metrics and datasets currently used to measure the performance of segmentation models are reviewed. While reviewing the progress, the challenges of current research, such as data imbalance, model generalization, and model interpretability, are emphasized and future research directions are suggested. The application of DL in multimodal image segmentation for liver cancer is transforming the field of medical imaging and is expected to further enhance the accuracy and efficiency of clinical decision making. This review provides useful insights and guidance for medical practitioners.

Radiation image reconstruction and uncertainty quantification using a Gaussian process prior

We propose a complete framework for Bayesian image reconstruction and uncertainty quantification based on a Gaussian process prior (GPP) to overcome limitations of maximum likelihood expectation maximization (ML-EM) image reconstruction algorithm. The prior distribution is constructed with a zero-mean Gaussian process (GP) with a choice of a covariance function, and a link function is used to map the Gaussian process to an image. Unlike many other maximum a posteriori approaches, our method offers highly interpretable hyperparamters that are selected automatically with the empirical Bayes method. Furthermore, the GP covariance function can be modified to incorporate a priori structural priors, enabling multi-modality imaging or contextual data fusion. Lastly, we illustrate that our approach lends itself to Bayesian uncertainty quantification techniques, such as the preconditioned Crank–Nicolson method and the Laplace approximation. The proposed framework is general and can be employed in most radiation image reconstruction problems, and we demonstrate it with simulated free-moving single detector radiation source imaging scenarios. We compare the reconstruction results from GPP and ML-EM, and show that the proposed method can significantly improve the image quality over ML-EM, all the while providing greater understanding of the source distribution via the uncertainty quantification capability. Furthermore, significant improvement of the image quality by incorporating a structural prior is illustrated.

MHW-GAN: Multidiscriminator Hierarchical Wavelet Generative Adversarial Network for Multimodal Image Fusion.

Image fusion technology aims to obtain a comprehensive image containing a specific target or detailed information by fusing data of different modalities. However, many deep learning-based algorithms consider edge texture information through loss functions instead of specifically constructing network modules. The influence of the middle layer features is ignored, which leads to the loss of detailed information between layers. In this article, we propose a multidiscriminator hierarchical wavelet generative adversarial network (MHW-GAN) for multimodal image fusion. First, we construct a hierarchical wavelet fusion (HWF) module as the generator of MHW-GAN to fuse feature information at different levels and scales, which avoids information loss in the middle layers of different modalities. Second, we design an edge perception module (EPM) to integrate edge information from different modalities to avoid the loss of edge information. Third, we leverage the adversarial learning relationship between the generator and three discriminators for constraining the generation of fusion images. The generator aims to generate a fusion image to fool the three discriminators, while the three discriminators aim to distinguish the fusion image and edge fusion image from two source images and the joint edge image, respectively. The final fusion image contains both intensity information and structure information via adversarial learning. Experiments on public and self-collected four types of multimodal image datasets show that the proposed algorithm is superior to the previous algorithms in terms of both subjective and objective evaluation.

MixFuse: An iterative mix-attention transformer for multi-modal image fusion

Multi-modal image fusion plays a crucial role in various visual systems. However, existing methods typically involve a multi-stage pipeline, i.e., feature extraction, integration, and reconstruction, which limits the effectiveness and efficiency of feature interaction and aggregation. In this paper, we propose MixFuse, a compact multi-modal image fusion framework based on Transformers. It smoothly unifies the process of feature extraction and integration, As its core, the Mix Attention Transformer Block (MATB) integrates the Cross-Attention Transformer Module (CATM) and the Self-Attention Transformer Module (SATM). The CATM introduces a symmetrical cross-attention mechanism to identify modality-specific and general features, filtering out irrelevant and redundant information. Meanwhile, the SATM is designed to refine the combined features via a self-attention mechanism, enhancing the internal consistency and proper preservation of the features. This successive cross and self-attention modules work together to enhance the generation of more accurate and refined feature maps, which are essential for later reconstruction. Extensive evaluation of MixFuse on five public datasets shows its superior performance and adaptability over state-of-the-art methods. The code and model will be released at https://github.com/Bitlijinfu/MixFuse.

Enhanced Preoperative Planning for Intracranial Aneurysms Through Multimodal Image Fusion of Silent/Time-of-Magnetic Resonance Angiography and Computed Tomography Using 3DSlicer

Objective: This study evaluates the effectiveness of multimodal image fusion (MIF) using silent and time-of-flight (TOF) magnetic resonance angiography (MRA) and computed tomography (CT) for preoperative planning in patients with intracranial aneurysms who have contraindications to contrast media. Materials and Methods: A retrospective study included 40 patients with intracranial aneurysms, diagnosed using three-dimensional computed tomography angiography (CTA). These patients underwent both Silent and TOF MRA scans, followed by a CTA scan. The multi-image fusion (MIF) technique, applied using 3DSlicer software, integrated the silent/TOF-MRA with CT images for preoperative assessment. This study compared the image quality, aneurysm detection sensitivity, and anatomic accuracy of the MIF images with those of three-dimensional CTA. Results: Silent-MRA-CT fusion images demonstrated higher sensitivity (95.5%) and lower false negative rates (4.5%) compared with TOF-MRA-CT. Furthermore, silent-MRA-CT fusion images outperformed TOF-MRA-CT in terms of signal homogeneity, venous signal interference suppression, and aneurysm visibility (all P &lt; 0.05). The interclass correlation coefficient and kappa values for aneurysm morphology and shape indicated superior measurement consistency and shape concordance of silent-MRA-CT with CTA compared with TOF-MRA-CT (all P &lt; 0.01). Conclusion: This study supports the use of silent/TOF-MRA-CT fusion imaging as a reliable alternative to CTA, noting that silent-MRA-CT closely mirrors CTA. Contrast-free MRA-CT fusion images have the potential to be used for preoperative planning in patients with intracranial aneurysms who have contraindications to contrast.

Multimodal registration network with multi-scale feature-crossing.

A critical piece of information for prostate intervention and cancer treatment is provided by the complementary medical imaging modalities of ultrasound (US) and magnetic resonance imaging (MRI). Therefore, MRI-US image fusion is often required during prostate examination to provide contrast-enhanced TRUS, in which image registration is a key step in multimodal image fusion. We propose a novel multi-scale feature-crossing network for the prostate MRI-US image registration task. We designed a feature-crossing module to enhance information flow in the hidden layer by integrating intermediate features between adjacent scales. Additionally, an attention block utilizing three-dimensional convolution interacts information between channels, improving the correlation between different modal features. We used 100 cases randomly selected from The Cancer Imaging Archive (TCIA) for our experiments. A fivefold cross-validation method was applied, dividing the dataset into five subsets. Four subsets were used for training, and one for testing, repeating this process five times to ensure each subset served as the test set once. We test and evaluate our technique using fivefold cross-validation. The cross-validation trials result in a median target registration error of 2.20 mm on landmark centroids and a median Dice of 0.87 on prostate glands, both of which were better than the baseline model. In addition, the standard deviation of the dice similarity coefficient is 0.06, which suggests that the model is stable. We propose a novel multi-scale feature-crossing network for the prostate MRI-US image registration task. A random selection of 100 cases from The Cancer Imaging Archive (TCIA) was used to test and evaluate our approach using fivefold cross-validation. The experimental results showed that our method improves the registration accuracy. After registration, MRI and TURS images were more similar in structure and morphology, and the location and morphology of cancer were more accurately reflected in the images.

The value of multimodal imaging fusion in preoperative visualization assessment of neurovascular relationship in hemifacial spasm: a single-center retrospective study.

The neurovascular conflict (NVC) at the brainstem exit zone of the facial nerve is considered the primary etiology of primary hemifacial spasm (HFS). Therefore, microvascular decompression (MVD) has become the preferred treatment for HFS. Successful neurovascular decompression can achieve significant therapeutic effects, and accurately identifying the site of compression is crucial for the success of this surgery. Detailed diagnostic neuroimaging plays an important role in accurately identifying the site of compression.The purpose of this study is to explore the feasibility and predictive value of preoperative visualization assessment of the neurovascular relationship in HFS using 3D Slicer software based on multimodal imaging fusion. This aims to reduce the omission of responsible vessels and lower the incidence of postoperative complications, thereby potentially improving the efficacy and safety of the surgery. This study retrospectively analyzed 80 patients with HFS who underwent MVD surgery. All patients underwent preoperative cranial MRI scans, including the 3D-FIESTA and the 3D-TOF MRA sequences. Three-dimensional models were reconstructed from the multimodal MRI images using 3D Slicer software. Independent observers, who were blinded to the surgical outcomes, evaluated the neurovascular relationships using both the three-dimensional models and multimodal MRI images. The assessment results were compared with intraoperative findings, and statistical analysis was conducted using SPSS 22.0 software. The agreement between preoperative assessment using the 3D-TOF MRA sequence combined with the 3D-FIESTA sequence and intraoperative findings was represented by a Kappa value of 0.343, while the Kappa value for agreement between three-dimensional reconstruction and intraoperative findings was 0.637. There was a statistically significant difference between the two methods ( X2 = 18.852, P = 0.001 ). The sensitivity and specificity of the 3D-TOF MRA sequence combined with the 3D-FIESTA sequence for evaluating neurovascular relationships were 92.4% and 100%, respectively, while for three-dimensional reconstruction, both were 100%. The Kappa value for agreement between preoperative the 3D-TOF MRA sequence combined with the 3D-FIESTA sequence prediction of offending vessels and intraoperative findings was 0.625, while the Kappa value for agreement between three-dimensional reconstruction and intraoperative findings was 0.938, showing a statistically significant difference ( X2 = 317.798, P = 0.000 ). The Kappa value for agreement between preoperative the 3D-TOF MRA sequence combined with the 3D-FIESTA sequence assessment of the anatomical location of facial nerve involvement in neurovascular compression and intraoperative findings was 0.608, while the Kappa value for agreement between three-dimensional reconstruction and intraoperative findings was 0.918, also showing a statistically significant difference ( X2 = 504.647, P = 0.000 ). The preoperative visualization assessment of neurovascular relationships in HFS using 3D Slicer software based on multimodal imaging fusion has been demonstrated to be reliable. It is more accurate than combining the 3D-TOF MRA sequence with the 3D-FIESTA sequence and shows higher consistency with intraoperative findings. This method provides guidance for surgical procedures and thereby potentially enhances the efficacy and safety of surgeries to a certain extent.

Comparative study of CTA/CTV and MRTA for preoperative simulation of microvascular decompression in neurovascular compression syndromes.

Neurovascular compression syndrome (NVCS), characterized by cranial nerve compression due to adjacent blood vessels at the root entry zone, frequently presents as trigeminal neuralgia (TN), hemifacial spasm (HFS), or glossopharyngeal neuralgia (GN). Despite its prevalence in NVCS assessment, Magnetic Resonance Tomographic Angiography (MRTA)'s limited sensitivity to small vessels and veins poses challenges. This study aims to refine vessel localization and surgical planning for NVCS patients using a novel 3D multimodal fusion imaging (MFI) technique incorporating computed tomography angiography and venography (CTA/CTV). A retrospective analysis was conducted on 76 patients who underwent MVD surgery and were diagnosed with single-site primary TN, HFS, or GN. Imaging was obtained from MRTA and CTA/CTV sequences, followed by image processing and 3D-MFI using FastSurfer and 3DSlicer. The CTA/CTV-3D-MFI showed higher sensitivity than MRTA-3D-MFI in predicting responsible vessels (98.6% vs. 94.6%) and NVC severity (98.6% vs. 90.8%). Kappa coefficients revealed strong agreement with MRTA-3D-MFI (0.855 for vessels, 0.835 for NVC severity) and excellent agreement with CTA/CTV-3D-MFI (0.951 for vessels, 0.952 for NVC). Resident neurosurgeons significantly preferred CTA/CTV-3D-MFI due to its better correlation with surgical reality, clearer depiction of surgical anatomy, and optimized visualization of approaches (p < 0.001). Implementing CTA/CTV-3D-MFI significantly enhanced diagnostic accuracy and surgical planning for NVCS, outperforming MRTA-3D-MFI in identifying responsible vessels and assessing NVC severity. This innovative imaging modality can potentially improve outcomes by guiding safer and more targeted surgeries, particularly in cases where MRTA may not adequately visualize crucial neurovascular structures.

Multimodal medical image fusion based on interval gradients and convolutional neural networks

Many image fusion methods have been proposed to leverage the advantages of functional and anatomical images while compensating for their shortcomings. These methods integrate functional and anatomical images while presenting physiological and metabolic organ information, making their diagnostic efficiency far greater than that of single-modal images. Currently, most existing multimodal medical imaging fusion methods are based on multiscale transformation, which involves obtaining pyramid features through multiscale transformation. Low-resolution images are used to analyse approximate image features, and high-resolution images are used to analyse detailed image features. Different fusion rules are applied to achieve feature fusion at different scales. Although these fusion methods based on multiscale transformation can effectively achieve multimodal medical image fusion, much detailed information is lost during multiscale and inverse transformation, resulting in blurred edges and a loss of detail in the fusion images. A multimodal medical image fusion method based on interval gradients and convolutional neural networks is proposed to overcome this problem. First, this method uses interval gradients for image decomposition to obtain structure and texture images. Second, deep neural networks are used to extract perception images. Three methods are used to fuse structure, texture, and perception images. Last, the images are combined to obtain the final fusion image after colour transformation. Compared with the reference algorithms, the proposed method performs better in multiple objective indicators of QEN\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{EN}$$\\end{document}, QNIQE\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{NIQE}$$\\end{document}, QSD\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{SD}$$\\end{document}, QSSEQ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{SSEQ}$$\\end{document} and QTMQI\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{TMQI}$$\\end{document}.

Multi-Source Image Fusion Based on BEMD and Region Sharpness Guidance Region Overlapping Algorithm

Multi-focal image and multi-modal image fusion technology can fully take advantage of different sensors or different times, retaining the image feature information and improving the image quality. A multi-source image fusion algorithm based on bidimensional empirical mode decomposition (BEMD) and a region sharpness-guided region overlapping algorithm are studied in this article. Firstly, source images are decomposed into multi-layer bidimensional intrinsic mode functions (BIMFs) and residuals from high-frequency layer to low-frequency layer by BEMD. Gaussian bidimensional intrinsic mode functions (GBIMFs) are obtained by applying Gaussian filtering operated on BIMF and calculating the sharpness value of segmented regions using an improved weighted operator based on the Tenengrad function, which is the key to comparison selection and fusion. Then, the GBIMFs and residuals selected by sharpness comparison strategy are fused by the region overlapping method, and the stacked layers are weighted to construct the final fusion image. Finally, based on qualitative evaluation and quantitative evaluation indicators, the proposed algorithm is compared with six typical image fusion algorithms. The comparison results show that the proposed algorithm can effectively capture the feature information of images in different states and reduce the redundant information.

HDCCT: Hybrid Densely Connected CNN and Transformer for Infrared and Visible Image Fusion

Multi-modal image fusion is a methodology that combines image features from multiple types of sensors, effectively improving the quality and content of fused images. However, most existing deep learning fusion methods need to integrate global or local features, restricting the representation of feature information. To address this issue, a hybrid densely connected CNN and transformer (HDCCT) fusion framework is proposed. In the proposed HDCCT framework, the network of the CNN-based blocks obtain the local structure of the input data, and the transformer-based blocks obtain the global structure of the original data, significantly improving the feature representation. In the fused image, the proposed encoder–decoder architecture is designed for both the CNN and transformer blocks to reduce feature loss while preserving the characterization of all-level features. In addition, the cross-coupled framework facilitates the flow of feature structures, retains the uniqueness of information, and makes the transform model long-range dependencies based on the local features already extracted by the CNN. Meanwhile, to retain the information in the source images, the hybrid structural similarity (SSIM) and mean square error (MSE) loss functions are introduced. The qualitative and quantitative comparisons of grayscale images with infrared and visible image fusion indicate that the suggested method outperforms related works.

Enhancing three-source cross-modality image fusion with improved DenseNet for infrared polarization and visible light images

The fusion of multi-modal images to create an image that preserves the unique features of each modality as well as the features shared across modalities is a challenging task, particularly in the context of infrared (IR)-visible image fusion. In addition, the presence of polarization and IR radiation information in images obtained from IR polarization sensors further complicates the multi-modal image-fusion process. This study proposes a fusion network designed to overcome the challenges associated with the integration of low-resolution IR, IR polarization, and high-resolution visible (VIS) images. By introducing cross attention modules and a multi-stage fusion approach, the network can effectively extract and fuse features from different modalities, fully expressing the diversity of the images. This network learns end-to-end mapping from sourced to fused images using a loss function, eliminating the need for ground-truth images for fusion. Experimental results using public datasets and remote-sensing field-test data demonstrate that the proposed methodology achieves commendable results in qualitative and quantitative evaluations, with gradient based fusion performance QAB/F, mutual information (MI), and QCB values higher than the second-best values by 0.20, 0.94, and 0.04, respectively. This study provides a comprehensive representation of target scene information that results in enhanced image quality and improved object identification capabilities. In addition, outdoor and VIS image datasets are produced, providing a data foundation and reference for future research in related fields.

A cross-temporal multimodal fusion system based on deep learning for orthodontic monitoring

IntroductionIn the treatment of malocclusion, continuous monitoring of the three-dimensional relationship between dental roots and the surrounding alveolar bone is essential for preventing complications from orthodontic procedures. Cone-beam computed tomography (CBCT) provides detailed root and bone data, but its high radiation dose limits its frequent use, consequently necessitating an alternative for ongoing monitoring. ObjectivesWe aimed to develop a deep learning-based cross-temporal multimodal image fusion system for acquiring root and jawbone information without additional radiation, enhancing the ability of orthodontists to monitor risk. MethodsUtilizing CBCT and intraoral scans (IOSs) as cross-temporal modalities, we integrated deep learning with multimodal fusion technologies to develop a system that includes a CBCT segmentation model for teeth and jawbones. This model incorporates a dynamic kernel prior model, resolution restoration, and an IOS segmentation network optimized for dense point clouds. Additionally, a coarse-to-fine registration module was developed. This system facilitates the integration of IOS and CBCT images across varying spatial and temporal dimensions, enabling the comprehensive reconstruction of root and jawbone information throughout the orthodontic treatment process. ResultsThe experimental results demonstrate that our system not only maintains the original high resolution but also delivers outstanding segmentation performance on external testing datasets for CBCT and IOSs. CBCT achieved Dice coefficients of 94.1 % and 94.4 % for teeth and jawbones, respectively, and it achieved a Dice coefficient of 91.7 % for the IOSs. Additionally, in the context of real-world registration processes, the system achieved an average distance error (ADE) of 0.43 mm for teeth and 0.52 mm for jawbones, significantly reducing the processing time. ConclusionWe developed the first deep learning-based cross-temporal multimodal fusion system, addressing the critical challenge of continuous risk monitoring in orthodontic treatments without additional radiation exposure. We hope that this study will catalyze transformative advancements in risk management strategies and treatment modalities, fundamentally reshaping the landscape of future orthodontic practice.

Enhanced river suspended sediment concentration identification via multimodal video image, optical flow, and water temperature data fusion

Monitoring suspended sediment concentration (SSC) in rivers is pivotal for water quality management and sustainable river ecosystem development. However, achieving continuous and precise SSC monitoring is fraught with challenges, including low automation, lengthy measurement processes, and high cost. This study proposes an innovative approach for SSC identification in rivers using multimodal data fusion. We developed a robust model by harnessing colour features from video images, motion characteristics from the Lucas–Kanade (LK) optical flow method, and temperature data. By integrating ResNet with a mixed density network (MDN), our method fused the image and optical flow fields, and temperature data to enhance accuracy and reliability. Validated at a hydropower station in the Xinjiang Uygur Autonomous Region, China, the results demonstrated that while the image field alone offers a baseline level of SSC identification, it experiences local errors under specific conditions. The incorporation of optical flow and water temperature information enhanced model robustness, particularly when coupling the image and optical flow fields, yielding a Nash–Sutcliffe efficiency (NSE) of 0.91. Further enhancement was observed with the combined use of all three data types, attaining an NSE of 0.93. This integrated approach offers a more accurate SSC identification solution, enabling non-contact, low-cost measurements, facilitating remote online monitoring, and supporting water resource management and river water–sediment element monitoring.

Intelligent fault diagnosis of bearings driven by double-level data fusion based on multichannel sample fusion and feature fusion under time-varying speed conditions

Bearing fault diagnosis holds paramount importance in guaranteeing the smooth operation of mechanical systems. However, single-channel vibration sensor data poses limitations in providing a comprehensive representation of bearing fault characteristics, and the fault diagnosis model extracts fault features with insufficient representativeness, resulting in low reliability and poor generalization capability of the model in the face of variable operating conditions. Therefore, a novel intelligent fault diagnosis model of bearings driven by double-level data fusion (DLDF) is developed in this paper. Initially, the fusion weights of each channel's data are obtained using the information entropy method. Then, a novel multi-modal image fusion strategy is proposed to integrate the time-frequency representations of vibration data from channels X, Y, and Z at a specific measurement point. Finally, the shallow and deep feature components extracted from the CNN model and the feature components obtained through different mapping methods are fused to extract more representative fault features, achieving a reliable diagnosis of bearing faults under time-varying speed conditions. The efficacy and reliability of the proposed approach are demonstrated through the fault diagnosis outcomes of two sets of bearing experimental data under time-varying speed conditions.