Multimodal Diagnosis Research Articles

Parkinson's Disease Prediction: An Attention-Based Multimodal Fusion Framework Using Handwriting and Clinical Data.

Neurodegenerative diseases (NGD) encompass a range of progressive neurological conditions, such as Alzheimer's disease (AD) and Parkinson's disease (PD), characterised by the gradual deterioration of neuronal structure and function. This degeneration manifests as cognitive decline, movement impairment, and dementia. Our focus in this investigation is on PD, a neurodegenerative disorder characterized by the loss of dopamine-producing neurons in the brain, leading to motor disturbances. Early detection of PD is paramount for enhancing quality of life through timely intervention and tailored treatment. However, the subtle nature of initial symptoms, like slow movements, tremors, muscle rigidity, and psychological changes, often reduce daily task performance and complicate early diagnosis. To assist medical professionals in timely diagnosis of PD, we introduce a cutting-edge Multimodal Diagnosis framework (PMMD). Based on deep learning techniques, the PMMD framework integrates imaging, handwriting, drawing, and clinical data to accurately detect PD. Notably, it incorporates cross-modal attention, a methodology previously unexplored within the area, which facilitates the modelling of interactions between different data modalities. The proposed method exhibited an accuracy of 96% on the independent tests set. Comparative analysis against state-of-the-art models, along with an in-depth exploration of attention mechanisms, highlights the efficacy of PMMD in PD classification. The obtained results highlight exciting new prospects for the use of handwriting as a biomarker, along with other information, for optimal model performance. PMMD's success in integrating diverse data sources through cross-modal attention underscores its potential as a robust diagnostic decision support tool for accurately diagnosing PD.

3D Multimodal Fusion Network with Disease-induced Joint Learning for Early Alzheimer's Disease Diagnosis.

Multimodal neuroimaging provides complementary information critical for accurate early diagnosis of Alzheimer's disease (AD). However, the inherent variability between multimodal neuroimages hinders the effective fusion of multimodal features. Moreover, achieving reliable and interpretable diagnoses in the field of multimodal fusion remains challenging. To address them, we propose a novel multimodal diagnosis network based on multi-fusion and disease-induced learning (MDL-Net) to enhance early AD diagnosis by efficiently fusing multimodal data. Specifically, MDL-Net proposes a multi-fusion joint learning (MJL) module, which effectively fuses multimodal features and enhances the feature representation from global, local, and latent learning perspectives. MJL consists of three modules, global-aware learning (GAL), local-aware learning (LAL), and outer latent-space learning (LSL) modules. GAL via a self-adaptive Transformer (SAT) learns the global relationships among the modalities. LAL constructs local-aware convolution to learn the local associations. LSL module introduces latent information through outer product operation to further enhance feature representation. MDL-Net integrates the disease-induced region-aware learning (DRL) module via gradient weight to enhance interpretability, which iteratively learns weight matrices to identify AD-related brain regions. We conduct the extensive experiments on public datasets and the results confirm the superiority of our proposed method. Our code will be available at: https://github.com/qzf0320/MDL-Net.

Multimodal diagnosis of Alzheimer’s disease based on volumetric and cognitive assessments

Multimodal diagnosis of Alzheimer’s disease based on volumetric and cognitive assessments

BIOM-14. UTILITY OF MIRNA-BASED LIQUID BIOPSY AND RADIOGRAPHIC IMAGING IN PEDIATRIC BRAIN TUMOR PATIENTS

Abstract BACKGROUND MRI is the current gold standard imaging technique for diagnostic evaluation and monitoring of pediatric CNS tumors; however, MRI measures can only provide inference into tumor biology and molecular stratification. miRNAs are an abundant and stable nucleic acid found circulating in blood and cerebrospinal fluid (CSF) and can be utilized as a tumor biomarker. METHODS We hypothesized that correlating miRNA biomarkers with radiological tumor measurements may provide improved diagnostic and monitoring for patients with pediatric brain tumors. Using a cohort of 54 pediatric brain tumors including low grade glioma, ependymoma, germ cell tumor, medulloblastoma, atypical teratoid rhabdoid tumor and high-grade glioma we attempted to combine MRI findings and circulating miRNA data. The miRNA expression was profiled in 33 CSF and 52 plasma samples using the HTG EdgeSeq platform. Clinically acquired, multi-parametric MRI scans at time-points close to liquid biopsy collection were collected retrospectively and used to generate volumetric tumor segmentations. Machine learning methods were implemented to evaluate combinatory features. RESULTS We identified unique miRNA targets that significantly correlated with MRI features, clinical findings, and patient outcomes. In both CSF and plasma, miRNA expression was found to correlate with diagnosis and clinical features including tumor grade and disease burden (p < 0.05). In CSF, miRNA expression was correlated with MRI features including cystic core presence, edema, leptomeningeal disease and tumor location (p <0.05). In plasma, differences in miRNA expression were related to MRI measurements including cystic core volume, location, and leptomeningeal disease (p < 0.05). Combination of miRNA targets for plasma or CSF with imaging-based features significantly improved histology specific multimodal diagnosis predictions. CONCLUSIONS These results demonstrate the potential utility of miRNAs as a pediatric brain tumor biomarker which when combined with imaging features can improve minimally and non-invasive diagnostics and the subsequent management of pediatric brain tumor patients.

Multimodal diagnosis of cerebrospinal fluid rhinorrhea: State of the art review and emerging concepts.

Currently, diagnosis of cerebrospinal fluid (CSF) rhinorrhea relies on a multimodal approach, increasing costs and ultimately delaying diagnosis. In the United States and internationally, the crux of such a diagnosis relies on confirmation testing (via biomarkers) and localization (e.g., imaging). Biomarker testing may require analysis at an outside facility, resulting in delays diagnosis and treatment. In addition, specialized imaging may be nonspecific and often requires an active leak for diagnosis. There remains a clear need for innovative new technology. A comprehensive review was conducted on both foundational and innovative scholarly articles regarding current and emerging diagnosis modalities for CSF. Current modalities in CSF rhinorrhea diagnosis and localization include laboratory tests (namely, B2T immunofixation), imaging (CT and/or MRI) with or without intrathecal administration, and surgical exploration. Each of these modalities carry flaws, risks, and benefits, ultimately contributing to delays in diagnosis and morbidity. Promising emerging technologies include lateral flow immunoassays (LFI) and biologically functionalized field-effect transistors (BioFET). Nevertheless, these carry some drawbacks of their own, and require further validation. CSF rhinorrhea remains a challenging diagnosis, requiring a multimodal approach to differentiate from nonpathologic causes of rhinorrhea. Current methods in diagnosis are imperfect, as the ideal test would be a readily accessible, inexpensive, rapid, highly accurate point-of-care test without the need for excess fluid or specialized processing. Critical work is being done to develop promising, new, improved tests, though a clear successor has not yet emerged. N/A.

Validating the accuracy of deep learning for the diagnosis of pneumonia on chest x-ray against a robust multimodal reference diagnosis: a post hoc analysis of two prospective studies.

Artificial intelligence (AI) seems promising in diagnosing pneumonia on chest x-rays (CXR), but deep learning (DL) algorithms have primarily been compared with radiologists, whose diagnosis can be not completely accurate. Therefore, we evaluated the accuracy of DL in diagnosing pneumonia on CXR using a more robust reference diagnosis. We trained a DL convolutional neural network model to diagnose pneumonia and evaluated its accuracy in two prospective pneumonia cohorts including 430 patients, for whom the reference diagnosis was determined a posteriori by a multidisciplinary expert panel using multimodal data. The performance of the DL model was compared with that of senior radiologists and emergency physicians reviewing CXRs and that of radiologists reviewing computed tomography (CT) performed concomitantly. Radiologists and DL showed a similar accuracy on CXR for both cohorts (p ≥ 0.269): cohort 1, radiologist 1 75.5% (95% confidence interval 69.1-80.9), radiologist 2 71.0% (64.4-76.8), DL 71.0% (64.4-76.8); cohort 2, radiologist 70.9% (64.7-76.4), DL 72.6% (66.5-78.0). The accuracy of radiologists and DL was significantly higher (p ≤ 0.022) than that of emergency physicians (cohort 1 64.0% [57.1-70.3], cohort 2 63.0% [55.6-69.0]). Accuracy was significantly higher for CT (cohort 1 79.0% [72.8-84.1], cohort 2 89.6% [84.9-92.9]) than for CXR readers including radiologists, clinicians, and DL (all p-values < 0.001). When compared with a robust reference diagnosis, the performance of AI models to identify pneumonia on CXRs was inferior than previously reported but similar to that of radiologists and better than that of emergency physicians. The clinical relevance of AI models for pneumonia diagnosis may have been overestimated. AI models should be benchmarked against robust reference multimodal diagnosis to avoid overestimating its performance. NCT02467192 , and NCT01574066 . • We evaluated an openly-access convolutional neural network (CNN) model to diagnose pneumonia on CXRs. • CNN was validated against a strong multimodal reference diagnosis. • In our study, the CNN performance (area under the receiver operating characteristics curve 0.74) was lower than that previously reported when validated against radiologists' diagnosis (0.99 in a recent meta-analysis). • The CNN performance was significantly higher than emergency physicians' (p ≤ 0.022) and comparable to that of board-certified radiologists (p ≥ 0.269).

Identifying High Gleason Score Prostate Cancer by Prostate Fluid Metabolic Fingerprint-Based Multi-Modal Recognition.

Prostate cancer (PCa) is the second most common cancer in males worldwide. The Gleason scoring system, which classifies the pathological growth pattern of cancer, is considered one of the most important prognostic factors for PCa. Compared to indolent PCa, PCa with high Gleason score (h-GS PCa, GS ≥ 8) has greater clinical significance due to its high aggressiveness and poor prognosis. It is crucial to establish a rapid, non-invasive diagnostic modality to decipher patients with h-GS PCa as early as possible. In this study, ferric nanoparticle-assisted laser desorption/ionization mass spectrometry (FeNPALDI-MS) to extract prostate fluid metabolic fingerprint (PSF-MF) is employed and combined with the clinical features of patients, such as prostate-specific antigen (PSA), to establish a multi-modal diagnosis assisted by machine learning. This approach yields an impressive area under the curve (AUC) of 0.87 to diagnose patients with h-GS, surpassing the results of single-modal diagnosis using only PSF-MF or PSA, respectively. Additionally, using various screening methods, six key metabolites that exhibit greater diagnostic efficacy (AUC = 0.96) are identified. These findings also provide insights into related metabolic pathways, which may provide valuable information for further elucidation of the pathological mechanisms underlying h-GS PCa.

Multimodal diagnosis model of Alzheimer’s disease based on improved Transformer

PurposeRecent technological advancements in data acquisition tools allowed neuroscientists to acquire different modality data to diagnosis Alzheimer’s disease (AD). However, how to fuse these enormous amount different modality data to improve recognizing rate and find significance brain regions is still challenging.MethodsThe algorithm used multimodal medical images [structural magnetic resonance imaging (sMRI) and positron emission tomography (PET)] as experimental data. Deep feature representations of sMRI and PET images are extracted by 3D convolution neural network (3DCNN). An improved Transformer is then used to progressively learn global correlation information among features. Finally, the information from different modalities is fused for identification. A model-based visualization method is used to explain the decisions of the model and identify brain regions related to AD.ResultsThe model attained a noteworthy classification accuracy of 98.1% for Alzheimer’s disease (AD) using the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset. Upon examining the visualization results, distinct brain regions associated with AD diagnosis were observed across different image modalities. Notably, the left parahippocampal region emerged consistently as a prominent and significant brain area.ConclusionsA large number of comparative experiments have been carried out for the model, and the experimental results verify the reliability of the model. In addition, the model adopts a visualization analysis method based on the characteristics of the model, which improves the interpretability of the model. Some disease-related brain regions were found in the visualization results, which provides reliable information for AD clinical research.

MI-DenseCFNet: deep learning–based multimodal diagnosis models for Aureus and Aspergillus pneumonia

ObjectiveTo build and merge a diagnostic model called multi-input DenseNet fused with clinical features (MI-DenseCFNet) for discriminating between Staphylococcus aureus pneumonia (SAP) and Aspergillus pneumonia (ASP) and to evaluate the significant correlation of each clinical feature in determining these two types of pneumonia using a random forest dichotomous diagnosis model. This will enhance diagnostic accuracy and efficiency in distinguishing between SAP and ASP.MethodsIn this study, 60 patients with clinically confirmed SAP and ASP, who were admitted to four large tertiary hospitals in Kunming, China, were included. Thoracic high-resolution CT lung windows of all patients were extracted from the picture archiving and communication system, and the corresponding clinical data of each patient were collected.ResultsThe MI-DenseCFNet diagnosis model demonstrates an internal validation set with an area under the curve (AUC) of 0.92. Its external validation set demonstrates an AUC of 0.83. The model requires only 10.24s to generate a categorical diagnosis and produce results from 20 cases of data. Compared with high-, mid-, and low-ranking radiologists, the model achieves accuracies of 78% vs. 75% vs. 60% vs. 40%. Eleven significant clinical features were screened by the random forest dichotomous diagnosis model.ConclusionThe MI-DenseCFNet multimodal diagnosis model can effectively diagnose SAP and ASP, and its diagnostic performance significantly exceeds that of junior radiologists. The 11 important clinical features were screened in the constructed random forest dichotomous diagnostic model, providing a reference for clinicians.Clinical relevance statementMI-DenseCFNet could provide diagnostic assistance for primary hospitals that do not have advanced radiologists, enabling patients with suspected infections like Staphylococcus aureus pneumonia or Aspergillus pneumonia to receive a quicker diagnosis and cut down on the abuse of antibiotics.Key points• MI-DenseCFNet combines deep learning neural networks with crucial clinical features to discern between Staphylococcus aureus pneumonia and Aspergillus pneumonia.• The comprehensive group had an area under the curve of 0.92, surpassing the proficiency of junior radiologists.• This model can enhance a primary radiologist’s diagnostic capacity.

Multi-Modal Diagnosis of Alzheimer's Disease using Interpretable Graph Convolutional Networks.

The interconnection between brain regions in neurological disease encodes vital information for the advancement of biomarkers and diagnostics. Although graph convolutional networks are widely applied for discovering brain connection patterns that point to disease conditions, the potential of connection patterns that arise from multiple imaging modalities has yet to be fully realized. In this paper, we propose a multi-modal sparse interpretable GCN framework (SGCN) for the detection of Alzheimer's disease (AD) and its prodromal stage, known as mild cognitive impairment (MCI). In our experimentation, SGCN learned the sparse regional importance probability to find signature regions of interest (ROIs), and the connective importance probability to reveal disease-specific brain network connections. We evaluated SGCN on the Alzheimer's Disease Neuroimaging Initiative database with multi-modal brain images and demonstrated that the ROI features learned by SGCN were effective for enhancing AD status identification. The identified abnormalities were significantly correlated with AD-related clinical symptoms. We further interpreted the identified brain dysfunctions at the level of large-scale neural systems and sex-related connectivity abnormalities in AD/MCI. The salient ROIs and the prominent brain connectivity abnormalities interpreted by SGCN are considerably important for developing novel biomarkers. These findings contribute to a better understanding of the network-based disorder via multi-modal diagnosis and offer the potential for precision diagnostics. The source code is available at https://github.com/Houliang-Zhou/SGCN.

PH-Responsive magnetic nanocarriers for chelator-free bimodal (MRI/SPECT-CT) image-guided chemo-hyperthermia therapy in human breast carcinoma.

Although chemotherapy with magnetic nanocarriers has witnessed significant advancement in the field of cancer treatment, multimodal diagnosis and combinatorial therapy using a single nanoplatform will have much better efficacy in achieving superior results. Herein, we constructed a smart theranostic system by combining pH-sensitive tartaric acid-stabilized Fe3O4 magnetic nanocarriers (TMNCs) with SPECT imaging and a chemotherapeutic agent for image-guided chemo-hyperthermia therapy. The carboxyl-enriched exteriors of TMNCs provided sites for the conjugation of a chemotherapeutic drug (doxorubicin hydrochloride, DOX) and radiolabeling (141Ce). The usage of 145.4 keV gamma rays made this platform an ideal choice for in vivo SPECT-CT imaging, showing the retention of the nanoformulation in the tumor site even after 28 days. Further, TMNCs showed a very high transverse relaxation rate (r2) of 171 mM-1 s-1, which is higher than that of clinically approved magnetic resonance imaging (MRI) contrast agents such as ferumoxtran (65 mM-1 s-1) and ferumoxides (120 mM-1 s-1). Further, the developed drug-loaded hybrid platform showed significantly higher cytotoxicity towards breast cancer cells, which was augmented by in vitro magnetic hyperthermia. Bright-field microscopy and cell cycle analysis suggested that cell death occurred through induction of G2-M arrest and subsequent apoptosis. These findings clearly suggest the potential of the developed hybrid nanoplatform for image-guided combination therapy.

Nano-technological advancements in multimodal diagnosis and treatment

&lt;p&gt;Nanomedicine, which is blazing new trails in modern healthcare, exploits the distinctive attributes of nanoparticles (NPs) to reimagine diagnosis and therapy. With imaging, this “magic ammunition” enhances the capability of MRI, a noninvasive, high-resolution technology that can pinpoint cancers with astonishing precision. Unlike traditional treatments, which muddle the distinction between diseased and healthy tissue, gold nanoparticles (AuNPs) directly target tumors, using increased permeability and retention (EPR) effects for crystal-clear, early-stage detection. While gadolinium (Gd (III)) T1 agents are the most widely used in clinical practice, iron oxide T2 agents have fallen out of favor due to failing to perform. Beyond imaging, NPs offer a “one-two punch” in the treatment of complicated disorders by enabling photothermal (PTT), photodynamic (PDT), and multimodal therapies (MMT), as well as nano drug delivery systems (NDDS). This nanotechnology-based technique personalizes medicines for diseases such as tuberculosis (TB) and cardiovascular disease (CVD), focusing on damaged tissues to improve therapeutic response. In tuberculosis, NPs circumvent medication resistance by delivering medicines directly to infected cells, thus avoiding biological barriers that impede standard treatments. Similarly, in CVD, NPs target arterial plaques to improve treatment accuracy while reducing systemic adverse effects. Furthermore, the genomic “blueprint” is evolving parallel to nanotechnology, with India's “Genome India” initiative mapping 10,000 genomes, lighting the path for genetic chips suited for specific disorders and moment tests. The convergence of nanotechnology and genomes is an arcade modification, bringing accuracy and personalization to the forefront of cancer and tuberculosis treatment.&lt;/p&gt;

A Dual-Branch Cross-Modality-Attention Network for Thyroid Nodule Diagnosis Based on Ultrasound Images and Contrast-Enhanced Ultrasound Videos.

Contrast-enhanced ultrasound (CEUS) has been extensively employed as an imaging modality in thyroid nodule diagnosis due to its capacity to visualise the distribution and circulation of micro-vessels in organs and lesions in a non-invasive manner. However, current CEUS-based thyroid nodule diagnosis methods suffered from: 1) the blurred spatial boundaries between nodules and other anatomies in CEUS videos, and 2) the insufficient representations of the local structural information of nodule tissues by the features extracted only from CEUS videos. In this paper, we propose a novel dual-branch network with a cross-modality-attention mechanism for thyroid nodule diagnosis by integrating the information from tow related modalities, i.e., CEUS videos and ultrasound image. The mechanism has two parts: US-attention-from-CEUS transformer (UAC-T) and CEUS-attention-from-US transformer (CAU-T). As such, this network imitates the manner of human radiologists by decomposing the diagnosis into two correlated tasks: 1) the spatio-temporal features extracted from CEUS are hierarchically embedded into the spatial features extracted from US with UAC-T for the nodule segmentation; 2) the US spatial features are used to guide the extraction of the CEUS spatio-temporal features with CAU-T for the nodule classification. The two tasks are intertwined in the dual-branch end-to-end network and optimized with the multi-task learning (MTL) strategy. The proposed method is evaluated on our collected thyroid US-CEUS dataset. Experimental results show that our method achieves the classification accuracy of 86.92%, specificity of 66.41%, and sensitivity of 97.01%, outperforming the state-of-the-art methods. As a general contribution in the field of multi-modality diagnosis of diseases, the proposed method has provided an effective way to combine static information with its related dynamic information, improving the quality of deep learning based diagnosis with an additional benefit of explainability.

MICDnet: Multimodal information processing networks for Crohn’s disease diagnosis

Crohn’s disease (CD) is a chronic inflammatory disease with increasing incidence worldwide and unclear etiology. Its clinical manifestations vary depending on location, extent, and severity of the lesions. In order to diagnose Crohn’s disease, medical professionals need to comprehensively analyze patients’ multimodal examination data, which includes medical imaging such as colonoscopy, pathological, and text information from clinical records. The processes of multimodal data analysis require collaboration among medical professionals from different departments, which wastes a lot of time and human resources. Therefore, a multimodal medical assisted diagnosis system for Crohn’s disease is particularly significant. Existing network frameworks find it hard to effectively capture multimodal patient data for diagnosis, and multimodal data for Crohn’s disease is currently lacking. In addition,a combination of data from patients with similar symptoms could serve as an effective reference for disease diagnosis. Thus, we propose a multimodal information diagnosis network (MICDnet) to learn CD feature representations by integrating colonoscopy, pathology images and clinical texts. Specifically, MICDnet first preprocesses each modality data, then uses encoders to extract image and text features separately. After that, multimodal feature fusion is performed. Finally, CD classification and diagnosis are conducted based on the fused features. Under the authorization, we build a dataset of 136 hospitalized inspectors, with colonoscopy images of seven areas, pathology images, and clinical record text for each individual. Training MICDnet on this dataset shows that multimodal diagnosis can improve the diagnostic accuracy of CD, and the diagnostic performance of MICDnet is superior to other models.

Parathyroid adenoma: multimodal diagnosis capabilities: A retrospective study

Background. Primary hyperparathyroidism is a common endocrinological disease caused mainly by parathyroid adenoma. The main treatment method is surgery (parathyroidectomy). Therefore, the exact determination of adenoma localization is crucial.&#x0D; Aim. To evaluate the current possibilities of multimodal diagnosis of parathyroid adenomas.&#x0D; Materials and methods. A retrospective analysis of 49 patients with primary hyperparathyroidism aged 24 to 82 (median 57.9 years) was performed. Modern radionuclide and hybrid technologies were used for topical diagnosis and metabolic assessment of parathyroid adenomas: scanning, single-photon emission computed tomography, single-photon emission computed tomography combined with computed tomography, positron emission tomography combined with computed tomography with 18F-deoxyglucose and 18F-choline. The diagnosis of primary hyperparathyroidism was confirmed by a biochemical blood test: the level of parathyroid hormone and ionized and total calcium.&#x0D; Results. The study included 43 (87.8%) females and 6 (12.2%) males. The female/male ratio was 7.2:1. Most cases (78.1%) were the hypercalcemic type of primary hyperparathyroidism, and the normocalcemic type was diagnosed in 21.9% of patients. The mean parathyroid hormone level was 145.43 pg/mL, exceeding the reference values by 2.2 times. Parathyroid hormone concentration in patients with primary hyperparathyroidism was 156.38 pg/mL, and mean ionized and total blood calcium levels were 1.43 and 3.04 mmol/L, respectively. The asymptomatic type occurred in 76.7% of patients. The symptomatic type of hyperparathyroidism had 23.3%, manifested with nephrolithiasis, pancreatitis, and bone lesions. Parathyroid adenomas were more often located in the left lobe (42.9%). In 77.6% of patients with primary hyperparathyroidism, solitary adenomas were detected. Ectopia of the parathyroid glands was detected in 16.3% of patients, with intrathyroidal location in the left lobe being the most common. Rare locations include the anterior and posterior mediastinum and the esophageal wall.&#x0D; Conclusion. Modern diagnostic multimodal options based on radionuclide and hybrid technologies are crucial in the personalized treatment of primary hyperparathyroidism.

Information Fusion and Data Augmentation for Risk-based Maintenance Optimization of Hydrogen Gas Pipelines

Demand for energy is increasing every year and hydrogen is being seen as a good alternative to conventional natural gas. The current focus is on the use of existing pipeline infrastructure for the transport of hydrogen gas, and it is necessary for us to ensure the safe and efficient operation of the pipeline infrastructure given the risks posed by hydrogen. Pipeline integrity management is critical for hydrogen transport and there are knowledge gaps for the impact of hydrogen on the pipeline integrity and operational considerations, thus hindering the pipeline operators from adopting hydrogen into their networks. To realize the concept of transporting hydrogen through existing pipeline systems, it is necessary to have reliable risk assessment and maintenance optimization frameworks in place. A Bayesian network methodology is proposed to fuse information from multiple sources obtained by multimodality diagnosis of pipe materials and Bayesian updating will be incorporated to reduce the uncertainty arising from different random variables. Risk assessment of the pipeline systems will be carried out based on the posterior distributions of the random variables. Given the predicted risk level, we then propose a risk-based maintenance optimization framework to minimize the maintenance costs while ensuring the safe operation of the pipeline systems.

Comprehensive learning and adaptive teaching: Distilling multi-modal knowledge for pathological glioma grading

The fusion of multi-modal data, e.g., pathology slides and genomic profiles, can provide complementary information and benefit glioma grading. However, genomic profiles are difficult to obtain due to the high costs and technical challenges, thus limiting the clinical applications of multi-modal diagnosis. In this work, we investigate the realistic problem where paired pathology-genomic data are available during training, while only pathology slides are accessible for inference. To solve this problem, a comprehensive learning and adaptive teaching framework is proposed to improve the performance of pathological grading models by transferring the privileged knowledge from the multi-modal teacher to the pathology student. For comprehensive learning of the multi-modal teacher, we propose a novel Saliency-Aware Masking (SA-Mask) strategy to explore richer disease-related features from both modalities by masking the most salient features. For adaptive teaching of the pathology student, we first devise a Local Topology Preserving and Discrepancy Eliminating Contrastive Distillation (TDC-Distill) module to align the feature distributions of the teacher and student models. Furthermore, considering the multi-modal teacher may include incorrect information, we propose a Gradient-guided Knowledge Refinement (GK-Refine) module that builds a knowledge bank and adaptively absorbs the reliable knowledge according to their agreement in the gradient space. Experiments on the TCGA GBM-LGG dataset show that our proposed distillation framework improves the pathological glioma grading and outperforms other KD methods. Notably, with the sole pathology slides, our method achieves comparable performance with existing multi-modal methods. The code is available at https://github.com/CUHK-AIM-Group/MultiModal-learning.

Multi-modal graph neural network for early diagnosis of Alzheimer's disease from sMRI and PET scans

In recent years, deep learning models have been applied to neuroimaging data for early diagnosis of Alzheimer's disease (AD). Structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) images provide structural and functional information about the brain, respectively. Combining these features leads to improved performance than using a single modality alone in building predictive models for AD diagnosis. However, current multi-modal approaches in deep learning, based on sMRI and PET, are mostly limited to convolutional neural networks, which do not facilitate integration of both image and phenotypic information of subjects. We propose to use graph neural networks (GNN) that are designed to deal with problems in non-Euclidean domains. In this study, we demonstrate how brain networks are created from sMRI or PET images and can be used in a population graph framework that combines phenotypic information with imaging features of the brain networks. Then, we present a multi-modal GNN framework where each modality has its own branch of GNN and a technique that combines the multi-modal data at both the level of node vectors and adjacency matrices. Finally, we perform late fusion to combine the preliminary decisions made in each branch and produce a final prediction. As multi-modality data becomes available, multi-source and multi-modal is the trend of AD diagnosis. We conducted explorative experiments based on multi-modal imaging data combined with non-imaging phenotypic information for AD diagnosis and analyzed the impact of phenotypic information on diagnostic performance. Results from experiments demonstrated that our proposed multi-modal approach improves performance for AD diagnosis. Our study also provides technical reference and support the need for multivariate multi-modal diagnosis methods.

Gradient modulated contrastive distillation of low-rank multi-modal knowledge for disease diagnosis.

The fusion of multi-modal data, e.g., medical images and genomic profiles, can provide complementary information and further benefit disease diagnosis. However, multi-modal disease diagnosis confronts two challenges: (1) how to produce discriminative multi-modal representations by exploiting complementary information while avoiding noisy features from different modalities. (2) how to obtain an accurate diagnosis when only a single modality is available in real clinical scenarios. To tackle these two issues, we present a two-stage disease diagnostic framework. In the first multi-modal learning stage, we propose a novel Momentum-enriched Multi-Modal Low-Rank (M3LR) constraint to explore the high-order correlations and complementary information among different modalities, thus yielding more accurate multi-modal diagnosis. In the second stage, the privileged knowledge of the multi-modal teacher is transferred to the unimodal student via our proposed Discrepancy Supervised Contrastive Distillation (DSCD) and Gradient-guided Knowledge Modulation (GKM) modules, which benefit the unimodal-based diagnosis. We have validated our approach on two tasks: (i) glioma grading based on pathology slides and genomic data, and (ii) skin lesion classification based on dermoscopy and clinical images. Experimental results on both tasks demonstrate that our proposed method consistently outperforms existing approaches in both multi-modal and unimodal diagnoses.

Advances in multi-modal non-invasive imaging techniques in the diagnosis and treatment of polypoidal choroidal vasculopathy.

Polypoidal choroidal vasculopathy (PCV) is a disease characterized by subretinal pigment epithelium (RPE) orange-red polypoidal lesions and abnormal branching neovascular networks (BNNs). In recent years, various non-invasive imaging technologies have rapidly developed, especially the emergence of optical coherence tomography angiography (OCTA), multi-spectral imaging, and other technologies, which enable the observation of more features of PCV. In addition, these technologies are faster and less invasive compared to indocyanine green angiography (ICGA). Multi-modal imaging, which combined multiple imaging techniques, provides important references for the diagnosis and treatment of PCV with the assistance of regression models, deep learning, and other algorithms. In this study, we reviewed the non-invasive imaging techniques, multi-modal imaging diagnosis, and multi-scene therapeutic applications of PCV, with the aim of providing a reference for non-invasive multi-modal diagnosis and treatment of PCV.