Non-imaging Data Research Articles

Identification of mild cognitive impairment using multimodal 3D imaging data and graph convolutional networks

Abstract Objective. Mild cognitive impairment (MCI) is a precursor stage of dementia characterized by mild cognitive decline in one or more cognitive domains, without meeting the criteria for dementia. MCI is considered a prodromal form of Alzheimer's disease (AD). Early identification of MCI is crucial for both intervention and prevention of AD. To accurately identify MCI, a novel multimodal 3D imaging data integration graph convolutional network model is designed in this paper. Approach. The proposed model utilizes 3D-VGGNet to extract three-dimensional features from multimodal imaging data (such as sMRI and FDG-PET), which are then fused into feature vectors as the node features of a population graph. Non-imaging features of participants are combined with the multimodal imaging data to construct a population sparse graph. Additionally, in order to optimize the connectivity of the graph, we employed the Pairwise Attribute Estimation (PAE) method to compute the edge weights based on non-imaging data, thereby enhancing the effectiveness of the graph structure. Subsequently, a population-based graph convolutional network (GCN) integrates the structural and functional features of different modal images into the features of each participant for MCI classification. Main results. Experiments on the ADNI demonstrated accuracies of 98.57%, 96.03%, and 96.83% for the NC-EMCI, NC-LMCI, and EMCI-LMCI classification tasks, respectively. The AUC, specificity, sensitivity, and F1-score are also superior to state-of-the-art models, demonstrating the effectiveness of the proposed model. Furthermore, the proposed model is applied to the ABIDE dataset for autism diagnosis, achieving an accuracy of 91.43% and outperforming the state-of-the-art models, indicating excellent generalization capabilities of the proposed model. Significance. This study demonstrate the proposed model’s ability to integrate multimodal imaging data and its excellent ability to recognize MCI. This will help achieve early warning for AD and intelligent diagnosis of other brain neurodegenerative diseases.

Detection and risk stratification of cardiac amyloidosis patients by integration of imaging and non-imaging data using a machine learning approach

Abstract Background With the advent of amyloid-targeting therapies, early and reliable diagnosis as well as precise risk estimation of cardiac amyloidosis (CA) have become of substantial importance. While the current diagnostic approach relies on difficult-to-standardize, visual interpretation of 99mTc-scintigraphy, risk assessment is largely based on a combination of blood and imaging parameters. Purpose Aim of this study is to assess the possibility of machine learning-based integration of scintigraphy, echocardiography, and non-imaging parameters such as blood parameters and comorbidities for the detection and risk stratification of patients with CA. Methods This study included all patients who underwent 99mTc-scintigraphy at a university affiliated tertiary care centre between 2010 and 2023. Two machine learning models were developed, and the relative predictive importance of their parameters were assessed (Figure 1): First, a model to detect patients with CA-suggestive uptake (Perugini grade 2/3) on 99mTc-scintigraphy scans; Second, a model to assess the risk of patients with CA-suggestive uptake for future heart failure hospitalization (HFH). A total of 58 features were extracted from electronic health records including blood, echocardiographic, demographic parameters and comorbidities. Scintigraphy imaging features were extracted from the raw imaging data using a deep learning approach. For the time to HFH prediction, a random survival forest machine learning model was employed. Results Overall, 12 380 consecutively enrolled patients were included, 279 (2.3%) of which were affected by CA-suggestive uptake. The machine learning model showed higher accuracy for the detection of CA (AUC 0.96 [95% CI 0.95-0.97], sensitivity 0.78 [95% CI 0.77-0.80] and specificity 0.99 [95% CI 0.99-0.99]. In the outcome analysis, the machine model showed good accuracy in predicting future HFH (C-index 0.71 [95% CI 0.68-0.75]). Parameter importance for the time to event analysis revealed right ventricular diameter, previous diagnosis of chronic heart failure and creatine kinase as the three most important predictors (Figure 2). Conclusions Detection of patients with cardiac amyloidosis and their risk estimation for heart failure can be aided using machine learning-based integration of imaging and non-imaging data. Our findings may guide novel approaches for assessing disease progression and treatment response.

Comparison of deep-learning multimodality data fusion strategies in mandibular osteoradionecrosis NTCP modelling using clinical variables and radiation dose distribution volumes

Objective.Normal tissue complication probability (NTCP) modelling is rapidly embracing deep learning (DL) methods, acknowledging the importance of spatial dose information. Finding effective ways to combine information from radiation dose distribution maps (dosiomics) and clinical data involves technical challenges and requires domain knowledge. We propose different multi-modality data fusion strategies to facilitate future DL-based NTCP studies.Approach.Early, joint and late DL multi-modality fusion strategies were compared using clinical and mandibular radiation dose distribution volumes. These were contrasted with single-modality models: a random forest trained on non-image data (clinical, demographic and dose-volume metrics) and a 3D DenseNet-40 trained on image data (mandibular dose distribution maps). The study involved a matched cohort of 92 osteoradionecrosis cases and 92 controls from a single institution.Main results.The late fusion model exhibited superior discrimination and calibration performance, while the join fusion achieved a more balanced distribution of the predicted probabilities. Discrimination performance did not significantly differ between strategies. Late fusion, though less technically complex, lacks crucial inter-modality interactions for NTCP modelling. In contrast, joint fusion, despite its complexity, resulted in a single network training process which included intra- and inter-modality interactions in its model parameter optimisation.Significance.This study is a pioneering effort in comparing different strategies for including image data into DL-based NTCP models in combination with lower dimensional data such as clinical variables. The discrimination performance of such multi-modality NTCP models and the choice of fusion strategy will depend on the distribution and quality of both types of data. Multiple data fusion strategies should be compared and reported in multi-modality NTCP modelling using DL.

Combining Computational Fluid Dynamics, Structural Analysis, and Machine Learning to Predict Cerebrovascular Events: A Mild ML Approach

Background/Objectives: Cerebrovascular events, such as strokes, are often preceded by the rupture of atherosclerotic plaques in the carotid arteries. This work introduces a novel approach to predict the occurrence of such events by integrating computational fluid dynamics (CFD), structural analysis, and machine learning (ML) techniques. The objective is to develop a predictive model that combines both imaging and non-imaging data to assess the risk of carotid atherosclerosis and subsequent cerebrovascular events, ultimately improving clinical decision-making. Methods: A multidisciplinary approach was employed, utilizing 3D reconstruction techniques and blood-flow simulations to extract key plaque characteristics. These were combined with patient-specific clinical data for risk evaluation. The study involved 134 asymptomatic individuals diagnosed with carotid artery disease. Data imbalance was addressed using two distinct approaches, with the optimal method chosen for training a Gradient Boosting Tree (GBT) classifier. The model’s performance was evaluated in terms of accuracy, sensitivity, specificity, and ROC AUC. Results: The best-performing GBT model achieved a balanced accuracy of 88%, with a ROC AUC of 0.92, a sensitivity of 0.88, and a specificity of 0.91. This demonstrates the model’s high predictive power in identifying patients at risk for cerebrovascular events. Conclusions: The proposed method effectively combines CFD, structural analysis, and ML to predict cerebrovascular event risk in patients with carotid artery disease. By providing clinicians with a tool for better risk assessment, this approach has the potential to significantly enhance clinical decision-making and patient outcomes.

Multimodal Convolutional Neural Network-Based Algorithm for Real-Time Detection and Differentiation of Malignant and Inflammatory Biliary Strictures in Cholangioscopy: a Proof of Concept Study with Video

Deep learning algorithms gained attention for detection (CADe) of biliary tract cancer (BTC) in digital single-operator cholangioscopy (dSOC). We developed a multimodal convolutional neural network (CNN) for detection (CADe) characterization and discriminating (CADx) between malignant, inflammatory and normal biliary tissue in raw dSOC videos. In addition, clinical metadata was included in the CNN algorithm to overcome limitations of image-only models. Based on dSOC videos and images of 111 patients (total of 15,158 still frames), we developed and validated a real-time CNN-based algorithm for CADe and CADx. We established an image-only model and metadata injection approach. In addition, we validated frame-wise and case-based predictions on complete dSOC video sequences. Model embeddings were visualized and class-activation maps highlighted relevant image regions. The concatenation-based CADx approach achieved a per-frame AUC of 0.871, sensitivity of 0.809 (95% CI: [0.784-0.832]), specificity of 0.773 [0.761-0.785], PPV of 0.450 [0.423-0.467], and NPV of 0.946 [0.940-0.954] with respect to malignancy on 5,715 test frames from complete videos of 20 patients. For case-based diagnosis using average prediction scores, six out of eight malignant cases and all twelve benign cases were identified correctly. Our algorithm distinguishes malignant and inflammatory bile duct lesions in dSOC videos, indicating the potential of CNN-based diagnostic support systems for both, CADe and CADx. The integration of non-image data can improve CNN based support systems, targeting current challenges in the assessment of biliary strictures.

Deep learning-based authentication for insider threat detection in critical infrastructure

In today’s cyber environment, threats such as data breaches, cyberattacks, and unauthorized access threaten national security, critical infrastructure, and financial stability. This research addresses the challenging task of protecting critical infrastructure from insider threats because of the high level of trust and access these individuals typically receive. Insiders may obtain a system administrator’s password through close observation or by deploying software to gather the information. To solve this issue, an innovative artificial intelligence-based methodology is proposed to identify a user by their password’s keystroke dynamics. This paper also introduces a new Gabor Filter Matrix Transformation method to transform numerical values into images by revealing the behavioral pattern of password typing. A siamese neural network (SNN) with the branches of convolutional neural networks is utilized for image comparison, aiming to detect unauthorized attempts to access critical infrastructure systems. The network analyzes the unique features of a user’s password timestamps transformed into images and compares them with previously submitted user passwords. The obtained results indicate that transforming the numerical values of keystroke dynamics into images and training an SNN leads to a lower equal error rate (EER) and higher user authentication accuracy than those previously reported in other studies. The methodology is validated on publicly available keystroke dynamics collections, the CMU and GREYC-NISLAB datasets, which collectively comprise over 30,000 password samples. It achieves the lowest EER value of 0.04545 compared to state-of-the-art methods for transforming non-image data into images. The paper concludes with a discussion of findings and potential future directions.

FC-HGNN: A heterogeneous graph neural network based on brain functional connectivity for mental disorder identification

Rapid and accurate diagnosis of mental disorders has long been an essential challenge in clinical medicine. Due to the advantage in addressing non-Euclidean structures, graph neural networks have been increasingly used to study brain networks. Among the existing methods, the population graph models have achieved high predictive accuracy by considering intersubject relationships but weak interpretability limits its clinical applicability. The individual graph approach models functional brain networks and identifies abnormal brain regions that cause diseases but has poor accuracy. To address these issues, we propose a heterogeneous graph neural network based on brain functional connectivity (FC-HGNN), which is an end-to-end model with a two-stage process. In the first phase, the brain connectomic graph is used to extract individual brain features. An integrated intrahemispheric and interhemispheric convolutional graph layer is used to learn brain region features, and a local–global dual-channel pooling layer is used to identify biomarkers. In the second stage, a heterogeneous population graph is constructed based on sex and the fusion of imaging and non-imaging data from subjects. The feature embeddings of same-sex and opposite-sex neighbours are learned separately according to a hierarchical feature aggregation approach. Subsequently, they are adaptively fused to generate the final node embedding, which is then utilized for obtaining classification predictions. The cross-validation and transduction learning results show that FC-HGNN achieves state-of-the-art performance in classification prediction experiments using two public datasets. Moreover, FC-HGNN identifies crucial biomarker regions relevant for disease classification, aligning with existing studies and exhibiting outstanding predictive performance on actual clinical data. The code is available at https://github.com/Guuuu11/FC-HGNN_Pytorch.

A transformer-based unified multimodal framework for Alzheimer's disease assessment

In Alzheimer's disease (AD) assessment, traditional deep learning approaches have often employed separate methodologies to handle the diverse modalities of input data. Recognizing the critical need for a cohesive and interconnected analytical framework, we propose the AD-Transformer, a novel transformer-based unified deep learning model. This innovative framework seamlessly integrates structural magnetic resonance imaging (sMRI), clinical, and genetic data from the extensive Alzheimer's Disease Neuroimaging Initiative (ADNI) database, encompassing 1651 subjects. By employing a Patch-CNN block, the AD-Transformer efficiently transforms image data into image tokens, while a linear projection layer adeptly converts non-image data into corresponding tokens. As the core, a transformer block learns comprehensive representations of the input data, capturing the intricate interplay between modalities. The AD-Transformer sets a new benchmark in AD diagnosis and Mild Cognitive Impairment (MCI) conversion prediction, achieving remarkable average area under curve (AUC) values of 0.993 and 0.845, respectively, surpassing those of traditional image-only models and non-unified multimodal models. Our experimental results confirmed the potential of the AD-Transformer as a potent tool in AD diagnosis and MCI conversion prediction. By providing a unified framework that jointly learns holistic representations of both image and non-image data, the AD-Transformer paves the way for more effective and precise clinical assessments, offering a clinically adaptable strategy for leveraging diverse data modalities in the battle against AD.

Computer-aided prognosis of tuberculous meningitis combining imaging and non-imaging data

Tuberculous meningitis (TBM) is the most lethal form of tuberculosis. Clinical features, such as coma, can predict death, but they are insufficient for the accurate prognosis of other outcomes, especially when impacted by co-morbidities such as HIV infection. Brain magnetic resonance imaging (MRI) characterises the extent and severity of disease and may enable more accurate prediction of complications and poor outcomes. We analysed clinical and brain MRI data from a prospective longitudinal study of 216 adults with TBM; 73 (34%) were HIV-positive, a factor highly correlated with mortality. We implemented an end-to-end framework to model clinical and imaging features to predict disease progression. Our model used state-of-the-art machine learning models for automatic imaging feature encoding, and time-series models for forecasting, to predict TBM progression. The proposed approach is designed to be robust to missing data via a novel tailored model optimisation framework. Our model achieved a 60% balanced accuracy in predicting the prognosis of TBM patients over the six different classes. HIV status did not alter the performance of the models. Furthermore, our approach identified brain morphological lesions caused by TBM in both HIV and non-HIV-infected, associating lesions to the disease staging with an overall accuracy of 96%. These results suggest that the lesions caused by TBM are analogous in both populations, regardless of the severity of the disease. Lastly, our models correctly identified changes in disease symptomatology and severity in 80% of the cases. Our approach is the first attempt at predicting the prognosis of TBM by combining imaging and clinical data, via a machine learning model. The approach has the potential to accurately predict disease progression and enable timely clinical intervention.

Multimodal Imaging Approach for Tumor Treatment Response Evaluation in the Era of Immunotherapy.

Immunotherapy is likely the most remarkable advancement in lung cancer treatment during the past decade. Although immunotherapy provides substantial benefits, their therapeutic responses differ from those of conventional chemotherapy and targeted therapy, and some patients present unique immunotherapy response patterns that cannot be judged under the current measurement standards. Therefore, the response monitoring of immunotherapy can be challenging, such as the differentiation between real response and pseudo-response. This review outlines the various tumor response patterns to immunotherapy and discusses methods for quantifying computed tomography (CT) and 18F-fluorodeoxyglucose positron emission tomography (PET) in the field of lung cancer. Emerging technologies in magnetic resonance imaging (MRI) and non-FDG PET tracers are also explored. With immunotherapy responses, the role for imaging is essential in both anatomical radiological responses (CT/MRI) and molecular changes (PET imaging). Multiple aspects must be considered when assessing treatment responses using CT and PET. Finally, we introduce multimodal approaches that integrate imaging and nonimaging data, and we discuss future directions for the assessment and prediction of lung cancer responses to immunotherapy.

Towards semi-supervised multi-modal rectal cancer segmentation: A large-scale dataset and a multi-teacher uncertainty-aware network

Rectal cancer is one of the most common malignant tumors of the digestive tract. Recently, deep learning has attracted significant attention in computer-aided cancerous region segmentation, offering better adjuvant treatment options for colorectal cancer patients. However, the size of existing rectal cancer datasets is insufficient, which severely limits the training of data-hungry segmentation models based on deep learning. Moreover, the existing datasets only contain image modality, while non-image modality information such as blood test reports is also important in practical clinical diagnosis. Hence, in this paper, we first collect the largest public dataset named RC4 for rectal cancer segmentation by far. RC4 contains 51,097 MRI slices from 657 patients of colorectal cancer, including MRI data and non-image data. We further propose a multi-modal multi-teacher uncertainty-aware network (M-MTUNet) for semi-supervised rectal cancer segmentation. Inspired by the human learning process, multi-teacher distillation is designed to transfer a variety of knowledge to a single student, which provides the student model with richer information. Besides, we propose a multi-modal uncertainty-aware consistency loss, which facilitates the knowledge transfer process from the two teacher networks to the student network and reduces the learning uncertainty within the student network. Extensive experiments demonstrate that M-MTUNet outperforms the current widely-used fully- and semi-supervised segmentation models.

Self-Report Tool for Identification of Individuals With Coronary Atherosclerosis: The Swedish CardioPulmonary BioImage Study.

Coronary atherosclerosis detected by imaging is a marker of elevated cardiovascular risk. However, imaging involves large resources and exposure to radiation. The aim was, therefore, to test whether nonimaging data, specifically data that can be self-reported, could be used to identify individuals with moderate to severe coronary atherosclerosis. We used data from the population-based SCAPIS (Swedish CardioPulmonary BioImage Study) in individuals with coronary computed tomography angiography (n=25 182) and coronary artery calcification score (n=28 701), aged 50 to 64 years without previous ischemic heart disease. We developed a risk prediction tool using variables that could be assessed from home (self-report tool). For comparison, we also developed a tool using variables from laboratory tests, physical examinations, and self-report (clinical tool) and evaluated both models using receiver operating characteristic curve analysis, external validation, and benchmarked against factors in the pooled cohort equation. The self-report tool (n=14 variables) and the clinical tool (n=23 variables) showed high-to-excellent discriminative ability to identify a segment involvement score ≥4 (area under the curve 0.79 and 0.80, respectively) and significantly better than the pooled cohort equation (area under the curve 0.76, P<0.001). The tools showed a larger net benefit in clinical decision-making at relevant threshold probabilities. The self-report tool identified 65% of all individuals with a segment involvement score ≥4 in the top 30% of the highest-risk individuals. Tools developed for coronary artery calcification score ≥100 performed similarly. We have developed a self-report tool that effectively identifies individuals with moderate to severe coronary atherosclerosis. The self-report tool may serve as prescreening tool toward a cost-effective computed tomography-based screening program for high-risk individuals.

O-026 Development of deep learning-based model for differential diagnosis of ovarian neoplasm using pelvic ultrasonography

Abstract Study question Can the deep learning-based diagnostic algorithms of pelvic ultrasonography be a useful clinical tool for evaluating the malignancy risk of ovarian neoplasms? Summary answer AI-based deep learning diagnostic algorithm for pelvic ultrasonography is a potential clinical aid, as it exhibits excellent performance in the differential diagnosis of ovarian neoplasm. What is known already Transvaginal ultrasonography serves as a primary diagnostic tool for adnexal neoplasms due to its widespread availability and user-friendly nature within gynecologic clinics. There have been attempts to employ artificial intelligence algorithms for identifying ovarian malignancy through sonographic evaluations, yet the existing literature is limited, and the optimal processing method for constructing ultrasound-based AI images is not known. This study aims to address this gap by constructing a comprehensive AI-based model and comparing the various input methods for the development of the ultrasound-based diagnostic model. Study design, size, duration A retrospective cohort study was conducted on the clinical records of 1,171 patients diagnosed with ovarian neoplasms through pelvic ultrasonography from May 2002 to August 2021 at our institution. Inclusion criteria mandated that patients undergo pelvic ultrasonographic examination in our department, followed by surgical excision, with a requisite availability of pathologic diagnosis for the identified ovarian neoplasm. Participants/materials, setting, methods The dataset was divided into training, validation, and testing subsets with a distribution ratio of 8:1:1 (2373, 297, 298 images). Pre-processing of ultrasound input was done after removing text annotations and unnecessary non-image data. Deep learning methods with different architectures were utilized, including Swin Transformer, ConvNeXt, MobileNet, ResNet18, EfficientNet V2, and VGG16. Main results and the role of chance This performance analysis encompasses a comparison of various deep learning architectures applied to classification tasks within ultrasound imaging data. The metrics evaluated include Sensitivity, Specificity, Positive Predictive Value (PPV, also known as Precision), Negative Predictive Value (NPV), F1 Score, Accuracy, and the Area Under the Receiver Operating Characteristic curve (AUROC). For Sensitivity, the sequence MobileNet, ResNet-18, EfficientNet-V2, VGG-16, Swin Transformer, and ConvNext is 0.830, 0.851, 0.879, 0.835, 0.883, 0.877; Specificity values follow as 0.848, 0.840, 0.877, 0.866, 0.893, 0.906; PPV (Precision) is reported as 0.848, 0.840, 0.877, 0.866, 0.893, 0.906; NPV metrics are 0.830, 0.851, 0.877, 0.835, 0.879, 0.883; F1 Score progression is 0.838, 0.845, 0.878, 0.848, 0.888, 0.890; Accuracy rates are 0.865, 0.865, 0.895, 0.875, 0.906, 0.909; and finally, AUROC scores span 0.920, 0.920, 0.930, 0.920, 0.930, 0.960. These metrics collectively highlight the nuanced efficacy of each model in ultrasound image classification, with ConvNext demonstrating exceptional performance across the board. Limitations, reasons for caution An external validation of our model using datasets outside our institution has not been carried out yet. Also, the clinical validity of our model in real-life practice has not been investigated. These issues will be addressed subsequently through our continued investigation. Wider implications of the findings A deep-learning model utilizing ConvNeXt yielded excellent diagnostic performance in the differential diagnosis of ovarian neoplasms. The integration of AI-based deep learning models in clinical settings has the potential to significantly enhance the diagnostic accuracy of ultrasonography for ovarian neoplasms. Trial registration number not applicable

PARKINSON'S DISEASE CLASSIFICATION USING CONTRASTIVE GRAPH CROSS-VIEW LEARNING WITH MULTIMODAL FUSION OF SPECT IMAGES AND CLINICAL FEATURES.

Parkinson's Disease (PD) affects millions globally, impacting movement. Prior research utilized deep learning for PD prediction, primarily focusing on medical images, neglecting the data's underlying manifold structure. This work proposes a multimodal approach encompassing both image and non-image features, leveraging contrastive cross-view graph fusion for PD classification. We introduce a novel multimodal co-attention module, integrating embeddings from separate graph views derived from low-dimensional representations of images and clinical features. This enables more robust and structured feature extraction for improved multi-view data analysis. Additionally, a simplified contrastive loss-based fusion method is devised to enhance cross-view fusion learning. Our graph-view multimodal approach achieves an accuracy of 91% and an area under the receiver operating characteristic curve (AUC) of 92.8% in five-fold cross-validation. It also demonstrates superior predictive capabilities on non-image data compared to solely machine learning-based methods.

Non-destructive classification of sturgeon stress using cross-modal data fusion and multi-input deep learning models

In the aquaculture phase, ensuring the safe transportation of sturgeon is crucial. The stress levels experienced during transit directly impact the fish quality and the economic returns for farmers. To address this, distributors enlist fishery farming experts to evaluate sturgeon stress. Our investigation identified three critical parameters for grading: sturgeon physiological, environmental, and visual characteristics. This study aims to develop a Cross-Modal Stress Classification Network (CM-SCN) model. It integrates information from three sensing systems to assess sturgeon stress levels. The model is built upon the architectures of AlexNet and ANN, skillfully combining both image and non-image data sources. This integration enables the model to effectively categorize sturgeon stress into four classes: A, B, C, D (representing mild, minor, moderate, and severe stress levels). The results demonstrated the model's high performance with an accuracy of 88.96%, precision of 90.06%, recall of 89.43%, and an F1 score of 89.49%. Notably, the CM-SCN model surpassed both the unimodal visual stress model and the bimodal physiological-environmental stress model. This study introduces an efficient and dependable method for monitoring sturgeon health, offering promising advancements in the field.

Self-Supervised Random Forest on Transformed Distribution for Anomaly Detection.

Anomaly detection, the task of differentiating abnormal data points from normal ones, presents a significant challenge in the realm of machine learning. Numerous strategies have been proposed to tackle this task, with classification-based methods, specifically those utilizing a self-supervised approach via random affine transformations (RATs), demonstrating remarkable performance on both image and non-image data. However, these methods encounter a notable bottleneck, the overlap of constructed labeled datasets across categories, which hampers the subsequent classifiers' ability to detect anomalies. Consequently, the creation of an effective data distribution becomes the pivotal factor for success. In this article, we introduce a model called "self-supervised forest (sForest)", which leverages the random Fourier transform (RFT) and random orthogonal rotations to craft a controlled data distribution. Our model utilizes the RFT to map input data into a new feature space. With this transformed data, we create a self-labeled training dataset using random orthogonal rotations. We theoretically prove that the data distribution formulated by our methodology is more stable compared to one derived from RATs. We then use the self-labeled dataset in a random forest (RF) classifier to distinguish between normal and anomalous data points. Comprehensive experiments conducted on both real and artificial datasets illustrate that sForest outperforms other anomaly detection methods, including distance-based, kernel-based, forest-based, and network-based benchmarks.

Enhancing the Accuracy of Lymph-Node-Metastasis Prediction in Gynecologic Malignancies Using Multimodal Federated Learning: Integrating CT, MRI, and PET/CT.

Gynecological malignancies, particularly lymph node metastasis, have presented a diagnostic challenge, even with traditional imaging techniques such as CT, MRI, and PET/CT. This study was conceived to explore and, subsequently, to bridge this diagnostic gap through a more holistic and innovative approach. By developing a comprehensive framework that integrates both non-image data and detailed MRI image analyses, this study harnessed the capabilities of a multimodal federated-learning model. Employing a composite neural network within a federated-learning environment, this study adeptly merged diverse data sources to enhance prediction accuracy. This was further complemented by a sophisticated deep convolutional neural network with an enhanced U-NET architecture for meticulous MRI image processing. Traditional imaging yielded sensitivities ranging from 32.63% to 57.69%. In contrast, the federated-learning model, without incorporating image data, achieved an impressive sensitivity of approximately 0.9231, which soared to 0.9412 with the integration of MRI data. Such advancements underscore the significant potential of this approach, suggesting that federated learning, especially when combined with MRI assessment data, can revolutionize lymph-node-metastasis detection in gynecological malignancies. This paves the way for more precise patient care, potentially transforming the current diagnostic paradigm and resulting in improved patient outcomes.

Multimodal Deep Learning for Integrating Chest Radiographs and Clinical Parameters: A Case for Transformers.

Background Clinicians consider both imaging and nonimaging data when diagnosing diseases; however, current machine learning approaches primarily consider data from a single modality. Purpose To develop a neural network architecture capable of integrating multimodal patient data and compare its performance to models incorporating a single modality for diagnosing up to 25 pathologic conditions. Materials and Methods In this retrospective study, imaging and nonimaging patient data were extracted from the Medical Information Mart for Intensive Care (MIMIC) database and an internal database comprised of chest radiographs and clinical parameters inpatients in the intensive care unit (ICU) (January 2008 to December 2020). The MIMIC and internal data sets were each split into training (n = 33 893, n = 28 809), validation (n = 740, n = 7203), and test (n = 1909, n = 9004) sets. A novel transformer-based neural network architecture was trained to diagnose up to 25 conditions using nonimaging data alone, imaging data alone, or multimodal data. Diagnostic performance was assessed using area under the receiver operating characteristic curve (AUC) analysis. Results The MIMIC and internal data sets included 36 542 patients (mean age, 63 years ± 17 [SD]; 20 567 male patients) and 45 016 patients (mean age, 66 years ± 16; 27 577 male patients), respectively. The multimodal model showed improved diagnostic performance for all pathologic conditions. For the MIMIC data set, the mean AUC was 0.77 (95% CI: 0.77, 0.78) when both chest radiographs and clinical parameters were used, compared with 0.70 (95% CI: 0.69, 0.71; P < .001) for only chest radiographs and 0.72 (95% CI: 0.72, 0.73; P < .001) for only clinical parameters. These findings were confirmed on the internal data set. Conclusion A model trained on imaging and nonimaging data outperformed models trained on only one type of data for diagnosing multiple diseases in patients in an ICU setting. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Kitamura and Topol in this issue.

Interpretability of a Deep Learning-Based Prediction Model for Mandibular Osteoradionecrosis.

The development of radiation-induced toxicities is a multifactorial process. Existing DVH-based prediction models use traditional multivariate analysis to combine all the potential risk factors. Recently, deep learning (DL) has been proposed for predicting mandibular osteoradionecrosis (ORN) directly from 3D dose distribution maps [1]. However, with this approach, incorporating non-imaging data such as potential risk factors presents challenges. We investigate the use of DL-based multimodality fusion for the purpose of radiation-induced ORN toxicity prediction. This study explores early and late fusion strategies for combining 3D radiation dose distribution maps and clinical and demographic variables in the prediction of mandibular ORN incidence in head and neck cancer patients treated with radiotherapy. The results are compared to single-modality predictions with a random forest (RF) trained only on clinical variables and a 3D DenseNet40 trained on dose maps alone. We investigate two different fusion approaches. In the first, the image features extracted from the radiation dose maps using a 3D DenseNet40 were concatenated with the clinical variables into one single vector using a type II early fusion strategy. The combined feature vector was input into a fully connected layer for classification of ORN vs. controls. A final softmax activation layer was added to obtain the class predicted probabilities. The second approach used a late fusion strategy, in which the outputs from the 3D DenseNet40 and the RF model were combined by averaging the predicted classification probabilities for each of the two classes (ORN and no ORN) to obtain the final class decision on a case-by-case basis. The AUROC values obtained for the late and early fusion models and the single-modality 3D DenseNet40 and RF models were 0.70, 0.68, 0.69 and 0.60, respectively. The highest AUC ROC was observed with the late fusion approach, which was statistically significantly different to that of the RF single-modality model with a significance level of 0.05. However, after Bonferroni correction (Altman 1999) for multiple comparisons was applied, resulting in a corrected significance level of 0.05/6 = 0.008 for each comparison, no statistically significant difference was observed between any of the models' AUROC values. This is most likely due to the lack of discriminative contribution observed from clinical variables, which on their own resulted in a poorly predictive RF model. To our knowledge, no previous work has been published on the use of multimodal fusion DL methods to combine dose distribution maps and clinical variables in the prediction of mandibular ORN. Although non-conclusive results were obtained, this study demonstrates the potential of DL in the prediction of the multifactorial side effects resulting from radiotherapy treatments. [1] Humbert-Vidan L et al. Prediction of Mandibular ORN Incidence from 3D Radiation Dose Distribution Maps Using DL (2022).

Evaluation of OMOP CDM, i2b2 and ICGC ARGO for supporting data harmonization in a breast cancer use case of a multicentric European AI project

ObjectiveObservational research in cancer poses great challenges regarding adequate data sharing and consolidation based on a homogeneous data semantic base. Common Data Models (CDMs) can help consolidate health data repositories from different institutions minimizing loss of meaning by organizing data into a standard structure. This study aims to evaluate the performance of the Observational Medical Outcomes Partnership (OMOP) CDM, Informatics for Integrating Biology & the Bedside (i2b2) and International Cancer Genome Consortium, Accelerating Research in Genomic Oncology (ICGC ARGO) for representing non-imaging data in a breast cancer use case of EuCanImage. MethodsWe used ontologies to represent metamodels of OMOP, i2b2, and ICGC ARGO and variables used in a cancer use case of a European AI project. We selected four evaluation criteria for the CDMs adapted from previous research: content coverage, simplicity, integration, implementability. Resultsi2b2 and OMOP exhibited higher element completeness (100% each) than ICGC ARGO (58.1%), while the three achieved 100% domain completeness. ICGC ARGO normalizes only one of our variables with a standard terminology, while i2b2 and OMOP use standardized vocabularies for all of them. In terms of simplicity, ICGC ARGO and i2b2 proved to be simpler both in terms of ontological model (276 and 175 elements, respectively) and in the queries (7 and 20 lines of code, respectively), while OMOP required a much more complex ontological model (615 elements) and queries similar to those of i2b2 (20 lines). Regarding implementability, OMOP had the highest number of mentions in articles in PubMed (130) and Google Scholar (1,810), ICGC ARGO had the highest number of updates to the CDM since 2020 (4), and i2b2 is the model with more tools specifically developed for the CDM (26). ConclusionICGC ARGO proved to be rigid and very limited in the representation of oncologic concepts, while i2b2 and OMOP showed a very good performance. i2b2′s lack of a common dictionary hinders its scalability, requiring sites that will share data to explicitly define a conceptual framework, and suggesting that OMOP and its Oncology extension could be the more suitable choice. Future research employing these CDMs with actual datasets is needed.