Non-imaging Data Research Articles

Predicting oxygen requirements in patients with coronavirus disease 2019 using an artificial intelligence-clinician model based on local non-image data.

When facing unprecedented emergencies such as the coronavirus disease 2019 (COVID-19) pandemic, a predictive artificial intelligence (AI) model with real-time customized designs can be helpful for clinical decision-making support in constantly changing environments. We created models and compared the performance of AI in collaboration with a clinician and that of AI alone to predict the need for supplemental oxygen based on local, non-image data of patients with COVID-19. We enrolled 30 patients with COVID-19 who were aged >60 years on admission and not treated with oxygen therapy between December 1, 2020 and January 4, 2021 in this 50-bed, single-center retrospective cohort study. The outcome was requirement for oxygen after admission. The model performance to predict the need for oxygen by AI in collaboration with a clinician was better than that by AI alone. Sodium chloride difference >33.5 emerged as a novel indicator to predict the need for oxygen in patients with COVID-19. To prevent severe COVID-19 in older patients, dehydration compensation may be considered in pre-hospitalization care. In clinical practice, our approach enables the building of a better predictive model with prompt clinician feedback even in new scenarios. These can be applied not only to current and future pandemic situations but also to other diseases within the healthcare system.

NAVIGATOR: an Italian regional imaging biobank to promote precision medicine for oncologic patients

NAVIGATOR is an Italian regional project boosting precision medicine in oncology with the aim of making it more predictive, preventive, and personalised by advancing translational research based on quantitative imaging and integrative omics analyses. The project’s goal is to develop an open imaging biobank for the collection and preservation of a large amount of standardised imaging multimodal datasets, including computed tomography, magnetic resonance imaging, and positron emission tomography data, together with the corresponding patient-related and omics-related relevant information extracted from regional healthcare services using an adapted privacy-preserving model. The project is based on an open-source imaging biobank and an open-science oriented virtual research environment (VRE). Available integrative omics and multi-imaging data of three use cases (prostate cancer, rectal cancer, and gastric cancer) will be collected. All data confined in NAVIGATOR (i.e., standard and novel imaging biomarkers, non-imaging data, health agency data) will be used to create a digital patient model, to support the reliable prediction of the disease phenotype and risk stratification. The VRE that relies on a well-established infrastructure, called D4Science.org, will further provide a multiset infrastructure for processing the integrative omics data, extracting specific radiomic signatures, and for identification and testing of novel imaging biomarkers through big data analytics and artificial intelligence.

Use of convolutional neural networks in skin lesion analysis using real world image and non-image data.

Understanding performance of convolutional neural networks (CNNs) for binary (benign vs. malignant) lesion classification based on real world images is important for developing a meaningful clinical decision support (CDS) tool. We developed a CNN based on real world smartphone images with histopathological ground truth and tested the utility of structured electronic health record (EHR) data on model performance. Model accuracy was compared against three board-certified dermatologists for clinical validity. At a classification threshold of 0.5, the sensitivity was 79 vs. 77 vs. 72%, and specificity was 64 vs. 65 vs. 57% for image-alone vs. combined image and clinical data vs. clinical data-alone models, respectively. The PPV was 68 vs. 69 vs. 62%, AUC was 0.79 vs. 0.79 vs. 0.69, and AP was 0.78 vs. 0.79 vs. 0.64 for image-alone vs. combined data vs. clinical data-alone models. Older age, male sex, and number of prior dermatology visits were important positive predictors for malignancy in the clinical data-alone model. Additional clinical data did not significantly improve CNN image model performance. Model accuracy for predicting malignant lesions was comparable to dermatologists (model: 71.31% vs. 3 dermatologists: 77.87, 69.88, and 71.93%), validating clinical utility. Prospective validation of the model in primary care setting will enhance understanding of the model's clinical utility.

Deep learning of longitudinal chest X-ray and clinical variables predicts duration on ventilator and mortality in COVID-19 patients

ObjectivesTo use deep learning of serial portable chest X-ray (pCXR) and clinical variables to predict mortality and duration on invasive mechanical ventilation (IMV) for Coronavirus disease 2019 (COVID-19) patients.MethodsThis is a retrospective study. Serial pCXR and serial clinical variables were analyzed for data from day 1, day 5, day 1–3, day 3–5, or day 1–5 on IMV (110 IMV survivors and 76 IMV non-survivors). The outcome variables were duration on IMV and mortality. With fivefold cross-validation, the performance of the proposed deep learning system was evaluated by receiver operating characteristic (ROC) analysis and correlation analysis.ResultsPredictive models using 5-consecutive-day data outperformed those using 3-consecutive-day and 1-day data. Prediction using data closer to the outcome was generally better (i.e., day 5 data performed better than day 1 data, and day 3–5 data performed better than day 1–3 data). Prediction performance was generally better for the combined pCXR and non-imaging clinical data than either alone. The combined pCXR and non-imaging data of 5 consecutive days predicted mortality with an accuracy of 85 ± 3.5% (95% confidence interval (CI)) and an area under the curve (AUC) of 0.87 ± 0.05 (95% CI) and predicted the duration needed to be on IMV to within 2.56 ± 0.21 (95% CI) days on the validation dataset.ConclusionsDeep learning of longitudinal pCXR and clinical data have the potential to accurately predict mortality and duration on IMV in COVID-19 patients. Longitudinal pCXR could have prognostic value if these findings can be validated in a large, multi-institutional cohort.

Multi-Output Regression with Generative Adversarial Networks (MOR-GANs)

Regression modelling has always been a key process in unlocking the relationships between independent and dependent variables that are held within data. In recent years, machine learning has uncovered new insights in many fields, providing predictions to previously unsolved problems. Generative Adversarial Networks (GANs) have been widely applied to image processing producing good results, however, these methods have not often been applied to non-image data. Seeing the powerful generative capabilities of the GANs, we explore their use, here, as a regression method. In particular, we explore the use of the Wasserstein GAN (WGAN) as a multi-output regression method. The resulting method we call Multi-Output Regression GANs (MOR-GANs) and its performance is compared to a Gaussian Process Regression method (GPR)—a commonly used non-parametric regression method that has been well tested on small datasets with noisy responses. The WGAN regression model performs well for all types of datasets and exhibits substantial improvements over the performance of the GPR for certain types of datasets, demonstrating the flexibility of the GAN as a model for regression.

Automatic 3D+t four-chamber CMR quantification of the UK biobank: integrating imaging and non-imaging data priors at scale.

Accurate 3D modelling of cardiac chambers is essential for clinical assessment of cardiac volume and function, including structural, and motion analysis. Furthermore, to study the correlation between cardiac morphology and other patient information within a large population, it is necessary to automatically generate cardiac mesh models of each subject within the population. In this study, we introduce MCSI-Net (Multi-Cue Shape Inference Network), where we embed a statistical shape model inside a convolutional neural network and leverage both phenotypic and demographic information from the cohort to infer subject-specific reconstructions of all four cardiac chambers in 3D. In this way, we leverage the ability of the network to learn the appearance of cardiac chambers in cine cardiac magnetic resonance (CMR) images, and generate plausible 3D cardiac shapes, by constraining the prediction using a shape prior, in the form of the statistical modes of shape variation learned a priori from a subset of the population. This, in turn, enables the network to generalise to samples across the entire population. To the best of our knowledge, this is the first work that uses such an approach for patient-specific cardiac shape generation. MCSI-Net is capable of producing accurate 3D shapes using just a fraction (about 23% to 46%) of the available image data, which is of significant importance to the community as it supports the acceleration of CMR scan acquisitions. Cardiac MR images from the UK Biobank were used to train and validate the proposed method. We also present the results from analysing 40,000 subjects of the UK Biobank at 50 time-frames, totalling two million image volumes. Our model can generate more globally consistent heart shape than that of manual annotations in the presence of inter-slice motion and shows strong agreement with the reference ranges for cardiac structure and function across cardiac ventricles and atria.

Deep learning in periodontology and oral implantology: Ascoping review.

Deep learning (DL) has been employed for a wide range of tasks in dentistry. We aimed to systematically review studies employing DL for periodontal and implantological purposes. A systematic electronic search was conducted on four databases (Medline via PubMed, Google Scholar, Scopus, and Embase) and a repository (ArXiv) for publications after 2010, without any limitation on language. In the present review, we included studies that reported deep learning models' performance on periodontal or oral implantological tasks. Given the heterogeneities in the included studies, no meta-analysis was performed. The risk of bias was assessed using the QUADAS-2 tool. We included 47 studies: focusing on imaging data (n=20) and non-imaging data in periodontology (n=12), or dental implantology (n=15). The detection of periodontitis and gingivitis or periodontal bone loss, the classification of dental implant systems, or the prediction of treatment outcomes in periodontology and implantology were major use cases. The performance of the models was generally high. However, it varied given the employed methods (which includes various types of convolutional neural networks (CNN) and multi-layered perceptron (MLP)), the variety in specific modeling tasks, as well as the chosen and reported outcomes, outcome measures and outcome level. Only a few studies (n=7) showed a low risk of bias across all assessed domains. A growing number of studies evaluated DL for periodontal or implantological objectives. Heterogeneity in study design, poor reporting and a high risk of bias severely limit the comparability of studies and the robustness of the overall evidence.

Mortality Prediction Analysis among COVID-19 Inpatients Using Clinical Variables and Deep Learning Chest Radiography Imaging Features.

The emergence of the COVID-19 pandemic over a relatively brief interval illustrates the need for rapid data-driven approaches to facilitate clinical decision making. We examined a machine learning process to predict inpatient mortality among COVID-19 patients using clinical and chest radiographic data. Modeling was performed with a de-identified dataset of encounters prior to widespread vaccine availability. Non-imaging predictors included demographics, pre-admission clinical history, and past medical history variables. Imaging features were extracted from chest radiographs by applying a deep convolutional neural network with transfer learning. A multi-layer perceptron combining 64 deep learning features from chest radiographs with 98 patient clinical features was trained to predict mortality. The Local Interpretable Model-Agnostic Explanations (LIME) method was used to explain model predictions. Non-imaging data alone predicted mortality with an ROC-AUC of 0.87 ± 0.03 (mean ± SD), while the addition of imaging data improved prediction slightly (ROC-AUC: 0.91 ± 0.02). The application of LIME to the combined imaging and clinical model found HbA1c values to contribute the most to model prediction (17.1 ± 1.7%), while imaging contributed 8.8 ± 2.8%. Age, gender, and BMI contributed 8.7%, 8.2%, and 7.1%, respectively. Our findings demonstrate a viable explainable AI approach to quantify the contributions of imaging and clinical data to COVID mortality predictions.

Bitbox: A Cloud-based data sharing solution for medical images.

With the modernization and digitisation of the healthcare system, the need for exchanging medical data has become increasingly compelling. Biomedical imaging has been no exception, where the gathering of medical imaging acquisitions from multi-site collaborations have enabled to reach data sizes never imaginable until few years ago. Usually, medical imaging data have very large volume and diverse complexity, requiring bespoken transfer systems that protect personal information as well as data integrity. Despite many digital innovations, there are still technical and regulatory bottlenecks that make biomedical imaging data exchange challenging. Here we present Bitbox, a web-based application which provides a reliable yet straightforward service to securely exchange medical imaging data. With Bitbox, both imaging and non-imaging data of any type can be transferred from any external and independent site into a centralized server. A showcase of the system will be illustrated for the COVID-19 Clinical Neuroscience Study (COVID-CNS) project, a UK-wide experimental medicine study to investigate the neurological and neuropsychiatric effects of COVID-19 infections in hundreds of patients.

MAMF-GCN: Multi-scale adaptive multi-channel fusion deep graph convolutional network for predicting mental disorder

PurposeExisting diagnoses of mental disorders rely on symptoms, patient descriptions, and scales, which are not objective enough. We attempt to explore an objective diagnostic method on fMRI data. Graph neural networks (GNN) have been paid more attention recently because of their advantages in processing unstructured relational data, especially for fMRI data. However, how to deeply embed and well-integrate with different modalities and scales on GNN is still a challenge. Instead of reaching a high degree of fusion, existing GCN methods simply combine image and non-image data. Most graph convolutional network (GCN) models use shallow structures, making it challenging to learn about potential information. Furthermore, current graph construction approaches usually use a single specific brain atlas, limiting the analysis and results. MethodIn this paper, a multi-scale adaptive multi-channel fusion deep graph convolutional network based on an attention mechanism (MAMF-GCN) is proposed to better integrate features of modalities and different atlas by exploiting multi-channel correlation. An encoder automatically combines one channel with non-imaging data to generate similarity weights between subjects using a similarity perception mechanism. Other channels generate multi-scale imaging features of fMRI data after processing in the different atlas. Multi-modal information is fused using an adaptive convolution module that applies a deep graph convolutional network (GCN) to extract information from richer hidden layers. ResultsTo demonstrate the effectiveness of our approach, we evaluate the performance of the proposed method on the Autism Brain Imaging Data Exchange (ABIDE) dataset and the Major Depressive Disorder (MDD) dataset. The experimental result shows that the proposed method outperforms many state-of-the-art methods in node classification performance. An extensive group of experiments on two disease prediction tasks demonstrates that the performance of the proposed MAMF-GCN on MDD/ABIDE dataset is improved by 3.37%–39.83% and 12.59%–32.92%, respectively. Moreover, our proposed method has also shown very effective performance in real-life clinical diagnosis. The comprehensive experiments demonstrate that our method is effective for node classification with brain disorders diagnosis. ConclusionThe proposed MAMF-GCN method simultaneously extracts specific and common embeddings from the topology composed of multi-scale imaging features, phenotypic information, and their combinations, then learning adaptive embedding weights by attention mechanism, which can capture and fuse the multi-scale essential embeddings to improve the classification performance of brain disorder diagnosis.

Artificially-reconstructed brain images with stroke lesions from non-imaging data: modeling in categorized patients based on lesion occurrence and sparsity

Brain imaging is necessary for understanding disease symptoms, including stroke. However, frequent imaging procedures encounter practical limitations. Estimating the brain information (e.g., lesions) without imaging sessions is beneficial for this scenario. Prospective estimating variables are non-imaging data collected from standard tests. Therefore, the current study aims to examine the variable feasibility for modelling lesion locations. Heterogeneous variables were employed in the multivariate logistic regression. Furthermore, patients were categorized (i.e., unsupervised clustering through k-means method) by the charasteristics of lesion occurrence (i.e., ratio between the lesioned and total regions) and sparsity (i.e., density measure of lesion occurrences across regions). Considering those charasteristics in models improved estimation performances. Lesions (116 regions in Automated Anatomical Labeling) were adequately predicted (sensitivity: 80.0–87.5% in median). We confirmed that the usability of models was extendable to different resolution levels in the brain region of interest (e.g., lobes, hemispheres). Patients’ charateristics (i.e., occurrence and sparsity) might also be explained by the non-imaging data as well. Advantages of the current approach can be experienced by any patients (i.e., with or without imaging sessions) in any clinical facilities (i.e., with or without imaging instrumentation).

A novelty detection approach to effectively predict conversion from mild cognitive impairment to Alzheimer’s disease

Accurately recognising patients with progressive mild cognitive impairment (pMCI) who will develop Alzheimer’s disease (AD) in subsequent years is very important, as early identification of those patients will enable interventions to potentially reduce the number of those transitioning from MCI to AD. Most studies in this area have concentrated on high-dimensional neuroimaging data with supervised binary/multi-class classification algorithms. However, neuroimaging data is more costly to obtain than non-imaging, and healthcare datasets are normally imbalanced which may reduce classification performance and reliability. To address these challenges, we proposed a new strategy that employs unsupervised novelty detection (ND) techniques to predict pMCI from the AD neuroimaging initiative non-imaging data. ND algorithms, including the k-nearest neighbours (kNN), k-means, Gaussian mixture model (GMM), isolation forest (IF) and extreme learning machine (ELM), were employed and compared with supervised binary support vector machine (SVM) and random forest (RF). We introduced optimisation with nested cross-validation and focused on maximising the adjusted F measure to ensure maximum generalisation of the proposed system by minimising false negative rates. Our extensive experimental results show that ND algorithms (0.727 ± 0.029 kNN, 0.7179 ± 0.0523 GMM, 0.7276 ± 0.0281 ELM) obtained comparable performance to supervised binary SVM (0.7359 ± 0.0451) with 20% stable MCI misclassification tolerance and were significantly better than RF (0.4771 ± 0.0167). Moreover, we found that the non-invasive, readily obtainable, and cost-effective cognitive and functional assessment was the most efficient predictor for predicting the pMCI within 2 years with ND techniques. Importantly, we presented an accessible and cost-effective approach to pMCI prediction, which does not require labelled data.

Machine Learning Coronary Artery Disease Prediction Based on Imaging and Non-Imaging Data.

The prediction of obstructive atherosclerotic disease has significant clinical meaning for the decision making. In this study, a machine learning predictive model based on gradient boosting classifier is presented, aiming to identify the patients of high CAD risk and those of low CAD risk. The machine learning methodology includes five steps: the preprocessing of the input data, the class imbalance handling applying the Easy Ensemble algorithm, the recursive feature elimination technique implementation, the implementation of gradient boosting classifier, and finally the model evaluation, while the fine tuning of the presented model was implemented through a randomized search optimization of the model’s hyper-parameters over an internal 3-fold cross-validation. In total, 187 participants with suspicion of CAD previously underwent CTCA during EVINCI and ARTreat clinical studies and were prospectively included to undergo follow-up CTCA. The predictive model was trained using imaging data (geometrical and blood flow based) and non-imaging data. The overall predictive accuracy of the model was 0.81, using both imaging and non-imaging data. The innovative aspect of the proposed study is the combination of imaging-based data with the typical CAD risk factors to provide an integrated CAD risk-predictive model.

Feature space transformation of user-clicks and deep transfer learning framework for fraudulent publisher detection in online advertising

The concept drift is a challenge in click fraud detection wherein frequent changes in the actual status label of publishers complicate the identification of publishers’ fraudulent behavior. However, using transfer learning by leveraging the knowledge from previously learned domains to newer domains can make these differences more accessible while saving training time and improving the model’s performance. But the absence of other user-click datasets available publicly poses complexity in using transfer learning. Therefore, to use transfer learning towards predicting the publisher’s conduct concerning change in their labels, this work aims to transform 1D user-click non-image features into a 2D graphical image. The work proposes a deep convolution neural network-based transfer learning (DCNNTr) framework that utilizes different pre-trained Deep Convolutional Neural Network (DCNN) models as powerful feature extractors that leverage prior learnings to avert learning from scratch. The robust features extracted by the feature extractors help identify the conduct of publishers and classify them as fraudulent or non-fraudulent from 2D graphical images using machine learning models. By leveraging the weighted layers in extracting features, DCNN models utilize their special properties of being computationally efficient and locally focused. We evaluated the designed model on the FDMA2012 user-click dataset using precision, recall, F1-score, and AUC. Current work uniquely transforms the time series user-click non-image data into an image form. The experimental results demonstrate that features extracted with DenseNet121 followed by GTB have identified the fraudulent publishers with an average precision score of 79.8%.

Generative adversarial neural networks augment marker and pathway analysis of treatment resistant HPV+ head and neck squamous cell carcinoma.

e18059 Background: Biological behaviors of cancers are extremely complex and governed via nonlinear interplay of thousands of genes. This complexity has become a major hurdle to generating mechanistic conclusions in rare clinical scenarios, such as treatment resistant HPV+ head and neck squamous cell carcinoma, when observations are limited. Recently, data augmentation algorithms have served to increase the number and diversity of training observations while preserving the underlying structure and nonlinear relationships of data features. Deep learning techniques such as generative adversarial neural networks (GANs) have shown promise as robust tools for data augmentation of nonimage data. This study aims to evaluate the utility of employing separate GANs to highlight gene expression differences between treatment sensitive (TS) vs treatment resistant (TR) HPV+ head and neck squamous cell carcinoma. Methods: We performed RNAseq on 17 patients with HPV-positive oropharyngeal squamous cell carcinoma (OPSCC). We defined TR patients as those with locally advanced OPSCC at presentation, who rapidly recurred following receipt of standard of care therapy (n = 11). In most cases, patients failed to respond to salvage therapies and rapidly died. In contrast, we defined TS patients as those with locally advanced OPSCC at presentation, who received standard of care therapy, and were followed for 5 years without any evidence of recurrence (n = 6). These two cohorts were used to train two separate GANs, which synthesized 500 robust synthetic samples of each phenotype for downstream analysis. Results: Analysis of 17 samples yielded less than 100 differentially expressed (DE) genes passing FDR cutoffs, inhibiting pathway enrichment analysis (PAE). DE testing between synthetic samples yielded 1196 DE genes. Validating our deep learning approach, these genes belong to coherent pathways as determined by PAE. Namely, the VEGFA-VEGFR2 pathway was strongly enriched in the TR population (p = 1.5 x 10-4). Further, the NKT cell signature (p = 3.0x10-62), and the gene library of T-cell receptor induction of apoptosis (p = 5.4 x 10-9), was enriched in the TS population. Lastly, to validate the experimental method, DE testing on GAN augmented cohorts of renal papillary vs renal clear cell samples from The Cancer Genome Atlas, trained on 10 samples per condition, reliably predicted DE genes when comparing the full cohort of renal papillary n = 534 vs renal clear cell n = 295 samples, with an accuracy of 93% and F1 score of 88%. Conclusions: To our knowledge this is the first study to utilize separate GANs to augment structural gene expression changes occurring in rare clinical scenarios such as treatment resistant HPV+ OPSCC. Utilizing this method uncovers changes in angiogenesis and T-cell function between treatment sensitive and resistant HPV+ OPSCC.

Prediction of treatment outcome in neovascular age-related macular degeneration using a novel convolutional neural network

While prognosis and risk of progression are crucial in developing precise therapeutic strategy in neovascular age-related macular degeneration (nAMD), limited predictive tools are available. We proposed a novel deep convolutional neural network that enables feature extraction through image and non-image data integration to seize imperative information and achieve highly accurate outcome prediction. The Heterogeneous Data Fusion Net (HDF-Net) was designed to predict visual acuity (VA) outcome (improvement ≥ 2 line or not) at 12th months after anti-VEGF treatment. A set of pre-treatment optical coherence tomography (OCT) image and non-image demographic features were employed as input data and the corresponding 12th-month post-treatment VA as the target data to train, validate, and test the HDF-Net. This newly designed HDF-Net demonstrated an AUC of 0.989 (95% CI 0.970–0.999), accuracy of 0.936 [95% confidence interval (CI) 0.889–0.964], sensitivity of 0.933 (95% CI 0.841–0.974), and specificity of 0.938 (95% CI 0.877–0.969). By simulating the clinical decision process with mixed pre-treatment information from raw OCT images and numeric data, HDF-Net demonstrated promising performance in predicting individualized treatment outcome. The results highlight the potential of deep learning to simultaneously process a broad range of clinical data to weigh and leverage the complete information of the patient. This novel approach is an important step toward real-world personalized therapeutic strategy for typical nAMD.

Disease prediction with edge-variational graph convolutional networks.

The need for computational models that can incorporate imaging data with non-imaging data while investigating inter-subject associations arises in the task of population-based disease analysis. Although off-the-shelf deep convolutional neural networks have empowered representation learning from imaging data, incorporating data of different modalities complementarily in a unified model to improve the disease diagnostic quality is still challenging. In this work, we propose a generalizable graph-convolutional framework for population-based disease prediction on multi-modal medical data. Unlike previous methods constructing a static affinity population graph in a hand-crafting manner, the proposed framework can automatically learn to build a population graph with variational edges, which we show can be optimized jointly with spectral graph convolutional networks. In addition, to estimate the predictive uncertainty related to the constructed graph, we propose Monte-Carlo edge dropout uncertainty estimation. Experimental results on four multi-modal datasets demonstrate that the proposed method can substantially improve the predictive accuracy for Autism Spectrum Disorder, Alzheimer's disease, and ocular diseases. A sufficient ablation study with in-depth discussion is conducted to evaluate the effectiveness of each component and the choice of algorithmic details of the proposed method. The results indicate the potential and extendability of the proposed framework in leveraging multi-modal data for population-based disease prediction.

Artificial Intelligence-Driven Prediction Modeling and Decision Making in Spine Surgery Using Hybrid Machine Learning Models.

Healthcare systems worldwide generate vast amounts of data from many different sources. Although of high complexity for a human being, it is essential to determine the patterns and minor variations in the genomic, radiological, laboratory, or clinical data that reliably differentiate phenotypes or allow high predictive accuracy in health-related tasks. Convolutional neural networks (CNN) are increasingly applied to image data for various tasks. Its use for non-imaging data becomes feasible through different modern machine learning techniques, converting non-imaging data into images before inputting them into the CNN model. Considering also that healthcare providers do not solely use one data modality for their decisions, this approach opens the door for multi-input/mixed data models which use a combination of patient information, such as genomic, radiological, and clinical data, to train a hybrid deep learning model. Thus, this reflects the main characteristic of artificial intelligence: simulating natural human behavior. The present review focuses on key advances in machine and deep learning, allowing for multi-perspective pattern recognition across the entire information set of patients in spine surgery. This is the first review of artificial intelligence focusing on hybrid models for deep learning applications in spine surgery, to the best of our knowledge. This is especially interesting as future tools are unlikely to use solely one data modality. The techniques discussed could become important in establishing a new approach to decision-making in spine surgery based on three fundamental pillars: (1) patient-specific, (2) artificial intelligence-driven, (3) integrating multimodal data. The findings reveal promising research that already took place to develop multi-input mixed-data hybrid decision-supporting models. Their implementation in spine surgery may hence be only a matter of time.

Transfer learning for non-image data in clinical research: A scoping review.

Transfer learning is a form of machine learning where a pre-trained model trained on a specific task is reused as a starting point and tailored to another task in a different dataset. While transfer learning has garnered considerable attention in medical image analysis, its use for clinical non-image data is not well studied. Therefore, the objective of this scoping review was to explore the use of transfer learning for non-image data in the clinical literature. We systematically searched medical databases (PubMed, EMBASE, CINAHL) for peer-reviewed clinical studies that used transfer learning on human non-image data. We included 83 studies in the review. More than half of the studies (63%) were published within 12 months of the search. Transfer learning was most often applied to time series data (61%), followed by tabular data (18%), audio (12%) and text (8%). Thirty-three (40%) studies applied an image-based model to non-image data after transforming data into images (e.g. spectrograms). Twenty-nine (35%) studies did not have any authors with a health-related affiliation. Many studies used publicly available datasets (66%) and models (49%), but fewer shared their code (27%). In this scoping review, we have described current trends in the use of transfer learning for non-image data in the clinical literature. We found that the use of transfer learning has grown rapidly within the last few years. We have identified studies and demonstrated the potential of transfer learning in clinical research in a wide range of medical specialties. More interdisciplinary collaborations and the wider adaption of reproducible research principles are needed to increase the impact of transfer learning in clinical research.

Semantic Integration of Multi-Modal Data and Derived Neuroimaging Results Using the Platform for Imaging in Precision Medicine (PRISM) in the Arkansas Imaging Enterprise System (ARIES).

Neuroimaging is among the most active research domains for the creation and management of open-access data repositories. Notably lacking from most data repositories are integrated capabilities for semantic representation. The Arkansas Imaging Enterprise System (ARIES) is a research data management system which features integrated capabilities to support semantic representations of multi-modal data from disparate sources (imaging, behavioral, or cognitive assessments), across common image-processing stages (preprocessing steps, segmentation schemes, analytic pipelines), as well as derived results (publishable findings). These unique capabilities ensure greater reproducibility of scientific findings across large-scale research projects. The current investigation was conducted with three collaborating teams who are using ARIES in a project focusing on neurodegeneration. Datasets included magnetic resonance imaging (MRI) data as well as non-imaging data obtained from a variety of assessments designed to measure neurocognitive functions (performance scores on neuropsychological tests). We integrate and manage these data with semantic representations based on axiomatically rich biomedical ontologies. These instantiate a knowledge graph that combines the data from the study cohorts into a shared semantic representation that explicitly accounts for relations among the entities that the data are about. This knowledge graph is stored in a triple-store database that supports reasoning over and querying these integrated data. Semantic integration of the non-imaging data using background information encoded in biomedical domain ontologies has served as a key feature-engineering step, allowing us to combine disparate data and apply analyses to explore associations, for instance, between hippocampal volumes and measures of cognitive functions derived from various assessment instruments.