Large Multimodal Dataset Research Articles

Multi-modal representation learning in retinal imaging using self-supervised learning for enhanced clinical predictions

Self-supervised learning has become the cornerstone of building generalizable and transferable artificial intelligence systems in medical imaging. In particular, contrastive representation learning techniques trained on large multi-modal datasets have demonstrated impressive capabilities of producing highly transferable representations for different downstream tasks. In ophthalmology, large multi-modal datasets are abundantly available and conveniently accessible as modern retinal imaging scanners acquire both 2D fundus images and 3D optical coherence tomography (OCT) scans to assess the eye. In this context, we introduce a novel multi-modal contrastive learning-based pipeline to facilitate learning joint representations for the two retinal imaging modalities. After self-supervised pre-training on 153,306 scan pairs, we show that such a pre-training framework can provide both a retrieval system and encoders that produce comprehensive OCT and fundus image representations that generalize well for various downstream tasks on three independent external datasets, explicitly focusing on clinically pertinent prediction tasks. In addition, we show that interchanging OCT with lower-cost fundus imaging can preserve the predictive power of the trained models.

Multimodal subspace independent vector analysis effectively captures the latent relationships between brain structure and function.

A key challenge in neuroscience is to understand the structural and functional relationships of the brain from high-dimensional, multimodal neuroimaging data. While conventional multivariate approaches often simplify statistical assumptions and estimate one-dimensional independent sources shared across modalities, the relationships between true latent sources are likely more complex - statistical dependence may exist within and between modalities, and span one or more dimensions. Here we present Multimodal Subspace Independent Vector Analysis (MSIVA), a methodology to capture both joint and unique vector sources from multiple data modalities by defining both cross-modal and unimodal subspaces with variable dimensions. In particular, MSIVA enables flexible estimation of varying-size independent subspaces within modalities and their one-to-one linkage to corresponding sub-spaces across modalities. As we demonstrate, a main benefit of MSIVA is the ability to capture subject-level variability at the voxel level within independent subspaces, contrasting with the rigidity of traditional methods that share the same independent components across subjects. We compared MSIVA to a unimodal initialization baseline and a multimodal initialization baseline, and evaluated all three approaches with five candidate subspace structures on both synthetic and neuroimaging datasets. We show that MSIVA successfully identified the ground-truth subspace structures in multiple synthetic datasets, while the multimodal baseline failed to detect high-dimensional subspaces. We then demonstrate that MSIVA better detected the latent subspace structure in two large multimodal neuroimaging datasets including structural MRI (sMRI) and functional MRI (fMRI), compared with the unimodal baseline. From subsequent subspace-specific canonical correlation analysis, brain-phenotype prediction, and voxelwise brain-age delta analysis, our findings suggest that the estimated sources from MSIVA with optimal subspace structure are strongly associated with various phenotype variables, including age, sex, schizophrenia, lifestyle factors, and cognitive functions. Further, we identified modality- and group-specific brain regions related to multiple phenotype measures such as age (e.g., cerebellum, precentral gyrus, and cingulate gyrus in sMRI; occipital lobe and superior frontal gyrus in fMRI), sex (e.g., cerebellum in sMRI, frontal lobe in fMRI, and precuneus in both sMRI and fMRI), schizophrenia (e.g., cerebellum, temporal pole, and frontal operculum cortex in sMRI; occipital pole, lingual gyrus, and precuneus in fMRI), shedding light on phenotypic and neuropsychiatric biomarkers of linked brain structure and function.

MosaicMPI: a framework for modular data integration across cohorts and -omics modalities.

Advances in molecular profiling have facilitated generation of large multi-modal datasets that can potentially reveal critical axes of biological variation underlying complex diseases. Distilling biological meaning, however, requires computational strategies that can perform mosaic integration across diverse cohorts and datatypes. Here, we present mosaicMPI, a framework for discovery of low to high-resolution molecular programs representing both cell types and states, and integration within and across datasets into a network representing biological themes. Using existing datasets in glioblastoma, we demonstrate that this approach robustly integrates single cell and bulk programs across multiple platforms. Clinical and molecular annotations from cohorts are statistically propagated onto this network of programs, yielding a richly characterized landscape of biological themes. This enables deep understanding of individual tumor samples, systematic exploration of relationships between modalities, and generation of a reference map onto which new datasets can rapidly be mapped. mosaicMPI is available at https://github.com/MorrissyLab/mosaicMPI.

Image-text crisis tweet categorization:a caption-based approach

The growth of social media usage this last decade has made available a massive and valuable volume of multimedia data. However, the lack of large multimodal annotated datasets, along with the inherent noise and the diversity of multimodal relations in this type of data presents challenges for machine learning methods. Unlike classic multimodal data, social media data comes with a large diversity of relations between image and text making the interaction between the two modalities more difficult.Previous research concentrated on fusion strategies with separate encoders for each modality. This paper introduces CMB (Caption-based Multimodal BERT), a method of classifying crisis-related social media posts by utilizing information from both images and texts. CMB translates the image modality into a text-compatible space, facilitating intermodal interaction. Furthermore, CMB presents training opportunities to enhance the model's robustness to missing modalities. Experimental results show that CMB is competitive with well-established, costly, and manually crafted multimodal models.

Semi-Supervised Multimodal Representation Learning Through a Global Workspace.

Recent deep learning models can efficiently combine inputs from different modalities (e.g., images and text) and learn to align their latent representations or to translate signals from one domain to another (as in image captioning or text-to-image generation). However, current approaches mainly rely on brute-force supervised training over large multimodal datasets. In contrast, humans (and other animals) can learn useful multimodal representations from only sparse experience with matched cross-modal data. Here, we evaluate the capabilities of a neural network architecture inspired by the cognitive notion of a "global workspace" (GW): a shared representation for two (or more) input modalities. Each modality is processed by a specialized system (pretrained on unimodal data and subsequently frozen). The corresponding latent representations are then encoded to and decoded from a single shared workspace. Importantly, this architecture is amenable to self-supervised training via cycle-consistency: encoding-decoding sequences should approximate the identity function. For various pairings of vision-language modalities and across two datasets of varying complexity, we show that such an architecture can be trained to align and translate between two modalities with very little need for matched data (from four to seven times less than a fully supervised approach). The GW representation can be used advantageously for downstream classification and cross-modal retrieval tasks and for robust transfer learning. Ablation studies reveal that both the shared workspace and the self-supervised cycle-consistency training are critical to the system's performance.

Graph Neural Networks in Cancer and Oncology Research: Emerging and Future Trends.

Next-generation cancer and oncology research needs to take full advantage of the multimodal structured, or graph, information, with the graph data types ranging from molecular structures to spatially resolved imaging and digital pathology, biological networks, and knowledge graphs. Graph Neural Networks (GNNs) efficiently combine the graph structure representations with the high predictive performance of deep learning, especially on large multimodal datasets. In this review article, we survey the landscape of recent (2020-present) GNN applications in the context of cancer and oncology research, and delineate six currently predominant research areas. We then identify the most promising directions for future research. We compare GNNs with graphical models and "non-structured" deep learning, and devise guidelines for cancer and oncology researchers or physician-scientists, asking the question of whether they should adopt the GNN methodology in their research pipelines.

Mental health prediction from social media connections

ABSTRACT The relation between mental health and the language employed by social media users has been the focus of a wide range of studies in Natural Language Processing and related fields intended to predict early signs of a mental disorder. In many cases, however, text data alone is considered to be a sufficient source of learning features, and other possible sources also available from social media are often neglected. To shed light on these issues, the present work addresses the use of social media connections information — represented by the identities of Twitter friends, followers and mentions — to predict depression and anxiety disorder. Using a large multimodal dataset of over 31K unique Twitter timelines (555 million words) and associated network data, we built a number of network-based models (using network embeddings representations), and text-based alternatives (bag-of-words and LIWC baseline models) to these tasks. Results suggest that not only network connections may act as strong predictors of mental health, but that these models may actually outperform standard baseline alternatives that rely on text data alone, which may be seen as a first step towards more sophisticated architectures that may actually combine textual and non-textual information.

Cross-attractor modeling of resting-state functional connectivity in psychiatric disorders

Resting-state functional connectivity (RSFC) is altered across various psychiatric disorders. Brain network modeling (BNM) has the potential to reveal the neurobiological underpinnings of such abnormalities by dynamically modeling the structure-function relationship and examining biologically relevant parameters after fitting the models with real data. Although innovative BNM approaches have been developed, two main issues need to be further addressed. First, previous BNM approaches are primarily limited to simulating noise-driven dynamics near a chosen attractor (or a stable brain state). An alternative approach is to examine multi(or cross)-attractor dynamics, which can be used to better capture non-stationarity and switching between states in the resting brain. Second, previous BNM work is limited to characterizing one disorder at a time. Given the large degree of co-morbidity across psychiatric disorders, comparing BNMs across disorders might provide a novel avenue to generate insights regarding the dynamical features that are common across (vs. specific to) disorders. Here, we address these issues by (1) examining the layout of the attractor repertoire over the entire multi-attractor landscape using a recently developed cross-attractor BNM approach; and (2) characterizing and comparing multiple disorders (schizophrenia, bipolar, and ADHD) with healthy controls using an openly available and moderately large multimodal dataset from the UCLA Consortium for Neuropsychiatric Phenomics. Both global and local differences were observed across disorders. Specifically, the global coupling between regions was significantly decreased in schizophrenia patients relative to healthy controls. At the same time, the ratio between local excitation and inhibition was significantly higher in the schizophrenia group than the ADHD group. In line with these results, the schizophrenia group had the lowest switching costs (energy gaps) across groups for several networks including the default mode network. Paired comparison also showed that schizophrenia patients had significantly lower energy gaps than healthy controls for the somatomotor and visual networks. Overall, this study provides preliminary evidence supporting transdiagnostic multi-attractor BNM approaches to better understand psychiatric disorders' pathophysiology.

How to establish and maintain a multimodal animal research dataset using DataLad

Sharing of data, processing tools, and workflows require open data hosting services and management tools. Despite FAIR guidelines and the increasing demand from funding agencies and publishers, only a few animal studies share all experimental data and processing tools. We present a step-by-step protocol to perform version control and remote collaboration for large multimodal datasets. A data management plan was introduced to ensure data security in addition to a homogeneous file and folder structure. Changes to the data were automatically tracked using DataLad and all data was shared on the research data platform GIN. This simple and cost-effective workflow facilitates the adoption of FAIR data logistics and processing workflows by making the raw and processed data available and providing the technical infrastructure to independently reproduce the data processing steps. It enables the community to collect heterogeneously acquired and stored datasets not limited to a specific category of data and serves as a technical infrastructure blueprint with rich potential to improve data handling at other sites and extend to other research areas.

The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation.

People spend a substantial portion of their lives engaged in conversation, and yet, our scientific understanding of conversation is still in its infancy. Here, we introduce a large, novel, and multimodal corpus of 1656 conversations recorded in spoken English. This 7+ million word, 850-hour corpus totals more than 1 terabyte of audio, video, and transcripts, with moment-to-moment measures of vocal, facial, and semantic expression, together with an extensive survey of speakers' postconversation reflections. By taking advantage of the considerable scope of the corpus, we explore many examples of how this large-scale public dataset may catalyze future research, particularly across disciplinary boundaries, as scholars from a variety of fields appear increasingly interested in the study of conversation.

Machine Learning and MRI-based Diagnostic Models for ADHD: Are We There Yet?

Objective: Machine learning (ML) has been applied to develop magnetic resonance imaging (MRI)-based diagnostic classifiers for attention-deficit/hyperactivity disorder (ADHD). This systematic review examines this literature to clarify its clinical significance and to assess the implications of the various analytic methods applied. Methods: A comprehensive literature search on MRI-based diagnostic classifiers for ADHD was performed and data regarding the utilized models and samples were gathered. Results: We found that, although most studies reported the classification accuracies, they varied in choice of MRI modalities, ML models, cross-validation and testing methods, and sample sizes. We found that the accuracies of cross-validation methods inflated the performance estimation compared with those of a held-out test, compromising the model generalizability. Test accuracies have increased with publication year but were not associated with training sample sizes. Improved test accuracy over time was likely due to the use of better ML methods along with strategies to deal with data imbalances. Conclusion: Ultimately, large multi-modal imaging datasets, and potentially the combination with other types of data, like cognitive data and/or genetics, will be essential to achieve the goal of developing clinically useful imaging classification tools for ADHD in the future.

Clustering of single-cell multi-omics data with a multimodal deep learning method

Single-cell multimodal sequencing technologies are developed to simultaneously profile different modalities of data in the same cell. It provides a unique opportunity to jointly analyze multimodal data at the single-cell level for the identification of distinct cell types. A correct clustering result is essential for the downstream complex biological functional studies. However, combining different data sources for clustering analysis of single-cell multimodal data remains a statistical and computational challenge. Here, we develop a novel multimodal deep learning method, scMDC, for single-cell multi-omics data clustering analysis. scMDC is an end-to-end deep model that explicitly characterizes different data sources and jointly learns latent features of deep embedding for clustering analysis. Extensive simulation and real-data experiments reveal that scMDC outperforms existing single-cell single-modal and multimodal clustering methods on different single-cell multimodal datasets. The linear scalability of running time makes scMDC a promising method for analyzing large multimodal datasets.

HARMONIC: A multimodal dataset of assistive human–robot collaboration

We present the Human And Robot Multimodal Observations of Natural Interactive Collaboration (HARMONIC) dataset. This is a large multimodal dataset of human interactions with a robotic arm in a shared autonomy setting designed to imitate assistive eating. The dataset provides human, robot, and environmental data views of 24 different people engaged in an assistive eating task with a 6-degree-of-freedom (6-DOF) robot arm. From each participant, we recorded video of both eyes, egocentric video from a head-mounted camera, joystick commands, electromyography from the forearm used to operate the joystick, third-person stereo video, and the joint positions of the 6-DOF robot arm. Also included are several features that come as a direct result of these recordings, such as eye gaze projected onto the egocentric video, body pose, hand pose, and facial keypoints. These data streams were collected specifically because they have been shown to be closely related to human mental states and intention. This dataset could be of interest to researchers studying intention prediction, human mental state modeling, and shared autonomy. Data streams are provided in a variety of formats such as video and human-readable CSV and YAML files.

Artificial Intelligence: An Interprofessional Perspective on Implications for Geriatric Mental Health Research and Care.

Artificial intelligence (AI) in healthcare aims to learn patterns in large multimodal datasets within and across individuals. These patterns may either improve understanding of current clinical status or predict a future outcome. AI holds the potential to revolutionize geriatric mental health care and research by supporting diagnosis, treatment, and clinical decision-making. However, much of this momentum is driven by data and computer scientists and engineers and runs the risk of being disconnected from pragmatic issues in clinical practice. This interprofessional perspective bridges the experiences of clinical scientists and data science. We provide a brief overview of AI with the main focus on possible applications and challenges of using AI-based approaches for research and clinical care in geriatric mental health. We suggest future AI applications in geriatric mental health consider pragmatic considerations of clinical practice, methodological differences between data and clinical science, and address issues of ethics, privacy, and trust.

A multimodal IVA fusion approach to identify linked neuroimaging markers.

In this study, we introduce a method to perform independent vector analysis (IVA) fusion to estimate linked independent sources and apply to a large multimodal dataset of over 3000 subjects in the UK Biobank study, including structural (gray matter), diffusion (fractional anisotropy), and functional (amplitude of low frequency fluctuations) magnetic resonance imaging data from each subject. The approach reveals a number of linked sources showing significant and meaningful covariation with subject phenotypes. One such mode shows significant linear association with age across all three modalities. Robust age-associated reductions in gray matter density were observed in thalamus, caudate, and insular regions, as well as visual and cingulate regions, with covarying reductions of fractional anisotropy in the periventricular region, in addition to reductions in amplitude of low frequency fluctuations in visual and parietal regions. Another mode identified multimodal patterns that differentiated subjects in their time-to-recall during a prospective memory test. In sum, the proposed IVA-based approach provides a flexible, interpretable, and powerful approach for revealing links between multimodal neuroimaging data.

The Role of Large-Scale Data Infrastructure in Developing Next-Generation Deep Brain Stimulation Therapies.

Advances in neuromodulation technologies hold the promise of treating a patient’s unique brain network pathology using personalized stimulation patterns. In service of these goals, neuromodulation clinical trials using sensing-enabled devices are routinely generating large multi-modal datasets. However, with the expansion of data acquisition also comes an increasing difficulty to store, manage, and analyze the associated datasets, which integrate complex neural and wearable time-series data with dynamic assessments of patients’ symptomatic state. Here, we discuss a scalable cloud-based data platform that enables ingestion, aggregation, storage, query, and analysis of multi-modal neurotechnology datasets. This large-scale data infrastructure will accelerate translational neuromodulation research and enable the development and delivery of next-generation deep brain stimulation therapies.

AW3M: An auto-weighting and recovery framework for breast cancer diagnosis using multi-modal ultrasound.

Recently, more clinicians have realized the diagnostic value of multi-modal ultrasound in breast cancer identification and began to incorporate Doppler imaging and Elastography in the routine examination. However, accurately recognizing patterns of malignancy in different types of sonography requires expertise. Furthermore, an accurate and robust diagnosis requires proper weights of multi-modal information as well as the ability to process missing data in practice. These two aspects are often overlooked by existing computer-aided diagnosis (CAD) approaches. To overcome these challenges, we propose a novel framework (called AW3M) that utilizes four types of sonography (i.e. B-mode, Doppler, Shear-wave Elastography, and Strain Elastography) jointly to assist breast cancer diagnosis. It can extract both modality-specific and modality-invariant features using a multi-stream CNN model equipped with self-supervised consistency loss. Instead of assigning the weights of different streams empirically, AW3Mautomatically learns the optimal weights using reinforcement learning techniques. Furthermore, we design a light-weight recovery block that can be inserted to a trained model to handle different modality-missing scenarios. Experimental results on a large multi-modal dataset demonstrate that our method can achieve promising performance compared with state-of-the-art methods. The AW3Mframework is also tested on another independent B-mode dataset to prove its efficacy in general settings. Results show that the proposed recovery block can learn from the joint distribution of multi-modal features to further boost the classification accuracy given single modality input during the test.

Abstract PO-030: Radiomics for head and neck cancer prognostication: results from the RADCURE machine learning challenge

Abstract Recently, there has been significant interest in using rich, multi-modal data routinely collected in the clinic, including imaging, demographic and clinical information for prognostic factor discovery in cancer. In particular, the emerging field of radiomics makes use of computational tools to extract quantitative features from radiological images, with the aim of capturing the morphological and biological characteristics of tumors. Previous studies have demonstrated the potential of computed tomography (CT) imaging features as independent prognostic factors for overall survival in multiple types of cancer, including head and neck (HNC). However, poor reproducibility and lack of large, rigorous validation studies have hindered widespread clinical use of radiomics so far. We conducted a HNC survival prediction challenge with the aim of 1) developing an accurate prognostic model for HNC survival using clinical, demographic and routinely collected CT imaging data and 2) evaluating the true added value of CT radiomics compared to other prognostic factors. Using a large, retrospective cohort of 2552 patients, we assessed prognostic performance of 12 different approaches developed by several research groups at University Health Network in Toronto, making use of engineered radiomics, deep learning, clinical information and combinations of those. To allow for unbiased comparison between different approaches, all participants had access to a public training dataset of 1802 patients, while 750 were held out for evaluation. The best challenge submission used a deep multi-task learning framework on clinical data and tumour volume, achieving area under the ROC curve (AUROC) of 0.812 [95% CI 0.763–0.858] for 2-year survival prediction and concordance (C) index for lifetime risk prediction of 0.795 [0.751–0.838], outperforming the best clinical-only model (AUROC=0.800 [0.749–0.848], C=0.708 [0.661–0.754]), best radiomics-only model (AUROC=0.766 [0.719–0.811], C=0.748 [0.704–0.790]), as well as the best model combining deep radiomics with clinical features (AUROC=0.786 [0.733–0.836], C=0.774 [0.726–0.820]). We also used a ‘wisdom of the crowds’ ensemble approach to combine the predictions of all challenge submissions to determine whether The ensemble achieved stronger performance than any individual model (AUROC=0.823 [0.778–0.864], C=0.810 [0.772–0.845]) indicating that there might be complementary information between the different data modalities. Our rigorous challenge framework allowed us to evaluate a diverse collection of prognostic models in a large multi-modal dataset, demonstrating the value of machine learning in HNC prognostication, as well as the advantages of simple imaging features over several hand-engineered and deep radiomics approaches. Furthemore, our ensemble approach achieves excellent performance for both 2-year and lifetime risk prediction, establishing new state-of-the-art in HNC prognostic modelling. Citation Format: Michal Kazmierski, Mattea Welch, Benjamin Haibe-Kains. Radiomics for head and neck cancer prognostication: results from the RADCURE machine learning challenge [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-030.

Consistent cross-modal identification of cortical neurons with coupled autoencoders.

Consistent identification of neurons in different experimental modalities is a key problem in neuroscience. Although methods to perform multimodal measurements in the same set of single neurons have become available, parsing complex relationships across different modalities to uncover neuronal identity is a growing challenge. Here we present an optimization framework to learn coordinated representations of multimodal data and apply it to a large multimodal dataset profiling mouse cortical interneurons. Our approach reveals strong alignment between transcriptomic and electrophysiological characterizations, enables accurate cross-modal data prediction, and identifies cell types that are consistent across modalities.

AlgoLabel: A Large Dataset for Multi-Label Classification of Algorithmic Challenges

While semantic parsing has been an important problem in natural language processing for decades, recent years have seen a wide interest in automatic generation of code from text. We propose an alternative problem to code generation: labelling the algorithmic solution for programming challenges. While this may seem an easier task, we highlight that current deep learning techniques are still far from offering a reliable solution. The contributions of the paper are twofold. First, we propose a large multi-modal dataset of text and code pairs consisting of algorithmic challenges and their solutions, called AlgoLabel. Second, we show that vanilla deep learning solutions need to be greatly improved to solve this task and we propose a dual text-code neural model for detecting the algorithmic solution type for a programming challenge. While the proposed text-code model increases the performance of using the text or code alone, the improvement is rather small highlighting that we require better methods to combine text and code features.