Multimodal Machine Research Articles

Multimodal machine learning models identify chemotherapy drugs with prospective clinical efficacy in dogs with relapsed B-cell lymphoma.

Dogs with B-cell lymphoma typically respond well to first-line CHOP-based chemotherapy, but there is no standard of care for relapsed patients. To help veterinary oncologists select effective drugs for dogs with lymphoid malignancies such as B-cell lymphoma, we have developed multimodal machine learning models that integrate data from multiple tumor profiling modalities and predict the likelihood of a positive clinical response for 10 commonly used chemotherapy drugs. Here we report on clinical outcomes that occurred after oncologists received a prediction report generated by our models. Remarkably, we found that dogs that received drugs predicted to be effective by the models experienced better clinical outcomes by every metric we analyzed (overall response rate, complete response rate, duration of complete response, patient survival times) relative to other dogs in the study and relative to historical controls.

Multimodal machine learning for modeling infant head circumference, mothers’ milk composition, and their shared environment

Links between human milk (HM) and infant development are poorly understood and often focus on individual HM components. Here we apply multi-modal predictive machine learning to study HM and head circumference (a proxy for brain development) among 1022 mother-infant dyads of the CHILD Cohort. We integrated HM data (19 oligosaccharides, 28 fatty acids, 3 hormones, 28 chemokines) with maternal and infant demographic, health, dietary and home environment data. Head circumference was significantly predictable at 3 and 12 months. Two of the most associated features were HM n3-polyunsaturated fatty acid C22:6n3 (docosahexaenoic acid, DHA; p = 9.6e−05) and maternal intake of fish (p = 4.1e−03), a key dietary source of DHA with established relationships to brain function. Thus, using a systems biology approach, we identified meaningful relationships between HM and brain development, which validates our statistical approach, gives credence to the novel associations we observed, and sets the foundation for further research with additional cohorts and HM analytes.

A hybrid multimodal machine learning model for Detecting Alzheimer's disease

Alzheimer's disease (AD) diagnosis utilizing single modality neuroimaging data has limitations. Multimodal fusion of complementary biomarkers may improve diagnostic performance. This study proposes a multimodal machine learning framework integrating magnetic resonance imaging (MRI), positron emission tomography (PET) and cerebrospinal fluid (CSF) assays for enhanced AD characterization. The model incorporates a hybrid algorithm combining enhanced Harris Hawks Optimization (HHO) algorithm referred to as ILHHO, with Kernel Extreme Learning Machine (KELM) classifier for simultaneous feature selection and classification. ILHHO enhances HHO's search efficiency by integrating iterative mapping (IM) to improve population diversity and local escaping operator (LEO) to balance exploration-exploitation. Comparative analysis with other improved HHO algorithms, classic meta-heuristic algorithms (MHAs), and state-of-the-art MHAs on IEEE CEC2014 benchmark functions indicates that ILHHO achieves superior optimization performance compared to other comparative algorithms. The synergistic ILHHO-KELM model is evaluated on 202 AD Neuroimaging Initiative (ADNI) subjects. Results demonstrate superior multimodal classification accuracy over single modalities, validating the importance of fusing heterogeneous biomarkers. MRI + PET + CSF achieves 99.2 % accuracy for AD vs. normal control (NC), outperforming conventional and proposed methods. Discriminative feature analysis provides further insights into differential AD-related neurodegeneration patterns detected by MRI and PET. The differential PET and MRI features demonstrate how the two modalities provide complementary biomarkers. The neuroanatomical relevance of selected features supports ILHHO-KELM's potential for extracting sensitive AD imaging signatures. Overall, the study showcases the advantages of capitalizing on complementary multimodal data through advanced feature learning techniques for improving AD diagnosis.

P502 Multimodal profiling of peripheral blood identifies proliferating circulating effector CD4 T cells as predictors for response to integrin alpha4 beta7-blocking therapy in patients with Inflammatory Bowel Disease

Abstract Background Despite the success of biological therapies in inflammatory bowel disease (IBD), patient management remains challenging due to a lack of therapy response predictors. Methods Here we prospectively sampled two cohorts of IBD patient cohorts receiving the anti-integrin α4β7 antibody vedolizumab. Samples were subjected to mass cytometry, single-cell RNA sequencing, single-cell V(D)J sequencing, serum proteomics, and multidimensional flow cytometry to comprehensively assess vedolizumab-induced immunological changes in the peripheral blood and their potential associations with treatment response. Results Vedolizumab induced changes in the abundance of both circulating innate and adaptive immune cell compartments and modified the T cell receptor diversity of circulating gut-homing CD4+ memory T cells. Through integration of multimodal parameters and machine learning, we identify that pretreatment activated proliferating CD4+ memory T cell abundance is associated with treatment failure, independent of clinical variables, thereby providing a reliable predictive classifier with significant implications for the personalized management of IBD patients. Conclusion Our study provides a comprehensive framework for assessing therapy response and understanding the mechanisms underlying resistance in chronic immune-mediated inflammatory diseases such as IBD. Identifying personalized treatment strategies based on individual patient characteristics, as exemplified by our "stratify to target" approach, can significantly improve IBD management.

Progressive modality-complement aggregative multitransformer for domain multi-modal neural machine translation

Domain-specific Multi-modal Neural Machine Translation (DMNMT) aims to translate domain-specific sentences from a source language to a target language by incorporating text-related visual information. Generally, domain-specific text-image data often complement each other and have the potential to collaboratively enhance the representation of domain-specific information. Unfortunately, there is a considerable modality gap between image and text in data format and semantic expression, which leads to distinctive challenges in domain-text translation tasks. Narrowing the modality gap and improving domain-aware representation are two critical challenges in DMNMT. To this end, this paper proposes a progressive modality-complement aggregative MultiTransformer, which aims to simultaneously narrow the modality gap and capture domain-specific multi-modal representation. We first adopt a bidirectional progressive cross-modal interactive strategy to effectively narrow the text-to-text, text-to-visual, and visual-to-text semantics in the multi-modal representation space by integrating visual and text information layer-by-layer. Subsequently, we introduce a modality-complement MultiTransformer based on progressive cross-modal interaction to extract the domain-related multi-modal representation, thereby enhancing machine translation performance. Experiment results on the Fashion-MMT and Multi-30k datasets are conducted, and the results show that the proposed approach outperforms the compared state-of-the-art (SOTA) methods on the En-Zh task in E-commerce domain, En-De, En-Fr and En-Cs tasks of Multi-30k in general domain. The in-depth analysis confirms the validity of the proposed modality-complement MultiTransformer and bidirectional progressive cross-modal interactive strategy for DMNMT.

Multimodal data-driven machine learning for the prediction of surface topography in end milling

Multimodal data-driven machine learning for the prediction of surface topography in end milling

Leverage manipulation and strategic disclosure: evidence from non-financial information in annual reports based on multimodal machine learning

ABSTRACT This study adopts multi-modal machine learning to deeply interpret corporate non-financial information and explore the influence of corporate leverage manipulation on non-financial information disclosure. It is found that enterprises will strategically disclose non-financial information when implementing leverage manipulation, that is, the emotion of non-financial information text is more positive, farther away and more difficult. Based on the motivation of leverage manipulation, when enterprises face greater pressure of ‘deleveraging’ policy and stronger financing constraints, enterprises are more inclined to disclose non-financial information strategically when implementing leverage manipulation. Further, the dual motives of leverage manipulation are explored deeply. For the samples with high ‘deleveraging’ policy pressure such as state-owned enterprises and high media attention, and the samples with strong financing constraints such as high short-term debt repayment pressure and high debt financing cost, the positive relationship between leverage manipulation and strategic disclosure of non-financial information is more significant.

Stress appraisal in the workplace and its associations with productivity and mood: Insights from a multimodal machine learning analysis.

Previous studies have primarily focused on predicting stress arousal, encompassing physiological, behavioral, and psychological responses to stressors, while neglecting the examination of stress appraisal. Stress appraisal involves the cognitive evaluation of a situation as stressful or non-stressful, and as a threat/pressure or a challenge/opportunity. In this study, we investigated several research questions related to the association between states of stress appraisal (i.e., boredom, eustress, coexisting eustress-distress, distress) and various factors such as stress levels, mood, productivity, physiological and behavioral responses, as well as the most effective ML algorithms and data signals for predicting stress appraisal. The results support the Yerkes-Dodson law, showing that a moderate stress level is associated with increased productivity and positive mood, while low and high levels of stress are related to decreased productivity and negative mood, with distress overpowering eustress when they coexist. Changes in stress appraisal relative to physiological and behavioral features were examined through the lenses of stress arousal, activity engagement, and performance. An XGBOOST model achieved the best prediction accuracies of stress appraisal, reaching 82.78% when combining physiological and behavioral features and 79.55% using only the physiological dataset. The small accuracy difference of 3% indicates that physiological data alone may be adequate to accurately predict stress appraisal, and the feature importance results identified electrodermal activity, skin temperature, and blood volume pulse as the most useful physiologic features. Implementing these models within work environments can serve as a foundation for designing workplace policies, practices, and stress management strategies that prioritize the promotion of eustress while reducing distress and boredom. Such efforts can foster a supportive work environment to enhance employee well-being and productivity.

Multimodal Machine Translation

In recent years, neural network machine translation, especially in the field of multimodality, has developed rapidly. It has been widely used in natural languages processing tasks such as event detection and sentiment classification. The existing multimodal neural network machine translation is mostly based on the autoencoder framework of the attention mechanism, which further integrates spatial-visual features. However, due to the ubiquitous lack of corpus and the semantic interaction between multimodalities, the quality of machine translation is difficult to guarantee. Therefore, this paper proposes a multi-modal machine translation model that integrates external linguistic knowledge. Specifically, on the encoder side, we adopt the pre-trained Bert model to be used as an additional encoder to integrate with the original text encoder and picture encoder. Under the cooperation of the three encoders, a better text representation and picture representation at the source end is generated. Besides, the decoder decodes and generates a translation based on the image and text representation of the source. To sum up, this paper studies the visual-text semantic interaction on the encoder side and the visual-text semantic interaction on the decoder side, and further improves the quality of translation by introducing external linguistic knowledge. We compared the performance of the multimodal neural network machine translation model with pre-trained Bert and other baseline models in English German translation tasks on the multi30k data sets. The results show that the model can significantly improve the quality of multimodal neural network machine translation, which also verifies the importance of integrating external knowledge and visual text semantic interaction.

Power of Multi-Modality Variables in Predicting Parkinson's Disease Progression.

Parkinson's disease (PD) is one of the most common neurodegenerative disorders. The increasing demand for high-accuracy forecasts of disease progression has led to a surge in research employing multi-modality variables for prediction. In this review, we selected articles published from 2016 through June 2024, adhering strictly to our exclusion-inclusion criteria. These articles employed a minimum of two types of variables, including clinical, genetic, biomarker, and neuroimaging modalities. We conducted a comprehensive review and discussion on the application of multi-modality approaches in predicting PD progression. The predictive mechanisms, advantages, and shortcomings of relevant key modalities in predicting PD progression are discussed in the paper. The findings suggest that integrating multiple modalities resulted in more accurate predictions compared to those of fewer modalities in similar conditions. Furthermore, we identified some limitations in the existing field. Future studies that harness advancements in multi-modality variables and machine learning algorithms can mitigate these limitations and enhance predictive accuracy in PD progression.

ISCA: Intelligent Sense-Compute Adaptive Co-optimization of Multimodal Machine Learning Kernels for Resilient mHealth Services on Wearables

ISCA: Intelligent Sense-Compute Adaptive Co-optimization of Multimodal Machine Learning Kernels for Resilient mHealth Services on Wearables

Graph Attention-Based Fusion of Pathology Images and Gene Expression for Prediction of Cancer Survival.

Multimodal machine learning models are being developed to analyze pathology images and other modalities, such as gene expression, to gain clinical and biological insights. However, most frameworks for multimodal data fusion do not fully account for the interactions between different modalities. Here, we present an attention-based fusion architecture that integrates a graph representation of pathology images with gene expression data and concomitantly learns from the fused information to predict patient-specific survival. In our approach, pathology images are represented as undirected graphs, and their embeddings are combined with embeddings of gene expression signatures using an attention mechanism to stratify tumors by patient survival. We show that our framework improves the survival prediction of human non-small cell lung cancers, outperforming existing state-of-the-art approaches that leverage multimodal data. Our framework can facilitate spatial molecular profiling to identify tumor heterogeneity using pathology images and gene expression data, complementing results obtained from more expensive spatial transcriptomic and proteomic technologies.

Multimodal Machine Learning-Based Ductal Carcinoma in situ Prediction from Breast Fibromatosis.

To develop a clinical-radiomics model using a multimodal machine learning method for distinguishing ductal carcinoma in situ (DCIS) from breast fibromatosis. The clinical factors, ultrasound features, and related ultrasound images of 306 patients (198 DCIS patients) were retrospectively collected. Patients in the development and validation cohort were 184 and 122, respectively. The independent clinical and ultrasound factors identified by the multivariable logistic regression analysis were used for the clinical-ultrasound model construction. Then, the region of interest of breast lesions was delineated and radiomics features were extracted. Six machine learning algorithms were trained to develop a radiomics model. The algorithm with higher and more stable prediction ability was chosen to convert the output of the results into the Radscore. Further, the independent clinical predictors and Radscore were enrolled into the logistic regression analysis to generate a combined clinical-radiomics model. The receiver operating characteristic curve analysis, DeLong test, and decision curve analysis were adopted to compare the prediction ability and clinical efficacy of three different models. Among the six classifiers, logistic regression model was selected as the final radiomics model. Besides, the combined clinical-radiomics model exhibited a superior ability in distinguishing DCIS from breast fibromatosis to the clinical-ultrasound model and the radiomics model. The combined model by integrating clinical-ultrasound factors and radiomics features performed well in predicting DCIS, which might promote prompt interventions to improve the early diagnosis and prognosis of the patients.

Predicting disease severity in multiple sclerosis using multimodal data and machine learning.

Multiple sclerosis patients would benefit from machine learning algorithms that integrates clinical, imaging and multimodal biomarkers to define the risk of disease activity. We have analysed a prospective multi-centric cohort of 322 MS patients and 98 healthy controls from four MS centres, collecting disability scales at baseline and 2years later. Imaging data included brain MRI and optical coherence tomography, and omics included genotyping, cytomics and phosphoproteomic data from peripheral blood mononuclear cells. Predictors of clinical outcomes were searched using Random Forest algorithms. Assessment of the algorithm performance was conducted in an independent prospective cohort of 271 MS patients from a single centre. We found algorithms for predicting confirmed disability accumulation for the different scales, no evidence of disease activity (NEDA), onset of immunotherapy and the escalation from low- to high-efficacy therapy with intermediate to high-accuracy. This accuracy was achieved for most of the predictors using clinical data alone or in combination with imaging data. Still, in some cases, the addition of omics data slightly increased algorithm performance. Accuracies were comparable in both cohorts. Combining clinical, imaging and omics data with machine learning helps identify MS patients at risk of disability worsening.

DEVELOPMENT OF AN AI-POWERED EMOTIONALLY INTELLIGENT, INTERACTIVE AGENT PROTOTYPE TO SUPPORT INFORMAL CAREGIVERS

Abstract Long-term informal caregiving for older adults with dementia is exhausting and poses risks to caregivers’ health outcomes. Most informal caregivers are not well prepared for their new role, facing various challenges and receiving limited support. With the advent of artificial intelligence (AI) and information and communication technologies (ICTs), assistive technology is increasingly useful for supporting caregivers. However, limited research assesses the potential for AI to facilitate a privacy-preserving computation environment to provide proactive assistance to caregivers. This project aims to fill this gap by leveraging the tremendous potential of Multimodal Machine Learning and Large Language Models (LLMs) to develop a prototype of an AI-powered emotionally intelligent, interactive agent for informal caregivers. We introduce an effective context-driven instruction tuning technique that enables LLMs to finetune various caregiving-related tasks (tips generation, answering questions, elaborating a specific training module) with instructions. The project will also deliver a first-of-its-kind custom dataset augmented with manualized instructions for relevant caregiving tasks, which is used for finetuning several state-of-the-art LLM models. Our intelligent interactive agent prototype currently has the ability to evaluate caregivers’ unique affective states at query times to provide personalized assistance, addressing various types of caregivers’ situation-specific queries (e.g., managing agitation, wandering, or medications). The prototype is capable of understanding and responding to the user’s natural language queries and designing short, machine-driven persuasive conversations on a pre-defined set of topics. Our prototype has the potential to provide individualized and contextualized support for caregiving tasks which in turn may greatly reduce caregiver burden.

MissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis

Abstract When applying multimodal machine learning in downstream inference, both joint and coordinated multimodal representations rely on the complete presence of modalities as in training. However, modal-incomplete data, where certain modalities are missing, greatly reduces performance in Multimodal Sentiment Analysis (MSA) due to varying input forms and semantic information deficiencies. This limits the applicability of the predominant MSA methods in the real world, where the completeness of multimodal data is uncertain and variable. The generation-based methods attempt to generate the missing modality, yet they require complex hierarchical architecture with huge computational costs and struggle with the representation gaps across different modalities. Diversely, we propose a novel representation learning approach named MissModal, devoting to increasing robustness to missing modality in a classification approach. Specifically, we adopt constraints with geometric contrastive loss, distribution distance loss, and sentiment semantic loss to align the representations of modal-missing and modal-complete data, without impacting the sentiment inference for the complete modalities. Furthermore, we do not demand any changes in the multimodal fusion stage, highlighting the generality of our method in other multimodal learning systems. Extensive experiments demonstrate that the proposed method achieves superior performance with minimal computational costs in various missing modalities scenarios (flexibility), including severely missing modality (efficiency) on two public MSA datasets.

Prediction of anxious depression using multimodal neuroimaging and machine learning

Anxious depression is a common subtype of major depressive disorder (MDD) associated with adverse outcomes and severely impaired social function. It is important to clarify the underlying neurobiology of anxious depression to refine the diagnosis and stratify patients for therapy. Here we explored associations between anxiety and brain structure/function in MDD patients. A total of 260 MDD patients and 127 healthy controls underwent three-dimensional T1-weighted structural scanning and resting-state functional magnetic resonance imaging. Demographic data were collected from all participants. Differences in gray matter volume (GMV), (fractional) amplitude of low-frequency fluctuation ((f)ALFF), regional homogeneity (ReHo), and seed point-based functional connectivity were compared between anxious MDD patients, non-anxious MDD patients, and healthy controls. A random forest model was used to predict anxiety in MDD patients using neuroimaging features. Anxious MDD patients showed significant differences in GMV in the left middle temporal gyrus and ReHo in the right superior parietal gyrus and the left precuneus than HCs. Compared with non-anxious MDD patients, patients with anxious MDD showed significantly different GMV in the left inferior temporal gyrus, left superior temporal gyrus, left superior frontal gyrus (orbital part), and left dorsolateral superior frontal gyrus; fALFF in the left middle temporal gyrus; ReHo in the inferior temporal gyrus and the superior frontal gyrus (orbital part); and functional connectivity between the left superior temporal gyrus(temporal pole) and left medial superior frontal gyrus. A diagnostic predictive random forest model built using imaging features and validated by 10-fold cross-validation distinguished anxious from non-anxious MDD with an AUC of 0.802. Patients with anxious depression exhibit dysregulation of brain regions associated with emotion regulation, cognition, and decision-making, and our diagnostic model paves the way for more accurate, objective clinical diagnosis of anxious depression.

Multi-Modal Feature Integration in Machine Learning Predictions for Cardiovascular Diseases

Early detection and prevention of cardiovascular illnesses rely heavily on phonocardiogram (PCG) and electrocardiogram (ECG). A novel multi-modal machine learning strategy based on ECG and PCG data is presented in this work for predicting cardiovascular diseases (CVD). ECG and PCG features are combined for optimal feature subset selection using a genetic algorithm (GA). Then, machine learning classifiers are implemented to do the classification of abnormal and normal signals

Explainable stress type classification captures physiologically relevant responses in the Maastricht Acute Stress Test.

Current stress detection methods concentrate on identification of stress and non-stress states despite the existence of various stress types. The present study performs a more specific, explainable stress classification, which could provide valuable information on the physiological stress reactions. Physiological responses were measured in the Maastricht Acute Stress Test (MAST), comprising alternating trials of cold pressor (inducing physiological stress and pain) and mental arithmetics (eliciting cognitive and social-evaluative stress). The responses in these subtasks were compared to each other and to the baseline through mixed model analysis. Subsequently, stress type detection was conducted with a comprehensive analysis of several machine learning components affecting classification. Finally, explainable artificial intelligence (XAI) methods were applied to analyze the influence of physiological features on model behavior. Most of the investigated physiological reactions were specific to the stressors, and the subtasks could be distinguished from baseline with up to 86.5% balanced accuracy. The choice of the physiological signals to measure (up to 25%-point difference in balanced accuracy) and the selection of features (up to 7%-point difference) were the two key components in classification. Reflection of the XAI analysis to mixed model results and human physiology revealed that the stress detection model concentrated on physiological features relevant for the two stressors. The findings confirm that multimodal machine learning classification can detect different types of stress reactions from baseline while focusing on physiologically sensible changes. Since the measured signals and feature selection affected classification performance the most, data analytic choices left limited input information uncompensated.

Precision Health Monitoring: Exploring the Fusion of Wearable IoT Sensors, Multimodal Data, and ML

The current state of health detection in IoT sensors is affected by several limitations. Existing systems often struggle with inaccurate and unreliable readings, leading to potential misdiagnoses and patient discomfort. These systems also tend to lack adaptability and robustness when dealing with various modalities of data, hindering their overall effectiveness. Furthermore, the reliance on traditional algorithms in the absence of machine learning hampers their ability to provide precise and real-time heartbeat information. In light of these shortcomings, this work seeks to address these issues by studying the enhanced approach, emphasizing the integration of multimodal data fusion techniques and machine learning algorithms. The aim is to identify the drawbacks associated with existing systems and provide the more accurate and responsive solution for heath detection offered by IoT sensors through the application of data fusion and machine learning.