Multimodal Machine Research Articles

CLIP-enhanced multimodal machine translation: integrating visual and label features with transformer fusion

CLIP-enhanced multimodal machine translation: integrating visual and label features with transformer fusion

Integrative Single-Plaque Analysis Reveals Signature Aβ and Lipid Profiles in the Alzheimer's Brain.

Cerebral accumulation of amyloid-β (Aβ) initiates molecular and cellular cascades that lead to Alzheimer's disease (AD). However, amyloid deposition does not invariably lead to dementia. Amyloid-positive but cognitively unaffected (AP-CU) individuals present widespread amyloid pathology, suggesting that molecular signatures more complex than the total amyloid burden are required to better differentiate AD from AP-CU cases. Motivated by the essential role of Aβ and the key lipid involvement in AD pathogenesis, we applied multimodal mass spectrometry imaging (MSI) and machine learning (ML) to investigate amyloid plaque heterogeneity, regarding Aβ and lipid composition, in AP-CU versus AD brain samples at the single-plaque level. Instead of focusing on a population mean, our analytical approach allowed the investigation of large populations of plaques at the single-plaque level. We found that different (sub)populations of amyloid plaques, differing in Aβ and lipid composition, coexist in the brain samples studied. The integration of MSI data with ML-based feature extraction further revealed that plaque-associated gangliosides GM2 and GM1, as well as Aβ1-38, but not Aβ1-42, are relevant differentiators between the investigated pathologies. The pinpointed differences may guide further fundamental research investigating the role of amyloid plaque heterogeneity in AD pathogenesis/progression and may provide molecular clues for further development of emerging immunotherapies to effectively target toxic amyloid assemblies in AD therapy. Our study exemplifies how an integrative analytical strategy facilitates the unraveling of complex biochemical phenomena, advancing our understanding of AD from an analytical perspective and offering potential avenues for the refinement of diagnostic tools.

Follower attraction in live streaming: Knowledge driven by PKM and data driven by EF-LSTM

In this study, we examine variables affecting follower attraction and determine the optimal model for its prediction. We identify 12 variables from the persuasion attempt perspective in the persuasion knowledge model and accordingly perform two progressive studies (study 1: knowledge driven, with coarse-grained variables, with unimodal models, without data fusion; study 2: knowledge-and-data driven, with fine-grained features, with multimodal models, with data fusion). In study 1, we compute 12 variables in accordance with hypotheses from nonfused trimodal data (verbal, vocal, and visual). XGBoost and Shapley additive explanations are used to evaluate the importance of these 12 variables, and economic models are adopted to test their statistical significance. In study 2, we reduce the data granularity of the 12 coarse-grained variables by generating 323 fine-grained features. We then process trimodal data fusion to consider intra- and intermodal interactions. Long Short-Term Memory (LSTM) models are applied to learn a mapping from the input sequences of trimodality to output values of follower increment. The results demonstrate that early fusion LSTM with fused trimodal features has a 66.3% prediction accuracy, outperforming LSTM with unimodal features and late fusion LSTM with nonfused trimodal features. To our knowledge, these results are among the first to unfold follower attraction strategies with unstructured video data, and this is the first study to provide knowledge-and-data-driven multimodal machine learning models for predicting follower attraction at different granularity levels.

A clinical-radiomic-pathomic model for prognosis prediction in patients with hepatocellular carcinoma after radical resection.

Radical surgery, the first-line treatment for patients with hepatocellular cancer (HCC), faces the dilemma of high early recurrence rates and the inability to predict effectively. We aim to develop and validate a multimodal model combining clinical, radiomics, and pathomics features to predict the risk of early recurrence. We recruited HCC patients who underwent radical surgery and collected their preoperative clinical information, enhanced computed tomography (CT) images, and whole slide images (WSI) of hematoxylin and eosin (H & E) stained biopsy sections. After feature screening analysis, independent clinical, radiomics, and pathomics features closely associated with early recurrence were identified. Next, we built 16 models using four combination data composed of three type features, four machine learning algorithms, and 5-fold cross-validation to assess the performance and predictive power of the comparative models. Between January 2016 and December 2020, we recruited 107 HCC patients, of whom 45.8% (49/107) experienced early recurrence. After analysis, we identified two clinical features, two radiomics features, and three pathomics features associated with early recurrence. Multimodal machine learning models showed better predictive performance than bimodal models. Moreover, the SVM algorithm showed the best prediction results among the multimodal models. The average area under the curve (AUC), accuracy (ACC), sensitivity, and specificity were 0.863, 0.784, 0.731, and 0.826, respectively. Finally, we constructed a comprehensive nomogram using clinical features, a radiomics score and a pathomics score to provide a reference for predicting the risk of early recurrence. The multimodal models can be used as a primary tool for oncologists to predict the risk of early recurrence after radical HCC surgery, which will help optimize and personalize treatment strategies.

A multimodal machine learning approach to generate news articles from geo-tagged images

Classical machine learning algorithms typically operate on unimodal data and hence it can analyze and make predictions based on data from a single source (modality). Whereas multimodal machine learning algorithm, learns from information across multiple modalities, such as text, images, audio, and sensor data. The paper leverages the functionalities of multimodal machine learning (ML) application for generating text from images. The proposed work presents an innovative multimodal algorithm that automates the creation of news articles from geo-tagged images by leveraging cutting-edge developments in machine learning, image captioning, and advanced text generation technologies. Employing a multimodal approach that integrates machine learning and transformer algorithms, such as visual geometry group network16 (VGGNet16), convolutional neural network (CNN) and a long short-term memory (LSTM) based system, the algorithm initiates by extracting the location from exchangeable image file format (Exif) data from the image. The features are extracted from the image and corresponding news headline is generated. The headlines are used for generating a comprehensive article with contemporary large language model (LLM). Further, the algorithm generates the news article big-science large open-science open-access multilingual language model (BLOOM). The algorithm was tested on real time photographs as well as images from the internet. In both the cases the news articles generated were validated with ROUGE and BULE score. The proposed work is found to be successful attempt in journalism field.

Anhedonia as a Potential Transdiagnostic Phenotype With Immune-Related Changes in Recent-Onset Mental Health Disorders

BackgroundChronic low-grade inflammation is observed across mental disorders and is associated with difficult-to-treat-symptoms of anhedonia and functional brain changes, reflecting a potential transdiagnostic dimension. Previous investigations have focused on distinct illness categories in people with enduring illness, but few have explored inflammatory changes. We sought to identify an inflammatory signal and the associated brain function underlying anhedonia among young people with recent-onset psychosis and recent-onset depression. MethodsResting-state functional magnetic resonance imaging, inflammatory markers, and anhedonia symptoms were collected from 108 (mean [SD] age = 26.2 [6.2] years; female = 50) participants with recent-onset psychosis (n = 53) and recent-onset depression (n = 55) from the European Union/Seventh Framework Programme–funded PRONIA (Personalised Prognostic Tools for Early Psychosis Management) study. Time series were extracted using the Schaefer atlas, defining 100 cortical regions of interest. Using advanced multimodal machine learning, an inflammatory marker model and a functional connectivity model were developed to classify participants into an anhedonic group or a normal hedonic group. ResultsA repeated nested cross-validation model using inflammatory markers classified normal hedonic and anhedonic recent-onset psychosis/recent-onset depression groups with a balanced accuracy of 63.9% and an area under the curve of 0.61. The functional connectivity model produced a balanced accuracy of 55.2% and an area under the curve of 0.57. Anhedonic group assignment was driven by higher levels of interleukin 6, S100B, and interleukin 1 receptor antagonist and lower levels of interferon gamma, in addition to connectivity within the precuneus and posterior cingulate. ConclusionsWe identified a potential transdiagnostic anhedonic subtype that was accounted for by an inflammatory profile and functional connectivity. Results have implications for anhedonia as an emerging transdiagnostic target across emerging mental disorders.

Multiscale knowledge distillation with attention based fusion for robust human activity recognition

Knowledge distillation is an effective approach for training robust multi-modal machine learning models when synchronous multimodal data are unavailable. However, traditional knowledge distillation techniques have limitations in comprehensively transferring knowledge across modalities and models. This paper proposes a multiscale knowledge distillation framework to address these limitations. Specifically, we introduce a multiscale semantic graph mapping (SGM) loss function to enable more comprehensive knowledge transfer between teacher and student networks at multiple feature scales. We also design a fusion and tuning (FT) module to fully utilize correlations within and between different data types of the same modality when training teacher networks. Furthermore, we adopt transformer-based backbones to improve feature learning compared to traditional convolutional neural networks. We apply the proposed techniques to multimodal human activity recognition and compared with the baseline method, it improved by 2.31% and 0.29% on the MMAct and UTD-MHAD datasets. Ablation studies validate the necessity of each component.

A multimodal machine learning model for predicting dementia conversion in Alzheimer’s disease

Alzheimer’s disease (AD) accounts for 60–70% of the population with dementia. Mild cognitive impairment (MCI) is a diagnostic entity defined as an intermediate stage between subjective cognitive decline and dementia, and about 10–15% of people annually convert to AD. We aimed to investigate the most robust model and modality combination by combining multi-modality image features based on demographic characteristics in six machine learning models. A total of 196 subjects were enrolled from four hospitals and the Alzheimer’s Disease Neuroimaging Initiative dataset. During the four-year follow-up period, 47 (24%) patients progressed from MCI to AD. Volumes of the regions of interest, white matter hyperintensity, and regional Standardized Uptake Value Ratio (SUVR) were analyzed using T1, T2-weighted-Fluid-Attenuated Inversion Recovery (T2-FLAIR) MRIs, and amyloid PET (αPET), along with automatically provided hippocampal occupancy scores (HOC) and Fazekas scales. As a result of testing the robustness of the model, the GBM model was the most stable, and in modality combination, model performance was further improved in the absence of T2-FLAIR image features. Our study predicts the probability of AD conversion in MCI patients, which is expected to be useful information for clinician’s early diagnosis and treatment plan design.

Bridging realities: training visuo-haptic object recognition models for robots using 3D virtual simulations

This paper proposes an approach for training visuo-haptic object recognition models for robots using synthetic datasets generated by 3D virtual simulations. In robotics, where visual object recognition has witnessed considerable progress due to an abundance of image datasets, the scarcity of diverse haptic samples has resulted in a noticeable gap in research on machine learning incorporating the haptic sense. Our proposed methodology addresses this challenge by utilizing 3D virtual simulations to create realistic synthetic datasets, offering a scalable and cost-effective solution to integrate haptic and visual cues for object recognition seamlessly. Acknowledging the importance of multimodal perception, particularly in robotic applications, our research not only closes the existing gap but envisions a future where intelligent agents possess a holistic understanding of their environment derived from both visual and haptic senses. Our experiments show that synthetic datasets can be used for training object recognition in haptic and visual modes by incorporating noise, performing some preprocessing, data augmentation, or domain adaptation. This work contributes to the advancement of multimodal machine learning toward a more nuanced and comprehensive robotic perception.

Multimodal Machine Translation Approaches for Indian Languages: A Comprehensive Survey

Multimodal machine translation (MMT) is a challenging task in the linguistically diverse Indian landscape. Machine translation refers to the task of automatically converting content from one language to another without human involvement. Within the realm of natural language processing, a significant challenge arises from the inherent ambiguity present in human language. Translation ambiguity is a cross-lingual phenomenon that can manifest itself for various reasons, including lexical ambiguity, the occasional need to impute missing words, the presence of gen-der ambiguity, and word-sense ambiguities. These factors can lead to a decrease in translation accuracy. The integration of multiple modalities, such as images, videos, and audio, in addition to text, plays a pivotal role in improving the robustness and precision of translation systems. Over the past five years, extensive research has been dedicated to incorporating secondary modalities alongside text to improve language translation and comprehension. In this comprehensive study, our objective was to identify and explore promising MMT approaches, available corpora, eval-uation metrics, research challenges, and the future direction of research specifically for Indian languages. We evaluated 81 papers, including MMT models, MMT dataset in Indian languages, survey on MMT approach, and the effects of multiple modalities in machine translation. The performance of the different proposed approaches has also been briefly analyzed on the basis of the claimed results and comparative evaluations. Finally, the challenges associated with the MMT task for India and some possible directions for future research in this domain are highlighted.

Multimodal fusion of brain signals for robust prediction of psychosis transition

The prospective study of youths at clinical high risk (CHR) for psychosis, including neuroimaging, can identify neural signatures predictive of psychosis outcomes using algorithms that integrate complex information. Here, to identify risk and psychosis conversion, we implemented multiple kernel learning (MKL), a multimodal machine learning approach allowing patterns from each modality to inform each other. Baseline multimodal scans (n = 74, 11 converters) included structural, resting-state functional imaging, and diffusion-weighted data. Multimodal MKL outperformed unimodal models (AUC = 0.73 vs. 0.66 in predicting conversion). Moreover, patterns learned by MKL were robust to training set variations, suggesting it can identify cross-modality redundancies and synergies to stabilize the predictive pattern. We identified many predictors consistent with the literature, including frontal cortices, cingulate, thalamus, and striatum. This highlights the advantage of methods that leverage the complex pathophysiology of psychosis.

Using Multimodal Methods and Machine Learning to Recognize Mental Workload: Distinguishing Between Underload, Moderate Load, and Overload

Mental workload recognition is of great significance in preventing human errors and accidents. This study constructed a multimodal recognition scheme to recognize three mental workload states: underload, moderate load, and overload. Based on driving scenarios, these three states were induced in this study by changing the driving modes and situations. Multimodal recognition of underload, moderate load, and overload was performed using electroencephalography (EEG), electrocardiography (ECG), and pupillometry. In addition, various machine learning methods were used to evaluate the recognition performance of different feature combinations. The results showed that the random forest method, trained using spectral power, pupil diameter, and heart rate variability, achieved the highest recognition accuracy of 83.13% for the three mental workload states. This study provides valuable reference information for multimodal recognition of mental workload states.

Comparative analysis of machine learning algorithms for Alzheimer's disease classification using EEG signals and genetic information

Alzheimer's disease (AD) is a progressive neurodegenerative disorder characterized by cognitive decline, memory impairments, and behavioral changes. The presence of abnormal beta-amyloid plaques and tau protein tangles in the brain is known to be associated with AD. However, current limitations of imaging technology hinder the direct detection of these substances. Consequently, researchers are exploring alternative approaches, such as indirect assessments involving monitoring brain signals, cognitive decline levels, and blood biomarkers. Recent studies have highlighted the potential of integrating genetic information into these approaches to enhance early detection and diagnosis, offering a more comprehensive understanding of AD pathology beyond the constraints of existing imaging methods. Our study utilized electroencephalography (EEG) signals, genotypes, and polygenic risk scores (PRSs) as features for machine learning models. We compared the performance of gradient boosting (XGB), random forest (RF), and support vector machine (SVM) to determine the optimal model. Statistical analysis revealed significant correlations between EEG signals and clinical manifestations, demonstrating the ability to distinguish the complexity of AD from other diseases by using genetic information. By integrating EEG with genetic data in an SVM model, we achieved exceptional classification performance, with an accuracy of 0.920 and an area under the curve of 0.916. This study presents a novel approach of utilizing real-time EEG data and genetic background information for multimodal machine learning. The experimental results validate the effectiveness of this concept, providing deeper insights into the actual condition of patients with AD and overcoming the limitations associated with single-oriented data.

Sanskrit to Hindi language translation using multimodal neural machine translation

Machine translation (MT) is a subfield of computer features that focuses on the automatic translation from one natural language into another without any human involvement. Due to native people interacting in a variety of languages, there is a great need for translating information between languages to send and communicate thoughts. However, they disregard the significance of semantic data encoded in the text features. In this paper, multimodal neural machine translation (MNMT) is proposed for Sanskrit-Hindi translation. The main goal of the proposed method is to fully utilize semantic text features on NMT architecture and to minimize testing and training time. The MNMT is validated on two different NMT architectures: recurrent neural network (RNN) and self-attention network (SAN). The MNMT method’s efficacy is demonstrated by employing the dataset of Sanskrit-Hindi Corpora. Extensive experimental outcomes represent the proposed method’s enhancement over baselines on both architectures. The existing methods, namely, English-to-Indian MT system, Sanskrit-Hindi MT system, and hybrid MT system are used to justify the efficacy of the MNMT method. When compared to the above-mentioned existing methods, RA-RNN respectively achieves a superior BLEU and METEOR of 80.5% and 75.3%, while the RA-SAN respectively achieves a superior BLEU and METEOR of 78.2% and 77.1%.

Bridging Modalities: A Multimodal Machine Learning Approach for Parkinson’s Disease Diagnosis Using EEG and MRI Data

Parkinson’s disease (PD) is a slowly progressing neurological disorder with symptoms that overlap with those of other conditions, making early detection and accurate diagnosis vital for effective treatment and a patient’s quality of life. Symptoms such as tremors, stiffness, slow movements, and balance issues, along with psychiatric manifestations, are typical of PD. This study introduces a groundbreaking approach to PD diagnosis, utilizing a multimodal machine learning framework that integrates Electroencephalography (EEG) and Magnetic Resonance Imaging (MRI) data. Focusing on the early detection and accurate classification of PD, the proposed research leverages the distinct yet complementary nature of EEG and MRI datasets to enhance diagnostic precision. We employed a robust algorithmic strategy, including LightGBM and machine learning techniques, to analyze the complex patterns inherent in neurological data. The key steps of the proposed research are preprocessing and feature extraction from both EEG and MRI modalities, followed by their fusion using Principal Component Analysis (PCA) for dimensionality reduction. The fused dataset was then analyzed using a LightGBM model and validated through a 10-fold cross-validation process to ensure reliability and stability. The model’s efficacy was further tested on independent datasets, demonstrating its robustness across diverse patient demographics. The obtained results showcased an accuracy of 97.17%, sensitivity of 96.58%, and specificity of 96.82% in PD classification, outperforming traditional multimodal as well as single-modality diagnostic methods. The integration of EEG and MRI data provided a more comprehensive view of the neurophysiological and neuroanatomical changes associated with PD. Additionally, the use of advanced machine learning algorithms allowed for a nuanced analysis, capturing subtle patterns indicative of early PD stages.

Dose multimodal machine translation can improve translation performance?

Dose multimodal machine translation can improve translation performance?

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

Machine learning (ML) applications in medical artificial intelligence (AI) systems have shifted from traditional and statistical methods to increasing application of deep learning models. This survey navigates the current landscape of multimodal ML, focusing on its profound impact on medical image analysis and clinical decision support systems. Emphasizing challenges and innovations in addressing multimodal representation, fusion, translation, alignment, and co-learning, the paper explores the transformative potential of multimodal models for clinical predictions. It also highlights the need for principled assessments and practical implementation of such models, bringing attention to the dynamics between decision support systems and healthcare providers and personnel. Despite advancements, challenges such as data biases and the scarcity of “big data” in many biomedical domains persist. We conclude with a discussion on principled innovation and collaborative efforts to further the mission of seamless integration of multimodal ML models into biomedical practice.

1005 Improved Sleep Apnea Screening After Stroke Using Multimodal Wearable Sensors and Machine Learning

1005 Improved Sleep Apnea Screening After Stroke Using Multimodal Wearable Sensors and Machine Learning

Radiomic- and dosiomic-based clustering development for radio-induced neurotoxicity in pediatric medulloblastoma

BackgroundTexture analysis extracts many quantitative image features, offering a valuable, cost-effective, and non-invasive approach for individual medicine. Furthermore, multimodal machine learning could have a large impact for precision medicine, as texture biomarkers can underlie tissue microstructure. This study aims to investigate imaging-based biomarkers of radio-induced neurotoxicity in pediatric patients with metastatic medulloblastoma, using radiomic and dosiomic analysis.MethodsThis single-center study retrospectively enrolled children diagnosed with metastatic medulloblastoma (MB) and treated with hyperfractionated craniospinal irradiation (CSI). Histological confirmation of medulloblastoma and baseline follow-up magnetic resonance imaging (MRI) were mandatory. Treatment involved helical tomotherapy (HT) delivering a dose of 39 Gray (Gy) to brain and spinal axis and a posterior fossa boost up to 60 Gy. Clinical outcomes, such as local and distant brain control and neurotoxicity, were recorded. Radiomic and dosiomic features were extracted from tumor regions on T1, T2, FLAIR (fluid-attenuated inversion recovery) MRI-maps, and radiotherapy dose distribution. Different machine learning feature selection and reduction approaches were performed for supervised and unsupervised clustering.ResultsForty-eight metastatic medulloblastoma patients (29 males and 19 females) with a mean age of 12 ± 6 years were enrolled. For each patient, 332 features were extracted. Greater level of abstraction of input data by combining selection of most performing features and dimensionality reduction returns the best performance. The resulting one-component radiomic signature yielded an accuracy of 0.73 with sensitivity, specificity, and precision of 0.83, 0.64, and 0.68, respectively.ConclusionsMachine learning radiomic-dosiomic approach effectively stratified pediatric medulloblastoma patients who experienced radio-induced neurotoxicity. Strategy needs further validation in external dataset for its potential clinical use in ab initio management paradigms of medulloblastoma.

Unsupervised Multimodal Machine Translation for Low-resource Distant Language Pairs

Unsupervised machine translation (UMT) has recently attracted more attention from researchers, enabling models to translate when languages lack parallel corpora. However, the current works mainly consider close language pairs (e.g., English-German and English-French), and the effectiveness of visual content for distant language pairs has yet to be investigated. This article proposes an unsupervised multimodal machine translation model for low-resource distant language pairs. Specifically, we first employ adequate measures such as transliteration and re-ordering to bring distant language pairs closer together. We then use visual content to extend masked language modeling and generate visual masked language modeling for UMT. Finally, empirical experiments are conducted on our distant language pair dataset and the public Multi30k dataset. Experimental results demonstrate the superior performance of our model, with BLEU score improvements of 2.5 and 2.6 on translation for distant language pairs English-Uyghur and Chinese-Uyghur. Moreover, our model also brings remarkable results for close language pairs, improving 2.3 BLEU compared with the existing models in English-German.