Multimodal Model Research Articles

QOL-18. MANAGING CANCER AND LIVING MEANINGFULLY (CALM) IN PRIMARY AND SECONDARY MALIGNANT BRAIN TUMOR PATIENTS: INTERIM ANALYSIS OF A PHASE IIC RANDOMIZED CONTROL PILOT TRIAL

Abstract BACKGROUND Patients with both primary and secondary brain tumors demonstrate substantial psychological distress but have been historically excluded from psychological research. Managing Cancer and Living Meaningfully (CALM), a supportive- expressive- psychotherapy, has proven to reduce depression and death-anxiety in advanced cancer patients but trials in neuro-oncology have been limited. Interim analyses (50% accrual) of the first randomized control pilot trial (RCT) of CALM in neuro-oncology are presented here. METHODS In this ongoing NIH ORBIT model Phase IIc pilot RCT, 60 adults with primary or secondary malignant brain tumors and heightened distress (PHQ/DADDS) are randomized (2:1) to a six-session CALM intervention or usual care. Data collection includes established feasibility and acceptability metrics, intervention satisfaction surveys and interviews, and clinically meaningful changes in distress severity scores across three timepoints (pre-, post-CALM, follow-up). RESULTS Of 64 patients screened, 36 were eligible and 32 enrolled (n=20 CALM, n=12 Control; 65% female; 90% White; Mage=46yrs). A multimodal recruitment model was used. Currently, assessment completion rates are: post-CALM (80%) and follow-up (67%). Reasons for withdrawal include disinterest in intervention topics (n=1) and lost to follow-up (n=1). Intervention retention is currently at 90%, with 8 participants still in active treatment. No adverse events have been documented. Perceived benefit appears high (4.1/5) and all participants report they would recommend CALM to others. Pre- to post-CALM, depression severity reductions appear equal across arms (CALM PHQ-9=30%; Control=33%) but death anxiety severity reductions appear more prevalent in the treatment group (CALM DADDS=70%; Control=50%). CONCLUSIONS Interim data suggests promising feasibility and acceptability results in this pilot RCT. Enrollment, retention, and assessment completion rates appear adequate. Participants are reporting high benefit and recommendation of CALM. Improvement in death anxiety appears greater in those treated with CALM compared to controls. These interim RCT findings are promising and suggest completion of the current phase.

Open-world disaster information identification from multimodal social media

The application of multimodal deep learning for emergency response and recovery, specifically in disaster social media analysis, is of utmost importance. It is worth noting that in real-world scenarios, sudden disaster events may differ from the training data, which may require the multimodal network to predict them as unknown classes instead of misclassifying them to known ones. Previous studies have primarily focused on model accuracy in a closed environment and may not be able to directly detect unknown classes. Thus, we propose a novel multimodal model for categorizing social media related to disasters in an open-world environment. Our methodology entails utilizing pre-trained unimodal models as encoders for each modality and performing information fusion with a cross-attention module to obtain the joint representation. For open-world detection, we use a multitask classifier that encompasses both a closed-world and an open-world classifier. The closed-world classifier is trained on the original data to classify known classes, whereas the open-world classifier is used to determine whether the input belongs to a known class. Furthermore, we propose a sample generation strategy that models the distribution of unknown samples using known data, which allows the open-world classifier to identify unknown samples. Our experiments were conducted on two public datasets, namely CrisisMMD and MHII. According to the experimental results, the proposed method outperforms other baselines and approaches in crisis information classification.

Comparison of the Robustness of Multimodal Models and Unimodal Models under Text-based Adversarial Attacks

Abstract. With the popularization of artificial intelligence technology, adversarial attacks have become a major challenge in the field of machine learning. This paper explores the robustness of multimodal and unimodal models under textual adversarial attacks, and probes to understand their differences and commonalities. By comparing and analyzing the performance of the CLIP multimodal model and BERT unimodal model under different textual datasets, it is pointed out that the multimodal model does not perform better than the unimodal model under unimodal adversarial attack when the multimodal fusion advantage cannot be reflected. On the contrary, the CLIP model, which is a multimodal model, exhibits larger robustness fluctuations similar to the BERT model under single-modal adversarial attacks. The advantages of multimodal models do not automatically translate into better robustness in all scenarios but need to be optimized for specific tasks and adversarial strategies, and the multimodal models do not have better accuracy than the unimodal models without task-specific pre-training. Both exhibit significant robustness fluctuations in the face of textual adversarial attacks. The research in this paper provides research value and further research directions for future studies

Damage‐level classification considering both correlation between image and text data and confidence of attention map

AbstractIn damage‐level classification, deep learning. models are more likely to focus on regions unrelated to classification targets because of the complexities inherent in real data, such as the diversity of damages (e.g., crack, efflorescence, and corrosion). This causes performance degradation. To solve this problem, it is necessary to handle data complexity and uncertainty. This study proposes a multimodal deep learning model that can focus on damaged regions using text data related to damage in images, such as materials and components. Furthermore, by adjusting the effect of attention maps on damage‐level classification performance based on the confidence calculated when estimating these maps, the proposed method realizes an accurate damage‐level classification. Our contribution is the development of a model with an end‐to‐end multimodal attention mechanism that can simultaneously consider both text and image data and the confidence of the attention map. Finally, experiments using real images validate the effectiveness of the proposed method.

Performance prediction applying different reduced turbulence models to the SMART tokamak

Abstract The SMall Aspect Ratio Tokamak (SMART) is currently being commissioned at the University of Seville and will be able to compare the performance of positive and negative triangularity plasmas at low aspect ratio. Predictive simulations have been performed for different machine scenarios and heating schemes using the TRANSP code. The objectives of these simulations are to predict the parameters expected in positive triangularity plasmas, to guide diagnostic development, and to validate transport models. Several reduced turbulence models have been used to predict electron and ion temperatures for the operational phase 2. All models provide similar results from approximately mid-radius to the separatrix but important discrepancies are found in the core region. These positive triangularity results are compared with experiments from a similar size machine like GLOBUS-M2. The multi-mode model (MMM) shows the best agreement. Simulations with different boundary conditions have been performed and no strong differences have been observed between them. The impact of neutral beam injection (NBI) on the predicted profiles has also been addressed. Rotation reduces turbulence levels so higher temperatures are achieved when included in the simulations. Studying the different contributions to the thermal diffusivities, it is observed that electron temperature gradient (ETG) turbulence dominates at the plasma core while micro-tearing modes (MTM) dominate at the edge in the electron channel. In the ion channel, the neoclassical contribution is dominant at the core and at the very edge while the Weiland component, which includes ion temperature gradient mode (ITG), trapped electron mode (TEM), kinetic ballooning mode (KBM), peeling mode (PM) and collisionless and collision dominated magnetohydrodynamic (MHD) modes governs the mid-radius region. For phase 3, two plasmas with different electron densities have been studied. The case with lower density matches well a specific discharge of GLOBUS-M2. The higher density plasma shows high performance with β N ≈ 3.8 .

Unveiling the distinctive variations in multi-omics triggered by TP53 mutation in lung cancer subtypes: An insight from interaction among intratumoral microbiota, tumor microenvironment, and pathology

The TP53 mutation is one of the most common gene mutations in non-small cell lung cancer (NSCLC) and plays a significant role in the occurrence, development, and prognosis of both lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC). Recent studies have also suggested the predictive value of TP53 mutations in the response to immunotherapy for NSCLC. It is known that intratumoral microbiota, tumor immune microenvironment (TIME) and histology are associated with the roles of TP53 mutation in NSCLC. However, the intrinsic associations among these three factors and their underlying interaction with TP53 mutation are not well understood. Additionally, the potential of predicting TP53 mutations using deep learning methods has not yet been fully evaluated. In this paper, we comprehensively evaluated the tissue microbiome, host gene expression characteristics, and histopathological slides of 992 NSCLC patients obtained from the cancer genome atlas (TCGA) and validated the findings using multi-omics data of 332 NSCLC patients from the Clinical Proteomic Tumor Analysis Consortium (CPTAC). Compared to LUSC, LUAD exhibited more substantial differences between patients with and without TP53 mutation in all three aspects. In LUAD, our results imply underlying links between the tissue microbiome and immune cell components in the TIME, and show that the abundance of immune cells is reflected in histology slides. Furthermore, we propose a novel multimodal deep learning model that focuses on histopathology images, which achieves an area under the curve (AUC) of 0.84 in LUAD. In summary, TP53 mutation of LUAD resulted more significant changes in intratumoral microbiota, TIME and histology than that of LUSC. And histopathology images can be used to predict TP53 mutation in LUAD with reasonable accuracy.

An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer

Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (ΔAUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP’s clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.

Gut-Brain Nexus: Mapping Multi-Modal Links to Neurodegeneration at Biobank Scale.

Co-occurring disorders of the gut-brain axis combined with genetic and proteomic data can better predict neurodegenerative risk.

CSKINet: A multimodal network model integrating conceptual semantic knowledge injection for relation extraction of Chinese corporate reports

Recognizing the associations among entities in corporate reports accurately is crucial for market regulation and policy development. Nevertheless, confronted with massive corporate information, the traditional manual screening approach is cumbersome, struggling to match the demand. Consequently, we propose a multimodal network model incorporating conceptual semantic knowledge injection, CSKINet, for accurately extracting relations from Chinese corporate reports. The essential highlights in the design of the CSKINet model are the following: (1) Integrate the conceptual descriptions of corporations from external resources to construct the semantic knowledge repository of corporate concepts, which provides a solid semantic foundation for the model. (2) Multimodal features are extracted from the documents by various means and corporate conceptual knowledge is integrated into the model representation to enhance the representation capability of the model. (3) The multimodal self-attention mechanism that captures cross-modal associations and the biaffine classifier with Taylor polynomial loss function that optimizes training iterations further improve the learning efficiency and prediction accuracy. The results on the real corporate report dataset show that our proposed model can more accurately extract the relations from Chinese corporate reports compared to other baseline models, where the F1 score reaches 85.76%.

The correlation between multi-mode ultrasonographic features of breast cancer and axillary lymph node metastasis.

This study aimed to explore the correlation between multi-mode ultrasonographic features of breast cancer and axillary lymph node metastasis. A total of 196 patients with surgically confirmed breast cancer between September 2019 and December 2023 were included. Data on preoperative B-mode ultrasound (US), color Doppler, and shear wave elastography (SWE) features of breast cancer masses were collected and analyzed to determine their correlation with axillary lymph node metastasis. The area under the receiver operating characteristic curve (AUC) of B-mode US, color Doppler, SWE, and the multi-mode predictive model for evaluating axillary lymph node metastasis were compared. Among the 196 patients, 70 had positive axillary lymph nodes, while 126 had negative axillary lymph nodes. There was no significant difference in the color features between the negative and positive axillary lymph node groups. Multifocality/multicentricity, architectural distortion, microcalcifications, and the "stiff rim" sign in SWE were identified as independent risk factors to predict axillary lymph node metastasis according to binary logistic regression analysis. The AUC of the predictive model based on these independent risk factors was 0.803 (95% CI: 0.739-0.867), which was significantly higher than that of B-mode US or SWE alone. Multifocality/multicentricity, architectural distortion, microcalcifications, and the "stiff rim" sign in SWE were found to be valuable for predicting axillary lymph node metastasis in patients with breast cancer. The predictive model developed in this study, combining the multi-mode ultrasonographic features of breast cancer masses, could serve as a noninvasive and convenient method to predict axillary lymph node status. This approach could aid in clinical decision-making and individualized treatment to improve the prognosis of breast cancer patients.

Constructing multi-modal emotion recognition model based on convolutional neural network

Constructing multi-modal emotion recognition model based on convolutional neural network

Wheat growth stage identification method based on multimodal data

Accurate identification of crop growth stages is a crucial basis for implementing effective cultivation management. With the development of deep learning techniques in image understanding, research on intelligent real-time recognition of crop growth stages based on RGB images has garnered significant attention. However, the small differences and high similarity in crop morphological characteristics during the transition between adjacent growth stages pose challenges for accurate identification. To address this issue, this study proposes a multi-scale convolutional neural network model, termed MultiScalNet-Wheat (MSN-W), which enhances the algorithm's ability to learn complex features by utilizing multi-scale convolution and attention mechanisms. This model extracts key information from redundant data to identify winter wheat growth stages in complex field environments. Experimental results show that the MSN-W model achieves a recognition accuracy of 97.6 %, outperforming typical convolutional neural network models such as VGG19, ResNet50, MobileNetV3, and DenseNet. To further address the difficulty in recognizing growth stages during transition periods, where canopy morphological features are highly similar and show small differences, this paper introduces an innovative approach by incorporating sequential environmental data related to wheat growth stages. By extracting these features and performing multi-modal collaborative inference, a multi-modal feature-based wheat growth stage recognition model, termed MultiModalNet-Wheat (MMN-W), is constructed on the basis of the MSN-W model. Experimental results indicate that the MMN-W model achieves a recognition accuracy of 98.5 %, improving by 0.9 % over the MSN-W model. Both the MSN-W and MMN-W models provide accurate methods for observing wheat growth stages, thereby supporting the scientific management of winter wheat at different growth stages.

Multimodal Deep Learning for Differentiating Bacterial and Fungal Keratitis Using Prospective, Representative Data

ObjectiveThis study develops and evaluates multimodal machine learning models for differentiating bacterial and fungal keratitis using a prospective, representative dataset from South India. DesignMachine learning classifier training and validation study. Participants599 subjects diagnosed with acute infectious keratitis at Aravind Eye Hospital in Madurai, India. MethodsWe developed and compared three prediction models to distinguish bacterial and fungal keratitis using a prospective, consecutively-collected, representative dataset gathered over a full calendar year (the MADURAI dataset). These models included a clinical data model, a computer vision model using the EfficientNet architecture, and a multimodal model combining both imaging and clinical data. We partitioned the MADURAI dataset into 70% train/validation and 30% test sets. Model training was performed with 5-fold cross-validation. We also compared the performance of the MADURAI-trained computer vision model against a model with identical architecture but trained on a pre-existing dataset collated from multiple prior bacterial and fungal keratitis randomized clinical trials (the RCT-trained computer vision model). Main Outcome MeasuresThe primary evaluation metric was the area under the precision-recall curve (AUPRC). Secondary metrics included area under the receiver operating curve (AUROC), accuracy, and F1 score. ResultsThe MADURAI-trained computer vision model outperformed the clinical data model and the RCT-trained computer vision model on the hold-out test set, with an AUPRC 0.94 (95% CI: 0.92-0.96), AUROC 0.81 (0.76-0.85), accuracy 77%, and F1 score 0.85. The multimodal model did not substantially improve performance compared to the computer vision model. ConclusionsThe best-performing machine learning classifier for infectious keratitis was a computer vision model trained using the MADURAI dataset. These findings suggest that image-based deep learning could significantly enhance diagnostic capabilities for infectious keratitis, and emphasize the importance of using prospective, consecutively-collected, representative data for machine learning model training and evaluation.

Sentiment Analysis: Comprehensive Reviews, Recent Advances, and Open Challenges.

Sentiment analysis (SA) aims to understand the attitudes and views of opinion holders with computers. Previous studies have achieved significant breakthroughs and extensive applications in the past decade, such as public opinion analysis and intelligent voice service. With the rapid development of deep learning, SA based on various modalities has become a research hotspot. However, only individual modality has been analyzed separately, lacking a systematic carding of comprehensive SA methods. Meanwhile, few surveys covering the topic of multimodal SA (MSA) have been explored yet. In this article, we first take the modality as the thread to design a novel framework of SA tasks to provide researchers with a comprehensive understanding of relevant advances in SA. Then, we introduce the general workflows and recent advances of single-modal in detail, discuss the similarities and differences of single-modal SA in data processing and modeling to guide MSA, and summarize the commonly used datasets to provide guidance on data and methods for researchers according to different task types. Next, a new taxonomy is proposed to fill the research gaps in MSA, which is divided into multimodal representation learning and multimodal data fusion. The similarities and differences between these two methods and the latest advances are described in detail, such as dynamic interaction between multimodalities, and the multimodal fusion technologies are further expanded. Moreover, we explore the advanced studies on multimodal alignment, chatbots, and Chat Generative Pre-trained Transformer (ChatGPT) in SA. Finally, we discuss the open research challenges of MSA and provide four potential aspects to improve future works, such as cross-modal contrastive learning and multimodal pretraining models.

Decision-making optimization for post-disaster restoration of multimodal transport networks in terms of resilience

IntroductionPublic health emergencies can have a ripple effect on the resilience of multimodal transport network, which will lead to problems such as route disruptions or blockages, route selection change and information transmission delay spreading to the whole network, further hindering the transportation planning and operational efficiency of the network. MethodsThis study constructs a multimodal transport route optimization model under uncertainty with the objective of the sum of transportation cost, transshipment cost, penalty cost and carbon emission cost. To enhance the computational efficiency of the model, a novel invasive weed optimization with memory and encoding value clustering capabilities is proposed. In addition, by fusing the Q-learning algorithm in reinforcement learning with the novel invasive weed algorithm, the action-value function table obtained from the training facilitates the selection of optimal routes. Based on empirical data, explore the sensitivity analysis of node disruptions, time windows, and fuzzy demand on route decision-making under public health emergencies. ResultsThe transport network is affected by public health emergencies, which makes the optimal route deviate from the expected goal, resulting in an increase in the total cost. The proportion of total cost is determined by the position of nodes in the network, with critical nodes suffering more losses than ordinary nodes. Reasonable setting of time windows and fuzzy demand intervals is an effective way to improve the resilience and transportation efficiency of multimodal transport network. ConclusionsThis study provides more applicable decision-making references for enterprises to prevent the risk of supply chain disruptions caused by public health emergencies.

Vision language models in ophthalmology.

Vision Language Models are an emerging paradigm in artificial intelligence that offers the potential to natively analyze both image and textual data simultaneously, within a single model. The fusion of these two modalities is of particular relevance to ophthalmology, which has historically involved specialized imaging techniques such as angiography, optical coherence tomography, and fundus photography, while also interfacing with electronic health records that include free text descriptions. This review then surveys the fast-evolving field of Vision Language Models as they apply to current ophthalmologic research and practice. Although models incorporating both image and text data have a long provenance in ophthalmology, effective multimodal Vision Language Models are a recent development exploiting advances in technologies such as transformer and autoencoder models. Vision Language Models offer the potential to assist and streamline the existing clinical workflow in ophthalmology, whether previsit, during, or post-visit. There are, however, also important challenges to be overcome, particularly regarding patient privacy and explainability of model recommendations.

Cross-Modal self-supervised vision language pre-training with multiple objectives for medical visual question answering

Medical Visual Question Answering (VQA) is a task that aims to provide answers to questions about medical images, which utilizes both visual and textual information in the reasoning process. The absence of large-scale annotated medical VQA datasets presents a formidable obstacle to training a medical VQA model from scratch in an end-to-end manner. Existing works have been using image captioning dataset in the pre-training stage and fine-tuning to downstream VQA tasks. Following the same paradigm, we use a collection of public medical image captioning datasets to pre-train multimodality models in a self-supervised setup, and fine-tune to downstream medical VQA tasks. In the work, we propose a method that featured with Cross-Modal pre-training with Multiple Objectives (CMMO), which includes masked image modelling, masked language modelling, image-text matching, and image-text contrastive learning. The proposed method is designed to associate the visual features of medical images with corresponding medical concepts in captions, for learning aligned vision and language feature representations, and multi-modal interactions. The experimental results reveal that our proposed CMMO method outperforms state-of-the-art methods on three public medical VQA datasets, showing absolute improvements of 2.6%, 0.9%, and 4.0% on the VQA-RAD, PathVQA, and SLAKE dataset, respectively. We also conduct comprehensive ablation studies to validate our method, and visualize the attention maps which show a strong interpretability. The code and pre-trained weights will be released at https://github.com/pengfeiliHEU/CMMO.

Exploring structured molecular landscape from single-cell multi-omics data by an explainable multimodal model

Exploring structured molecular landscape from single-cell multi-omics data by an explainable multimodal model

Large multimodal model-based standardisation of pathology reports with confidence and its prognostic significance.

Despite the existence of established standards and guidelines for pathology reporting, many pathology reports are still written in unstructured free text. Extracting information from these reports and formatting it according to a standard is crucial for consistent interpretation. Automated information extraction from unstructured pathology reports is a challenging task, as it requires accurately interpreting medical terminologies and context-dependent details. In this work, we present a practical approach for automatically extracting information from unstructured pathology reports or scanned paper reports utilising a large multimodal model. This framework uses context-aware prompting strategies to extract values of individual fields, such as grade, size, etc. from pathology reports. A unique feature of the proposed approach is that it assigns a confidence value indicating the correctness of the model's extraction for each field and generates a structured report in line with national pathology guidelines in human and machine-readable formats. We have analysed the extraction performance in terms of accuracy and kappa scores, and the quality of the confidence scores assigned by the model. We have also evaluated the prognostic value of the extracted fields and feature embeddings of the raw text. Results showed that the model can accurately extract information with an accuracy and kappa score up to 0.99 and 0.98, respectively. Our results indicate that confidence scores are an effective indicator of the correctness of the extracted information achieving an area under the receiver operating characteristic curve up to 0.93 thus enabling automatic flagging of extraction errors. Our analysis further reveals that, as expected, information extracted from pathology reports is highly prognostically relevant. The framework demo is available at: https://labieb.dcs.warwick.ac.uk/. Information extracted from pathology reports of colorectal cancer cases in the cancer genome atlas using the proposed approach and its code are available at: https://github.com/EtharZaid/Labieb.

A Stacked Multimodality Model Based on Functional MRI Features and Deep Learning Radiomics for Predicting the Early Response to Radiotherapy in Nasopharyngeal Carcinoma

BackgroundThis study aimed to construct and assess a comprehensive model that integrates MRI-derived deep learning radiomics, functional imaging (fMRI), and clinical indicators to predict early efficacy of radiotherapy in nasopharyngeal carcinoma (NPC). MethodsThis retrospective study recruited NPC patients with radiotherapy from two Chinese hospitals between October 2018 and July 2022, divided into a training set (hospital I, 194 cases), an internal validation set (hospital I, 82 cases), and an external validation set (hospital II, 40 cases). We extracted 3404 radiomic features and 2048 deep learning characteristics from MRI modalities - T1WI, CE-T1WI, T2WI, and T2WI/FS. Additionally, both the ADC and its maximum (ADCmax) from DWI imaging, along with TBF and its maximum (TBFmax) from ASL imaging were derived. We used four classifiers (LR, XGBoost, SVM and KNN) and stacked algorithm as model construction methods. The receiver operating characteristic (ROC) curve (AUC) and decision curve analysis (DCA) was used to assess models. ResultsThe manual radiomics model leveraging XGBoost and the deep learning model based on KNN (the AUCs in the training set: 0.909, 0.823, respectively) showed better predictive efficacy than other machine learning algorithms. The stacked model that integrated MRI-based deep learning radiomics, fMRI, and hematological indicators, has a large improvement of AUC in the training set [0.984 (95%CI: 0.972 - 0.996)], the internal validation set [0.936 (95%CI: 0.885 - 0.987)], and the external validation set [0.959 (95%CI: 0.901-1)]. ConclusionsOur research has developed a clinical-radiomics integrated model based on MRI which can predict early radiotherapy response in NPC and provide guidance for personalized treatment.