Multimodal Sources Research Articles

Pathway-dependent brain stimulation responses indicate motion processing integrity after stroke.

Homonymous Hemianopia (HH), a common visual impairment resulting from occipital lobe lesions, affects a significant number of stroke survivors. Intensive perceptual training can foster recovery, possibly by enhancing surviving visual pathways. This study employed cortico-cortical paired associative stimulation (ccPAS) to induce associative plasticity within the residual and bi-directional primary visual cortex (V1) - middle temporal area (MT) pathways in stroke patients. We used ccPAS, which is thought to tap into Hebbian-like spike-timing dependent plasticity, over a motion processing pathway in stroke patients to transiently improve visual motion discrimination in their blind field. Sixteen stroke patients participated in this double-blind, crossover study comparing the effects of bidirectional ccPAS (V1-to-MT or MT-to-V1) on motion discrimination and EEG-Granger Causality. Additionally, we explored potential multimodal sources of inter-individual variability. Results showed that MT-to-V1 ccPAS enhanced motion direction discrimination, but the expected electrophysiological increase in top-down MT-to-V1 inputs was observed only in patients who showed improvement in motion discrimination. Good responders to MT-V1 ccPAS also demonstrated improved functional coupling between the cortical motion pathway and other relevant areas in the visual network, as well as more preserved ipsilesional V1-MT structural integrity. These findings indicate that targeted ccPAS can effectively engage functionally relevant residual visual pathways in stroke-affected brains, potentially offering new avenues for patient stratification and visual recovery strategies.

Advancing Blast Furnace Thermal State Prediction: A Data‐Driven Approach Using Thermocouple Integration and Multimodal Modeling

This study develops a data‐driven framework to predict the thermal state of blast furnaces using feature fusion from thermocouple data and spatial temperature distribution. The article proposes a hybrid framework based on multimodal integration and clustering algorithms, utilizing data extracted from thermocouples and the temperature distribution features around the furnace hearth. Through these fused features, multiple ensemble models are constructed to predict the thermal state of the blast furnace, with a focus on the thermocouple readings at the hearth. This method enhances understanding of the thermal state of the blast furnace, aiming to improve prediction accuracy and operational reliability. By validating the model with actual industrial data, its effectiveness in thermal state monitoring is demonstrated. The integration of multimodal data sources allows for the extraction of rich information from the thermocouple data, significantly enhancing the model's predictive performance.

Teachers’ Perspectives and Experiences in Higher Education Vocabulary Instruction

The selection of vocabulary learning strategies (VLS) is a pivotal action in the English teaching and learning process. This study aimed to explore what teachers perceive and experience in vocabulary instruction in higher education. A narrative approach was employed in this study. Using the purposive sampling technique under the criteria of a minimum of four years of experience as permanent English teachers and limited the area of study only to one of religious higher education in Central Kalimantan, five English teachers were enrolled as participants. Unstructured, open-ended interviews were used as the instrument for collecting the primary data. After collecting the stories, the data were reported and validated using triangulation. The results revealed that teachers believed that using a variety of learning strategies to engage learners to be interested in vocabulary instruction was useful. Then, the challenges both external and internal factors faced by teachers in vocabulary instruction could be handled by employing the strategies that fit the students’ characteristics, such as learning vocabulary through songs, games, and social media platforms such as TikTok, YouTube, Instagram, and so forth. Therefore, it can be drawn to the conclusion that teachers must always be creative and innovative in action by utilizing technology advancement. Besides, the use of differentiated instruction approach could be better employed by teachers for solutions to problems, particularly in facilitating diverse student characteristics. Next, the use of various modes of learning or multimodal sources can also help students in higher education to get very meaningful vocabulary learning.

DOP098 Development of BEN8744, a phase 2 ready, peripherally restricted PDE10 inhibitor targeting a new mechanism of action for the treatment of ulcerative colitis.

Abstract Background To identify novel therapeutic targets for ulcerative colitis (UC), a disease-agnostic Knowledge Graph1, comprising multi-modal data sources was used as input to a machine learning gene/protein prioritisation model2. Following in-depth target assessment, phosphodiesterase 10 (PDE10) was selected as a promising novel target for further investigation. Methods Our aim was to develop a novel therapeutic target for UC that will have not just anti-inflammatory properties but improve barrier function. PDE10 hydrolyses both cAMP and cGMP. In UC patients, reduced cAMP levels promote inflammation, while guanylate cyclase-C and its endogenous activators are downregulated in the colon, with lower levels correlating to higher disease severity. Our transcriptional analysis revealed that PDE10 is upregulated in UC patients’ colonic mucosa, correlating with disease severity, suggesting that PDE10 inhibition could potentially restore normal cyclic nucleotide signalling. Following successful experimental target validation, we developed BEN8744 a potent, selective, peripherally restricted oral PDE10 inhibitor. Supporting its therapeutic potential, BEN8744 significantly reduced TNFα, IL-6, and IL-8 in ex vivo inflamed UC patient colonic mucosa biopsy samples, and improved barrier integrity in an in vitro intestinal epithelial cell model. In a Phase 1, randomised, double-blind, placebo-controlled trial with 74 healthy subjects, we assessed the safety, tolerability, pharmacokinetics (PK), and food effect of BEN8744 (NCT06118385). The trial was in 3 parts: Part A investigated single ascending oral doses of BEN8744; Part B was a 2-way crossover assessment of the effect of food on the PK of BEN8744; and Part C investigated multiple ascending oral doses of BEN8744 for 14 days. Results All treatment emergent adverse events (TEASs) were mild or moderate in severity with no clinically significant findings in physical examination, laboratory variables, ECGs, OAAS/S or VAS results, during the study. No class effects previously seen with centrally active PDE10 inhibitors were observed in the present trial. Neither were there any dose dependent nervous system related TEAEs. Conclusion BEN8744 is a potent, peripherally restricted, well tolerated Phase 2 ready PDE10 inhibitor targeting a novel mechanism of action for the treatment of UC. The Phase 1 study showed a favourable safety, tolerability, and PK profile for BEN8744, supporting its further development. We are now planning a study in patients with moderate to severely active ulcerative colitis.

Multimodal approach to public health interventions using EGG and mobile health technologies

IntroductionPublic health interventions increasingly integrate multimodal data sources, such as Electroencephalogram (EEG) data, to enhance monitoring and predictive capabilities for mental health conditions. However, traditional models often face challenges with the complexity and high dimensionality of EEG signals. While recent advancements like Contrastive Language-lmage Pre-training(CLIP) models excel in cross-modal understanding, their application to EEG-based tasks remains limited due to the unique characteristics of EEG data.MethodsIn response, we introduce PH-CLIP (Public Health Contrastive Language-lmage Pretraining), a novel framework that combines CLIP's representational power with a multi-scale fusion mechanism designed specifically for EEG data within mobile health technologies. PH-CLIP employs hierarchical feature extraction to capture the temporal dynamics of EEG signals, aligning them with contextually relevant textual descriptions for improved public health insights. Through a multi-scale fusion layer, PH-CLIP enhances interpretability and robustness in EEG embeddings, thereby supporting more accurate and scalable interventions across diverse public health applications.Results and discussionExperimental results indicate that PH-CLIP achieves significant improvements in EEG classification accuracy and mental health prediction efficiency compared to leading EEG analysis models. This framework positions PH-CLIP as a transformative tool in public health monitoring, with the potential to advance large-scale mental health interventions through integrative mobile health technologies.

Refugee Learning Center in Indonesia: A Habitat for Refugee Well‐Being in Transit

ABSTRACTAlthough students' well‐being is widely researched in educational settings, little is known about how educational contexts influence the sense of well‐being experienced by refugee students in transit countries. This case study examined how a refugee learning center in Indonesia helps cultivate refugee students' sense of well‐being in different forms of capital. Data derived from public‐facing multimodal sources included the center's annual reports, staff and students' written narratives, digital stories on social media, and other artefacts. A qualitative content analysis (QCA) illuminated the interconnected forms of resources enacted in the refugee learning center for facilitating refugee students' subjective well‐being, such as accredited education programs, English‐medium instruction, and extracurricular activities for cultural capital; inclusive support systems and teacher support for social capital; and job‐related skills through volunteer opportunities for economic capital. This study not only broadens understanding of the connection between acquiring capital and the subjective well‐being of refugees but also contributes to the dearth of literature on refugee education in Indonesia. The implications for research, policy, and practice emphasise how stakeholders can implement comprehensive programs and strategies that integrate different resources to cultivate refugee students' well‐being and to further examine, sustain, and scale such programs in Indonesia and possibly elsewhere.

Emotion-Aware Ensemble Learning (EAEL): Revolutionizing Mental Health Diagnosis of Corporate Professionals via Intelligent Integration of Multi-Modal Data Sources and Ensemble Techniques

Emotion-Aware Ensemble Learning (EAEL): Revolutionizing Mental Health Diagnosis of Corporate Professionals via Intelligent Integration of Multi-Modal Data Sources and Ensemble Techniques

Development of a universal multimodal prediction method to optimise process parameters for improving densification during laser powder bed fusion

ABSTRACT This work develops a universal machine learning method that enhances the densification of metallic materials fabricated by laser powder bed fusion (LPBF) by optimising process parameters using multimodal data. By integrating ResNet-50 neural network and LightGBM model, this method combines process parameters with surface morphology of LPBF-fabricated samples at the feature scale, enabling extraction of multidimensional feature information and improving prediction accuracy of relative density. Statistical analysis demonstrates exceptional performance of the model on both training and testing datasets, yielding coefficient of determination (R²) values of 0.9821 and 0.8169, and mean absolute error values of 0.0178 and 0.0108, respectively. Bayesian optimisation was employed using the multimodal model as a surrogate model to identify potential optimal combinations within the parameter space. These results validate the effectiveness of establishing a multimodal machine learning method based on multimodal data sources to optimise LPBF process parameters and improve relative density of fabricated new materials.

Automated Nutrient Deficiency Detection and Recommendation Systems Using Deep Learning in Nutrition Science

Nutrient deficiencies affect millions globally, contributing to severe health issues and reduced quality of life. Traditional methods of diagnosing these deficiencies and recommending dietary adjustments are often time-intensive, prone to error, and lack personalization. The advent of deep learning has revolutionized nutrition science, offering automated, accurate, and scalable solutions. This paper delves into the development and application of automated nutrient deficiency detection and recommendation systems powered by deep learning. Key components of such systems include advanced data processing techniques that analyze multimodal datasets, such as biomarkers, dietary records, and food images. Convolutional Neural Networks (CNNs) excel in recognizing and quantifying nutrients from food images, while Recurrent Neural Networks (RNNs) handle time-series dietary data. Generative Adversarial Networks (GANs) and Natural Language Processing (NLP) facilitate data augmentation and textual analysis of dietary logs, respectively. These systems enable precise detection of deficiencies and generate tailored dietary plans based on individual needs, considering demographic and lifestyle factors. This article highlights case studies and practical implementations of deep learning models in real-world applications, such as AI-powered nutrition apps and biomarker-based deficiency prediction. It also addresses significant challenges, including data quality, algorithmic bias, and ethical concerns related to privacy and equity. Furthermore, the study explores future opportunities, such as integrating explainable AI, leveraging multi-modal data sources, and enhancing IoT-based tracking devices to improve recommendation systems. By bridging the gap between AI technology and nutrition science, these systems hold the potential to revolutionize global dietary health, offering scalable, personalized, and efficient solutions to combat nutrient deficiencies.

ILDIM-MFAM: interstitial lung disease identification model with multi-modal fusion attention mechanism.

This study aims to address the potential and challenges of multimodal medical information in the diagnosis of interstitial lung disease (ILD) by developing an ILD identification model (ILDIM) based on the multimodal fusion attention mechanism (MFAM) to improve the accuracy and reliability of ILD. Large-scale multimodal medical information data, including chest CT image slices, physiological indicator time series data, and patient history text information were collected. These data are professionally cleaned and normalized to ensure data quality and consistency. Convolutional Neural Network (CNN) is used to extract CT image features, Bidirectional Long Short-Term Memory Network (Bi-LSTM) model is used to learn temporal physiological metrics data under long-term dependency, and Self-Attention Mechanism is used to encode textual semantic information in patient's self-reporting and medical prescriptions. In addition, the multimodal perception mechanism uses a Transformer-based model to improve the diagnostic performance of ILD by learning the importance weights of each modality's data to optimally fuse the different modalities. Finally, the ablation test and comparison results show that the model performs well in terms of comprehensive performance. By combining multimodal data sources, the model not only improved the Precision, Recall and F1 score, but also significantly increased the AUC value. This suggests that the combined use of different modal information can provide a more comprehensive assessment of a patient's health status, thereby improving the diagnostic comprehensiveness and accuracy of ILD. This study also considered the computational complexity of the model, and the results show that ILDIM-MFAM has a relatively low number of model parameters and computational complexity, which is very favorable for practical deployment and operational efficiency.

Big Data Analytics And Ai-Driven Solutions For Financial Fraud Detection: Techniques, Applications, And Challenges

Financial fraud is an ever-evolving threat that poses significant challenges to the stability and integrity of financial systems. This study systematically reviews the application of big data analytics and advanced technologies, including machine learning and natural language processing (NLP), in detecting and mitigating financial fraud. Adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, 189 peer-reviewed articles were meticulously analyzed to explore state-of-the-art methodologies and identify critical gaps in the literature. The findings underscore the transformative role of big data analytics in uncovering patterns and anomalies that traditional systems often miss, enabling real-time fraud detection with enhanced accuracy and efficiency. Machine learning techniques, particularly ensemble models and deep learning algorithms, were found to adapt effectively to dynamic fraud patterns, while NLP extended the scope of detection to unstructured data, such as emails, contracts, and social media. Furthermore, real-time fraud detection systems emerged as vital tools for immediate mitigation, yet challenges persist in integrating multimodal data sources like audio and video and shifting from reactive to proactive fraud prevention strategies. This comprehensive review provides a holistic understanding of current advancements, highlighting both achievements and areas requiring further exploration, and sets the stage for the development of robust, scalable, and ethical fraud detection frameworks.

Regional Analysis Of Extreme Weather Events Using Deep Learning

The increasing frequency and severity of extreme weather events necessitate advanced predictive models that can effectively analyze complex meteorological phenomena. This study conducts a systematic review of 120 peer-reviewed articles to explore the application of deep learning techniques in weather prediction, adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The review highlights the transformative potential of hybrid deep learning models, which integrate Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to capture both spatial and temporal dependencies, significantly improving the accuracy of extreme weather forecasting. The study also examines the integration of multi-modal data sources, such as satellite imagery, IoT sensors, and ground-based observations, to enable comprehensive analyses of weather systems. Additionally, the role of Explainable Artificial Intelligence (XAI) in enhancing the interpretability of predictions and fostering stakeholder trust is critically analyzed. Findings reveal that while deep learning approaches offer substantial advancements, challenges related to data quality, computational demands, and resource disparities remain significant. This review underscores the need for global collaboration and innovation to address these limitations, paving the way for more reliable and equitable applications of AI-driven weather forecasting systems.

Enhancing Movie Recommendation Systems Through CNN-Based Feature Extraction and Optimized Collaborative Filtering

Abstract. This paper presents the design and implementation of a movie recommendation model that combines collaborative filtering with Convolutional Neural Networks (CNNs) to tackle the challenge of information overload in extensive movie databases. The primary objective is to enhance the user experience by developing an efficient and personalized recommendation system. The proposed model integrates CNNs to extract detailed image features from movie posters, enriching the movie representations used in the recommendation process. This feature extraction is coupled with an optimized collaborative filtering algorithm to deliver more accurate and tailored movie suggestions. The study is based on a dataset that includes movie information, user ratings, and user behavior data. Experimental results demonstrate that the proposed model significantly improves the accuracy and personalization of movie recommendations, thereby offering users a superior selection experience. The practical implications of this research are substantial, as it has the potential to boost user satisfaction and engagement on movie recommendation platforms. Future work will focus on further enhancing the model by incorporating long-term user behavior patterns and integrating multi-modal data sources for even more precise recommendations.

Multimodal Approaches Based on Microbial Data for Accurate Postmortem Interval Estimation.

Accurate postmortem interval (PMI) estimation is critical for forensic investigations, aiding case classification and providing vital trial evidence. Early postmortem signs, such as body temperature and rigor mortis, are reliable for estimating PMI shortly after death. However, these indicators become less useful as decomposition progresses, making late-stage PMI estimation a significant challenge. Decomposition involves predictable microbial activity, which may serve as an objective criterion for PMI estimation. During decomposition, anaerobic microbes metabolize body tissues, producing gases and organic acids, leading to significant changes in skin and soil microbial communities. These shifts, especially the transition from anaerobic to aerobic microbiomes, can objectively segment decomposition into pre- and post-rupture stages according to rupture point. Microbial communities change markedly after death, with anaerobic bacteria dominating early stages and aerobic bacteria prevalent post-rupture. Different organs exhibit distinct microbial successions, providing valuable PMI insights. Alongside microbial changes, metabolic and volatile organic compound (VOC) profiles also shift, reflecting the body's biochemical environment. Due to insufficient information, unimodal models could not comprehensively reflect the PMI, so a muti-modal model should be used to estimate the PMI. Machine learning (ML) offers promising methods for integrating these multimodal data sources, enabling more accurate PMI predictions. Despite challenges such as data quality and ethical considerations, developing human-specific multimodal databases and exploring microbial-insect interactions can significantly enhance PMI estimation accuracy, advancing forensic science.

Conversational Agents and Personal Privacy Harms Case Study

This fictional case study examines the question of whether a personal conversational agent/advisor called iSoph, encoding extensive personal information and having a fluent natural language interface, may raise privacy harms that are normally thought to attend to personal relationships, as opposed to harms associated with institutional databases and big data analytics. The case stipulates that (i) iSoph collects extensive personal data about its users from multiple, multimodal sources, (ii) can make inferences from this data in combination with its models, but (ii) cannot share information with its developer or any third party. It is shown that despite condition (iii), iSoph raises privacy risks. These privacy risks are of the type associated with personal relationships and direct observation. iSoph raises these risks because, as an advanced conversational agent, it is able to evoke anthropomorphizing responses from its users in ways that they are not fully conscious of or able to control.

Advancing lettuce physiological state recognition in IoT aeroponic systems: A meta-learning-driven data fusion approach

Automatically identifying key physiological factors in plants, such as leaf relative humidity (LRH), chlorophyll content (Chl), and nitrogen levels (N), is vital for effective aeroponic management and improving growth, yield, quality, and sustainability. Meta-learning (MetaL) solutions utilize data fusion and intelligent processing, ensuring fast and consistent outcomes. This paper aims to develop a novel MetaL framework that leverages multimodal data sources—including spectral, thermal, and IoT environmental data—to enable real-time, non-invasive identification of LRH, Chl, and N content in aeroponically grown lettuce. The research examined various spectral reflectance indices (SRIs) and thermal indicators from plant characteristics. Model-based feature selection was implemented using back-propagation neural networks (BPNN), decision trees (DT), and gradient boosting machines (GBM) to identify key attributes and optimize hyperparameters. The experimental findings indicated that deploying GBM-based top variables as the foundational model, combined with BPNN as the meta-model, significantly improved the accuracy of analyzing the assigned factors. The prediction scores (R²) for LRH, Chl, and N increased to 0.875 (RMSE=0.879), 0.886 (RMSE=0.694), and 0.930 (RMSE=0.184), respectively, compared to applying BPNN-based features alone as a standalone model. Overall, the designed methodology contributes to more accurate predictions of plant physiological states, enabling proactive steps toward sustainable aeroponic agriculture.

Multi-Modal Contextualization of Trajectory Data for Advanced Analysis

SummaryRising amounts of generated geospatial data, either trajectory-like tracking data, raster-like imagery, or vector-like mappings as in OpenStreetMap (OSM), grow the need for multi-modal algorithmic analysis. Existing machine-learning-based algorithms contradictly mainly focus on image and textual input representations and cannot deal with other modes of geospatial data. Therefore, we propose a novel method to contextualize vector-like trajectory data with surrounding data to create easy-to-be-analyzed image-like representations. Our approach includes the proposition of a chase-cam-like scanline over space according to the trajectory’s speed and possibly smoothed orientation. Thereby, surrounding pixels in the vicinity of the trajectory points are accumulated along the scanline and are combined into a visual representation of the trajectory. To show the potential effects of our work, we predict traffic regulations for trajectory sections in the vehicle speed dataset based on our proposed trajectory-based sampling of orthophotos in the same region. This proposes a new way of using multi-modal data sources (trajectories and airborne imagery) to extract road metadata.

Data-driven decision-making in public health: The role of advanced statistical models in epidemiology

This paper critically examines the transformative role of data-driven decision-making in public health, focusing on the integration of advanced statistical models in epidemiology. As the volume and complexity of health data increase, leveraging predictive analytics, machine learning, and real-time data integration has become essential for improving public health outcomes. The study explores how these technologies have shifted public health strategies from reactive to proactive approaches, particularly in areas such as disease surveillance, chronic disease management, and health equity. Through comprehensive analysis, the paper identifies key advancements, such as hybrid models that combine traditional epidemiological frameworks with AI, and the integration of multi-modal data sources that enhance predictive accuracy. The findings emphasize the potential of these models to optimize resource allocation, address health disparities, and provide timely interventions. However, challenges such as data quality, algorithmic bias, and the ethical implications of model transparency are highlighted as critical issues requiring ongoing research. The study concludes that for these models to be effectively adopted, there must be a balance between technological innovation and ethical considerations. Recommendations include the need for interdisciplinary collaboration, improved data governance frameworks, and the development of more inclusive models that are generalizable across diverse populations. This research underscores the necessity of combining robust analytical tools with ethical frameworks to enhance the reliability and equity of public health interventions.

1.5 million materials narratives generated by chatbots

The advent of artificial intelligence (AI) has enabled a comprehensive exploration of materials for various applications. However, AI models often prioritize frequently encountered material examples in the scientific literature, limiting the selection of suitable candidates based on inherent physical and chemical attributes. To address this imbalance, we generated a dataset consisting of 1,453,493 natural language-material narratives from OQMD, Materials Project, JARVIS, and AFLOW2 databases based on ab initio calculation results that are more evenly distributed across the periodic table. The generated text narratives were then scored by both human experts and GPT-4, based on three rubrics: technical accuracy, language and structure, and relevance and depth of content, showing similar scores but with human-scored depth of content being the most lagging. The integration of multimodal data sources and large language models holds immense potential for AI frameworks to aid the exploration and discovery of solid-state materials for specific applications of interest.

Investigating 24- to 48-hr Forecast of Offshore Worker Alertness and Vigilance Using Multimodal Sources: A Proactive Fatigue Monitoring Paradigm

Fatigue is a significant contributor to accidents in the high-risk oil and gas industry. This study developed and evaluated models for forecasting fatigue manifestation in offshore workers using the Psychomotor Vigilance Task (PVT). Seventy offshore workers participated in a four-week study, providing data on sleep, physiological, subjective, and performance measures. Various machine learning models (Ridge, Random Forest, Support Vector, Long Short-Term Memory (LSTM) regressions) were employed to predict PVT reaction times using different data normalization between generalized and personalized datasets. Results indicate that personalized Support Vector Regression models outperform other models in predicting short-term fatigue. Age and perceived exertion emerged as crucial predictors of fatigue. The findings underscore the potential of personalized fatigue forecasting for enhancing safety in the oil and gas industry.