Multimodal Model Research Articles

CECS-CLIP: Fusing Domain Knowledge for Rare Wildlife Detection Model.

Accurate and efficient wildlife monitoring is essential for conservation efforts. Traditional image-based methods often struggle to detect small, occluded, or camouflaged animals due to the challenges posed by complex natural environments. To overcome these limitations, an innovative multimodal target detection framework is proposed in this study, which integrates textual information from an animal knowledge base as supplementary features to enhance detection performance. First, a concept enhancement module was developed, employing a cross-attention mechanism to fuse features based on the correlation between textual and image features, thereby obtaining enhanced image features. Secondly, a feature normalization module was developed, amplifying cosine similarity and introducing learnable parameters to continuously weight and transform image features, further enhancing their expressive power in the feature space. Rigorous experimental validation on a specialized dataset provided by the research team at Northwest A&F University demonstrates that our multimodal model achieved a 0.3% improvement in precision over single-modal methods. Compared to existing multimodal target detection algorithms, this model achieved at least a 25% improvement in AP and excelled in detecting small targets of certain species, significantly surpassing existing multimodal target detection model benchmarks. This study offers a multimodal target detection model integrating textual and image information for the conservation of rare and endangered wildlife, providing strong evidence and new perspectives for research in this field.

GenAI for Scientific Discovery in Electrochemical Energy Storage: State-of-the-Art and Perspectives from Nano- and Micro-Scale.

The transition to electric vehicles (EVs) and the increased reliance on renewable energy sources necessitate significant advancements in electrochemical energy storage systems. Fuel cells, lithium-ion batteries, and flow batteries play a key role in enhancing the efficiency and sustainability of energy usage in transportation and storage. Despite their potential, these technologies face limitations such as high costs, material scarcity, and efficiency challenges. This research introduces a novel integration of Generative AI (GenAI) within electrochemical energy storage systems to address these issues. By leveraging advanced GenAI techniques like Generative Adversarial Networks, autoencoders, diffusion and flow-based models, and multimodal large language models, this paper demonstrates significant improvements in material discovery, battery design, performance prediction, and lifecycle management across different types of electrochemical storage systems. The research further emphasizes the importance of nano- and micro-scale interactions, providing detailed insights into optimizing these interactions for improved efficiency and longevity. Additionally, the paper discusses the challenges and future directions for integrating GenAI in energy storage research, highlighting the importance of data quality, model transparency, workflow integration, scalability, and ethical considerations. By addressing these aspects, this research sets a new benchmark for the use of GenAI in battery development, promoting sustainable, efficient, and safer energy solutions.

Intelligent Method of Identifying the Nonlinear Dynamic Model for Helicopter Turboshaft Engines

This research focused on the helicopter turboshaft engine dynamic model, identifying task solving in unsteady and transient modes (engine starting and acceleration) based on sensor data. It is known that about 85% of helicopter turboshaft engines operate in steady-state modes, while only around 15% operate in unsteady and transient modes. Therefore, developing dynamic multi-mode models that account for engine behavior during these modes is a critical scientific and practical task. The dynamic model for starting and acceleration modes has been further developed using on-board parameters recorded by sensors (gas-generator rotor r.p.m., free turbine rotor speed, gas temperature in front of the compressor turbine, fuel consumption) to achieve a 99.88% accuracy in identifying the dynamics of these parameters. An improved Elman recurrent neural network with dynamic stack memory was introduced, enhancing the robustness and increasing the performance by 2.7 times compared to traditional Elman networks. A theorem was proposed and proven, demonstrating that the total execution time for N Push and Pop operations in the dynamic stack memory does not exceed a certain value O(N). The training algorithm for the Elman network was improved using time delay considerations and Butterworth filter preprocessing, reducing the loss function from 2.5 to 0.12% over 120 epochs. The gradient diagram showed a decrease over time, indicating the model’s approach to the minimum loss function, with optimal settings ensuring the stable training.

Social Event Classification Based on Multimodal Masked Transformer Network

The key to multimodal social event classification is to fully and accurately utilize the features of both image and text modalities. However, most existing methods have the following limitations: (1) they simply concatenate the image features and text features of the event, and (2) there is irrelevant contextual information between different modalities, which leads to mutual interference. Therefore, it is not enough to only consider the relationship between the modalities of multimodal data, but also the irrelevant contextual information (i.e., regions or words) between the modalities. To overcome these limitations, a novel social event classification method based on multimodal masked transformer network (MMTN) is proposed. A better representation of text and image is learned through an image-text encoding network. Then, the obtained image and text representations are input into the multimodal masked transformer network to fuse the multimodal information, and the relationship between the modalities of multimodal information is modeled by calculating the similarity between the multimodal information, masking the irrelevant context between the modalities. Extensive experiments on two benchmark datasets show that the proposed multimodal masked transformer network model achieves state-of-the-art performance.

Tissue-based Profiling Techniques to Achieve Precision Medicine in Cancer: Opportunities and Challenges in Melanoma.

Immunotherapies targeting the programmed cell death 1 (PD-1) and cytotoxic T lymphocyte antigen 4 (CTLA-4) checkpoint receptors have revolutionised the treatment of metastatic melanoma. However, half of treated patients do not respond to or eventually progress on standard therapies and many experience adverse events as a result of drug toxicity. The identification of accurate biomarkers of clinical outcomes are required in order to move away from the one-size-fits-all treatment approach of standard clinical practice, and towards a more personalised approach to enable the administration of the optimal therapy for any given patient and further improve patient outcomes. Recent clinical trials have proven the potential of multi-omics analyses, including genomic, gene expression, and tumour immune profiling, of patients' tumour biopsies, to predict a patient's response to subsequently administered immunotherapies. However, reproducibility of such multi-omics analyses, tissue requirements, and clinical validation have limited the practical application of these approaches in routine clinical workflows. In this review, we discuss several pivotal tissue-based profiling techniques that can be utilised to identify potential genomic, transcriptomic and immune biomarkers predictive of clinical outcomes following treatment with immune checkpoint inhibitors in melanoma. Furthermore, we highlight the key opportunities and challenges associated with the use of each of these techniques. The development and implementation of multimodal predictive models which combine data derived from these various methods is the future for achieving precision medicine for patients with melanoma.

A multi-modal deep language model for contaminant removal from metagenome-assembled genomes

A multi-modal deep language model for contaminant removal from metagenome-assembled genomes

Developing a Robust Multi-Skill, Multi-Mode Resource-Constrained Project Scheduling Model with Partial Preemption, Resource Leveling, and Time Windows

Real-world projects encounter numerous issues, challenges, and assumptions that lead to changes in scheduling. This exposure has prompted researchers to develop new scheduling models, such as those addressing constrained resources, multi-skill resources, and activity pre-emption. Constrained resources arise from competition among projects for limited access to renewable resources. This research presents a scheduling model with constrained multi-skill and multi-mode resources, where activity durations vary under different scenarios and allow for partial pre-emption due to resource shortages. The main innovation is the pre-emption of activities when resources are unavailable, with defined minimum and maximum delivery time windows. For this purpose, a multi-objective mathematical programming model is developed that considers Bertsimas and Sim’s robust model in uncertain conditions. The model aims to minimize resource consumption, idleness, and project duration. The proposed model was solved using a multi-objective genetic algorithm and finally, its validation was completed and confirmed. Analysis shows that limited renewable resources can lead to increased activity pre-emption and extended project timelines. Additionally, higher demand raises resource consumption, reducing availability and prolonging project duration. Increasing the upper time window extends project time while decreasing the lower bound pressures resources, leading to higher consumption and resource scarcity.

The application of multimodal AI large model in the green supply chain of energy industry

With the accelerated advancements in artificial intelligence and the increasing emphasis on sustainable supply chain management, the integration of multimodal artificial intelligence (AI) into green supply chains has emerged as a critical research frontier. This study delves into the synergistic potential and challenges of combining multimodal AI, which leverages diverse data types such as text, images, and numerical data, to enhance decision-making processes in green supply chains. Through the meticulous design of a data strategy and model framework, this research establishes a sophisticated and efficient data processing and model training pipeline. The experimental results reveal that the comprehensive analysis and fusion of multimodal data significantly improve the prediction accuracy of key supply chain metrics, with observed increases in accuracy and recall rates by 12.4% and 9.8%, respectively. Additionally, the model's limitations are critically assessed, and targeted improvement strategies are proposed. The practical implications of this study are profound, offering actionable insights for the application of multimodal AI in real-world energy sector scenarios. The findings underscore the potential of this technology to optimize operations, reduce environmental impact, and drive sustainable growth in the energy industry.

Strategizing Low Carbon Urban Planning Through Environmental Impact Assessment by Artificial Intelligence Driven Carbon Foot-print Forecasting

Addressing the associated rise in Carbon Emissions (CE) as smart cities expand becomes paramount. Effective low-carbon urban planning demands robust, precise assessments. This research introduces a cutting-edge solution via an Artificial Intelligence (AI) -driven Carbon Footprint (CF) impact assessment. A detailed dataset, collected over 3 years, was harnessed to gather insights into vital urban factors, including CE, Energy Consumption (EC) patterns, variations in land use, transportation dynamics, and changes in air quality. The cornerstone of this research is developing the Multi-modal Stacked VAR-LSTM model. This model proposes to provide accurate CF predictions for urban environments by merging the capabilities of Vector Autoregression (VAR) with Long Short-Term Memory (LSTM) neural networks. The process encompasses dedicated assessments for each data segment, harnessing VAR to delineate interdependencies and refining these predictions with the LSTM network using the residuals from the VAR analysis. By interweaving AI-driven carbon footprint impact assessments into the urban planning discourse, this study underscores the vast potential in sculpting future urban development strategies that are sustainable and sensitive to carbon impact.

A multimodal multistream multilevel fusion network for finger joint angle estimation with hybrid sEMG and FMG sensing

Finger joint angle (FJA) estimation, as a dynamic and fine-grained decoding mode, can support intuitive and natural human–machine interactions. This study is pioneering work dedicated to achieving accurate and clinically friendly FJA estimation. In this context, a novel surface electromyography and force myography-based multimodal multistream multilevel fusion network (MMMFNet) is proposed. MMMFNet utilises gate-recurrent units (GRU) networks to capture spatial dependencies among features and adopts a multilevel fusion strategy to extract more comprehensive representations from information sources than a single-level fusion. A cross-stream interaction (CSI) block is added between the GRU networks to focus on important information in the prediction and optimise the learning process and representation capability by facilitating information interaction among branch networks. The experimental results suggest that the MMMFNet has higher estimation accuracy (RMSE: 6.225±0.276 vs. 6.953±0.282, R2: 0.879±0.019 vs 0.846±0.021), shorter training time (4.146±0.074 min vs. 16.536±0.430 min), and lower computational complexity (5.4×106 FLOPs vs 8.8×106 FLOPs) than state-of-the-art fusion methods. Moreover, MMMFNet, which quantifies feature importance using attention weights, is the first explainable multimodal model for hand movement intention decoding. In summary, compared with state-of-the-art methods, MMMFNet can not only estimate FJAs more accurately but is also more clinically friendly.

A copula-based approach for multi-modal demand dependence modeling: Temporal correlation between demand of subway and bike-sharing

As a representative mode of shared mobility, bike-sharing serves not only as a convenient way to conduct short-distance trips in urban areas, but also as a feeder mode to public transit, forming the Bike and Ride (BnR) system. Conducting management for such a hybrid multi-modal system faces various challenges, including the complex interactions between bike-sharing and other modes, highly dynamic passenger demand, and the difficulty of accessing direct transfer data. To overcome such difficulties, our study proposes a framework for assessing the dependency between the two usage modes. Firstly, a Dynamic-Time-Warping-based (DTW) method is utilized to determine the catchment area (CA) between the two modes, allowing the BnR-related tendency similarity under a given time scale to be considered. Then, the patterns of probabilistic dependence between travel demand of the two modes are obtained by a copula-based approach, which separates correlations under specific usage levels from single modal demands. A case study on the multi-modal system formed by docked bike-sharing and subway in New York is conducted to validate the proposed framework. The tendency similarity is found to be most pronounced within 500 m on average under a 4-hour interval. For each formed station group (SG), the best-fitted copula type is selected, capturing the strong tail correlations present only at specific usage levels. The results show a variety of different correlation patterns within SGs, despite the close geographic locations they may share. Areas of potential transfer resistance between the two modes are identified, which is more evident in first-mile-related (FMR) activities. In contrast, the two modes display more weak connections in last-mile-related (LMR) activities. The obtained results can be utilized by bike-sharing service providers to analyze demand distributions and conduct efficient station-level rebalancing. Compared to previous methods, our proposed framework is computationally inexpensive since no direct transfer of data or complex inference network is required. It incorporates statistically significant spatial–temporal information, allowing for a more accurate determination of the bi-modal assessment range. Moreover, considering that single-mode influences are mathematically removed, the resulting correlation in principle links to the strength of the connections between the two modes. Therefore, it can be assessed as an indicator of the reliability of the multi-modal system.

A Multimodal Recurrent Model for Driver Distraction Detection

A Multimodal Recurrent Model for Driver Distraction Detection

Establishing the Foundations of Emotional Intelligence in Care Companion Robots to Mitigate Agitation Among High-Risk Patients With Dementia: Protocol for an Empathetic Patient-Robot Interaction Study.

An estimated 6.7 million persons are living with dementia in the United States, a number expected to double by 2060. Persons experiencing moderate to severe dementia are 4 to 5 times more likely to fall than those without dementia, due to agitation and unsteady gait. Socially assistive robots fail to address the changing emotional states associated with agitation, and it is unclear how emotional states change, how they impact agitation and gait over time, and how social robots can best respond by showing empathy. This study aims to design and validate a foundational model of emotional intelligence for empathetic patient-robot interaction that mitigates agitation among those at the highest risk: persons experiencing moderate to severe dementia. A design science approach will be adopted to (1) collect and store granular, personal, and chronological data using Personicle (an open-source software platform developed to automatically collect data from phones and other devices), incorporating real-time visual, audio, and physiological sensing technologies in a simulation laboratory and at board and care facilities; (2) develop statistical models to understand and forecast the emotional state, agitation level, and gait pattern of persons experiencing moderate to severe dementia in real time using machine learning and artificial intelligence and Personicle; (3) design and test an empathy-focused conversation model, focused on storytelling; and (4) test and evaluate this model for a care companion robot (CCR) in the community. The study was funded in October 2023. For aim 1, architecture development for Personicle data collection began with a search for existing open-source data in January 2024. A community advisory board was formed and met in December 2023 to provide feedback on the use of CCRs and provide personal stories. Full institutional review board approval was received in March 2024 to place cameras and CCRs at the sites. In March 2024, atomic marker development was begun. For aim 2, after a review of open-source data on patients with dementia, the development of an emotional classifier was begun. Data labeling was started in April 2024 and completed in June 2024 with ongoing validation. Moreover, the team established a baseline multimodal model trained and validated on healthy-person data sets, using transformer architecture in a semisupervised manner, and later retrained on the labeled data set of patients experiencing moderate to severe dementia. In April 2024, empathy alignment of large language models was initiated using prompt engineering and reinforcement learning. This innovative caregiving approach is designed to recognize the signs of agitation and, upon recognition, intervene with empathetic verbal communication. This proposal has the potential to have a significant impact on an emerging field of computational dementia science by reducing unnecessary agitation and falls of persons experiencing moderate to severe dementia, while reducing caregiver burden. PRR1-10.2196/55761.

The future of multimodal artificial intelligence models for integrating imaging and clinical metadata: a narrative review.

With the ongoing revolution of artificial intelligence (AI) in medicine, the impact of AI in radiology is more pronounced than ever. An increasing number of technical and clinical AI-focused studies are published each day. As these tools inevitably affect patient care and physician practices, it is crucial that radiologists become more familiar with the leading strategies and underlying principles of AI. Multimodal AI models can combine both imaging and clinical metadata and are quickly becoming a popular approach that is being integrated into the medical ecosystem. This narrative review covers major concepts of multimodal AI through the lens of recent literature. We discuss emerging frameworks, including graph neural networks, which allow for explicit learning from non-Euclidean relationships, and transformers, which allow for parallel computation that scales, highlighting existing literature and advocating for a focus on emerging architectures. We also identify key pitfalls in current studies, including issues with taxonomy, data scarcity, and bias. By informing radiologists and biomedical AI experts about existing practices and challenges, we hope to guide the next wave of imaging-based multimodal AI research.

Reinforcement Learning-Based Multimodal Model for the Stock Investment Portfolio Management Task

Machine learning has been applied by more and more scholars in the field of quantitative investment, but traditional machine learning methods cannot provide high returns and strong stability at the same time. In this paper, a multimodal model based on reinforcement learning (RL) is constructed for the stock investment portfolio management task. Most of the previous methods based on RL have chosen the value-based RL methods. Policy gradient-based RL methods have been proven to be superior to value-based RL methods by a growing number of research. Commonly used policy gradient-based reinforcement learning methods are DDPG, TD3, SAC, and PPO. We conducted comparative experiments to select the most suitable method for the dataset in this paper. The final choice was DDPG. Furthermore, there will rarely be a way to refine the raw data before training the agent. The stock market has a large amount of data, and the data are complex. If the raw stock market data are fed directly to the agent, the agent cannot learn the information in the data efficiently and quickly. We use state representation learning (SRL) to process the raw stock data and then feed the processed data to the agent. It is not enough to train the agent using only stock data; we also added comment text data and image data. The comment text data comes from investors’ comments on stock bars. Image data are derived from pictures that can represent the overall direction of the market. We conducted experiments on three datasets and compared our proposed model with 11 other methods. We set up three evaluation indicators in the paper. Taken together, our proposed model works best.

Multicenter Development and Validation of a Multimodal Deep Learning Model to Predict Severe AKI

Multicenter Development and Validation of a Multimodal Deep Learning Model to Predict Severe AKI

Multimodal data-driven, vertical visualization prediction model for early prediction of atherosclerotic cardiovascular disease in patients with new-onset hypertension.

: Hypertension is an important contributing factor to atherosclerotic cardiovascular disease (ASCVD), and multiple risk factors, many of which are implicated in metabolic disorders, contribute to the cause of hypertension. Despite the promise of multimodal data-driven prediction model, no such prediction model was available to predict the risk of ASCVD in Chinese individuals with new-onset hypertension and no history of ASCVD. : A total of 514 patients were randomly allocated to training and verification cohorts (ratio, 7 : 3). We employed Boruta feature selection and conducted multivariate Cox regression analyses to identify variables associated with ASCVD in these patients, which were subsequently utilized for constructing the predictive model. The performance of prediction model was assessed in terms of discriminatory power (C-index), calibration (calibration curves), and clinical utility [decision curve analysis (DCA)]. : This model was derived from four clinical variables: 24-h SBP coefficient of variation, 24-h DBP coefficient of variation, urea nitrogen and the triglyceride-glucose (TyG) index. Bootstrapping with 500 iterations was conducted to adjust the C-indexes were C-index = 0.731, 95% confidence interval (CI) 0.620-0.794 and C-index: 0.799, 95% CI 0.677-0.892 in the training and verification cohorts, respectively. Calibration plots with 500 bootstrapping iterations exhibited a strong correlation between the predicted and observed occurrences of ASCVD in both the training and verification cohorts. DCA analysis confirmed the clinical utility of this prediction model. The constructed nomogram demonstrated significant additional prognostic utility for ASCVD, as evidenced by improvements in the C-index, net reclassification improvement, integrated discrimination improvement, and DCA compared with the overall ASCVD risk assessment. The developed longitudinal prediction model based on multimodal data can effectively predict ASCVD risk in individuals with an initial diagnosis of hypertension. : The trial was registered in the Chinese Clinical Trial Registry (ChiCTR2300074392).

Understanding mechanotransduction in the distal colon and rectum via multiscale and multimodal computational modeling

Visceral pain in the large bowel is a defining symptom of irritable bowel syndrome (IBS) and the primary reason that patients visit gastroenterologists. This pain is reliably triggered by mechanical distension of the distal colon and rectum (colorectum). Consequently, the process of mechanotransduction by sensory afferents, responsible for translating mechanical colorectal stimuli into neural action potentials, plays a central role in IBS-related bowel pain. In this study, we aim to enhance our understanding of colorectal mechanotransduction by combining experimental findings in colorectal biomechanics and afferent neural encoding within a comprehensive computational simulation framework. To achieve this, we implemented a three-layered, fiber-reinforced finite element model that accurately replicates the nonlinear, heterogeneous, and anisotropic mechanical characteristics of the mouse colorectum. This model facilitates the computation of local mechanical stresses and strains around individual afferent endings, which have diameters on the micron-scale. We then integrated a neural membrane model to simulate the encoding of action potentials by afferent nerves in response to microscopic stresses and strains along the afferent endings. Our multiscale simulation framework enables the assessment of three hypotheses regarding the mechanical gating of action potential generation: (1) axial stress dominates mechanical gating of mechanosensitive channels, (2) both axial and circumferential stresses contribute, and (3) membrane shear stress dominates. Additionally, we explore how the orientation of afferent endings impacts neural encoding properties. This computational framework not only allows for the virtual investigation of colorectal mechanotransduction in the context of prolonged visceral hypersensitivity but can also guide the development of new experimental studies aimed at uncovering the neural and biomechanical mechanisms underlying IBS-related bowel pain.

Text-image multimodal fusion model for enhanced fake news detection.

In the era of rapid internet expansion and technological progress, discerning real from fake news poses a growing challenge, exposing users to potential misinformation. The existing literature primarily focuses on analyzing individual features in fake news, overlooking multimodal feature fusion recognition. Compared to single-modal approaches, multimodal fusion allows for a more comprehensive and enriched capture of information from different data modalities (such as text and images), thereby improving the performance and effectiveness of the model. This study proposes a model using multimodal fusion to identify fake news, aiming to curb misinformation. The framework integrates textual and visual information, using early fusion, joint fusion and late fusion strategies to combine them. The proposed framework processes textual and visual information through data cleaning and feature extraction before classification. Fake news classification is accomplished through a model, achieving accuracy of 85% and 90% in the Gossipcop and Fakeddit datasets, with F1-scores of 90% and 88%, showcasing its performance. The study presents outcomes across different training periods, demonstrating the effectiveness of multimodal fusion in combining text and image recognition for combating fake news. This research contributes significantly to addressing the critical issue of misinformation, emphasizing a comprehensive approach for detection accuracy enhancement.

A lightweight finger multimodal recognition model based on detail optimization and perceptual compensation embedding

Multimodal biometric recognition technology has attracted the attention of many scholars due to its higher security and stability than single-modal recognition, but its additional parameter quantity and computational cost have brought challenges to the lightweight deployment of the model. In order to meet the needs of a wider range of application scenarios, this paper proposes a lightweight model DPNet using fingerprint and finger vein images for multimodal recognition, which adopts a double-branch lightweight feature extraction structure combining detail optimization and perception compensation. Among them, the detail extraction optimization branch uses multi-scale dimensionality reduction filtering to obtain low-redundant detail information, and combines the depth extension operation to enhance the generalization ability of detail features. The perception compensation branch expands and compensates the model's perceptual field of view through lightweight spatial location query and global information attention. In addition, this paper designs a perceptual feature embedding method to embed perceptual compensation information in the way of importance adjustment to improve the consistency of embedded features. The ABFM fusion module is proposed to carry out multi-level lightweight and deep interactive fusion of the extracted finger modal features from the global to the spatial region, so as to improve the degree and utilization rate of feature fusion. In this paper, the model recognition performance and lightweight advantages are verified on three multimodal datasets. Experimental results show that the proposed model achieves the most advanced lightweight effect and recognition performance in the experimental comparison of all datasets.