Multimodal Task Research Articles

Backpropagation-Free Multi-modal On-Device Model Adaptation via Cloud-Device Collaboration

In our increasingly interconnected world, where intelligent devices continually amass copious personalized multi-modal data, a pressing need arises to deliver high-quality, personalized device-aware services. However, this endeavor presents a multifaceted challenge to prevailing artificial intelligence (AI) systems primarily rooted in the cloud. As these systems grapple with shifting data distributions between the cloud and devices, the traditional approach of fine-tuning-based adaptation (FTA) exists the following issues: the costly and time-consuming data annotation required by FTA and the looming risk of model overfitting. To surmount these challenges, we introduce a Universal On-Device Multi-modal Model Adaptation Framework, revolutionizing on-device model adaptation by striking a balance between efficiency and effectiveness. The framework features the Fast Domain Adaptor (FDA) hosted in the cloud, providing tailored parameters for the Lightweight Multi-modal Model on devices. To enhance adaptability across multi-modal tasks, the AnchorFrame Distribution Reasoner (ADR) minimizes communication costs. Our contributions, encapsulated in the Cloud-Device Collaboration Multi-modal Parameter Generation (CDC-MMPG) framework, represent a pioneering solution for on-Device Multi-modal Model Adaptation (DMMA). Extensive experiments validate the efficiency and effectiveness of our method, particularly in video question answering and retrieval tasks, driving forward the integration of intelligent devices into our daily lives.

Capturing inter-individual variability in stress dynamics with heart rate traces reveals activity in the bilateral hippocampus, amygdala, and insula

Abstract Abnormalities in the neural mechanisms of the stress response have the potential to serve as a transdiagnostic marker for stress-related disorders. This potential is rooted in the highly individual and dynamic stress response, which poses a challenge to classical experimentally informed models that focus on specific phases, conditions, or stimuli during a stress task. Here, we integrate individual heart rates (HR) as an immanent index of the stress-response and combine these with functional magnetic resonance imaging (fMRI) data. In this study, 83 healthy participants completed a multimodal psychosocial imaging stress task comprising three different task phases (PreStress, Stress, and PostStress), with each phase consisting of five 60 s blocks of active task interleaved with 40 s of rest, and simultaneous recordings of pulse plethysmography. Participants were asked to solve mental calculations and were exposed to negative social feedback during the Stress phase. We estimated a general linear model (GLM) with individual heart rates averaged per active block as a single parametric modulator of the task regressor across all 15 active blocks irrespective of the task phase. Results revealed a negative correlation between HR and activation in the bilateral amygdala and anterior hippocampus as well as deactivation in the default mode network. Positive correlations with HR were detected in the bilateral insular cortex, bilateral angular gyrus, and parts of the inferior and superior parietal lobes. In summary, our findings emphasize the utility of integrating the commonly assessed cardiovascular stress response (here: HR) as an immediate index of the participant’s stress status. We conclude that by such integration, brain regions involved in regulating the acute stress response, such as the anterior hippocampus and amygdala, are detected more sensitively by tracking the individual’s “stress wave” rather than treating every experimental block uniformly. Our approach may serve as a complementary analysis to the task-regressor based model.

Integrating Neural-Symbolic Reasoning With Variational Causal Inference Network for Explanatory Visual Question Answering.

Recently, a novel multimodal reasoning task named Explanatory Visual Question Answering (EVQA) has been introduced, which combines answering visual questions with multimodal explanation generation to expound upon the underlying reasoning processes. In contrast to conventional Visual Question Answering (VQA) that merely concentrates on providing answers, EVQA aims to improve the explainability and verifiability of reasoning by providing user-friendly explanations. Despite the improved explainability of inferred results, the existing EVQA models still adopt black-box neural networks to infer results, lacking the explainability of the reasoning process. Moreover, existing EVQA models commonly predict answers and explanations in isolation, overlooking the inherent causal correlation between them. To handle these challenges, we propose a Program-guided Variational Causal Inference Network (Pro-VCIN) that integrates neural-symbolic reasoning with variational causal inference and constructs causal correlations between the predicted answers and explanations. First, we utilize pretrained models to extract visual features and convert questions into the corresponding programs. Second, we propose a multimodal program Transformer to translate programs and the related visual features into coherent and rational explanations of the reasoning processes. Finally, we propose a variational causal inference to construct the target structural causal model and predict answers based on the causal correlation to explanations. Comprehensive experiments conducted on EVQA benchmark datasets reveal the superiority of Pro-VCIN in terms of both performance and explainability over state-of-the-art EVQA methods.

Vision + X: A Survey on Multimodal Learning in the Light of Data.

We are perceiving and communicating with the world in a multisensory manner, where different information sources are sophisticatedly processed and interpreted by separate parts of the human brain to constitute a complex, yet harmonious and unified sensing system. To endow the machines with true intelligence, multimodal machine learning that incorporates data from various sources has become an increasingly popular research area with emerging technical advances in recent years. In this paper, we present a survey on multimodal machine learning from a novel perspective considering not only the purely technical aspects but also the intrinsic nature of different data modalities. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions, and then present the methodological advancements categorized by the combination of data modalities, such as Vision+Text, with slightly inclined emphasis on the visual data. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels, and provide an additional comparison in the light of their technical connections with the data nature, e.g., the semantic consistency between image objects and textual descriptions, and the rhythm correspondence between video dance moves and musical beats. We hope that the exploitation of the alignment as well as the existing gap between the intrinsic nature of data modality and the technical designs, will benefit future research studies to better address a specific challenge related to the concrete multimodal task, prompting a unified multimodal machine learning framework closer to a real human intelligence system.

Backdoor Attacks and Defenses Targeting Multi-Domain AI Models: A Comprehensive Review

Since the emergence of security concerns in artificial intelligence (AI), there has been significant attention devoted to the examination of backdoor attacks. Attackers can utilize backdoor attacks to manipulate model predictions, leading to significant potential harm. However, current research on backdoor attacks and defenses in both theoretical and practical fields still has many shortcomings. To systematically analyze these shortcomings and address the lack of comprehensive reviews, this paper presents a comprehensive and systematic summary of both backdoor attacks and defenses targeting multi-domain AI models. Simultaneously, based on the design principles and shared characteristics of triggers in different domains and the implementation stages of backdoor defense, this paper proposes a new classification method for backdoor attacks and defenses. We use this method to extensively review backdoor attacks in the fields of computer vision and natural language processing, and also examine the current applications of backdoor attacks in audio recognition, video action recognition, multimodal tasks, time series tasks, generative learning, and reinforcement learning, while critically analyzing the open problems of various backdoor attack techniques and defense strategies. Finally, this paper builds upon the analysis of the current state of AI security to further explore potential future research directions for backdoor attacks and defenses.

NeuroMechFly v2: simulating embodied sensorimotor control in adult Drosophila.

Discovering principles underlying the control of animal behavior requires a tight dialogue between experiments and neuromechanical models. Such models have primarily been used to investigate motor control with less emphasis on how the brain and motor systems work together during hierarchical sensorimotor control. NeuroMechFly v2 expands Drosophila neuromechanical modeling by enabling vision, olfaction, ascending motor feedback and complex terrains that can be navigated using leg adhesion. We illustrate its capabilities by constructing biologically inspired controllers that use ascending feedback to perform path integration and head stabilization. After adding vision and olfaction, we train a controller using reinforcement learning to perform a multimodal navigation task. Finally, we illustrate more bio-realistic modeling involving complex odor plume navigation, and fly-fly following using a connectome-constrained visual network. NeuroMechFly can be used to accelerate the discovery of explanatory models of the nervous system and to develop machine learning-based controllers for autonomous artificial agents and robots.

LRCN: Layer-residual Co-Attention Networks for visual question answering

Visual Question Answering (VQA) is a multimodal task requiring a collaborative understanding of fine-grained visual concepts and language semantics. The key to VQA research is constructing a framework for modeling inter and intramodal information interactions between intricate modalities. Due to the superior global view and the ability to capture the relationships within multimodal information, the Transformer and its variants have become the prime choice for VQA tasks. Despite this layer-by-layer architecture enabling answer reasoning by optimizing multimodal feature information, the information may still be lost when transferred from lower to higher layers. To solve such an issue, we propose a Layer-Residual Mechanism (LRM), a plug-and-play generic approach that can reduce the computation and memory overhead to almost negligible. By adding a residual straight-through line between adjacent layers that cascades the attention block in-depth, LRM mitigates the vanishing during information transfer, stabilizes training, and endeavors to overcome performance decline in VQA models at deeper layers. To verify the effectiveness and generality of the proposed LRM, we apply it to the Encoder–Decoder structure, the Pure-Stacking structure, and a specifically designed Co-Stacking structure that can simultaneously understand textual features and visual features called Layer-residual Co-Attention Networks (LRCN). Extensive ablation studies and comparative experiments on the benchmark VQA v2 and CLEVR datasets show that LRCN significantly outperforms the original architectures, demonstrating the superior effectiveness and compatibility of LRCN.

A Multimodal conceptual framework to achieve automated software evolution for context-rich intelligent applications

AbstractWhile AI is extensively transforming Software Engineering (SE) fields, SE is still in need of a framework to consider overall all phases to facilitate Automated Software Evolution (ASEv), particularly for intelligent applications that are context-rich instead of conquering each division independently. Its complexity comes from the intricacy of the intelligent applications, the heterogeneity of the data sources, and the constant changes in the context. This study proposes a conceptual framework for achieving automated software evolution, emphasizing the importance of multimodality learning. A Selective Sequential Scope Model (3 S) model is developed based on the conceptual framework, and it can be used to categorize existing and future research when it covers different SE phases and multimodal learning tasks. This research is a preliminary step toward the blueprint of a higher-level ASEv. The proposed conceptual framework can act as a practical guideline for practitioners to prepare themselves for diving into this area. Although the study is about intelligent applications, the framework and analysis methods may be adapted for other types of software as AI brings more intelligence into their life cycles.

Robot imitation from multimodal observation with unsupervised cross-modal representation

Abstract Imitation from Observation (IfO) prompts the robot to imitate tasks from unlabeled videos via reinforcement learning (RL). The performance of the IfO algorithm depends on its ability to extract task-relevant representations since images are informative. Existing IfO algorithms extract image representations by using a simple encoding network or pre-trained network. Due to the lack of action labels, it is challenging to design a supervised task-relevant proxy task to train the simple encoding network. Representations extracted by a pre-trained network such as Resnet are often task-irrelevant. In this article, we propose a new approach for robot IfO via multimodal observations. Different modalities describe the same information from different sides, which can be used to design an unsupervised proxy task. Our approach contains two modules: the unsupervised cross-modal representation (UCMR) module and a self-behavioral cloning (self-BC)-based RL module. The UCMR module learns to extract task-relevant representations via a multimodal unsupervised proxy task. The Self-BC for further offline policy optimization collects successful experiences during the RL training. We evaluate our approach on the real robot pouring water task, quantitative pouring task, and pouring sand task. The robot achieves state-of-the-art performance.

ME[formula omitted]A: A Multimodal Entity Entailment framework for multimodal Entity Alignment

Current methods for multimodal entity alignment (MEA) primarily rely on entity representation learning, which undermines entity alignment performance because of cross-KG interaction deficiency and multimodal heterogeneity. In this paper, we propose a Multimodal Entity Entailment framework of multimodal Entity Alignment task, ME3A, and recast the MEA task as an entailment problem about entities in the two KGs. This way, the cross-KG modality information directly interacts with each other in the unified textual space. Specifically, we construct the multimodal information in the unified textual space as textual sequences: for relational and attribute modalities, we combine the neighbors and attribute values of entities as sentences; for visual modality, we map the entity image as trainable prefixes and insert them into sequences. Then, we input the concatenated sequences of two entities into the pre-trained language model (PLM) as an entailment reasoner to capture the unified fine-grained correlation pattern of the multimodal tokens between entities. Two types of entity aligners are proposed to model the bi-directional entailment probability as the entity similarity. Extensive experiments conducted on nine MEA datasets with various modality combination settings demonstrate that our ME3A effectively incorporates multimodal information and surpasses the performance of the state-of-the-art MEA methods by 16.5% at most.

Black-box backdoor attack with everyday physical objects in mobile crowdsourcing

Mobile crowdsourcing (MCS) has emerged as a promising sensing paradigm in smart cities as it reduces the need of expensive static infrastructure and facilitates efficient data collection with many volunteers. However, malicious users may contribute tampered data deliberately, compromising the model trained with them. Prior neural adversarial attacks usually rely on several major assumptions, such as requiring expert knowledge of models, manipulating the training process, or generating specific patterns for poisoning, which are not always feasible for novice volunteers.In this paper, we identify an attack paradigm that can be easily executed without prior knowledge or extra manipulation of training. As an example, we conduct a crowdsourcing task for visual landmark classification and realize attacks with everyday objects (e.g., a cola cup). Extensive experimental results show that our approach achieves high attack success rates (over 90%) with marginal poisoning rates (less than 10%). To safeguard the trained model, we propose a defense framework that harnesses the tamper resistance between multi-modal sensors to erase the infectious model, termed MUSE. Besides visual landmark identification, we also apply MUSE to two multi-modal localization tasks. Experimental results show that MUSE substantially reduces attack success rate to less than 1%.

GeoPredict-LLM: Intelligent tunnel advanced geological prediction by reprogramming large language models

With the improvement of multisource information sensing and data acquisition capabilities inside tunnels, the availability of multimodal data in tunnel engineering has significantly increased. However, due to structural differences in multimodal data, traditional intelligent advanced geological prediction models have limited capacity for data fusion. Furthermore, the lack of pre-trained models makes it difficult for neural networks trained from scratch to deeply explore the features of multimodal data. To address these challenges, we utilize the fusion capability of knowledge graph for multimodal data and the pre-trained knowledge of large language models (LLMs) to establish an intelligent advanced geological prediction model (GeoPredict-LLM). First, we develop an advanced geological prediction ontology model, forming a knowledge graph database. Using knowledge graph embeddings, multisource and multimodal data are transformed into low-dimensional vectors with a unified structure. Secondly, pre-trained LLMs, through reprogramming, reconstruct these low-dimensional vectors, imparting linguistic characteristics to the data. This transformation effectively reframes the complex task of advanced geological prediction as a "language-based" problem, enabling the model to approach the task from a linguistic perspective. Moreover, we propose the prompt-as-prefix method, which enables output generation, while freezing the core of the LLM, thereby significantly reduces the number of training parameters. Finally, evaluations show that compared to neural network models without pre-trained models, GeoPredict-LLM significantly improves prediction accuracy. It is worth noting that as long as a knowledge graph database can be established, GeoPredict-LLM can be adapted to multimodal data mining tasks with minimal modifications.

WaterGPT: Training a Large Language Model to Become a Hydrology Expert

This paper introduces WaterGPT, a language model designed for complex multimodal tasks in hydrology. WaterGPT is applied in three main areas: (1) processing and analyzing data such as images and text in water resources, (2) supporting intelligent decision-making for hydrological tasks, and (3) enabling interdisciplinary information integration and knowledge-based Q&A. The model has achieved promising results. One core aspect of WaterGPT involves the meticulous segmentation of training data for the supervised fine-tuning phase, sourced from real-world data and annotated with high quality using both manual methods and GPT-series model annotations. These data are carefully categorized into four types: knowledge-based, task-oriented, negative samples, and multi-turn dialogues. Additionally, another key component is the development of a multi-agent framework called Water_Agent, which enables WaterGPT to intelligently invoke various tools to solve complex tasks in the field of water resources. This framework handles multimodal data, including text and images, allowing for deep understanding and analysis of complex hydrological environments. Based on this framework, WaterGPT has achieved over a 90% success rate in tasks such as object detection and waterbody extraction. For the waterbody extraction task, using Dice and mIoU metrics, WaterGPT’s performance on high-resolution images from 2013 to 2022 has remained stable, with accuracy exceeding 90%. Moreover, we have constructed a high-quality water resources evaluation dataset, EvalWater, which covers 21 categories and approximately 10,000 questions. Using this dataset, WaterGPT achieved the highest accuracy to date in the field of water resources, reaching 83.09%, which is about 17.83 points higher than GPT-4.

Multimodal dual perception fusion framework for multimodal affective analysis

The misuse of social platforms and the difficulty in regulating post contents have culminated in a surge of negative sentiments, sarcasms, and the rampant spread of fake news. In response, Multimodal sentiment analysis, sarcasm detection and fake news detection based on image and text have attracted considerable attention recently. Due to that these areas share semantic and sentiment features and confront related fusion challenges in deciphering complex human expressions across different modalities, integrating these multimodal classification tasks that share commonalities across different scenarios into a unified framework is expected to simplify research in sentiment analysis, and enhance the effectiveness of classification tasks involving both semantic and sentiment modeling. Therefore, we consider integral components of a broader spectrum of research known as multimodal affective analysis towards semantics and sentiment, and propose a novel multimodal dual perception fusion framework (MDPF). Specifically, MDPF contains three core procedures: (1) Generating bootstrapping language-image Knowledge to enrich origin modality space, and utilizing cross-modal contrastive learning for aligning text and image modalities to understand underlying semantics and interactions. (2) Designing dynamic connective mechanism to adaptively match image-text pairs and jointly employing gaussian-weighted distribution to intensify semantic sequences. (3) Constructing a cross-modal graph to preserve the structured information of both image and text data and share information between modalities, while introducing sentiment knowledge to refine the edge weights of the graph to capture cross-modal sentiment interaction. We evaluate MDPF on three publicly available datasets across three tasks, and the empirical results demonstrate the superiority of our proposed model.

Estrogen predicts multimodal emotion recognition accuracy across the menstrual cycle.

Researchers have proposed that variation in sex hormones across the menstrual cycle modulate the ability to recognize emotions in others. Existing research suggests that accuracy is higher during the follicular phase and ovulation compared to the luteal phase, but findings are inconsistent. Using a repeated measures design with a sample of healthy naturally cycling women (N = 63), we investigated whether emotion recognition accuracy varied between the follicular and luteal phases, and whether accuracy related to levels of estrogen (estradiol) and progesterone. Two tasks assessed recognition of a range of positive and negative emotions via brief video recordings presented in visual, auditory, and multimodal blocks, and non-linguistic vocalizations (e.g., laughter, sobs, and sighs). Multilevel models did not show differences in emotion recognition between cycle phases. However, coefficients for estrogen were significant for both emotion recognition tasks. Higher within-person levels of estrogen predicted lower accuracy, whereas higher between-person estrogen levels predicted greater accuracy. This suggests that in general having higher estrogen levels increases accuracy, but that higher-than-usual estrogen at a given time decreases it. Within-person estrogen further interacted with cycle phase for both tasks and showed a quadratic relationship with accuracy for the multimodal task. In particular, women with higher levels of estrogen were more accurate in the follicular phase and middle of the menstrual cycle. We propose that the differing role of within- and between-person hormone levels could explain some of the inconsistency in previous findings.

A novel mobile robot with origami wheels designed for navigating sandy terrains

Abstract Sand robots play a vital role as mobile tools for human exploration of desert regions, facilitating resource transportation and exploration. However, desert areas primarily consist of beaches or dunes, resulting in a highly diverse and complex terrain environment that demands enhanced adaptability from sandy mobile robots. Traditional wheeled robots currently face challenges such as skidding, limited climbing ability, and inadequate obstacle avoidance capabilities in sandy environments. To address these issues and enable effective adaptation to the intricate sand environment, we propose a novel sandy mobile robot equipped with Kresling origami wheels. The origami wheel can dynamically adjust its width and morphology through Kresling origami folding. Experimental tests were conducted to illustrate the impact of width variation on the robot’s mobility velocity, propulsive force, climbing performance, and carrying capacity. The self-folding malleability of the origami wheel empowers the robot to efficiently accomplish diverse tasks, including swift movement on flat sand surfaces, seamless crossing of narrow channels, and intelligent obstacle avoidance. By successfully completing these multimodal tasks while adapting to varying requirements, our robot demonstrates promising prospects for practical applications of mobile robots equipped with origami wheels – paving the way for wider adaptation and utilization of sand mobile robots.

Identifying Implicit Social Biases in Vision-Language Models

Vision-language models, like CLIP (Contrastive Language Image Pretraining), are becoming increasingly popular for a wide range of multimodal retrieval tasks. However, prior work has shown that large language and deep vision models can learn historical biases contained in their training sets, leading to perpetuation of stereotypes and potential downstream harm. In this work, we conduct a systematic analysis of the social biases that are present in CLIP, with a focus on the interaction between image and text modalities. We first propose a taxonomy of social biases called So-B-It, which contains 374 words categorized across ten types of bias. Each type can lead to societal harm if associated with a particular demographic group. Using this taxonomy, we examine images retrieved by CLIP from a facial image dataset using each word as part of a prompt. We find that CLIP frequently displays undesirable associations between harmful words and specific demographic groups, such as retrieving mostly pictures of Middle Eastern men when asked to retrieve images of a "terrorist". Finally, we conduct an analysis of the source of such biases, by showing that the same harmful stereotypes are also present in a large image-text dataset used to train CLIP models for examples of biases that we find. Our findings highlight the importance of evaluating and addressing bias in vision-language models, and suggest the need for transparency and fairness-aware curation of large pre-training datasets.

Wearable EEG-Based Brain-Computer Interface for Stress Monitoring.

Detecting stress is important for improving human health and potential, because moderate levels of stress may motivate people towards better performance at cognitive tasks, while chronic stress exposure causes impaired performance and health risks. We propose a Brain-Computer Interface (BCI) system to detect stress in the context of high-pressure work environments. The BCI system includes an electroencephalogram (EEG) headband with dry electrodes and an electrocardiogram (ECG) chest belt. We collected EEG and ECG data from 40 participants during two stressful cognitive tasks: the Cognitive Vigilance Task (CVT), and the Multi-Modal Integration Task (MMIT) we designed. We also recorded self-reported stress levels using the Dundee Stress State Questionnaire (DSSQ). The DSSQ results indicated that performing the MMIT led to significant increases in stress, while performing the CVT did not. Subsequently, we trained two different models to classify stress from non-stress states, one using EEG features, and the other using heart rate variability (HRV) features extracted from the ECG. Our EEG-based model achieved an overall accuracy of 81.0% for MMIT and 77.2% for CVT. However, our HRV-based model only achieved 62.1% accuracy for CVT and 56.0% for MMIT. We conclude that EEG is an effective predictor of stress in the context of stressful cognitive tasks. Our proposed BCI system shows promise in evaluating mental stress in high-pressure work environments, particularly when utilizing an EEG-based BCI.

A unified multimodal classification framework based on deep metric learning

Multimodal classification algorithms play an essential role in multimodal machine learning, aiming to categorize distinct data points by analyzing data characteristics from multiple modalities. Extensive research has been conducted on distilling multimodal attributes and devising specialized fusion strategies for targeted classification tasks. Nevertheless, current algorithms mainly concentrate on a specific classification task and process data about the corresponding modalities. To address these limitations, we propose a unified multimodal classification framework proficient in handling diverse multimodal classification tasks and processing data from disparate modalities. UMCF is task-independent, and its unimodal feature extraction module can be adaptively substituted to accommodate data from diverse modalities. Moreover, we construct the multimodal learning scheme based on deep metric learning to mine latent characteristics within multimodal data. Specifically, we design the metric-based triplet learning to extract the intra-modal relationships within each modality and the contrastive pairwise learning to capture the inter-modal relationships across various modalities. Extensive experiments on two multimodal classification tasks, fake news detection and sentiment analysis, demonstrate that UMCF can extract multimodal data features and achieve superior classification performance than task-specific benchmarks. UMCF outperforms the best fake news detection baselines by 2.3% on average regarding F1 scores.

Building a Multimodal Dataset of Academic Paper for Keyword Extraction

ABSTRACTUp to this point, keyword extraction task typically relies solely on textual data. Neglecting visual details and audio features from image and audio modalities leads to deficiencies in information richness and overlooks potential correlations, thereby constraining the model's ability to learn representations of the data and the accuracy of model predictions. Furthermore, the currently available multimodal datasets for keyword extraction task are particularly scarce, further hindering the progress of research on multimodal keyword extraction task. Therefore, this study constructs a multimodal dataset of academic paper consisting of 1,000 samples, with each sample containing paper text, images, audios and keywords. Based on unsupervised and supervised methods of keyword extraction, experiments are conducted using textual data from papers, as well as text extracted from images and audio. The aim is to investigate the differences in performance in keyword extraction task with respect to different modal information and the fusion of multimodal information. The experimental results indicate that text from different modalities exhibits distinct characteristics in the model. The concatenation of paper text, image text and audio text can effectively enhance the keyword extraction performance of academic papers.