Multimodal Signals Research Articles

Evaluating Desk-Assisted Standing Techniques for Simulated Pregnant Conditions: An Experimental Study Using a Maternity-Simulation Jacket.

Lower back pain, a common issue among pregnant women, often complicates daily activities like standing up from a chair. Therefore, research into the standing motion of pregnant women is important, and many research studies have already been conducted. However, many of these studies were conducted in highly controlled environments, overlooking everyday scenarios such as using a desk for support when standing up, and their effects have not been adequately tested. To address this gap, we measured multimodal signals for a sit-to-stand (STS) movement with hand assistance and verified the changes using a t-test. To avoid imposing strain on pregnant women, we used 10 non-diseased young adults who wore jackets designed to simulate pregnancy conditions, thus allowing for more comprehensive and rigorous experimentation. We attached surface electromyography (sEMG) sensors to the erector spinae muscles of participants and measured changes in muscle activity, skeletal positioning, and center of pressure both before and after wearing a Maternity-Simulation Jacket. Our analysis showed that the jacket successfully mimicked key aspects of the movement patterns typical in pregnant women. These results highlight the possibility of developing practical strategies that more accurately mirror the real-life scenarios met by pregnant women, enriching the current research on their STS movement.

Алгоритмы передачи информации в системах управления биологическими объектами

One of the problems of the theory of management of objects with a high degree of uncertainty of behavior is solved: the development of theoretical foundations of analysis, modeling methods and improvement of bio cybernetic systems, their algorithmic and software necessary to improve the efficiency of the management process is proposed. To solve this problem, it is proposed to use the mathematical apparatus of information theory, on the basis of which the information characteristics of the control signal transmission channel from the source to the control object through the external environment are calculated, taking into account the noise of the ecosystem of the control object. The existing systems using monomodal control signals that do not carry a semantic load are analyzed, and the disadvantages and limited possibilities of improving the control systems under consideration are described. Using the example of one of the insect behavior control systems with a trichromic view, a comparison of the results obtained using various technical means used as control signal sources was performed. For improving the efficiency of control systems, which use a multimodal control signal structure, proposed various algorithms for constructing control systems using a multimodal control signal structure are considered, namely: serial and parallel circuits, combined, combining serial and parallel algorithms for using control signal sources. As a method for further improving the efficiency of control systems for biological objects with a high degree of uncertainty, it is proposed to use biological signals containing a large semantic load for the control object, which determines its behavioral response. The directions of further research are formulated in order to maximize the amount of information transmitted to the control object.

Tactile texture recognition of multi-modal bionic finger based on multi-modal CBAM-CNN interpretable method

To solve the problem of lacking surface information while using a single-modal sensor to scan fine textures, a multi-modal bionic finger tactile sensor is used to perceive the information contained in textures, and a multi-modal tactile texture recognition method is proposed to address the issue of poor interpretability for tactile texture recognition models. First, the experimental platform of multi-modal bionic finger tactile texture information acquisition system is built, which can simultaneously collect 14 channels of tactile electrode signals, one channel of pressure signals and one channel of temperature signals. Second, the multi-modal tactile signals can be fused at the data level after signal processing, and then the wavelet transform method is carried out on the basis of data fusion. Third, a multi-modal CBAM-CNN network (Convolutional Block Attention Module, Convolutional Neural Network) is established to extract the multi-modal tactile features after the wavelet transform and the multi-modal tactile features are fused at feature level. To verify the performance of multi-modal CBAM-CNN model, 27(3*9) kinds of millimeter-level fine textures of 3 datasets are tested and the recognition accuracy of each dataset is 98.47%, 97.63%, 99.16%. And it can also be seen that the 14-channel modal data contributes the most to the performance of multi-modal CBAM-CNN model for various textures. The results of ablation experiments prove that the identification accuracy of the fused multi-modal CBAM-CNN is higher than the single-modal CBAM-CNN no matter in time domain or time–frequency domain. Moreover, the accuracy of the weighted fusion method based on correlation function is higher than horizontal fusion method in 14 channels of tactile electrode signals modal. Finally, to interpret the classification decision of the multi-modal CBAM-CNN network in each modal intuitively, the improved multi-modal CAM (Class Activation Map) algorithm is adopted to visualize the interest regions of the samples in detail, and the activated areas in the map make the identification result of CBAM-CNN in each modal more transparent and interpretable.

Levels of Airborne Sound And Substrate-borne Vibration Calling Are Negatively Related Across Neotropical False-leaf Katydids.

Animals often signal in multiple sensory modalities to attract mates, but the level of signaling investment in each modality can differ dramatically between individuals and across species. When functionally overlapping signals are produced in different modalities, their relative use can be influenced by many factors, including differences in signal active space, energetic costs, and predation risk. Characterizing differences in total signal investment across time can shed light on these factors, but requires long focal recordings of signal production. Neotropical pseudophylline katydids produce mate advertisement signals as airborne sound and substrate-borne vibration. Airborne calls, produced via stridulation, are extremely short, high-frequency, and longer-range signals. Conversely, substrate-borne calls produced via abdominal tremulation are longer, low-frequency, relatively more energetically costly, and shorter-range signals. To examine patterns of stridulation and tremulation across species and test hypotheses about the drivers of signal use in each modality, we recorded multimodal signaling activity over 24 hours for males from 10 pseudophylline species from a single Panamanian community. We also collected data on demographic and morphological species characteristics, and acoustic features of airborne calls, such as bandwidth, peak frequency, and duration. Finally, we generated a molecular phylogeny for these species and used phylogenetic generalized least squares models to test for relationships between variables while controlling for evolutionary relationships. We found a negative relationship between sound and vibration calling, indicating that substrate-borne vibrational signaling may compensate for reduced airborne signaling in these species. Sound call bandwidth and the proportion of males collected at lights, a proxy for the amount of male movement, also explained a significant amount of variation in sound calling across species, indicating that the overall relationship between the two types of calling signals may be mediated by the specific characteristics of the signals as well as other species traits.

AFLEMP: Attention-based Federated Learning for Emotion recognition using Multi-modal Physiological data

Automated emotion recognition systems utilizing physiological signals are essential for affective computing and intelligent interaction. Combining the multiple physiological signals is more precise and effective in accurately assessing a person’s emotional state. These automated emotion recognition systems using conventional machine learning techniques require complete access to the physiological data for emotion state classification, compromising sensitive data privacy. Federated Learning (FL) resolves this issue by preserving the user’s privacy and sensitive physiological data while recognizing emotions. However, existing FL methods have limitations in handling data heterogeneity in the physiological data and do not measure communication efficiency and scalability. In response to these challenges, this paper proposes a unique novel framework called AFLEMP (Attention-based Federated Learning for Emotion recognition using Multi-modal Physiological data) integrating attention mechanism-based Transformer with an Artificial Neural Network (ANN) model. The framework reduces two types of data heterogeneity: (1) Variation Heterogeneity (VH) in multi-modal EEG, GSR, and ECG physiological signal data using attention mechanisms and (2) Imbalanced Data Heterogeneity (IDH) in the FL environment using scaled weighted federated averaging. This paper validates the proposed AFLEMP framework on two publicly available emotion datasets, AMIGOS and DREAMER, achieving an average accuracy of 88.30% and 84.10%, respectively. The proposed AFLEMP framework proves robust, scalable, and efficient in communication. AFLEMP is the first FL framework to propose for emotion recognition using multi-modal physiological signals while reducing data heterogeneity and outperforming existing FL methods.

A multimodal shared network with a cross-modal distribution constraint for continuous emotion recognition

Continuous emotion recognition has been a compelling topic in affective computing because it can interpret human emotions subtly and continuously. Existing studies have achieved advanced emotion recognition performance using multimodal knowledge. However, these studies generally ignore the circumstances where some particular modalities are missing in the inference phase and thus become sensitive to the absence of modalities. To resolve this issue, we propose a novel multimodal shared network with a cross-modal distribution constraint, i.e. the DS-Net, which aims to improve the robustness of the model to missing modalities. The training process of the proposed network generally includes two components: multimodal shared space modeling and a cross-modal distribution matching constraint. The former utilizes the local and temporal information of multimodal signals for multimodal shared space modeling, while the latter further enhances the multimodal shared space via a loose constraint method. Coupled with the latter, the former can effectively exploit the complementarity between videos and peripheral physiological signals (PPSs), thus enhancing the discriminative capability of the shared space. Based on the shared space, the DS-Net works during the inference phase with only one modality input and can leverage multimodal knowledge to improve emotion recognition accuracy. Comprehensive experiments were conducted on two public datasets. Results demonstrate that the proposed method is competitive or superior to the current state-of-the-art methods. Further experiments indicate that the proposed method can be extended to handle other modalities and to deal with partially missing modalities, demonstrating its potential in real-world applications.

Chemical species recognition in an adaptive radiation of Hawaiian Tetragnatha spiders (Araneae: Tetragnathidae).

Studies of adaptive radiations have played a central role in our understanding of reproductive isolation. Yet the focus has been on human-biased visual and auditory signals, leaving gaps in our knowledge of other modalities. To date, studies on chemical signals in adaptive radiations have focused on systems with multimodal signalling, making it difficult to isolate the role chemicals play in reproductive isolation. In this study we examine the use of chemical signals in the species recognition and adaptive radiation of Hawaiian Tetragnatha spiders by focusing on entire communities of co-occurring species, and conducting behavioural assays in conjunction with chemical analysis of their silks using gas chromatography-mass spectrometry. Male spiders significantly preferred the silk extracts of conspecific mates over those of sympatric heterospecifics. The compounds found in the silk extracts, long chain alkyl methyl ethers, were remarkably species-specific in the combination and quantity. The differences in the profile were greatest between co-occurring species and between closely related sibling species. Lastly, there were significant differences in the chemical profile between two populations of a particular species. These findings provide key insights into the role chemical signals play in the attainment and maintenance of reproductive barriers between closely related co-occurring species.

Comparing multimodal physiological responses to social and physical pain in healthy participants.

Previous physiology-driven pain studies focused on examining the presence or intensity of physical pain. However, people experience various types of pain, including social pain, which induces negative mood; emotional distress; and neural activities associated with physical pain. In particular, comparison of autonomic nervous system (ANS) responses between social and physical pain in healthy adults has not been well demonstrated. We explored the ANS responses induced by two types of pain-social pain, associated with a loss of social ties; and physical pain, caused by a pressure cuff-based on multimodal physiological signals. Seventy-three healthy individuals (46 women; mean age = 20.67 ± 3.27 years) participated. Behavioral responses were assessed to determine their sensitivity to pain stimuli. Electrocardiogram, electrodermal activity, photoplethysmogram, respiration, and finger temperature (FT) were measured, and 12 features were extracted from these signals. Social pain induced increased heart rate (HR) and skin conductance (SC) and decreased blood volume pulse (BVP), pulse transit time (PTT), respiration rate (RR), and FT, suggesting a heterogeneous pattern of sympathetic-parasympathetic coactivation. Moreover, physical pain induced increased heart rate variability (HRV) and SC, decreased BVP and PTT, and resulted in no change in FT, indicating sympathetic-adrenal-medullary activation and peripheral vasoconstriction. These results suggest that changes in HR, HRV indices, RR, and FT can serve as markers for differentiating physiological responses to social and physical pain stimuli.

Enhanced multimodal emotion recognition in healthcare analytics: A deep learning based model-level fusion approach

Deep learning techniques have drawn considerable interest in emotion recognition due to recent technological developments in healthcare analytics. Automatic patient emotion recognition can assist healthcare analytics by providing feedback to the stakeholders of competent healthcare about the conditions of the patients and their satisfaction levels. In this paper, we propose a novel model-level fusion technique based on deep learning for enhanced emotion recognition from multimodal signals to monitor patients in connected healthcare. The representative visual features from the video signals are extracted through the Depthwise Separable Convolution Neural Network, and the optimized temporal attributes are derived from the multiple physiological data utilizing Bi-directional Long Short-Term Memory. A soft attention method fused the high multimodal features obtained from the two data modalities to retrieve the most significant features by focusing on emotionally salient parts of the features. We exploited two face detection methods, Histogram of Oriented Gradients and Convolutional Neural Network-based face detector (ResNet-34), to observe the effects of facial features on emotion recognition. Lastly, extensive experimental evaluations have been conducted using the widely used Bio Vid Emo DB multimodal dataset to verify the performance of the proposed architecture. Experimental results show that the developed fusion architecture improved the accuracy of emotion recognition from multimodal signals and outperformed the performance of both state-of-the-art techniques and baseline methods.

Event detection and classification from multimodal time series with application to neural data

The detection of events in time-series data is a common signal-processing problem. When the data can be modeled as a known template signal with an unknown delay in Gaussian noise, detection of the template signal can be done with a traditional matched filter. However, in many applications, the event of interest is represented in multimodal data consisting of both Gaussian and point-process time series. Neuroscience experiments, for example, can simultaneously record multimodal neural signals such as local field potentials (LFPs), which can be modeled as Gaussian, and neuronal spikes, which can be modeled as point processes. Currently, no method exists for event detection from such multimodal data, and as such our objective in this work is to develop a method to meet this need. Here we address this challenge by developing the multimodal event detector (MED) algorithm which simultaneously estimates event times and classes. To do this, we write a multimodal likelihood function for Gaussian and point-process observations and derive the associated maximum likelihood estimator of simultaneous event times and classes. We additionally introduce a cross-modal scaling parameter to account for model mismatch in real datasets. We validate this method in extensive simulations as well as in a neural spike-LFP dataset recorded during an eye-movement task, where the events of interest are eye movements with unknown times and directions. We show that the MED can successfully detect eye movement onset and classify eye movement direction. Further, the MED successfully combines information across data modalities, with multimodal performance exceeding unimodal performance. This method can facilitate applications such as the discovery of latent events in multimodal neural population activity and the development of brain-computer interfaces for naturalistic settings without constrained tasks or prior knowledge of event times.

Cross-modal credibility modelling for EEG-based multimodal emotion recognition

Objective. The study of emotion recognition through electroencephalography (EEG) has garnered significant attention recently. Integrating EEG with other peripheral physiological signals may greatly enhance performance in emotion recognition. Nonetheless, existing approaches still suffer from two predominant challenges: modality heterogeneity, stemming from the diverse mechanisms across modalities, and fusion credibility, which arises when one or multiple modalities fail to provide highly credible signals. Approach. In this paper, we introduce a novel multimodal physiological signal fusion model that incorporates both intra-inter modality reconstruction and sequential pattern consistency, thereby ensuring a computable and credible EEG-based multimodal emotion recognition. For the modality heterogeneity issue, we first implement a local self-attention transformer to obtain intra-modal features for each respective modality. Subsequently, we devise a pairwise cross-attention transformer to reveal the inter-modal correlations among different modalities, thereby rendering different modalities compatible and diminishing the heterogeneity concern. For the fusion credibility issue, we introduce the concept of sequential pattern consistency to measure whether different modalities evolve in a consistent way. Specifically, we propose to measure the varying trends of different modalities, and compute the inter-modality consistency scores to ascertain fusion credibility. Main results. We conduct extensive experiments on two benchmarked datasets (DEAP and MAHNOB-HCI) with the subject-dependent paradigm. For the DEAP dataset, our method improves the accuracy by 4.58%, and the F1 score by 0.63%, compared to the state-of-the-art baseline. Similarly, for the MAHNOB-HCI dataset, our method improves the accuracy by 3.97%, and the F1 score by 4.21%. In addition, we gain much insight into the proposed framework through significance test, ablation experiments, confusion matrices and hyperparameter analysis. Consequently, we demonstrate the effectiveness of the proposed credibility modelling through statistical analysis and carefully designed experiments. Significance. All experimental results demonstrate the effectiveness of our proposed architecture and indicate that credibility modelling is essential for multimodal emotion recognition.

MGEED: A Multimodal Genuine Emotion and Expression Detection Database

Multimodal emotion recognition has attracted increasing interest from academia and industry in recent years, since it enables emotion detection using various modalities, such as facial expression images, speech and physiological signals. Although research in this field has grown rapidly, it is still challenging to create a multimodal database containing facial electrical information due to the difficulty in capturing natural and subtle facial expression signals, such as optomyography (OMG) signals. To this end, we present a newly developed Multimodal Genuine Emotion and Expression Detection (MGEED) database in this paper, which is the first publicly available database containing the facial OMG signals. MGEED consists of 17 subjects with over 150K facial images, 140K depth maps and different modalities of physiological signals including OMG, electroencephalography (EEG) and electrocardiography (ECG) signals. The emotions of the participants are evoked by video stimuli and the data are collected by a multimodal sensing system. With the collected data, an emotion recognition method is developed based on multimodal signal synchronisation, feature extraction, fusion and emotion prediction. The results show that superior performance can be achieved by fusing the visual, EEG and OMG features. The database can be obtained from <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/YMPort/MGEED</uri> .

Multisensory Integration Underlies the Distinct Representation of Odor-Taste Mixtures in the Gustatory Cortex of Behaving Rats.

The perception of food relies on the integration of olfactory and gustatory signals originating from the mouth. This multisensory process generates robust associations between odors and tastes, significantly influencing the perceptual judgment of flavors. However, the specific neural substrates underlying this integrative process remain unclear. Previous electrophysiological studies identified the gustatory cortex as a site of convergent olfactory and gustatory signals, but whether neurons represent multimodal odor-taste mixtures as distinct from their unimodal odor and taste components is unknown. To investigate this, we recorded single-unit activity in the gustatory cortex of behaving female rats during the intraoral delivery of individual odors, individual tastes, and odor-taste mixtures. Our results demonstrate that chemoselective neurons in the gustatory cortex are broadly responsive to intraoral chemosensory stimuli, exhibiting time-varying multiphasic changes in activity. In a subset of these chemoselective neurons, odor-taste mixtures elicit nonlinear cross-modal responses that distinguish them from their olfactory and gustatory components. These findings provide novel insights into multimodal chemosensory processing by the gustatory cortex, highlighting the distinct representation of unimodal and multimodal intraoral chemosensory signals. Overall, our findings suggest that olfactory and gustatory signals interact nonlinearly in the gustatory cortex to enhance the identity coding of both unimodal and multimodal chemosensory stimuli.

DANNMCTG: Domain-Adversarial Training of Neural Network for multicenter antenatal cardiotocography signal classification

Intelligent classification of cardiotocography (CTG) based on machine learning (ML), a useful tool to improve the accuracy of fetal abnormality detection, can assist obstetricians with clinical decisions. With the advancement of information technologies and medical devices, there are development opportunities for multicenter clinical research and obtaining more digital CTG signals. However, most of the existing clinical multicenter CTG datasets are partially annotated and have discrepancies which do not satisfy the ML condition of independent identical distribution. Therefore, this paper focuses on an unsupervised domain adaptation (UDA) algorithm to realize cross-domain intelligent classification of multimodal CTG signals. We propose a method dubbed domain adversarial training of neural network for multicenter CTG (DANNMCTG), which mainly consists of a label classifier, a feature extractor and a domain discriminator. To match different distribution of fetal heart rate (FHR), uterine contraction (UC) and fetal movement (FetMov) signals, we condition the domain alignment on label predictions by defining the multi-linear map. For analysis, two datasets from the hospital central station and home monitoring devices were considered as the source and target domains. The results showed that the accuracy value, F1 value and area under the curve (AUC) value of the DANNMCTG were 71.25%, 76.08% and 0.7705, respectively. This method significantly improved the performance of the deep learning models without exploiting any information in the target domain, and outperformed the state-of-the-art UDA algorithms for CTG classification. In summary, the DANNMCTG can effectively mitigate the influence of domain shift for multicenter intelligent prenatal fetal monitoring.

MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators

Generating realistic human motion from given action descriptions has experienced significant advancements because of the emerging requirement of digital humans. While recent works have achieved impressive results in generating motion directly from textual action descriptions, they often support only a single modality of the control signal, which limits their application in the real digital human industry. This paper presents a Motion General-Purpose generaTor (MotionGPT) that can use multimodal control signals, e.g., text and single-frame poses, for generating consecutive human motions by treating multimodal signals as special input tokens in large language models (LLMs). Specifically, we first quantize multimodal control signals into discrete codes and then formulate them in a unified prompt instruction to ask the LLMs to generate the motion answer. Our MotionGPT demonstrates a unified human motion generation model with multimodal control signals by tuning a mere 0.4% of LLM parameters. To the best of our knowledge, MotionGPT is the first method to generate human motion by multimodal control signals, which we hope can shed light on this new direction. Visit our webpage at https://qiqiapink.github.io/MotionGPT/.

Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation

Recently, video object segmentation (VOS) referred by multi-modal signals, e.g., language and audio, has evoked increasing attention in both industry and academia. It is challenging for exploring the semantic alignment within modalities and the visual correspondence across frames. However, existing methods adopt separate network architectures for different modalities, and neglect the inter-frame temporal interaction with references. In this paper, we propose MUTR, a Multi-modal Unified Temporal transformer for Referring video object segmentation. With a unified framework for the first time, MUTR adopts a DETR-style transformer and is capable of segmenting video objects designated by either text or audio reference. Specifically, we introduce two strategies to fully explore the temporal relations between videos and multi-modal signals. Firstly, for low-level temporal aggregation before the transformer, we enable the multi-modal references to capture multi-scale visual cues from consecutive video frames. This effectively endows the text or audio signals with temporal knowledge and boosts the semantic alignment between modalities. Secondly, for high-level temporal interaction after the transformer, we conduct inter-frame feature communication for different object embeddings, contributing to better object-wise correspondence for tracking along the video. On Ref-YouTube-VOS and AVSBench datasets with respective text and audio references, MUTR achieves +4.2% and +8.7% J&amp;F improvements to state-of-the-art methods, demonstrating our significance for unified multi-modal VOS. Code is released at https://github.com/OpenGVLab/MUTR.

LGMRec: Local and Global Graph Learning for Multimodal Recommendation

The multimodal recommendation has gradually become the infrastructure of online media platforms, enabling them to provide personalized service to users through a joint modeling of user historical behaviors (e.g., purchases, clicks) and item various modalities (e.g., visual and textual). The majority of existing studies typically focus on utilizing modal features or modal-related graph structure to learn user local interests. Nevertheless, these approaches encounter two limitations: (1) Shared updates of user ID embeddings result in the consequential coupling between collaboration and multimodal signals; (2) Lack of exploration into robust global user interests to alleviate the sparse interaction problems faced by local interest modeling. To address these issues, we propose a novel Local and Global Graph Learning-guided Multimodal Recommender (LGMRec), which jointly models local and global user interests. Specifically, we present a local graph embedding module to independently learn collaborative-related and modality-related embeddings of users and items with local topological relations. Moreover, a global hypergraph embedding module is designed to capture global user and item embeddings by modeling insightful global dependency relations. The global embeddings acquired within the hypergraph embedding space can then be combined with two decoupled local embeddings to improve the accuracy and robustness of recommendations. Extensive experiments conducted on three benchmark datasets demonstrate the superiority of our LGMRec over various state-of-the-art recommendation baselines, showcasing its effectiveness in modeling both local and global user interests.

Interpretable prediction of brain activity during conversations from multimodal behavioral signals.

We present an analytical framework aimed at predicting the local brain activity in uncontrolled experimental conditions based on multimodal recordings of participants' behavior, and its application to a corpus of participants having conversations with another human or a conversational humanoid robot. The framework consists in extracting high-level features from the raw behavioral recordings and applying a dynamic prediction of binarized fMRI-recorded local brain activity using these behavioral features. The objective is to identify behavioral features required for this prediction, and their relative weights, depending on the brain area under investigation and the experimental condition. In order to validate our framework, we use a corpus of uncontrolled conversations of participants with a human or a robotic agent, focusing on brain regions involved in speech processing, and more generally in social interactions. The framework not only predicts local brain activity significantly better than random, it also quantifies the weights of behavioral features required for this prediction, depending on the brain area under investigation and on the nature of the conversational partner. In the left Superior Temporal Sulcus, perceived speech is the most important behavioral feature for predicting brain activity, regardless of the agent, while several features, which differ between the human and robot interlocutors, contribute to the prediction in regions involved in social cognition, such as the TemporoParietal Junction. This framework therefore allows us to study how multiple behavioral signals from different modalities are integrated in individual brain regions during complex social interactions.

Exploiting multimodal synthetic data for egocentric human-object interaction detection in an industrial scenario

In this paper, we tackle the problem of Egocentric Human-Object Interaction (EHOI) detection in an industrial setting. To overcome the lack of public datasets in this context, we propose a pipeline and a tool for generating synthetic images of EHOIs paired with several annotations and data signals (e.g., depth maps or segmentation masks). Using the proposed pipeline, we present EgoISM-HOI a new multimodal dataset composed of synthetic EHOI images in an industrial environment with rich annotations of hands and objects. To demonstrate the utility and effectiveness of synthetic EHOI data produced by the proposed tool, we designed a new method that predicts and combines different multimodal signals to detect EHOIs in RGB images. Our study shows that exploiting synthetic data to pre-train the proposed method significantly improves performance when tested on real-world data. Moreover, to fully understand the usefulness of our method, we conducted an in-depth analysis in which we compared and highlighted the superiority of the proposed approach over different state-of-the-art class-agnostic methods. To support research in this field, we publicly release the datasets, source code, and pre-trained models at https://iplab.dmi.unict.it/egoism-hoi.

Online multi-hypergraph fusion learning for cross-subject emotion recognition

Multimodal fusion for emotion recognition has received increasing attention from researchers because of its ability to effectively leverage multimodal complementary information. However, there are two main challenges lead to performance degradation of existing emotion recognition models, which limits the practical use of existing models. One is that multimodal signals are difficult to fuse effectively to respond to the complexity of emotions. The other is that the individual variability and non-stationarity of physiological signals lead to poor performance of the model on new subjects. In particular, existing methods will not work well when faced with emotion recognition of new subjects in online scenarios. In this paper, we propose a novel online multi-hypergraph fusion learning method (OnMHF) to effectively fuse multimodal information and to reduce the difference between training data and test data for online cross-subject emotion recognition. Specifically, in a training phase, a multi-hypergraph fusion is proposed to fuse multimodal physiological signals to effectively obtain emotion-aware information via leveraging multimodal complementary information and high-order correlations among multimodal signals. In an online recognizing phase, an online multi-hypergraph learning is designed to learn online multimodal information from online multimodal data by updating hypergraph structure. As a result, the proposed method can be more effective for emotion recognition of target subjects when target data arrive in an online manner. Experimental results have demonstrated that the proposed method outperforms the baselines and compared state-of-the-art methods in online emotion recognition tasks.