Interactive Behavior Research Articles

Do they like me?

Abstract People are frequently concerned about the impressions they make on others (referred to as metaperceptions), but their insights are often inaccurate. Illustrating the phenomenon called the liking gap, speakers interacting in their first language (L1) and second language (L2) tend to underestimate how much they are liked by their interlocutor, and these judgments often predict their desire to engage in future interaction and collaboration. To understand the scope of this bias and its consequences, we focused on L1–L2 dyadic interaction, examining metaperception as a potential barrier to conversations between university students. We recruited 58 previously unacquainted university students to perform a 10-min academic discussion task between one L1 and one L2 speaker. Afterward, the speakers (a) assessed each other’s interpersonal liking, speaking skill, and interactional behavior; (b) provided their metaperceptions of their interlocutor’s assessments of the same dimensions; and (c) estimated their interest in future interaction with the same interlocutor. All speakers showed a reliable metaperception bias to underestimate their interpersonal liking, speaking skill, and interactional behavior. However, only L1 speakers’ desire to engage in future interaction was associated with their metaperceptions of interpersonal liking. We discuss implications of this finding for understanding and promoting academic communication.

Uncovering bacterial-mammalian cell interactions via single-cell tracking

BackgroundThe interactions between bacterial pathogens and host cells are characterized by a multitude of complexities, leading to a wide range of heterogeneous outcomes. Despite extensive research, we still have a limited understanding of how bacterial motility in complex environments impacts their ability to tolerate antibiotics and adhere to mammalian cell surfaces. The challenge lies in unraveling the complexity of these interactions and developing quantitative microscopy approaches to predict the behavior of bacterial populations.ResultsTo address this challenge, we directed our efforts towards Pseudomonas aeruginosa, a pathogenic bacterium known for producing thick films in the lungs of cystic fibrosis patients, and Escherichia coli, used as a proof of concept to develop and demonstrate our single-cell tracking approaches. Our results revealed that P. aeruginosa exhibits diverse and complex interactions on mammalian cell surfaces, such as adhesion, rotational motion, and swimming, unlike the less interactive behavior of Escherichia coli. Our analysis indicated that P. aeruginosa demonstrated lower mean-squared displacement (MSD) values and greater adherence to mammalian cells compared to E. coli, which showed higher MSD slopes and less frequent adherence. Genetic mutations in membrane proteins of P. aeruginosa resulted in altered displacement patterns and reduced adhesion, with the ΔfliD mutant displaying a more Gaussian displacement distribution and significantly less adherence to mammalian cells. Adhesion and tolerance mechanisms are diverse and complex, potentially involving distinct pathways; however, our findings highlight the therapeutic potential of targeting the fliD gene (encoding a critical flagellum protein), as its deletion not only reduced adherence but also antibiotic tolerance.ConclusionsOverall, our findings underscore the importance of single cell tracking in accurately assessing bacterial behavior over short time periods and highlight its significant potential in guiding effective intervention strategies.

Multimodal Locomotion and Dynamic Interaction of Hydrogel Microdisks at the Air-Water Interface under Magnetic and Light Stimuli.

The potential applications of hydrogel microrobots in biomedicine and environmental exploration have sparked significant interest in understanding their behavior under multiphysical fields. This study explores the multimodal locomotion and dynamic interaction of hydrogel microrobots at the air-water interface under magnetic and light stimuli. A pair of hydrogel microrobots at the air-water interface exhibits a transition from cooperative, combined rotation to interactive behavior, involving both rotation and revolution under the influence of a rotating magnetic field (RMF), and a shift from attraction to separation under near-infrared (NIR) light, demonstrating the dynamic modulation of their behaviors in response to different stimuli. Notably, the behavioral patterns of multiple hydrogel microrobots under multiphysical fields indicate that NIR light can enhance interactive motion behaviors under RMFs and extend the range of motion trajectories. Dynamic models for each condition are established and analyzed based on dynamic equilibrium, and their behavior can be modulated by parameters such as magnetic particle concentration, magnetic field frequency, and NIR light intensity. This work introduces a novel strategy for regulating and controlling the dynamic behaviors of hydrogel microrobots, offering new insights into their multiphysical field locomotion.

Evidence for interacting but decoupled controls of decisions and movements in nonhuman primates.

Many recent studies indicate that control of decisions and actions is integrated during interactive behavior. Among these, several carried out in humans and monkeys conclude that there is a coregulation of choices and movements. Another perspective, based on human data only, proposes a decoupled control of decision duration and movement speed, allowing, for instance, trading decision duration for movement duration when time pressure increases. Crucially, it is not currently known whether this ability to flexibly dissociate decision duration from movement speed is specific to humans, whether it can vary depending on the context in which a task is performed, and whether it is stable over time. These are important questions to address, especially to rely on monkey electrophysiology to infer the neural mechanisms of decision-action coordination in humans. To do so, we trained two macaque monkeys in a perceptual decision-making task and analyzed data collected over multiple behavioral sessions. Our findings reveal a strong and complex relationship between decision duration and movement vigor. Decision duration and action duration can covary but also "compensate" each other. Such integrated but decoupled control of decisions and actions aligns with recent studies in humans, validating the monkey model in electrophysiology as a means of inferring neural mechanisms in humans. Crucially, we demonstrate for the first time that this control can evolve with experience, in an adapted manner. Together, the present findings contribute to deepening our understanding of the integrated control of decisions and actions during interactive behavior.NEW & NOTEWORTHY The mechanism by which the integrated control of decisions and actions occurs, coupled or interactive but decoupled, is debated. In the present study, we show in monkeys that decisions and actions influence each other in a decoupled way. For the first time, we also demonstrate that this control can evolve depending the subject's experience, allowing the trade of movement time for decision time and limiting the temporal discounting of reward value.

Creep-fatigue interactive behavior and damage mechanism of TP321 stainless steel under hybrid-controlled conditions

In this study, Hybrid-controlled creep-fatigue (HCCF) tests were conducted on TP321 austenitic stainless steel under a variety of test conditions. The effects of strain amplitude, holding time, holding stress and temperature on the creep-fatigue behavior of TP321 austenitic stainless steel were not only analyzed in detail but also revealed the creep-fatigue damage interactive mechanism. The results demonstrated that a rise in test temperature and load holding time markedly induced creep deformation, resulting in a notable reduction in failure life. Additionally, an increase in test temperature led to the cessation of the cyclic hardening phenomenon. Secondly, the analysis of fracture morphology and X-ray computed tomography (X-CT) scanning results demonstrated that the transgranular cracks expanded inwards and connected with the intergranular voids under creep-fatigue interaction, forming a mixed intergranular and transcrystalline fracture mode. The presence of large creep cavities impeded the propagation of fatigue cracks when creep damage was the dominant phenomenon. Subsequently, the damage evolution mechanism was elucidated through microstructural analysis, which revealed that the impact of the slip bands on the triangular grain boundaries and the precipitation of carbides facilitated the nucleation of voids and the internal formation of intergranular microcracks, thereby causing creep-fatigue damage interaction. Finally, the TP321 austenitic stainless steel creep-fatigue damage interactive mechanism diagram was proposed in conjunction with the fracture morphological characteristics and microstructure.

Behavior and Identity Construction of Online Fan Community—Taking Pocket 48 as an Example

With the development of new media, online social platforms have gradually emerged and improved, greatly changing the interaction between idols and fans, and even between fans and fans. This article takes Pocket 48, an exclusive derivative application of the SNH48 idol group, as an example to explore the fan groups’ interactive behavior and identity construction. Through participant observation and in-depth interviews with 15 fans, the study found that fans’ interactions on the Pocket 48 platform not only significantly enhanced their emotional connection with their idols, but also experienced gradual development in self-expression and identity. process. Through long-term interaction and participation, fans’ emotional investment in idols and communication with other fans continue to deepen, gradually forming a solid identity. This research provides a new perspective for understanding the progressiveness and complexity of modern social media in individual identity construction, revealing how online communities influence users’ identity development through continuous interaction mechanisms.

A Study on the Influencing Factors of New Media Short Video Platforms on Adolescent Mental Health

With the rapid development of the Internet and mobile technology, new media short video platforms have rapidly become popular around the world and have become an important part of adolescents ‘s daily lives. Short video platforms are popular among adolescent for their rich content and strong interactivity. However, the popularity of short video platforms also brings potential impacts on the mental health of adolescents. This study aims to discuss the specific influencing factors of new media short video platforms on adolescents’ mental health. It uses a combination of questionnaire surveys and literature searches to study factors such as the frequency of adolescents’ short video use, content preferences, and interactive behaviors The results of this study show that new media short video platforms have a multi-faceted negative impact on the mental health of adolescents, mainly manifested in increased psychological stress, decreased sleep quality, and decreased academic performance. The research further concluded that when adolescents overuse short video platforms, their academic and social pressures increase significantly. In response to this type of problem, it is recommended that parents and educators provide more guidance, use short video platforms reasonably, and create a healthy online environment.

TIBA: A web application for the visual analysis of temporal occurrences, interactions, and transitions of animal behavior.

Data in behavioral research is often quantified with event-logging software, generating large data sets containing detailed information about subjects, recipients, and the duration of behaviors. Exploring and analyzing such large data sets can be challenging without tools to visualize behavioral interactions between individuals or transitions between behavioral states, yet software that can adequately visualize complex behavioral data sets is rare. TIBA (The Interactive Behavior Analyzer) is a web application for behavioral data visualization, which provides a series of interactive visualizations, including the temporal occurrences of behavioral events, the number and direction of interactions between individuals, the behavioral transitions and their respective transitional frequencies, as well as the visual and algorithmic comparison of the latter across data sets. It can therefore be applied to visualize behavior across individuals, species, or contexts. Several filtering options (selection of behaviors and individuals) together with options to set node and edge properties (in the network drawings) allow for interactive customization of the output drawings, which can also be downloaded afterwards. TIBA accepts data outputs from popular logging software and is implemented in Python and JavaScript, with all current browsers supported. The web application and usage instructions are available at tiba.inf.uni-konstanz.de. The source code is publicly available on GitHub: github.com/LSI-UniKonstanz/tiba.

Maternal childhood trauma, caregiving behavior, and child cognitive development in the context of drug addiction.

Exposure to childhood trauma confers intergenerational risk on child development. However, the mechanism linking a mother's childhood trauma with her child's cognitive development remains poorly understood. This study recruited 71 mother-child dyads affected by substance use disorder from local, community-based, outpatient substance use treatment programs. Maternal exposure to childhood trauma, caregiving behavior, and child cognitive development were assessed in each mother-child dyad. These were measured through a comprehensive biopsychosocial interview, an observed dyadic interaction coded using the Coding Interactive Behavior system, and the Bayley Scales of Infant and Toddler Development, Screening Test, respectively. We hypothesized that compromised caregiving behavior would mediate a negative relationship between maternal childhood trauma and child cognitive development. Analyses did not support this hypothesis. Specifically, maternal childhood trauma was not significantly associated with child cognitive development nor the four dimensions of maternal caregiving behavior. However, caregiving behavior (specifically maternal sensitivity and limit setting) was associated with child cognitive development, when controlling for maternal childhood trauma and child age. The lack of associations observed suggests that protective factors may buffer the intergenerational impact of childhood trauma. Furthermore, the relationship between quality of caregiving and child cognitive development highlights the importance of interventions which foster sensitive caregiving behaviors that may bolster child cognitive development in the context of maternal substance use disorder and maternal childhood trauma. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Computational joint action: Dynamical models to understand the development of joint coordination.

Coordinating with others is part of our everyday experience. Previous studies using sensorimotor coordination games suggest that human dyads develop coordination strategies that can be interpreted as Nash equilibria. However, if the players are uncertain about what their partner is doing, they develop coordination strategies which are robust to the actual partner's actions. This has suggested that humans select their actions based on an explicit prediction of what the partner will be doing-a partner model-which is probabilistic by nature. However, the mechanisms underlying the development of a joint coordination over repeated trials remain unknown. Very much like sensorimotor adaptation of individuals to external perturbations (eg force fields or visual rotations), dynamical models may help to understand how joint coordination develops over repeated trials. Here we present a general computational model-based on game theory and Bayesian estimation-designed to understand the mechanisms underlying the development of a joint coordination over repeated trials. Joint tasks are modeled as quadratic games, where each participant's task is expressed as a quadratic cost function. Each participant predicts their partner's next move (partner model) by optimally combining predictions and sensory observations, and selects their actions through a stochastic optimization of its expected cost, given the partner model. The model parameters include perceptual uncertainty (sensory noise), partner representation (retention rate and internale noise), uncertainty in action selection and its rate of decay (which can be interpreted as the action's learning rate). The model can be used in two ways: (i) to simulate interactive behaviors, thus helping to make specific predictions in the context of a given joint action scenario; and (ii) to analyze the action time series in actual experiments, thus providing quantitative metrics that describe individual behaviors during an actual joint action. We demonstrate the model in a variety of joint action scenarios. In a sensorimotor version of the Stag Hunt game, the model predicts that different representations of the partner lead to different Nash equilibria. In a joint two via-point (2-VP) reaching task, in which the actions consist of complex trajectories, the model captures well the observed temporal evolution of performance. For this task we also estimated the model parameters from experimental observations, which provided a comprehensive characterization of individual dyad participants. Computational models of joint action may help identifying the factors preventing or facilitating the development of coordination. They can be used in clinical settings, to interpret the observed behaviors in individuals with impaired interaction capabilities. They may also provide a theoretical basis to devise artificial agents that establish forms of coordination that facilitate neuromotor recovery.

Bayesian optimization of one-dimensional convolutional neural networks (1D CNN) for early diagnosis of Autistic Spectrum Disorder

Autistic Spectrum Disorder (ASD) is a challenging neurological development disorder, which involves poor social interaction, communication, and repetitive behaviours. If autism is identified early enough it can be treated with better outcomes but present diagnostic tests are dependent on subjective opinion, consume a lot of time, and are vague. This study is aimed at optimizing one-dimensional convolutional neural networks (1D CNN) to improve the precision and speed of early ASD diagnosis. Four ASD datasets representing different age groups — toddlers, children, adolescents, and adults were modelled using one-dimensional convolutional neural networks (1D CNN). These datasets are accessible to the public on the UCI Machine Learning Repository and Kaggle, they consist of behavioural features relevant to ASD diagnosis. Each dataset underwent feature selection, categorical encoding, and missing value handling. Then, baseline 1D CNN with predefined hyperparameters was modelled on each of the datasets. Subsequently, the baseline models were optimized using the Tree-structured Parzen Estimator (TPE). An interactive web-based ASD diagnostic tool was developed, where user inputs are processed through age-specific pre-trained optimized models to determine ASD probability. The optimized 1D CNN models significantly outperformed the baseline models across all age groups and achieved scores of 100% in accuracy, precision, recall, F1-score, MCC, and AUC ROC. This implies that the optimized models can reliably identify people in various age groups who have and do not have ASD. The development of an interactive web-based diagnostic tool extends the practical utility of the models, making them accessible for clinical and at-home use.

An approach of designing a child-height-assisted robot with semi-autonomous enhances social reciprocity for children with autism spectrum disorders

Individuals with autism spectrum disorder (ASD) often face significant challenges in social reciprocity, which involves the mutual exchange of social cues and interactions. This study investigates the potential of using a child-height assistive robot named Rudolf to enhance the social reciprocity skills of children with ASD. The research focused on two main questions: how the robot’s semi-autonomous functions influence the enhancement of social reciprocity and how its customizable features elicit and improve these skills. Rudolf was specifically designed to engage children with ASD in a structured, supportive environment, offering tailored interactions to practice social reciprocity. A multiple-probe design was employed over three months to evaluate the robot’s impact on the social skills of three participants with ASD. The study revealed significant improvements in the children’s social reciprocity, with participants demonstrating increased engagement, trust, and interaction within the robot-assisted environment. The customizable functionalities of Rudolf were particularly effective in eliciting and reinforcing interactive behaviors. This research highlights the potential of assistive robots, like Rudolf, in addressing the unique social challenges faced by individuals with ASD. The findings suggest that such robots, when designed with customizable, child-centered features, can significantly enhance social skills in this population. The study contributes to the development of tailored assistive technologies that improve social integration and overall quality of life for children with ASD.

Enhancing University English Teaching through Mobile Interactive Applications from an Educational Technology Perspective

In the context of globalization and rapid advancements in information technology (IT), the application of educational technology in higher education has become increasingly prevalent. Particularly in university English teaching, mobile interactive applications have emerged as a significant teaching model. These applications transcend the constraints of time and space, providing students with a flexible learning environment that promotes interaction between teachers and students, thereby enhancing learning outcomes. However, existing research has primarily focused on the technological development and effectiveness evaluation of mobile interactive applications, neglecting the impact of student interaction and trust on learning outcomes. Additionally, there is a lack of dynamism and personalization in the recommendation of educational resources and study partners. This study constructs global and local dynamic trust models for mobile interactions, calculates trustworthiness based on mobile interactive behaviors, and integrates mobile interaction trust relationships and behaviors with a dynamic English recommendation algorithm, thereby providing a new theoretical framework and a practical approach for university English teaching, aiming to enhance the teaching effectiveness and improve students’ learning experiences.

Residence time prediction in magnetically controlled biomolecular local rebinding-dissociation kinetics

The residence time of drug-target conjugates is a critical factor in drug screening and efficacy prediction. The local rebinding-dissociation kinetics gives insights into in-vivo drug-target interactions. A magnetic torque system (MTS) is designed to observe rebinding-dissociation kinetics for predicting residence time. The system utilizes an alternating magnetic field (AMF) to manipulate the magnetization motion of magnetically labeled biomolecules and the forces acting upon biomolecular bonds. The motion, sensed by a quartz crystal microbalance (QCM), reflects biomolecular interactions occurring at the particle surface. Meanwhile, the motion facilitates the separation of dissociated molecules from the surface, thereby obviating the necessity for fixed and mobile phases in common kinetics observations. The constant and static solution environment minimizes reagent consumption. The MTS was utilized to observe the local rebinding-dissociation of antibodies (PAB and MAB) to magnetic beads (MB) and to HER2 receptors. The residence times recorded by the MTS were larger than the results obtained via SPR method, due to the occurrences of rebinding-dissociation kinetics. Interaction behaviours can be meticulously regulated for varying affinities by modulating the intensity of magnetic field. A high intensity field (400 Oe) was applied for strong binding between antibody-MB (biotin-streptavidin), and a low intensity field (300 Oe) was applied for weak antigen-antibody interactions. An increase in AMF strength enhanced dissociation, with a shift from 300 Oe to 400 Oe resulting in a 1 ∼ 4-fold reduction in residence time. Overall, the MTS provides an interactive and customizable perspective on kinetics observations.

Research on the diffusion and control of unsafe behaviors among chemical industry park enterprises based on the SEIR evolutionary game model

In order to control the diffusion of unsafe behaviors among chemical industrial park enterprises, a Susceptible-Exposed-Infectious-Recovered (SEIR) evolutionary game model is constructed based on the theory of epidemiology and evolutionary game theory. By exploring the game relationship between emergency management departments and safety assessment agencies, the paper analyzes the diffusion paths and evolutionary patterns of unsafe behaviors in chemical industrial park enterprises under different behavioral strategies adopted by both parties. Furthermore, it examines the process and patterns of system evolution to a positive stable state, and conducts multi-scenario simulation and analysis. The results indicate that the interactive behaviors of "strict regulation" by emergency management departments and "high-quality safety assessment" by safety assessment agencies are more conducive to controlling the diffusion of unsafe behaviors among chemical industrial park enterprises. Improving the efficiency of emergency management departments, establishing an attractive and constraining reward and punishment system to guide safety assessment agencies in conducting high-quality safety assessments, and stimulating the initiative of chemical industrial park enterprises in safety behaviors will be more conducive to forming an integrated, coordinated, and mutually constraining mechanism for work safety governance.

Sensitive infant care tunes a frontotemporal interbrain network in adolescence

Caregiving plays a critical role in children’s cognitive, emotional, and psychological well-being. In the current longitudinal study, we investigated the enduring effects of early maternal behavior on processes of interbrain synchrony in adolescence. Mother-infant naturalistic interactions were filmed when infants were 3–4 months old and interactions were coded for maternal sensitivity and intrusiveness with the Coding Interactive Behavior Manual. In early adolescence (Mean = 12.30, SD = 1.25), mother-adolescent interbrain synchrony was measured using hyperscanning EEG during a naturalistic interaction of positive valence. Guided by previous hyperscanning studies, we focused on interbrain connections within the right frontotemporal interbrain network. Results indicate that maternal sensitivity in early infancy was longitudinally associated with neural synchrony in the right interbrain frontotemporal network. Post-hoc comparisons highlighted enhancement of mother-adolescent frontal-frontal connectivity, a connection that has been implicated in parent-child social communication. In contrast, maternal intrusiveness in infancy was linked with attenuation of interbrain synchrony in the right interbrain frontotemporal network. Sensitivity and intrusiveness are key maternal social orientations that have shown to be individually stable in the mother-child relationship from infancy to adulthood and foreshadow children’s positive and negative social-emotional outcomes, respectively. Our findings are the first to demonstrate that these two maternal orientations play a role in enhancing or attenuating the child’s interbrain frontotemporal network, which sustains social communication and affiliation. Results suggest that the reported long-term impact of maternal sensitivity and intrusiveness may relate, in part, to its effects on tuning the child’s brain to sociality.

Learning from watching dialog and monolog videos in online STEM courses

BackgroundInstructional videos have been increasingly critical to afford effective learning experiences in online courses, but most videos follow a lecturing monolog delivery format. This format tends to lead students to observe the videos passively. Previous laboratory studies have indicated that observing dialog videos of an instructor tutoring a tutee enhances student learning more than watching monolog videos of the same instructor lecturing. This paper describes two empirical studies that replicated the laboratory findings in large-scale college-level online STEM courses.ResultsThe results show that observing dialog videos of tutoring led to superior learning outcomes compared to monolog videos of lecturing, regardless of whether students observe the videos individually or collaboratively, as long as they engaged generatively with the materials. This finding was confirmed across two tested STEM domains.ConclusionsThe findings of these classroom studies suggest that observing dialog tutoring videos is a novel and robust online instructional format that can be generalized across STEM domains. The benefit of overhearing tutorial dialogs requires students to engage in Constructive and Interactive behaviors, as defined by the ICAP framework, rather than exposing students to long didactic lectures.

Ai-enhanced thermal modeling for integrated process-product-system optimization in zero-defect manufacturing chains

This paper proposes a novel approach for realising zero-defect manufacturing by integrating AI-based thermal modelling to perform multi-stage process, product, and system optimisation in complex manufacturing chains. It integrates advanced thermal simulation and sensor data feed in real-time as a multistage thermal prediction and management system via machine learning algorithms. This framework seeks to predict thermal behaviour using a combination of convolutional neural networks (CNNs), long short-term memory (LSTM) networks, and graph neural networks (GNNs) for encoding, predicting, and learning the spatial, temporal, and interactive thermal behaviour, respectively. It further embeds finite element analysis (FEA) simulations for high-fidelity thermal predictions using data fusion through Kalman filters. This helps obtain the optimal estimates of thermal states from sensor measurements involving different types of sensors and the characteristics of signals. A multistage multimodal optimization framework involves genetic algorithms (GA) for global thermal parameter optimisation, reinforcement learning (RL) for multi-stage dynamic process control optimisation, and multi-agent systems (MAS) for coordinated multi-stage multi-objective balance embedded in a digital twin architecture. Evaluation results show that the effectiveness of the proposed system in improving the overall production efficiency is 33%, reducing defects is 47%, and reducing energy utilisation is 22%, when compared to the current de facto approaches. There is also a 38% improvement in predictive capability in preventing, detecting, and predicting of cross-stage process faults.

Modeling and Optimization of Interaction Behavior between Right Turning Motor Vehicles and Pedestrians at Signalized Intersections based on Machine Learning

This project first conducted a comprehensive analysis of the interaction process between pedestrians and right turning motor vehicles and its influencing factors through a summary of existing literature and observation of actual situations. On the basis of comprehensive and accurate data, machine learning algorithms were used to model and predict the interaction behavior between pedestrians and right turning motor vehicles. By comparing different algorithms, the optimal models for predicting the yielding behavior of right turning motor vehicles and pedestrian crossing behavior in the current dataset were obtained. According to the model evaluation report, the obtained models have better predictive performance on the test set. And based on the obtained optimal model, the feature importance ranking was carried out for the pedestrian crossing model and the right turn motor vehicle yielding model, and the importance scores of the influencing factors of pedestrian crossing behavior and right turn motor vehicle yielding behavior were obtained respectively. Finally, a comparison was made between the modeling effects of the crossing gap feature group and the speed distance feature group, further clarifying the importance of the speed distance index for right turning motor vehicles in predicting interactive behavior.

EAV: EEG-Audio-Video Dataset for Emotion Recognition in Conversational Contexts

Understanding emotional states is pivotal for the development of next-generation human-machine interfaces. Human behaviors in social interactions have resulted in psycho-physiological processes influenced by perceptual inputs. Therefore, efforts to comprehend brain functions and human behavior could potentially catalyze the development of AI models with human-like attributes. In this study, we introduce a multimodal emotion dataset comprising data from 30-channel electroencephalography (EEG), audio, and video recordings from 42 participants. Each participant engaged in a cue-based conversation scenario, eliciting five distinct emotions: neutral, anger, happiness, sadness, and calmness. Throughout the experiment, each participant contributed 200 interactions, which encompassed both listening and speaking. This resulted in a cumulative total of 8,400 interactions across all participants. We evaluated the baseline performance of emotion recognition for each modality using established deep neural network (DNN) methods. The Emotion in EEG-Audio-Visual (EAV) dataset represents the first public dataset to incorporate three primary modalities for emotion recognition within a conversational context. We anticipate that this dataset will make significant contributions to the modeling of the human emotional process, encompassing both fundamental neuroscience and machine learning viewpoints.