Emotional Video Research Articles

Standardization of facial electromyographic responses

In behavioral studies, facial electromyographic (EMG) responses to external stimuli or internal events are usually quantified relative to the resting state, presumed to represent a neutral baseline condition. In the large majority of recent studies, EMG responses were expressed as a difference score in terms of microvolts with the resting state. We argue that since EMG activity is measured on a ratio scale rather than on an interval scale, percentage scores should be used instead of difference scores. Reanalyzing results from an earlier study on the relationships between facial EMG responses and affective empathic responses to emotional video clips, we found that the two different types of EMG response quantification were differently related to affective empathy. Relationships between EMG responses and affective empathy were more consistent or stronger for percentage scores than for difference scores. In another study, facial EMG mimicry responses to pictures of emotional facial expressions were stronger for percentage scores than for difference scores. The adequacy of percentage scores relative to difference scores as indices of psychological variables may be simply checked by comparing both types of scores

A closer look at the time course of bodily responses to awe experiences

The structure and function of awe have been examined by focusing on the average level of outcomes during awe experiences. In the present study, we tested the psychophysiological process of experiencing awe, focusing on time-series changes in skin conductance responses (SCRs), a moment-by-moment measure of sympathetic nervous responses, and pupil diameter, which is dilated or constricted through the sympathetic and parasympathetic nervous system. The responses were measured through an experiment where 77 Japanese university students watched emotional (awe, amusement, or neutral) videos while moving a joystick when they felt supernatural agency or non-agency, examining the underlying psychological processes. We found that experiencing awe was associated with frequent and steep changes in SCRs and frequent changes in pupil diameter. The joystick inclination, the perceptions of the supernatural, was kept at a high level from the start to the end of awe experiences. These results may reflect the psychophysiological processes of awe: the “fluctuation” of the sympathetic nervous system might underlie awe-specific experiences. Our findings shed new light on the mechanisms of the body-mind interaction in awe experiences.

Postural control in emotional states: An effect of biofeedback

BackgroundEmotional states in the short term and affective impairments, such as anxiety and depression in the long run, can have a significant impact on postural control. Individuals with these conditions often struggle with maintaining balance during emotionally charged situations, which can increase their risk of falls and injuries. Biofeedback has been proposed as a potential intervention to improve postural control during emotional induction. Research questionHow do emotional stimuli affect postural control performance in the presence of real-time postural feedback? MethodsForty-two participants took part in the study. The visual stimuli used in the experiment to induce emotions of various valence in the subjects were video clips, containing a series of pictures (24 positive, 24 neutral and 24 negative pictures). During the experiment, participants were required to maintain balance standing on a force platform while watching the videos on a computer screen. For each subject, three sessions of 480 s were generated, each of which included two conditions: with and without biofeedback (a target on the screen indicating the subject’s position relative to the center of pressure (CoP). ResultsThe No Feedback condition provoked a higher medial-lateral and anterior-posterior sway, but lower velocity of CoP than the Feedback condition. We did not find any effect of the valence of emotional videos on indicators of postural control in No Feedback condition. However, both negative and positive videos provoked a lower velocity of CoP in comparison to neutral videos in the Feedback condition. SignificanceChanges observed in postural control during the Feedback condition in the presence of emotional stimuli (both negative and positive) could be explained by the shift in attention from the task of maintaining posture to the emotional videos. The results of this study have important implications for clinicians involved in postural training and rehabilitation.

Improvement of continuous emotion recognition of temporal convolutional networks with incomplete labels

AbstractVideo‐based emotion recognition has been a long‐standing research topic for computer scientists and psychiatrists. In contrast to traditional discrete emotional models, emotion recognition based on continuous emotional models can better describe the progression of emotions. Quantitative analysis of emotions will have crucial impacts on promoting the development of intelligent products. The current solutions to continuous emotion recognition still have many issues. The original continuous emotion dataset contains incomplete data annotations, and the existing methods often ignore temporal information between frames. The following measures are taken in response to the above problems. Initially, aiming at the problem of incomplete video labels, the correlation between discrete and continuous video emotion labels is used to complete the dataset labels. This correlation is used to propose a mathematical model to fill the missing labels of the original dataset without adding data. Moreover, this paper proposes a continuous emotion recognition network based on an optimized temporal convolutional network, which adds a feature extraction submodule and a residual module to retain shallow features while improving the feature extraction ability. Finally, validation experiments on the Aff‐wild2 dataset achieved accuracies of 0.5159 and 0.65611 on the valence and arousal dimensions, respectively, by adopting the above measures.

229 Measuring the Effectiveness of Both Cognitive and Emotional Forms of Instructional Videos Related to the Beef Industry

Abstract US citizenries are becoming detached from the animal agricultural industry, so what is the best way to bridge the knowledge gap? Results of studies have shown that video messaging could be a key factor in decreasing this gap. This study focused on assessing perceptions of young adults about animal agriculture as well as the effectiveness of emotional and cognitive videos featuring local farmers and experts in the industry to alter perception as well as develop and create trust for the beef industry. An invitation to participate was distributed to 10,000 students at Auburn University. Responses were stopped after receiving 500 complete responses. Participants were directed to a survey in Qualtrics. Participants were given a survey with 5-point Likert scale and open-ended questions over aspects of their opinions about animal welfare, diet/health of consumers of red meat and environment/sustainability of the beef industry. After viewing two videos, participants re-took the survey. Data were subjected to paired t-test statistical analysis using SPSS. Results showed the views of the participants about the beef industry improved by 82% after watching the videos. ATLAS was used to code negative and positive key words within open response questions. The emotional video had a greater impact (P < 0.05) on participants perceptions with 190 people choosing that video. In contrast only 99 participants preferred the cognitively designed video. Compared with cognitive messaging, emotional video when used as communication modalities may serve to diminish the knowledge gap relationship between the public and the beef industry.

A - 51 The Interaction of Social Cognition Deficits and Personality Change in Progressive Supranuclear Palsy.

Small studies have suggested individuals with progressive supranuclear palsy (PSP) may have socioemotional deficits (i.e., poor emotion recognition and mentalizing). However, consideration has not been given to the interplay of social cognition and personality changes. The present study used multiple personality and social cognition measures to identify neurologically-driven socioemotional changes in PSP. Data was collected from 177 individuals: 77 with PSP and 100 older healthy controls (HC). The Awareness of Social Inference Test (TASIT) measures patients' comprehension of emotional videos (Emotion Evaluation Test: EET) and everyday social interactions (Social Inference Enriched: SI-E). Premorbid and current personality was reported by informants using the Interpersonal Adjectives Scale (IAS) and the Big Five Inventory (BFI). Groups were compared controlling for age and sex; statistical significance was set at p < 0.05. PSPs displayed impairment in reading emotions on the TASIT-EET (M:9.1 ± SD:2.66) and inferring others' intentions on the TASIT-SIE (44.8 ± 6.12) compared to HCs (11.5 ± 1.48; 53.8 ± 4.94). PSPs showed clinically (-1.5 SD) and statistically significant drops from premorbid levels in Extraversion (IAS current T: 37.0 ± 16.28; BFI: 20.1 ± 5.80) and IAS Dominance (39.9 ± 13.54). PSP patients showed significant deficits at multiple levels of emotion reading and theory of mind, impairing their comprehension of social communication. This may in turn explain the observed declines in interpersonal dominance and extraversion, possibly reflecting loss of agency and sociableness with disease progression. These results reveal more clinically significant socioemotional deficits in PSP than are generally recognized, and suggest face-to-face social testing and informant reports of personality change can be valuable testing approaches.

Examining the consistency of continuous affect annotations and psychophysiological measures in response to emotional videos

Despite the growing necessity of understanding the dynamics of emotion by naturalistic stimuli, averaging time-locked responses seems insufficient to capture emotional experiences that change over time. Intersubject correlation (ISC) has been implemented to examine dynamic emotional experiences by quantifying the consistency of responses across individuals. While previous research has shown that enhanced psychophysiological ISC can capture dynamic emotional experiences in response to long-lasting videos that evoke dimensional emotions, it is not yet fully understood how psychophysiological consistency varies during videos that elicit distinct emotions, such as fear. In this study, we re-analyzed publicly available data consisting of continuous affect annotations and psychophysiological signals, namely heart rate (HR), electrodermal activity (EDA), electromyographic signals from zygomaticus major (EMG-z), and corrugator supercilii (EMG-c), in response to categorical emotional videos, namely amusing, boring, relaxing, and fearful. Results showed an overall increase in ISC in multiple measures during fearful videos, indicating that emotional experiences during fearful videos were reliably consistent across participants. The effect of amusing and boring videos on ISC revealed varying results depending on the measurements. In particular, larger ISC in valence rating, EDA, and EMG-z was found for amusing than boring videos, whereas larger ISC in HR and EMG-c was observed for boring than amusing movies. Lastly, decreased ISC for relaxing videos was observed across multiple measurements, showing inconsistent emotional experiences during relaxing videos. This study builds on previous research on physiological consistency during emotional experiences by examining how the consistency of continuous affect annotations and psychophysiological measures differs in response to videos that elicit distinct emotions.

Learning facial expression and body gesture visual information for video emotion recognition

Recent research has shown that facial expressions and body gestures are two significant implications in identifying human emotions. However, these studies mainly focus on contextual information of adjacent frames, and rarely explore the spatio-temporal relationships between distant or global frames. In this paper, we revisit the facial expression and body gesture emotion recognition problems, and propose to improve the performance of video emotion recognition by extracting the spatio-temporal features via further encoding temporal information. Specifically, for facial expression, we propose a super image-based spatio-temporal convolutional model (SISTCM) and a two-stream LSTM model to capture the local spatio-temporal features and learn global temporal cues of emotion changes. For body gestures, a novel representation method and an attention-based channel-wise convolutional model (ACCM) are introduced to learn key joints features and independent characteristics of each joint. Extensive experiments on five common datasets are carried out to prove the superiority of the proposed method, and the results proved learning two visual information leads to significant improvement over the existing state-of-the-art methods.

Psychological, physiological, and biochemical correlations after negative emotional videos in college students with and without premenstrual syndrome.

Women with premenstrual syndrome (PMS) suffer heavily from emotional problems, the pathogenesis of which is believed to be related to the hypothalamic-pituitary-adrenal (HPA) axis, autonomic nervous system (ANS) and central nervous system (CNS). We took into account all 3 aspects to observed the psychological, physiological and biochemical correlations under anger and sadness in college students with and without PMS. 33 students with PMS and 24 healthy students participated in the emotion induction experiment, and were required to fill out self-report scales. Their salivary cortisol (SCort), skin conductivity level (SCL), heart rate variability (HRV), blood pressure (BP) and electroencephalogram (EEG) data were collected at the resting stage and 10-15 minutes after each video. Compared to healthy controls, students with PMS showed lower SCort level and higher VLF at rest, and no statistic difference in activities of ANS and HPA axis after emotional videos, but different results in EEG in all conditions. The decreases in SBP during angry video, SCort after angry and neutral videos, and increases in θ band power during sad video were moderately correlated with increases in PMS score. No intergroup differences were found in self-report emotions. Students with PMS had lower activity of HPA axis and possibly higher activity of PNS at rest, and different response patterns in CNS in all conditions. Several EEG frequencies, especially θ band, in specific encephalic regions during emotional videos, as well as declined HPA activities in dealing with angry and neutral stressors, in which γ activity in frontal lobe may play a role, showed moderate correlations with more severe PMS.

Spontaneous humanization of robots in passive observation of human-robot interaction: A path toward ethical consideration and human-robot cooperation

As humanoid robots become more prevalent, understanding how people form initial representations of and attitudes towards these agents is crucial. This study investigated whether observing a human's emotional reactions towards a robot in a passive human-robot interaction influences the spontaneous humanization and mental state attributions of the robot, and subsequent ethical considerations and cooperation. In Experiment 1, Japanese and British participants observed videos of a human interacting emotionally or neutrally with a Pepper robot and then assessed the robot. Results showed the emotional video elicited more humanization and intentionality attributions towards the robot, subsequently predicting greater ethical concern, especially among Japanese participants. In Experiment 2, observing human emotional engagement increased cooperation with the robot in a Prisoner's Dilemma game through enhanced humanization and intentionality attributions. The findings suggest that passive observation of human-robot interactions can shape the representation, attribution, and treatment of robots based on culture and situational cues. This highlights the need to consider social factors that may influence people's developing relationships with robots.

Emotional brain network decoded by biological spiking neural network.

Emotional disorders are essential manifestations of many neurological and psychiatric diseases. Nowadays, researchers try to explore bi-directional brain-computer interface techniques to help the patients. However, the related functional brain areas and biological markers are still unclear, and the dynamic connection mechanism is also unknown. To find effective regions related to different emotion recognition and intervention, our research focuses on finding emotional EEG brain networks using spiking neural network algorithm with binary coding. We collected EEG data while human participants watched emotional videos (fear, sadness, happiness, and neutrality), and analyzed the dynamic connections between the electrodes and the biological rhythms of different emotions. The analysis has shown that the local high-activation brain network of fear and sadness is mainly in the parietal lobe area. The local high-level brain network of happiness is in the prefrontal-temporal lobe-central area. Furthermore, the α frequency band could effectively represent negative emotions, while the α frequency band could be used as a biological marker of happiness. The decoding accuracy of the three emotions reached 86.36%, 95.18%, and 89.09%, respectively, fully reflecting the excellent emotional decoding performance of the spiking neural network with self- backpropagation. The introduction of the self-backpropagation mechanism effectively improves the performance of the spiking neural network model. Different emotions exhibit distinct EEG networks and neuro-oscillatory-based biological markers. These emotional brain networks and biological markers may provide important hints for brain-computer interface technique exploration to help related brain disease recovery.

Sentiment Analysis Using Deep Learning

Abstract: Sentiment is one of the few terms in the English language that lacks a clear definition yet is nonetheless comprehensible. Understanding others in society is greatly hampered by the inability to recognise facial expressions of emotion. We have opted to investigate video inputs and create a model that compiles data from sources and presents findings in an understandable manner. Modern artificial intelligence systems urgently require human sentiment detection to mimic and decipher face expressions. This might help people make informed decisions about things like purpose determination, offering advertising, and security-related dangers. The human eye can easily identify emotions in images or videos, but machines find it more challenging to do so, necessitating the employment of different image processing techniques for feature extraction.

Are Self-transcendent Emotions One Big Family? An Empirical Taxonomy of Positive Self-transcendent Emotion Labels

In the past decade, there has been a growing amount of research on so-called self-transcendent emotions, mainly in the domain of positive emotions. However, most candidate self-transcendent emotions (e.g., Awe, Gratitude, Being Moved, Wonder) have been studied in isolation, leaving the commonalities and the differences of their phenomenology unknown. In the present paper, we sought to identify the phenomenological nature of main families of self-transcendent emotions. We drew on two large datasets (N1 = 3,113; N2 = 1,443) in which participants had to recall an emotional episode or to watch emotional videos and had to report their emotions through a list of 40 emotion labels. Participants were also presented with a large list of items probing their cognitive appraisals, bodily feelings, and action tendencies. Using a principal component analysis, we identified three main dimensions of positive emotions: hedonic, social, and epistemic states. Candidate self-transcendent emotions were distributed across two dimensions, suggesting that at least two main different families of self-transcendent emotions should be distinguished. Our results also allowed us to identify self-reported cognitive appraisals, bodily feelings, and action tendencies characteristic of each family.

Randomized intervention to assess the effectiveness of an educational video on organ donation intent among Hispanics in the New York metropolitan area.

The Hispanic community has a high demand for organ donation but a shortage of donors. Studies investigating factors that could promote or hinder organ donation have examined emotional video interventions. Factors acting as barriers to organ donation registration have been classified as: (1) Bodily integrity; (2) medical mistrust; (3) "ick"-feelings of disgust towards organ donation; and (4) "jinx"-fear that registration may result in one dying due to premeditated plans. We predict that by providing necessary information and education about the donation process via a short video, individuals will be more willing to register as organ donors. To determine perceptions and attitudes regarding barriers and facilitators to organ donation intention among Hispanic residents in the New York metro politan area. This study was approved by the Institutional Review Board at Northwell Health. The approval reference number is No. 19-0009 (as presented in Supplementary material). Eligible participants included Hispanic New York City (NYC) residents, 18 years of age and above, who were recruited voluntarily through Cloud Research and participated in a larger randomized survey study of NYC residents. The survey an 85-item Redcap survey measured participant demographics, attitudes, and knowledge of organ donation as well as the intention to register as an organ donor. Attention checks were implemented throughout the survey, and responses were excluded for those who did fail. Participants were randomly assigned two-between subject conditions: To view a short video on organ donation and then proceed to complete the survey (i.e., video first) and view the same video at the end of the survey (video last). No intra-group activities were conducted. This study utilized an evidenced-based emotive educational intervention (video) which was previously utilized and was shown to increase organ donation registration rates at the Ohio Department of Motor Vehicles. Results were analyzed using Jamovi statistical software. Three hundred sixty-five Hispanic individuals were included in the analysis. Once consent was obtained and participants entered the survey (the survey sample is presented in Supplementary material), participants were asked to report on demographic variables and their general impression of organ donation after death. The video depicted stories regarding organ donation after death from various viewpoints, including from the loved ones of a deceased person who died waiting for a transplant; from the loved ones of a deceased person whose organs were donated upon death; and, from those who were currently waiting for a transplant. Using a binomial logistic regression, the analysis provides information about the relationship between the effects of an emotive video and the intention to donate among Hispanic participants who were not already registered as donors. The willingness to go back and register was found to be significantly more probable for those who watched the emotive video before being asked about their organ donation opinions (odds ratio: 2.05, 95% confidence interval: 1.06-3.97). Motivations for participation in organ donation were also captured with many stating the importance of messages coming from "people like me" and a message that highlights "the welfare of those in need". Overall, the findings suggest that using an emotive video that addresses organ donation barriers to prompt organ donation intentions can be effective among the Hispanic populous. Future studies should explore using targeted messaging that resonates with specific cultural groups, highlighting the welfare of others. This study suggests that an emotive educational intervention is likely to be effective in improving organ donation registration intent among the Hispanic population residing in NYC.

Application of visual communication in digital animation advertising design using convolutional neural networks and big data.

In the age of big data, visual communication has emerged as a critical means of engaging with customers. Among multiple modes of visual communication, digital animation advertising is an exceptionally potent tool. Advertisers can create lively and compelling ads by harnessing the power of digital animation technology. This article proposes a multimodal visual communication system (MVCS) model based on multimodal video emotion analysis. This model automatically adjusts video content and playback mode according to users' emotions and interests, achieving more personalized video communication. The MVCS model analyses videos from multiple dimensions, such as vision, sound, and text, by training on a large-scale video dataset. We employ convolutional neural networks to extract the visual features of videos, while the audio and text features are extracted and analyzed for emotions using recurrent neural networks. By integrating feature information, the MVCS model can dynamically adjust the video's playback mode based on users' emotions and interaction behaviours, which increases its playback volume. We conducted a satisfaction survey on 106 digitally corrected ads created using the MVCS method to evaluate our approach's effectiveness. Results showed that 92.6% of users expressed satisfaction with the adjusted ads, indicating the MVCS model's efficacy in enhancing digital ad design effectiveness.

Counterfactual Scenario-relevant Knowledge-enriched Multi-modal Emotion Reasoning

Multi-modal video emotion reasoning (MERV) has recently attracted increasing attention due to its potential application in human-computer interaction. This task needs to not only recognize utterance-level emotions for conspicuous speakers, but also perceive the emotions of non-speakers in videos. Existing methods focus on modeling multi-modal multi-level contexts to capture emotion-relevant clues from the complex scenarios in videos. However, the context information is far from enough to infer the emotion labels of non-speakers due to the large gap between the scenario situation and emotions labels. Inspired by the observation that humans can find solutions to complex problems with the leverage of experience and knowledge, we propose SK-MER , a Scenario-relevant Knowledge-enhanced Multi-modal Emotion Reasoning framework for MERV task, which can leverage external knowledge to enhance the video scenario understanding and emotion reasoning. Specifically, we use scenario concepts extracted from videos to build knowledge subgraphs from external knowledge bases. The knowledge subgraphs are then utilized to obtain scenario-relevant knowledge representations through dynamic knowledge graph attention. Next, we incorporate the knowledge representations into context modeling to enhance emotion reasoning with external scenario-relevant knowledge. In addition, we propose a counterfactual knowledge representation learning approach to obtain more effective scenario-relevant knowledge representations. Extensive experimental results on MEmoR dataset show that the proposed SK-MER framework achieves new state-of-the-art results.

Orthogonal-Moment-Based Attraction Measurement With Ocular Hints in Video-Watching Task

Pupil dilation and eye movements are closely related to human emotional and cognitive processes. Visual stimulus, especially video clips, is widely used in computer-assisted experimental paradigms as emotional inducers. However, the level of attraction as a critical factor to such visual stimulus still needs comprehensive investigation. This article conducts a novel study of attraction assessment with pupil diameter and eye movements. We collected high temporal resolution ocular variation data from 50 subjects while they viewed a variety of emotional video stimuli. Besides, this article proposes two orthogonal-moment-based feature extraction methods for emotion classification, i.e., Legendre moment and Krawtchouk moment. The results of experiments show that our proposed feature sets achieve better classification performance compared with conventional time-or frequency-domain feature sets. The accuracy of predicting attraction level reached 87.4% and 91.0% when new features were used alone and combined with conventional features, respectively. Compared with the conventional features with an accuracy of 86.6%, our proposed features can improve the accuracy by 4.4%.This study conducts a ground investigation of quantitative attraction assessment for video clips, which might provide reference for paradigm design of affective computing research.

Automated Video Classification System Driven by Characteristics of Emotional Human Brainwaves Caused by Audiovisual Stimuli

This paper proposes a method of automatically classifying videos into human emotional categories by imitating human neural processes, in which emotion-specific electroencephalography (EEG) characteristics are generated by audiovisual stimulation. In the proposed method, sophisticated emotional features are first extracted from EEG signals that are generated while a subject watches a video, using a sampleattention-based deep neural network encoder. Next, the direct mapping relationship between the extracted emotional EEG features and audiovisual features extracted from the contents of the corresponding videos are learned by a deep belief network. For practical application, the EEG features corresponding to the input video are automatically generated based on the machine&#x2019;s learned ability without measuring human EEG signals and subsequently applied to a segment-attention-based deep neural network decoder for emotion classification of the video. The experimental results demonstrated that the proposed method significantly outperforms existing methods with an average accuracy of about 95% for classifying four emotional classes of the video. As for automated emotional video classification, our artificial emotional EEG features-based approach obtains competitive performance, comparable to models that directly measure EEG, and can be generalized to various audiovisual datasets.

Recognition of Empathy from Synchronization between Brain Activity and Eye Movement.

In the era of user-generated content (UGC) and virtual interactions within the metaverse, empathic digital content has become increasingly important. This study aimed to quantify human empathy levels when exposed to digital media. To assess empathy, we analyzed brain wave activity and eye movements in response to emotional videos. Forty-seven participants watched eight emotional videos, and we collected their brain activity and eye movement data during the viewing. After each video session, participants provided subjective evaluations. Our analysis focused on the relationship between brain activity and eye movement in recognizing empathy. The findings revealed the following: (1) Participants were more inclined to empathize with videos depicting pleasant-arousal and unpleasant-relaxed emotions. (2) Saccades and fixation, key components of eye movement, occurred simultaneously with specific channels in the prefrontal and temporal lobes. (3) Eigenvalues of brain activity and pupil changes showed synchronization between the right pupil and certain channels in the prefrontal, parietal, and temporal lobes during empathic responses. These results suggest that eye movement characteristics can serve as an indicator of the cognitive empathic process when engaging with digital content. Furthermore, the observed changes in pupil size result from a combination of emotional and cognitive empathy elicited by the videos.

EEG-Based Emotional Video Classification via Learning Connectivity Structure

Electroencephalography (EEG) is a useful way to implicitly monitor the users perceptual state during multimedia consumption. One of the primary challenges for the practical use of EEG-based monitoring is to achieve a satisfactory level of accuracy in EEG classification. Connectivity between different brain regions is an important property for the classification of EEG. However, how to define the connectivity structure for a given task is still an open problem, because there is no ground truth about how the connectivity structure should be in order to maximize the classification performance. In this paper, we propose an end-to-end neural network model for EEG-based emotional video classification, which can extract an appropriate multi-layer graph structure and signal features directly from a set of raw EEG signals and perform classification using them. Experimental results demonstrate that our method yields improved performance in comparison to the existing approaches where manually defined connectivity structures and signal features are used. Furthermore, we show that the graph structure extraction process is reliable in terms of consistency, and the learned graph structures make much sense in the viewpoint of emotional perception occurring in the brain.