Affective Computing Studies Research Articles

Classification of Complex Emotions Using EEG and Virtual Environment: Proof of Concept and Therapeutic Implication.

During the last decades, neurofeedback training for emotional self-regulation has received significant attention from scientific and clinical communities. Most studies have investigated emotions using functional magnetic resonance imaging (fMRI), including the real-time application in neurofeedback training. However, the electroencephalogram (EEG) is a more suitable tool for therapeutic application. Our study aims at establishing a method to classify discrete complex emotions (e.g., tenderness and anguish) elicited through a near-immersive scenario that can be later used for EEG-neurofeedback. EEG-based affective computing studies have mainly focused on emotion classification based on dimensions, commonly using passive elicitation through single-modality stimuli. Here, we integrated both passive and active elicitation methods. We recorded electrophysiological data during emotion-evoking trials, combining emotional self-induction with a multimodal virtual environment. We extracted correlational and time-frequency features, including frontal-alpha asymmetry (FAA), using Complex Morlet Wavelet convolution. Thinking about future real-time applications, we performed within-subject classification using 1-s windows as samples and we applied trial-specific cross-validation. We opted for a traditional machine-learning classifier with low computational complexity and sufficient validation in online settings, the Support Vector Machine. Results of individual-based cross-validation using the whole feature sets showed considerable between-subject variability. The individual accuracies ranged from 59.2 to 92.9% using time-frequency/FAA and 62.4 to 92.4% using correlational features. We found that features of the temporal, occipital, and left-frontal channels were the most discriminative between the two emotions. Our results show that the suggested pipeline is suitable for individual-based classification of discrete emotions, paving the way for future personalized EEG-neurofeedback training.

Heart rate variability analysis for the assessment of immersive emotional arousal using virtual reality: Comparing real and virtual scenarios.

Many affective computing studies have developed automatic emotion recognition models, mostly using emotional images, audio and videos. In recent years, virtual reality (VR) has been also used as a method to elicit emotions in laboratory environments. However, there is still a need to analyse the validity of VR in order to extrapolate the results it produces and to assess the similarities and differences in physiological responses provoked by real and virtual environments. We investigated the cardiovascular oscillations of 60 participants during a free exploration of a real museum and its virtualisation viewed through a head-mounted display. The differences between the heart rate variability features in the high and low arousal stimuli conditions were analysed through statistical hypothesis testing; and automatic arousal recognition models were developed across the real and the virtual conditions using a support vector machine algorithm with recursive feature selection. The subjects' self-assessments suggested that both museums elicited low and high arousal levels. In addition, the real museum showed differences in terms of cardiovascular responses, differences in vagal activity, while arousal recognition reached 72.92% accuracy. However, we did not find the same arousal-based autonomic nervous system change pattern during the virtual museum exploration. The results showed that, while the direct virtualisation of a real environment might be self-reported as evoking psychological arousal, it does not necessarily evoke the same cardiovascular changes as a real arousing elicitation. These contribute to the understanding of the use of VR in emotion recognition research; future research is needed to study arousal and emotion elicitation in immersive VR.

The Ordinal Nature of Emotions: An Emerging Approach

Computational representation of everyday emotional states is a challenging task and, arguably, one of the most fundamental for affective computing. Standard practice in emotion annotation is to ask people to assign a value of intensity or a class value to each emotional behavior they observe. Psychological theories and evidence from multiple disciplines including neuroscience, economics and artificial intelligence, however, suggest that the task of assigning reference-based values to subjective notions is better aligned with the underlying representations. This paper draws together the theoretical reasons to favor ordinal labels for representing and annotating emotion, reviewing the literature across several disciplines. We go on to discuss good and bad practices of treating ordinal and other forms of annotation data and make the case for preference learning methods as the appropriate approach for treating ordinal labels. We finally discuss the advantages of ordinal annotation with respect to both reliability and validity through a number of case studies in affective computing, and address common objections to the use of ordinal data. More broadly, the thesis that emotions are by nature ordinal is supported by both theoretical arguments and evidence, and opens new horizons for the way emotions are viewed, represented and analyzed computationally.

Affective computing in education: A systematic review and future research

It is becoming a trend to apply an emotional lens and to position emotions as central to educational interactions. Recently, affective computing has been one of the most actively research topics in education, attracting much attention from both academics and practitioners. However, despite the increasing number of papers published, there still are deficiencies and gaps in the comprehensive literature review in the specific area of affective computing in education. Therefore, this study presents a review of the literature on affective computing in education by selecting articles published from 2010 to 2017. A review protocol consisting of both automatic and manual searches is used to ensure the retrieval of all relevant studies. The final 94 selected papers are reviewed and relevant information extracted based on a set of research questions. This study classifies selected articles according to the research purposes, learning domains, channels and methods of affective recognition and expression, and emotion theories/models as well as the emotional states. The findings show the increased number and importance of affective computing studies in education domain in recent years. The research purposes of most affective computing studies are found to be designing emotion recognition and expression systems/methods/instruments as well as examining the relationships among emotion, motivation, learning style, and cognition. Affective measurement channels are classified into textual, visual, vocal, physiological, and multimodal channels, while the textual channel is recognized as the most widely-used affective measurement channel. Meanwhile, integration of textual and visual channels is the most widely-used multimodal channel in affective computing studies. Dimensional theories/models are the most preferred models for description of emotional states. Boredom, anger, anxiety, enjoyment, surprise, sadness, frustration, pride, hopefulness, hopelessness, shame, confusion, happiness, natural emotion, fear, joy, disgust, interest, relief, and excitement are reported as the top 20 emotional states in education domain. Finally, this study provides recommendations for future research directions to help researchers, policymakers and practitioners in the education sector to apply affective computing technology more effectively and to expand educational practices.

Towards the Recognition of the Emotions of People with Visual Disabilities through Brain-Computer Interfaces.

A brain–computer interface is an alternative for communication between people and computers, through the acquisition and analysis of brain signals. Research related to this field has focused on serving people with different types of motor, visual or auditory disabilities. On the other hand, affective computing studies and extracts information about the emotional state of a person in certain situations, an important aspect for the interaction between people and the computer. In particular, this manuscript considers people with visual disabilities and their need for personalized systems that prioritize their disability and the degree that affects them. In this article, a review of the state of the techniques is presented, where the importance of the study of the emotions of people with visual disabilities, and the possibility of representing those emotions through a brain–computer interface and affective computing, are discussed. Finally, the authors propose a framework to study and evaluate the possibility of representing and interpreting the emotions of people with visual disabilities for improving their experience with the use of technology and their integration into today’s society.

A longitudinal database of Irish political speech with annotations of speaker ability

This paper presents the Irish Political Speech Database, an English-language database collected from Irish political recordings. The database is collected with automated indexing and content retrieval in mind, and thus is gathered from real-world recordings (such as television interviews and election rallies) which represent the nature and quality of recordings which will be encountered in practical applications. The database is labelled for six speaker attributes: boring; charismatic; enthusiastic; inspiring; likeable; and persuasive. Each of these traits is linked to the perceived ability or appeal of the speaker, and as such are relevant to a range of content retrieval and speech analysis tasks. The six base attributes are combined to form a metric of Overall Speaker Appeal. A set of baseline experiments is presented, which demonstrate the potential of this database for affective computing studies. Classification accuracies of up to 76% are achieved, with little feature or system optimisation.

Affective Computing and the Impact of Gender and Age.

Affective computing aims at the detection of users’ mental states, in particular, emotions and dispositions during human-computer interactions. Detection can be achieved by measuring multimodal signals, namely, speech, facial expressions and/or psychobiology. Over the past years, one major approach was to identify the best features for each signal using different classification methods. Although this is of high priority, other subject-specific variables should not be neglected. In our study, we analyzed the effect of gender, age, personality and gender roles on the extracted psychobiological features (derived from skin conductance level, facial electromyography and heart rate variability) as well as the influence on the classification results. In an experimental human-computer interaction, five different affective states with picture material from the International Affective Picture System and ULM pictures were induced. A total of 127 subjects participated in the study. Among all potentially influencing variables (gender has been reported to be influential), age was the only variable that correlated significantly with psychobiological responses. In summary, the conducted classification processes resulted in 20% classification accuracy differences according to age and gender, especially when comparing the neutral condition with four other affective states. We suggest taking age and gender specifically into account for future studies in affective computing, as these may lead to an improvement of emotion recognition accuracy.

Best of affective computing and intelligent interaction 2013 in multimodal interactions

The fifth biannual Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII 2013) was held in Geneva, Switzerland. This conference featured the recent advancement in affective computing and relevant applications in education, entertainment and health. A number of selected paperswere invited to submit an extended article to this special issue. The selectionwas done based on their highly favorable reviews at ACII 2013 and involvement of multimodality in their methods and applications. The review process of this special issue was performed in one cycle after which we either accepted or rejected the submissions based on the responses provided to the reviewers’ comments. This special issue features seven articles on multimodal applications or studies in affective computing. In the reminder of this editorial,we summarize the articles and their key findings in this special issue.

Affective computing vs. affective placebo: Study of a biofeedback-controlled game for relaxation training

Relaxation training is an application of affective computing with important implications for health and wellness. After detecting user׳s affective state through physiological sensors, a relaxation training application can provide the user with explicit feedback about his/her detected affective state. This process (biofeedback) can enable an individual to learn over time how to change his/her physiological activity for the purposes of improving health and performance. In this paper, we provide three contributions to the field of affective computing for health and wellness. First, we propose a novel application for relaxation training that combines ideas from affective computing and games. The game detects user׳s level of stress and uses it to influence the affective state and the behavior of a 3D virtual character as a form of embodied feedback. Second, we compare two algorithms for stress detection which follow two different approaches in the affective computing literature: a more practical and less costly approach that uses a single physiological sensor (skin conductance), and a potentially more accurate approach that uses four sensors (skin conductance, heart rate, muscle activity of corrugator supercilii and zygomaticus major). Third, as the central motivation of our research, we aim to improve the traditional methodology employed for comparisons in affective computing studies. To do so, we add to the study a placebo condition in which user׳s stress level, unbeknown to him/her, is determined pseudo-randomly instead of taking into account his/her physiological sensor readings. The obtained results show that only the feedback presented by the single-sensor algorithm was perceived as significantly more accurate than the placebo. If the placebo condition was not included in the study, the effectiveness of the two algorithms would have instead appeared similar. This outcome highlights the importance of using more thorough methodologies in future affective computing studies.