Восприятие бимодальных выражений эмоциональных состояний человека: механизмы интеграции

The Influence of Emotional Content on Event-Related Brain Potentials during Spoken Word Processing

Electrodermal activity in response to empathic statements in clinical interviews with fibromyalgia patients

Functional Near-Infrared spectroscopy (fNIRS) is a neuroimaging tool that has been recently used in a variety of cognitive paradigms. Yet, it remains unclear whether fNIRS is suitable to study complex cognitive processes such as categorization or discrimination. Previously, functional imaging has suggested a role of both inferior frontal cortices in attentive decoding and cognitive evaluation of emotional cues in human vocalizations. Here, we extended paradigms used in functional magnetic resonance imaging (fMRI) to investigate the suitability of fNIRS to study frontal lateralization of human emotion vocalization processing during explicit and implicit categorization and discrimination using mini-blocks and event-related stimuli. Participants heard speech-like but semantically meaningless pseudowords spoken in various tones and evaluated them based on their emotional or linguistic content. Behaviorally, participants were faster to discriminate than to categorize; and processed the linguistic faster than the emotional content of stimuli. Interactions between condition (emotion/word), task (discrimination/categorization) and emotion content (anger, fear, neutral) influenced accuracy and reaction time. At the brain level, we found a modulation of the Oxy-Hb changes in IFG depending on condition, task, emotion and hemisphere (right or left), highlighting the involvement of the right hemisphere to process fear stimuli, and of both hemispheres to treat anger stimuli. Our results show that fNIRS is suitable to study vocal emotion evaluation, fostering its application to complex cognitive paradigms.

ABSTRACTThe relation between affect-driven feedback and engagement on a given task has been largely investigated. This relation can be used to make personalised instructional decisions and/or modify the affect content within the feedback. However, although it is generally assumed that providing encouraging feedback to students should help them adopt a state of flow, there are instances where those messages might result counterproductive. In this paper, we present a case study with 48 secondary school students using an Intelligent Tutoring System for arithmetical word problem solving. This system, which makes some common assumptions on how to relate affective state with performance, takes into account subjective (user's affective state) and objective information (previous problem performance) to decide the upcoming difficulty levels and the type of affective feedback to be delivered. Surprisingly, results revealed that feedback was more effective when no emotional content was included, and lead to the conclusion that purely instructional and concise help messages are more important than the emotional reinforcement contained therein. This finding shows that this is still an open issue. Different settings present different constraints generating related compounding factors that affect obtained results. This research confirms that new approaches are required to determine when, how and where affect-driven feedback is needed. Affect-driven feedback, engagement and their mutual relation have been largely investigated. Student's interactions combined with their emotional state can be used to make personalised instructional decisions and/or modify the affect content within the feedback, aiming to entice engagement on the task. However, although it is generally assumed that providing encouraging feedback to the students should help them adopt a state of flow, there are instances where those encouraging messages might result counterproductive. In this paper, we analyze these issues in terms of a case study with 48 secondary school students using an Intelligent Tutoring System for arithmetical word problem solving. This system, which makes some common assumptions on how to relate affective state with performance, takes into account subjective (user's affective state) and objective (previous problem performance) information to decide the difficulty level of the next exercise and the type of affective feedback to be delivered. Surprisingly, findings revealed that feedback was more effective when no emotional content was included in the messages, and lead to the conclusion that purely instructional and concise help messages are more important than the emotional reinforcement contained therein. This finding, which coincides with related work, shows that this is still an open issue. Different settings present different constraints and there are related compounding factors that affect obtained results, such as the message's contents and their target, how to measure the effect of the message on engagement through affective variables considering other issues involved, and to what extent engagement can be manipulated solely in terms of affective feedback. The contribution here is that this research confirms that new approaches are needed to determine when, how and where affect-driven feedback is needed. In particular, based on our previous experience in developing educational recommender systems, we suggest the combination of user-centred design methodologies with data mining methods to yield a more effective feedback.

<p>The patterns of perception of a part and a whole of multimodal emotional dynamic states of people unfamiliar to observers are studied. Audio-video clips of fourteen key emotional states expressed by specially trained actors were randomly presented to two groups of observers. In one group (N=96, average age — 34, SD — 9.4l.), each audio—video image was shown in full, in the other (N=78, average age — 25, SD — 9.6l.), it was divided into two parts of equal duration from the beginning to the conditional middle (short phonetic pause) and from the middle to the end of the exposure. The stimulus material contained facial expressions, gestures, head and eye movements, changes in the position of the body of the sitters, who voiced pseudolinguistic statements accompanied by affective intonations. The accuracy of identification and the structure of categorical fields were evaluated depending on the modality and form (whole/part) of the exposure of affective states. After the exposure of each audio-video image from the presented list of emotions, observers were required to choose the one that best corresponds to what they saw. According to the data obtained, the accuracy of identifying the emotions of the initial and final fragments of audio-video images practically coincide, but significantly less than with full exposure. Functional differences in the perception of fragmented audio-video images of the same emotional states are revealed. The modes of transitions from the initial stage to the final one and the conditions affecting the relative speed of the perceptual process are shown. The uneven formation of the information basis of multimodal expressions and the heterochronous perceptogenesis of emotional states of actors are demonstrated.</p>

Adapting a Model of Duration

BackgroundBreast cancer affects the lives of not only those diagnosed but also the people around them. Many of those affected share their experiences on social media. However, these narratives may differ according to who the poster is and what their relationship with the patient is; a patient posting about their experiences may post different content from someone whose friends or family has breast cancer. Weibo is 1 of the most popular social media platforms in China, and breast cancer–related posts are frequently found there.ObjectiveWith the goal of understanding the different experiences of those affected by breast cancer in China, we aimed to explore how content and language used in relevant posts differ according to who the poster is and what their relationship with the patient is and whether there are differences in emotional expression and topic content if the patient is the poster themselves or a friend, family member, relative, or acquaintance.MethodsWe used Weibo as a resource to examine how posts differ according to the different poster-patient relationships. We collected a total of 10,322 relevant Weibo posts. Using a 2-step analysis method, we fine-tuned 2 Chinese Robustly Optimized Bidirectional Encoder Representations from Transformers (BERT) Pretraining Approach models on this data set with annotated poster-patient relationships. These models were lined in sequence, first a binary classifier (no_patient or patient) and then a multiclass classifier (post_user, family_members, friends_relatives, acquaintances, heard_relation), to classify poster-patient relationships. Next, we used the Linguistic Inquiry and Word Count lexicon to conduct sentiment analysis from 5 emotion categories (positive and negative emotions, anger, sadness, and anxiety), followed by topic modeling (BERTopic).ResultsOur binary model (F1-score=0.92) and multiclass model (F1-score=0.83) were largely able to classify poster-patient relationships accurately. Subsequent sentiment analysis showed significant differences in emotion categories across all poster-patient relationships. Notably, negative emotions and anger were higher for the “no_patient” class, but sadness and anxiety were higher for the “family_members” class. Focusing on the top 30 topics, we also noted that topics on fears and anger toward cancer were higher in the “no_patient” class, but topics on cancer treatment were higher in the “family_members” class.ConclusionsChinese users post different types of content, depending on the poster- poster-patient relationships. If the patient is family, posts are sadder and more anxious but also contain more content on treatments. However, if no patient is detected, posts show higher levels of anger. We think that these may stem from rants from posters, which may help with emotion regulation and gathering social support.

It is believed that stimulating the inspiration of short video consumers might be an effective way to attract and maintain the attention of consumers so that they are willing to respond positively to short video ads. Therefore, in order to explore the source of customer inspiration in short video and its cognitive psychological process, the text and grid data collected from an interview among 25 short video users have been qualitatively analyzed by Kelly Grid Technology in order to construct the formation path model of short video customer inspiration, and find out its source, triggering mechanism, and influencing factors. It is found that the inspiring informational content characteristics include richness, reliability, vividness, and fluency of emotional content characteristics, fun, novelty, and narrative. However, the characteristics of commercial content in short video ads hinder the inspiration of consumers. The study also reveals that an internal mechanism of inspiration stimulation is built on some cognitive processes (i.e., presence, processing fluency, perceived innovation, perceived convenience) generated by informational content, and emotional responses by emotional content (i.e., curiosity, surprise, enjoyment, etc.). In addition, it is shown that personal involvement enhances the relationship between the inspiring content characteristics and consumer inspiration. As a result, customer inspiration and engagement in short video ads are highly enriched. Findings provide implications for short video platforms and online marketers.

Purpose- Environmental concern is a key issue for today's society, and consumer behavior is one of the reflection areas of environmental concern in today's marketing. Understanding the behavior patterns of consumers acting with the awareness of protecting the environment, evaluating their practices, and examining their opinions will be beneficial for marketing research. This paper aims to explore an online community discussing the Zero Waste concept and evaluate the themes in the discussion content. With an exploratory aim, this study examines the posts on an online community on a social media platform (Reddit/ZeroWaste) to evaluate community discussion for the Zero Waste concept. Methodology- Data collected from Reddit community of ZeroWaste with the filter of “top posts” and “all time” is used for the sample of the study. Consistent with the exploratory purpose of the study, thematic analysis methodology is employed on community posts. Content is evaluated in two dimensions -post type and content categories- and four post types and thirteen content categories are concluded in the study. Findings- Post types include information, instruction, personal experience, and discussion with community. Content categories include zero waste cases, showcase of zerowaste lifestyle and applications, upcycling, repurposing, reusable items/reusing, recycling, food waste, do it yourself, news sharing, product instruction, discussion about responsibility, environmental awareness, and memes/humor. Conclusion- Post types can signal how the discussion is shaped in the community, while content categories are reflecting the different type of themes in the content. "Zero Waste cases" theme expresses the industrial applications, therefore companies can use this theme to benchmark themselves and understand the applications. "Showcase of Zerowaste Lifestyle and Applications" theme can signal the consumers’ behaviors about the topic. Other themes such as "Upcycling, Repurposing, Reusable items / Reusing, Recycling" also shows the different types of zero waste applications. Evaluation of the post types and content categories included in the conversation of Zero Waste community can help managerial decision-making process about sustainable marketing practises and consumer behavior. Keywords: Zero waste, sustainable consumption, consumer behavior, reddit JEL Codes: M31, Q56

In 2019, the Chinese Football Association (CFA) released a policy mandating all men’s clubs qualifying for the Chinese Super League (CSL) to integrate a women’s football team within their structure. Against this backdrop, this study aims to answer the research questions: What are the integration mechanisms and socialisation tactics adopted by men’s clubs to integrate women’s teams? What are the barriers and enablers of an effective integration process and positive integration experience for women players? Underpinned by a social constructionism philosophical position, we conducted 18 semi-structured interviews with the club directors (n = 4) responsible for women’s football development and women football players (n = 14) impacted by the integration policy. All data were analysed using the thematic analysis method. The findings highlighted a superficial integration process as most clubs resorted to partnering with an existing external football team, with little evidence of resource-sharing in physical assets, personnel, and knowledge and information during the integration process. Additionally, barriers such as inadequate organisational support for women players, as well as the perceived inequality and job insecurity associated with their affiliation status to the men’s team have further contributed to an ineffective integration process. Meanwhile, women players’ proactive tactics such as sensemaking and positive framing have enhanced their integration experience into their club. This study highlights the understudied negative impact associated with integration from women players’ perspectives. Policymakers are encouraged to create more financial incentives for clubs to support women’s teams; clubs should implement their integration strategies with careful consideration of the broader gendered, socio-cultural, and political context.

The research on the expression of emotion in human-computer dialogue can greatly improve the user experience. Existing research has paid a lot of attention to how to generate specific emotional content and how to improve the extraction rate of emotions, while ignoring the reduction of emotion expression caused by factors such as topics and emotions added to the encoder. This paper proposes a novel Topic-extended Emotional Conversation Generation Model Based on Joint Decoding (TECM-JD). The model embeds the specified emotion category as an additional input into the emotional independent unit of the decoder, in order to reduce the expression of the content affected by adding emotion into the model. The joint attention mechanism is used to obtain the input sequence content and the input sequence topic word content obtained by the Twitter LDA model, which ensures that the output topic and the input are under the same topic. The experimental results show that the proposed model can generate richer emotional content related to the topic and have good performance and are superior to traditional dialogue models.

This study presents a widespread analysis of affective vocal expression classification systems. In this study, the Hilbert–Huang–Hurst coefficient (HHHC) vector is proposed as a non-linear vocal source feature to represent the emotional states according to their effects on the speech production mechanism. Affective states are highlighted by the empirical mode decomposition-based method, which exploits the non-stationarity of the acoustic variations. Hurst coefficients are then estimated from the decomposition modes to form the feature vector. Additionally, a vector of the index of non-stationarity (INS) is introduced as dynamic information to the HHHC. The proposed feature vector is evaluated in speech emotion classification experiments with three databases in German and English languages. Three state-of-the-art acoustic feature vectors are adopted as a baseline. The -integrated Gaussian mixture model ( -GMM) is also introduced for the emotion representation and classification. Its performance is compared to competing for stochastic and machine learning classifiers. Results demonstrate that the HHHC leads to significant classification improvement when compared to the baseline acoustic feature vectors. Moreover, results also show that the -GMM outperforms the competing classification methods. Finally, the complementarity aspects of HHHC and INS are also evaluated for the GeMAPS and eGeMAPS feature sets.

Speech is a natural way of communication amongst humans and advancements in speech emotion recognition (SER) technology allow further improvement of human-computer interactions (HCI) with speech by understanding human emotions. SER systems are traditionally focused on categorizing emotions into discrete classes. However, discrete classes often overlook some subtleties between each emotion as they are prone to individual differences and cultures. In this study, we focused on the use of dimensional emotional values: valence, arousal, and dominance as outputs for an SER instead of the traditional categorical classification. An SER model is developed using largely pre-trained models Wav2Vec 2.0 and HuBERT as feature encoders as a feature extraction technique from raw audio input. The model’s performance is assessed using a mean concordance coefficient (CCC) score for models trained on an English language-based dataset called Interactive Emotional Dyadic Motion Capture (IEMOCAP) and a Korean language-based dataset called Korean Emotion Multimodal Database (KEMDy19). For the experiments done on the IEMOCAP dataset, we reported a mean CCC of 0.3673 on the Wav2Vec 2.0-based model with CCC values of 0.3004, 0.4585, and 0.3431 for the valence, arousal, and dominance values respectively trained on the “anger”, “happy”, “sad”, and “neutral” emotion classes. Meanwhile, a mean CCC of 0.3573 on the HuBERT-based model with CCC values of 0.2789, 0.3295, and 0.3361 for the respectively on the same set of emotional classes. For the experiments done on the KEMDy19 dataset, a mean CCC of 0.5473 on the Wav2Vec 2.0-based model with CCC values of 0.5804 and 0.5142 for the valence and arousal were achieved using all available emotional classes on the dataset, while a mean CCC of 0.5580 from CCC values of 0.5941 and 0.5219 on four emotional classes “anger”, “happy”, “sad”, and “neutral” were observed. For the HuBERT-based model, a mean CCC of 0.5271 with CCC values of 0.5429 and 0.5113 for the valence and arousal were recorded using all available emotional classes, while a mean CCC of 0.5392 from CCC values of 0.5765 and 0.5019 for the valence and arousal values on the four selected emotional classes. The proposed approach outperforms traditional machine learning methods and previously reported CCC values from other literature. Moreover, the use of dimensional emotional values provides a more fine-grained insight into the user’s emotional states allowing for a much deeper understanding of one’s affective state with reduced dimensionality. By applying such SER technologies to other areas such as HCI, affective computing, and psychological research, more personalized and adaptable user interfaces can be developed to suit the emotional needs of each individual. This could also contribute to the advancement of our understanding of human factors by developing emotion recognition systems.

Interest in human emotion recognition, regarding physiological signals, has recently risen. In this study, an efficient emotion recognition system, based on geometrical analysis of autonomic nervous system signals, is presented. The electrocardiogram recordings of 47 college students were obtained during rest condition and affective visual stimuli. Pictures with four emotional contents, including happiness, peacefulness, sadness, and fear were selected. Then, ten lags of Poincare plot were constructed for heart rate variability (HRV) segments. For each lag, five geometrical indices were extracted. Next, these features were fed into an automatic classification system for the recognition of the four affective states and rest condition. The results showed that the Poincare plots have different shapes for different lags, as well as for different affective states. Considering higher lags, the greatest increment in SD1 and decrements in SD2 occurred during the happiness stimuli. In contrast, the minimum changes in the Poincare measures were perceived during the fear inducements. Therefore, the HRV geometrical shapes and dynamics were altered by the positive and negative values of valence-based emotion dimension. Using a probabilistic neural network, a maximum recognition rate of 97.45% was attained. Applying the proposed methodology based on lagged Poincare indices, a valuable tool for discriminating the emotional states was provided.

Abstract. Currently, there is a rapid development in the techniques of the automated image based modelling (IBM), especially in advanced structure-from-motion (SFM) and dense image matching methods, and camera technology. One possibility is to use video imaging to create 3D reality based models of cultural heritage architectures and monuments. Practically, video imaging is much easier to apply when compared to still image shooting in IBM techniques because the latter needs a thorough planning and proficiency. However, one is faced with mainly three problems when video image sequences are used for highly detailed modelling and dimensional survey of cultural heritage objects. These problems are: the low resolution of video images, the need to process a large number of short baseline video images and blur effects due to camera shake on a significant number of images. In this research, the feasibility of using video images for efficient 3D modelling is investigated. A method is developed to find the minimal significant number of video images in terms of object coverage and blur effect. This reduction in video images is convenient to decrease the processing time and to create a reliable textured 3D model compared with models produced by still imaging. Two experiments for modelling a building and a monument are tested using a video image resolution of 1920×1080 pixels. Internal and external validations of the produced models are applied to find out the final predicted accuracy and the model level of details. Related to the object complexity and video imaging resolution, the tests show an achievable average accuracy between 1 – 5 cm when using video imaging, which is suitable for visualization, virtual museums and low detailed documentation.