Social Signal Processing Research Articles

CCNN-SVM: Automated Model for Emotion Recognition Based on Custom Convolutional Neural Networks with SVM

The expressions on human faces reveal the emotions we are experiencing internally. Emotion recognition based on facial expression is one of the subfields of social signal processing. It has several applications in different areas, specifically in the interaction between humans and computers. This study presents a simple CCNN-SVM automated model as a viable approach for FER. The model combines a Convolutional Neural Network for feature extraction, certain image preprocessing techniques, and Support Vector Machine (SVM) for classification. Firstly, the input image is preprocessed using face detection, histogram equalization, gamma correction, and resizing techniques. Secondly, the images go through custom single Deep Convolutional Neural Networks (CCNN) to extract deep features. Finally, SVM uses the generated features to perform the classification. The suggested model was trained and tested on four datasets, CK+, JAFFE, KDEF, and FER. These datasets consist of seven primary emotional categories, which encompass anger, disgust, fear, happiness, sadness, surprise, and neutrality for CK+, and include contempt for JAFFE. The model put forward demonstrates commendable performance in comparison to existing facial expression recognition techniques. It achieves an impressive accuracy of 99.3% on the CK+ dataset, 98.4% on the JAFFE dataset, 87.18% on the KDEF dataset, and 88.7% on the FER.

Innova4Health: an integrated approach for prevention of recurrence and personalized treatment of Major Depressive Disorder.

Major Depressive Disorder (MDD) is a prevalent mental health condition characterized by persistent low mood, cognitive and physical symptoms, anhedonia (loss of interest in activities), and suicidal ideation. The World Health Organization (WHO) predicts depression will become the leading cause of disability by 2030. While biological markers remain essential for understanding MDD's pathophysiology, recent advancements in social signal processing and environmental monitoring hold promise. Wearable technologies, including smartwatches and air purifiers with environmental sensors, can generate valuable digital biomarkers for depression assessment in real-world settings. Integrating these with existing physical, psychopathological, and other indices (autoimmune, inflammatory, neuroradiological) has the potential to improve MDD recurrence prevention strategies. This prospective, randomized, interventional, and non-pharmacological integrated study aims to evaluate digital and environmental biomarkers in adolescents and young adults diagnosed with MDD who are currently taking medication. The study implements a sensor-integrated platform built around an open-source "Pothos" air purifier system. This platform is designed for scalability and integration with third-party devices. It accomplishes this through software interfaces, a dedicated app, sensor signal pre-processing, and an embedded deep learning AI system. The study will enroll two experimental groups (10 adolescents and 30 young adults each). Within each group, participants will be randomly allocated to Group A or Group B. Only Group B will receive the technological equipment (Pothos system and smartwatch) for collecting digital biomarkers. Blood and saliva samples will be collected at baseline (T0) and endpoint (T1) to assess inflammatory markers and cortisol levels. Following initial age-based stratification, the sample will undergo detailed classification at the 6-month follow-up based on remission status. Digital and environmental biomarker data will be analyzed to explore intricate relationships between these markers, depression symptoms, disease progression, and early signs of illness. This study seeks to validate an AI tool for enhancing early MDD clinical management, implement an AI solution for continuous data processing, and establish an AI infrastructure for managing healthcare Big Data. Integrating innovative psychophysical assessment tools into clinical practice holds significant promise for improving diagnostic accuracy and developing more specific digital devices for comprehensive mental health evaluation.

Emotional Recognition Based on Faces through Deep Learning Algorithms

Facial expressions have long been a straightforward way for humans to determine emotions, but computer systems find it significantly more difficult to do the same. Emotion recognition from facial expressions, a subfield of social signal processing, is employed in many different circumstances, but is especially useful for human-computer interaction. Many studies have been conducted on automatic emotion recognition, with the majority utilizing machine learning techniques. However, the identification of basic emotions such as fear, sadness, surprise, anger, happiness, and contempt remains a challenging subject in computer vision. Recently, deep learning has gained more attention as potential solutions for a range of real-world problems, such as emotion recognition. In this work, we refined the convolutional neural network method to discern seven basic emotions and assessed several preprocessing approaches to illustrate their impact on CNN performance. The goal of this research is to enhance facial emotions and features by using emotional recognition. Computers may be able to forecast mental states more accurately and respond with more customised answers if they can identify or recognise the facial expressions that elicit human responses. Consequently, we investigate how a convolutional neural network-based deep learning technique may enhance the recognition of emotions from facial features (CNN). Consequently, we investigate how a convolutional neural network-based deep learning technique may enhance the recognition of emotions from facial features (CNN). Our dataset, which comprises of roughly 32,298 pictures for testing and training, includes multiple face expressions. After noise removal from the input image, the pretraining phase helps reveal face detection, including feature extraction. The preprocessing system helps with this.

Social Signal Processing in Affective Virtual Reality: Human-Shaped Agents Increase Electrodermal Activity in an Elicited Negative Environment.

Prior research on affect elicitation indicates that stimuli with social content (pictures or videos) are more arousing than nonsocial stimuli. In particular, they elicit stronger physiological arousal as measured by electrodermal activity (EDA; i.e., social EDA effect). However, it is unclear how this effect applies to virtual reality (VR), which enables an enhanced sense of presence (SoP) and ecological validity. The study here approached this question from a social-emotional VR framework. A sample of N = 72 participants (55 percent women) experienced a set of six virtual environments (VEs) in the form of emotional parks specifically designed to elicit positive, negative, or neutral affectivity. Half of these VEs included human-shaped agents (social context) and the other half omitted these agents (nonsocial context). The results supported the social EDA effect, which in addition was amplified by the reported SoP. Importantly, the VE featuring a social negative content qualified this observed social EDA effect. The finding is discussed in the light of a negativity bias reported in affect literature, through which negative stimuli typically mobilize attention and bodily activation as a mechanism linked to stress responses. The study's implications extend to the use of VR in both research and practical applications, emphasizing the role of social content in influencing affective and physiological responses.

Computational charisma—A brick by brick blueprint for building charismatic artificial intelligence

Charisma is considered as one's ability to attract and potentially influence others. Clearly, there can be considerable interest from an artificial intelligence's (AI) perspective to provide it with such skill. Beyond, a plethora of use cases opens up for computational measurement of human charisma, such as for tutoring humans in the acquisition of charisma, mediating human-to-human conversation, or identifying charismatic individuals in big social data. While charisma is a subject of research in its own right, a number of models exist that base it on various “pillars,” that is, dimensions, often following the idea that charisma is given if someone could and would help others. Examples of such pillars, therefore, include influence (could help) and affability (would help) in scientific studies, or power (could help), presence, and warmth (both would help) as a popular concept. Modeling high levels in these dimensions, i. e., high influence and high affability, or high power, presence, and warmth for charismatic AI of the future, e. g., for humanoid robots or virtual agents, seems accomplishable. Beyond, also automatic measurement appears quite feasible with the recent advances in the related fields of Affective Computing and Social Signal Processing. Here, we therefore present a brick by brick blueprint for building machines that can appear charismatic, but also analyse the charisma of others. We first approach the topic very broadly and discuss how the foundation of charisma is defined from a psychological perspective. Throughout the manuscript, the building blocks (bricks) then become more specific and provide concrete groundwork for capturing charisma through artificial intelligence (AI). Following the introduction of the concept of charisma, we switch to charisma in spoken language as an exemplary modality that is essential for human-human and human-computer conversations. The computational perspective then deals with the recognition and generation of charismatic behavior by AI. This includes an overview of the state of play in the field and the aforementioned blueprint. We then list exemplary use cases of computational charismatic skills. The building blocks of application domains and ethics conclude the article.

A Multimodal Social Signal Processing Approach to Team Interactions

Social signal processing develops automated approaches to detect, analyze, and synthesize social signals in human–human as well as human–machine interactions by means of machine learning and sensor data processing. Most works analyze individual or dyadic behavior, while the analysis of group or team interactions remains limited. We present a case study of an interdisciplinary work process for social signal processing that can develop automatized measures of complex team interaction dynamics, using team task and social cohesion as an example. In a field sample of 25 real project team meetings, we obtained sensor data from cameras, microphones, and a smart ID badge measuring acceleration. We demonstrate how fine-grained behavioral expressions of task and social cohesion in team meetings can be extracted and processed from sensor data by capturing dyadic coordination patterns that are then aggregated to the team level. The extracted patterns act as proxies for behavioral synchrony and mimicry of speech and body behavior which map onto verbal expressions of task and social cohesion in the observed team meetings. We reflect on opportunities for future interdisciplinary or collaboration that can move beyond a simple producer–consumer model.

Reward learning improves social signal processing in autism model mice

Social and reward signal processing and their association are critical elements of social motivation. Despite the use of reward learning to improve the social interactions of patients with autism spectrum disorder (ASD), the underlying neural mechanisms are unknown. Here, we found different yet conjunct neuronal representations of social and reward signals in the mouse medial prefrontal cortex (mPFC). We also found that social signal processing is selectively disrupted, whereas reward signal processing is intact in the mPFC of Shank2-knockout mice, a mouse model of ASD. Furthermore, reward learning not only allows Shank2-knockout mice to associate social stimuli with reward availability, but it also rescues the impaired social signal processing. These findings provide insights into the neural basis for the therapeutic use of reward learning in ASD.

Inferring Causal Factors of Core Affect Dynamics on Social Participation through the Lens of the Observer

A core endeavour in current affective computing and social signal processing research is the construction of datasets embedding suitable ground truths to foster machine learning methods. This practice brings up hitherto overlooked intricacies. In this paper, we consider causal factors potentially arising when human raters evaluate the affect fluctuations of subjects involved in dyadic interactions and subsequently categorise them in terms of social participation traits. To gauge such factors, we propose an emulator as a statistical approximation of the human rater, and we first discuss the motivations and the rationale behind the approach.The emulator is laid down in the next section as a phenomenological model where the core affect stochastic dynamics as perceived by the rater are captured through an Ornstein-Uhlenbeck process; its parameters are then exploited to infer potential causal effects in the attribution of social traits. Following that, by resorting to a publicly available dataset, the adequacy of the model is evaluated in terms of both human raters' emulation and machine learning predictive capabilities. We then present the results, which are followed by a general discussion concerning findings and their implications, together with advantages and potential applications of the approach.

Optimal Facial Feature Based Emotional Recognition Using Deep Learning Algorithm.

Humans have traditionally found it simple to identify emotions from facial expressions, but it is far more difficult for a computer system to do the same. The social signal processing subfield of emotion recognition from facial expression is used in a wide range of contexts, particularly for human-computer interaction. Automatic emotion recognition has been the subject of numerous studies, most of which use a machine learning methodology. The recognition of simple emotions like anger, happiness, contempt, fear, sadness, and surprise, however, continues to be a difficult topic in computer vision. Deep learning has recently drawn increased attention as a solution to a variety of practical issues, including emotion recognition. In this study, we improved the convolutional neural network technique to identify 7 fundamental emotions and evaluated several preprocessing techniques to demonstrate how they affected the CNN performance. This research focuses on improving facial features and expressions based on emotional recognition. By identifying or recognising facial expressions that elicit human responses, it is possible for computers to make more accurate predictions about a person's mental state and to provide more tailored responses. As a result, we examine how a deep learning technique that employs a convolutional neural network might improve the detection of emotions based on facial features (CNN). Multiple facial expressions are included in our dataset, which consists of about 32,298 photos for testing and training. The preprocessing system aids in removing noise from the input image, and the pretraining phase aids in revealing face detection after noise removal, including feature extraction. As a result, the existing paper generates the classification of multiple facial reactions like the seven emotions of the facial acting coding system (FACS) without using the optimization technique, but our proposed paper reveals the same seven emotions of the facial acting coding system.

Behavioral context affects social signal representations within single primate prefrontal cortex neurons.

We tested whether social signal processing in more traditional, head-restrained contexts is representative of the putative natural analog-social communication-by comparing responses to vocalizations within individual neurons in marmoset prefrontal cortex (PFC) across a series of behavioral contexts ranging from traditional to naturalistic. Although vocalization-responsive neurons were evident in all contexts, cross-context consistency was notably limited. A response to these social signals when subjects were head-restrained was not predictive of a comparable neural response to the identical vocalizations during natural communication. This pattern was evident both within individual neurons and at a population level, as PFC activity could be reliably decoded for the behavioral context in which vocalizations were heard. These results suggest that neural representations of social signals in primate PFC are not static but highly flexible and likely reflect hownuances of the dynamic behavioral contexts affect the perception of these signals and what they communicate.

DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing From Decentralized Data.

Deep neural speech and audio processing systems have a large number of trainable parameters, a relatively complex architecture, and require a vast amount of training data and computational power. These constraints make it more challenging to integrate such systems into embedded devices and utilize them for real-time, real-world applications. We tackle these limitations by introducing DeepSpectrumLite, an open-source, lightweight transfer learning framework for on-device speech and audio recognition using pre-trained image Convolutional Neural Networks (CNNs). The framework creates and augments Mel spectrogram plots on the fly from raw audio signals which are then used to finetune specific pre-trained CNNs for the target classification task. Subsequently, the whole pipeline can be run in real-time with a mean inference lag of 242.0 ms when a DenseNet121 model is used on a consumer-grade Motorola moto e7 plus smartphone. DeepSpectrumLite operates decentralized, eliminating the need for data upload for further processing. We demonstrate the suitability of the proposed transfer learning approach for embedded audio signal processing by obtaining state-of-the-art results on a set of paralinguistic and general audio tasks, including speech and music emotion recognition, social signal processing, COVID-19 cough and COVID-19 speech analysis, and snore sound classification. We provide an extensive command-line interface for users and developers which is comprehensively documented and publicly available at https://github.com/DeepSpectrum/DeepSpectrumLite.

Classification of endogenous and exogenous bursts in collective emotions based on Weibo comments during COVID-19

Bursts and collective emotion have been widely studied in social physics field where researchers use mathematical models to understand human social dynamics. However, few researches recognize and separately analyze the internal and external influence on burst behaviors. To bridge this gap, we introduce a non-parametric approach to classify an interevent time series into five scenarios: random arrival, endogenous burst, endogenous non-burst, exogenous burst and exogenous non-burst. In order to process large-scale social media data, we first segment the interevent time series into sections by detecting change points. Then we use the rule-based algorithm to classify the time series based on its distribution. To validate our model, we analyze 27.2 million COVID-19 related comments collected from Chinese social media between January to October 2020. We adopt the emotion category called Profile of Mood States which consists of six emotions: Anger, Depression, Fatigue, Vigor, Tension and Confusion. This enables us to compare the burst features of different collective emotions during the COVID-19 period. The burst detection and classification approach introduced in this paper can also be applied to analyzing other complex systems, including but not limited to social media, financial market and signal processing.

A Computational Look at Oral History Archives

Computational technologies have revolutionized the archival sciences field, prompting new approaches to process the extensive data in these collections. Automatic speech recognition and natural language processing create unique possibilities for analysis of oral history (OH) interviews, where otherwise the transcription and analysis of the full recording would be too time consuming. However, many oral historians note the loss of aural information when converting the speech into text, pointing out the relevance of subjective cues for a full understanding of the interviewee narrative. In this article, we explore various computational technologies for social signal processing and their potential application space in OH archives, as well as neighboring domains where qualitative studies is a frequently used method. We also highlight the latest developments in key technologies for multimedia archiving practices such as natural language processing and automatic speech recognition. We discuss the analysis of both visual (body language and facial expressions), and non-visual cues (paralinguistics, breathing, and heart rate), stating the specific challenges introduced by the characteristics of OH collections. We argue that applying social signal processing to OH archives will have a wider influence than solely OH practices, bringing benefits for various fields from humanities to computer sciences, as well as to archival sciences. Looking at human emotions and somatic reactions on extensive interview collections would give scholars from multiple fields the opportunity to focus on feelings, mood, culture, and subjective experiences expressed in these interviews on a larger scale.

COVID-19: Affect recognition through voice analysis during the winter lockdown in Scotland.

The COVID-19 pandemic has led to unprecedented restrictions in people's lifestyle which have affected their psychological wellbeing. In this context, this paper investigates the use of social signal processing techniques for remote assessment of emotions. It presents a machine learning method for affect recognition applied to recordings taken during the COVID-19 winter lockdown in Scotland (UK). This method is exclusively based on acoustic features extracted from voice recordings collected through home and mobile devices (i.e. phones, tablets), thus providing insight into the feasibility of monitoring people's psychological wellbeing remotely, automatically and at scale. The proposed model is able to predict affect with a concordance correlation coefficient of 0.4230 (using Random Forest) and 0.3354 (using Decision Trees) for arousal and valence respectively.Clinical relevance- In 2018/2019, 12% and 14% of Scottish adults reported depression and anxiety symptoms. Remote emotion recognition through home devices would support the detection of these difficulties, which are often underdiagnosed and, if untreated, may lead to temporal or chronic disability.

Using Social Signals to Predict Shoplifting: A Transparent Approach to a Sensitive Activity Analysis Problem.

Retail shoplifting is one of the most prevalent forms of theft and has accounted for over one billion GBP in losses for UK retailers in 2018. An automated approach to detecting behaviours associated with shoplifting using surveillance footage could help reduce these losses. Until recently, most state-of-the-art vision-based approaches to this problem have relied heavily on the use of black box deep learning models. While these models have been shown to achieve very high accuracy, this lack of understanding on how decisions are made raises concerns about potential bias in the models. This limits the ability of retailers to implement these solutions, as several high-profile legal cases have recently ruled that evidence taken from these black box methods is inadmissible in court. There is an urgent need to develop models which can achieve high accuracy while providing the necessary transparency. One way to alleviate this problem is through the use of social signal processing to add a layer of understanding in the development of transparent models for this task. To this end, we present a social signal processing model for the problem of shoplifting prediction which has been trained and validated using a novel dataset of manually annotated shoplifting videos. The resulting model provides a high degree of understanding and achieves accuracy comparable with current state of the art black box methods.

Nonverbal Social Sensing: What Social Sensing Can and Cannot Do for the Study of Nonverbal Behavior From Video.

The study of nonverbal behavior (NVB), and in particular kinesics (i.e., face and body motions), is typically seen as cost-intensive. However, the development of new technologies (e.g., ubiquitous sensing, computer vision, and algorithms) and approaches to study social behavior [i.e., social signal processing (SSP)] makes it possible to train algorithms to automatically code NVB, from action/motion units to inferences. Nonverbal social sensing refers to the use of these technologies and approaches for the study of kinesics based on video recordings. Nonverbal social sensing appears as an inspiring and encouraging approach to study NVB at reduced costs, making it a more attractive research field. However, does this promise hold? After presenting what nonverbal social sensing is and can do, we discussed the key challenges that researchers face when using nonverbal social sensing on video data. Although nonverbal social sensing is a promising tool, researchers need to be aware of the fact that algorithms might be as biased as humans when extracting NVB or that the automated NVB coding might remain context-dependent. We provided study examples to discuss these challenges and point to potential solutions.

Training public speaking with virtual social interactions: effectiveness of real-time feedback and delayed feedback

Social signal processing and virtual social interaction technologies have allowed the creation of social skills training applications, and initial studies have shown that such solutions can lead to positive training outcomes and could complement traditional teaching methods by providing cheap, accessible, safe tools for training social skills. However, these studies evaluated social skills training systems as a whole and it is unclear to what extent which components contributed to positive outcomes. In this paper, we describe an experimental study where we compared the relative efficacy of real-time interactive feedback and after-action feedback in the context of a public speaking training application. We observed that both components provide benefits to the overall training: the real-time interactive feedback made the experience more immersive and improved participants’ motivation in using the system, while the after-action feedback led to positive training outcomes when it contained personalized feedback elements. Taken in combination, these results confirm that both social signal processing technologies and virtual social interactions are both contributing to social skills training systems’ efficiency. Additionally, we observed that several individual factors, here the subjects’ initial level of public speaking anxiety, personality and tendency to immersion significantly influenced the training experience. This finding suggests that social skills training systems could benefit from being tailored to participants’ particular individual circumstances.

Lateralization of social signal brain processing correlates with the degree of social integration in a songbird

Group cohesion relies on the ability of its members to process social signals. Songbirds provide a unique model to investigate links between group functioning and brain processing of social acoustic signals. In the present study, we performed both behavioral observations of social relationships within a group of starlings and individual electrophysiological recordings of HVC neuronal activity during the broadcast of either familiar or unfamiliar individual songs. This allowed us to evaluate and compare preferred partnerships and individual electrophysiological profiles. The electrophysiological results revealed asymmetric neuronal activity in the HVC and higher responsiveness to familiar than to unfamiliar songs. However, most importantly, we found a correlation between strength of cerebral asymmetry and social integration in the group: the more preferred partners a bird had, the more its HVC neuronal activity was lateralized. Laterality is likely to give advantages in terms of survival. Our results suggest that these include social skill advantages. Better knowledge of links between social integration and lateralization of social signal processing should help understand why and how lateralization has evolved.

The Neural Correlates of Face-Voice-Integration in Social Anxiety Disorder.

Faces and voices are very important sources of threat in social anxiety disorder (SAD), a common psychiatric disorder where core elements are fears of social exclusion and negative evaluation. Previous research in social anxiety evidenced increased cerebral responses to negative facial or vocal expressions and also generally increased hemodynamic responses to voices and faces. But it is unclear if also the cerebral process of face-voice-integration is altered in SAD. Applying functional magnetic resonance imaging, we investigated the correlates of the audiovisual integration of dynamic faces and voices in SAD as compared to healthy individuals. In the bilateral midsections of the superior temporal sulcus (STS) increased integration effects in SAD were observed driven by greater activation increases during audiovisual stimulation as compared to auditory stimulation. This effect was accompanied by increased functional connectivity with the visual association cortex and a more anterior position of the individual integration maxima along the STS in SAD. These findings demonstrate that the audiovisual integration of facial and vocal cues in SAD is not only systematically altered with regard to intensity and connectivity but also the individual location of the integration areas within the STS. These combined findings offer a novel perspective on the neuronal representation of social signal processing in individuals suffering from SAD.

Introducing the Urdu-Sindhi Speech Emotion Corpus: A Novel Dataset of Speech Recordings for Emotion Recognition for Two Low-Resource Languages

Speech emotion recognition is one of the most active areas of research in the field of affective computing and social signal processing. However, most research is directed towards a select group of languages such as English, German, and French. This is mainly due to a lack of available datasets in other languages. Such languages are called low-resource languages given that there is a scarcity of publicly available datasets. In the recent past, there has been a concerted effort within the research community to create and introduce datasets for emotion recognition for low-resource languages. To this end, we introduce in this paper the Urdu-Sindhi Speech Emotion Corpus, a novel dataset consisting of 1,435 speech recordings for two widely spoken languages of South Asia, that is Urdu and Sindhi. Furthermore, we also trained machine learning models to establish a baseline for classification performance, with accuracy being measured in terms of unweighted average recall (UAR). We report that the best performing model for Urdu language achieves a UAR = 65.00% on the validation partition and a UAR = 56.96% on the test partition. Meanwhile, the model for Sindhi language achieved UARs of 66.50% and 55.29% on the validation and test partitions, respectively. This classification performance is considerably better than the chance level UAR of 16.67%. The dataset can be accessed via https://zenodo.org/record/3685274.