Social Signal Processing Research Articles

Towards a conceptual framework of research on social signal processing

Social Signal Processing (SSP) as an emerging research area can draw on material from many disciplines, but it needs effective ways to organise the material. We propose a framework that integrates concepts, drawn primarily from psychology, but with input from other disciplines, in a way that indicates how they relate to SSP. We identify seven core constructs: state; indicators that convey info about it; process by which indicators are generated; types of inference involved in identifying states from indicators; perceptual issues, including recovery of indicators from flux of activity and accuracy of impressions about the state; the role of macro-context; and the different levels at which analysis may be couched (the individual, the dyad, the group, the organisation). These may or may not be reflected in the structure of an SSP system, but the natural subtlety and context-sensitivity of human communication makes it important that people designing systems should consider how they relate to its task, and decide how best to take account of them. The analysis works from simple models which consider only states and indicators to models which embrace all seven constructs. At each stage it points to the literatures which discuss the relevant issues. It is fully acknowledged that attempting such a synthesis raises many problems (not least of terminology), and that alternative frameworks deal at least as well with subsets of the issues. It remains to be seen whether they could be extended to cover a comparable range.

Conceptual analysis of social signals: the importance of clarifying terminology

As a burgeoning field, Social Signal Processing (SSP) needs a solid grounding in the disciplines that have developed important concepts in the study of communication. However, the number and diversity of terms developed in linguistics, psychology, and the behavioural sciences may seem confusing for scholars who are not versed in the subtleties of conceptual analysis and theoretical developments. Indeed, different disciplines sometimes use the same term to mean different things or, conversely, use different terms to mean the same thing. The goals of this article are to present an overview of the different concepts developed in the various disciplines that studied animal and human communication, and to understand the differences and commonalities between concepts emerging from these disciplines. We conclude that such an understanding will greatly improve the efficiency of pluridisciplinary research projects, for the advancement of SSP requires that we look at the complexity of communication from different angles

An insight into multimodal databases for social signal processing: acquisition, efforts, and directions

The importance of context-aware computing in understanding social signals gave a rise to a new emerging domain, called social signal processing (SSP). SSP depends heavily on the existence of comprehensive multimodal databases containing the descriptors of social context and behaviors, such as situational environment, roles and gender of human participants. In the recent paper SSP community has emphasized how current research lacks of the adequate data, for the greatest part because acquisition and annotation of large multimodal datasets are time- and resource-consuming for the researchers. This paper aims to collect the existing work in this scope and to deliver the key aspects and clear directions for managing the multimodal behavior data. It reviews some of the existing databases, gives their important characteristics and draws the most important tools and methods conducted in capturing and managing social behavior signals. Summarizing the relevant findings it also addresses the existing issues and proposes fundamental topics that need to be investigated in the future research.

A psycho-ethological approach to social signal processing

The emerging field of social signal processing can benefit from a theoretical framework to guide future research activities. The present article aims at drawing attention to two areas of research that devoted considerable efforts to the understanding of social behaviour: ethology and social psychology. With a long tradition in the study of animal signals, ethology and evolutionary biology have developed theoretical concepts to account for the functional significance of signalling. For example, the consideration of divergent selective pressures responsible for the evolution of signalling and social cognition emphasized the importance of two classes of indicators: informative cues and communicative signals. Social psychology, on the other hand, investigates emotional expression and interpersonal relationships, with a focus on the mechanisms underlying the production and interpretation of social signals and cues. Based on the theoretical considerations developed in these two fields, we propose a model that integrates the processing of perceivable individual features (social signals and cues) with contextual information, and we suggest that output of computer-based processing systems should be derived in terms of functional significance rather than in terms of absolute conceptual meaning.

Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing

Social Signal Processing is the research domain aimed at bridging the social intelligence gap between humans and machines. This paper is the first survey of the domain that jointly considers its three major aspects, namely, modeling, analysis, and synthesis of social behavior. Modeling investigates laws and principles underlying social interaction, analysis explores approaches for automatic understanding of social exchanges recorded with different sensors, and synthesis studies techniques for the generation of social behavior via various forms of embodiment. For each of the above aspects, the paper includes an extensive survey of the literature, points to the most important publicly available resources, and outlines the most fundamental challenges ahead.

Pervasive Social Computing: augmenting five facets of human intelligence

Pervasive Social Computing is a novel collective paradigm, derived from pervasive computing, social media, social networking, social signal processing, etc. This paper reviews Pervasive Social Computing as an integrated computing environment, which promises to augment five facets of human intelligence: physical environment awareness, behavior awareness, community awareness, interaction awareness, and content awareness. Reviews of related studies are given, and their generic architectures are designed. The resulting architecture for Pervasive Social Computing is presented. A prototype is developed and examined, in order to investigate the characteristics exhibited by Pervasive Social Computing.

Social molecular pathways and the evolution of bee societies.

Bees provide an excellent model with which to study the neuronal and molecular modifications associated with the evolution of sociality because relatively closely related species differ profoundly in social behaviour, from solitary to highly social. The recent development of powerful genomic tools and resources has set the stage for studying the social behaviour of bees in molecular terms. We review 'ground plan' and 'genetic toolkit' models which hypothesize that discrete pathways or sets of genes that regulate fundamental behavioural and physiological processes in solitary species have been co-opted to regulate complex social behaviours in social species. We further develop these models and propose that these conserved pathways and genes may be incorporated into 'social pathways', which consist of relatively independent modules involved in social signal detection, integration and processing within the nervous and endocrine systems, and subsequent behavioural outputs. Modifications within modules or in their connections result in the evolution of novel behavioural patterns. We describe how the evolution of pheromonal regulation of social pathways may lead to the expression of behaviour under new social contexts, and review plasticity in circadian rhythms as an example for a social pathway with a modular structure.

Advances in Behavioral Science Using Automated Facial Image Analysis and Synthesis [Social Sciences

The face conveys information about a person's age, sex, background, and identity; what they are feeling, thinking, or likely to do next. Facial expression regulates face-to-face interactions, indicates reciprocity and interpersonal attraction or repulsion, and enables intersubjectivity between members of different cultures. Facial expression indexes neurological and psychiatric functioning and reveals personality and socioemotional development. Not surprisingly, the face has been of keen interest to behavioral scientists. About 15 years ago, computer scientists became increasingly interested in the use of computer vision and graphics to automatically analyze and synthesize facial expression. This effort was made possible in part by the development in psychology of detailed coding systems for describing facial actions and their relation to basic emotions, that is, emotions that are interpreted similarly in diverse cultures. The most detailed of these systems, the Facial Action Coding System (FACS), informed the development of the MPEG-4 facial animation parameters for video transmission and enabled progress toward automated measurement and synthesis of facial actions for research in affective computing, social signal processing, and behavioral science. This article reports key advances in behavioral science that are becoming possible through these developments. Before beginning, automated facial image analysis and synthesis (AFAS) is briefly described.

Techware: www.sspnet.eu: A Web Portal for Social Signal Processing [Best of the Web

In this issue, "Best of the Web" focuses on introducing the social signal processing network (SSPNet), a large European collaboration aimed at establishing a research community in social signal processing (SSP), the new, emerging domain aimed at bringing social intelligence in computers.

Automatic Detection of Dominance and Expected Interest

Social Signal Processing is an emergent area of research that focuses on the analysis of social constructs. Dominance and interest are two of these social constructs. Dominance refers to the level of influence a person has in a conversation. Interest, when referred in terms of group interactions, can be defined as the degree of engagement that the members of a group collectively display during their interaction. In this paper, we argue that only using behavioral motion information, we are able to predict the interest of observers when looking at face-to-face interactions as well as the dominant people. First, we propose a simple set of movement-based features from body, face, and mouth activity in order to define a higher set of interaction indicators. The considered indicators are manually annotated by observers. Based on the opinions obtained, we define an automatic binary dominance detection problem and amulticlass interest quantification problem. Error-Correcting Output Codes framework is used to learn to rank the perceived observer's interest in face-to-face interactions meanwhile Adaboost is used to solve the dominant detection problem. The automatic system shows good correlation between the automatic categorization results and the manual ranking made by the observers in both dominance and interest detection problems.

Social signal processing: Survey of an emerging domain

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence – the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement – in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.

Amygdala and Orbitofrontal Reactivity to Social Threat in Individuals with Impulsive Aggression

Converging evidence from animal and human lesion studies implicates the amygdala and orbitofrontal cortex (OFC) in emotional regulation and aggressive behavior. However, it remains unknown if functional deficits exist in these specific brain regions in clinical populations in which the cardinal symptom is impulsive aggression. We have previously shown that subjects diagnosed with intermittent explosive disorder (IED), a psychiatric disorder characterized by reactive aggressive behavior, perform poorly on facial emotion recognition tasks. In this study we employed a social-emotional probe of amygdala-OFC function in individuals with impulsive aggression. Ten unmedicated subjects with IED and 10 healthy, matched comparison subjects (HC) underwent functional magnetic resonance imaging while viewing blocks of emotionally salient faces. We compared amygdala and OFC reactivity to faces between IED and HC subjects, and examined the relationship between the extent of activation in these regions and extent of prior history of aggressive behavior. Relative to controls, individuals with IED exhibited exaggerated amygdala reactivity and diminished OFC activation to faces expressing anger. Extent of amygdala and OFC activation to angry faces were differentially related to prior aggressive behavior across subjects. Unlike controls, aggressive subjects failed to demonstrate amygdala-OFC coupling during responses to angry faces. These findings provide evidence of amygdala-OFC dysfunction in response to an ecologically-valid social threat signal (processing angry faces) in individuals with a history of impulsive aggressive behavior, and further substantiate a link between a dysfunctional cortico-limbic network and aggression.