Social Signal Processing Research Articles

Social signal processing for studying parent-infant interaction.

Studying early interactions is a core issue of infant development and psychopathology. Automatic social signal processing theoretically offers the possibility to extract and analyze communication by taking an integrative perspective, considering the multimodal nature and dynamics of behaviors (including synchrony). This paper proposes an explorative method to acquire and extract relevant social signals from a naturalistic early parent–infant interaction. An experimental setup is proposed based on both clinical and technical requirements. We extracted various cues from body postures and speech productions of partners using the IMI2S (Interaction, Multimodal Integration, and Social Signal) Framework. Preliminary clinical and computational results are reported for two dyads (one pathological in a situation of severe emotional neglect and one normal control) as an illustration of our cross-disciplinary protocol. The results from both clinical and computational analyzes highlight similar differences: the pathological dyad shows dyssynchronic interaction led by the infant whereas the control dyad shows synchronic interaction and a smooth interactive dialog. The results suggest that the current method might be promising for future studies.

Erratum to: An insight into multimodal databases for social signal processing: acquisition, efforts, and directions

Erratum to: An insight into multimodal databases for social signal processing: acquisition, efforts, and directions

Leveraging Social Judgment Theory to Examine the Relationship between Social Cues and Signals in Human-Robot Interactions

Human-robot interaction (HRI) research requires new techniques for understanding the social dynamics that occur at the interface between humans and robots. Prior work has focused on incorporating the social cues and social signals distinction from the field of social signal processing and complementing this with recent advances in understanding human social cognition that specify two primary types of cognitive processes. A related account, stemming from Social Judgment Theory (SJT), specifies a Lens Model for which cues can be interpreted as well as the task conditions that would induce either of the types of cognitive processes. Surprisingly, SJT-based research has not yet examined the social cue and signal relationship. We argue it provides an ideal path forward for such research and we integrate these related disciplines of study to provide a theoretically derived account that can be useful for both the design of humanhuman and HRI experiments focused on social interaction dynamics.

Structure-revealing data fusion.

BackgroundAnalysis of data from multiple sources has the potential to enhance knowledge discovery by capturing underlying structures, which are, otherwise, difficult to extract. Fusing data from multiple sources has already proved useful in many applications in social network analysis, signal processing and bioinformatics. However, data fusion is challenging since data from multiple sources are often (i) heterogeneous (i.e., in the form of higher-order tensors and matrices), (ii) incomplete, and (iii) have both shared and unshared components. In order to address these challenges, in this paper, we introduce a novel unsupervised data fusion model based on joint factorization of matrices and higher-order tensors.ResultsWhile the traditional formulation of coupled matrix and tensor factorizations modeling only shared factors fails to capture the underlying structures in the presence of both shared and unshared factors, the proposed data fusion model has the potential to automatically reveal shared and unshared components through modeling constraints. Using numerical experiments, we demonstrate the effectiveness of the proposed approach in terms of identifying shared and unshared components. Furthermore, we measure a set of mixtures with known chemical composition using both LC-MS (Liquid Chromatography - Mass Spectrometry) and NMR (Nuclear Magnetic Resonance) and demonstrate that the structure-revealing data fusion model can (i) successfully capture the chemicals in the mixtures and extract the relative concentrations of the chemicals accurately, (ii) provide promising results in terms of identifying shared and unshared chemicals, and (iii) reveal the relevant patterns in LC-MS by coupling with the diffusion NMR data.ConclusionsWe have proposed a structure-revealing data fusion model that can jointly analyze heterogeneous, incomplete data sets with shared and unshared components and demonstrated its promising performance as well as potential limitations on both simulated and real data.Electronic supplementary materialThe online version of this article (doi:10.1186/1471-2105-15-239) contains supplementary material, which is available to authorized users.

The application of biological motion research: biometrics, sport, and the military.

The body of research that examines the perception of biological motion is extensive and explores the factors that are perceived from biological motion and how this information is processed. This research demonstrates that individuals are able to use relative (temporal and spatial) information from a person's movement to recognize factors, including gender, age, deception, emotion, intention, and action. The research also demonstrates that movement presents idiosyncratic properties that allow individual discrimination, thus providing the basis for significant exploration in the domain of biometrics and social signal processing. Medical forensics, safety garments, and victim selection domains also have provided a history of research on the perception of biological motion applications; however, a number of additional domains present opportunities for application that have not been explored in depth. Therefore, the purpose of this paper is to present an overview of the current applications of biological motion-based research and to propose a number of areas where biological motion research, specific to recognition, could be applied in the future.

Real-time feedback on nonverbal clinical communication. Theoretical framework and clinician acceptance of ambient visual design.

This article is part of the focus theme of Methods of Information in Medicine on "Pervasive Intelligent Technologies for Health". Effective nonverbal communication between patients and clinicians fosters both the delivery of empathic patient-centered care and positive patient outcomes. Although nonverbal skill training is a recognized need, few efforts to enhance patient-clinician communication provide visual feedback on nonverbal aspects of the clinical encounter. We describe a novel approach that uses social signal processing technology (SSP) to capture nonverbal cues in real time and to display ambient visual feedback on control and affiliation--two primary, yet distinct dimensions of interpersonal nonverbal communication. To examine the design and clinician acceptance of ambient visual feedback on nonverbal communication, we 1) formulated a model of relational communication to ground SSP and 2) conducted a formative user study using mixed methods to explore the design of visual feedback. Based on a model of relational communication, we reviewed interpersonal communication research to map nonverbal cues to signals of affiliation and control evidenced in patient-clinician interaction. Corresponding with our formulation of this theoretical framework, we designed ambient real-time visualizations that reflect variations of affiliation and control. To explore clinicians' acceptance of this visual feedback, we conducted a lab study using the Wizard-of-Oz technique to simulate system use with 16 healthcare professionals. We followed up with seven of those participants through interviews to iterate on the design with a revised visualization that addressed emergent design considerations. Ambient visual feedback on non- verbal communication provides a theoretically grounded and acceptable way to provide clinicians with awareness of their nonverbal communication style. We provide implications for the design of such visual feedback that encourages empathic patient-centered communication and include considerations of metaphor, color, size, position, and timing of feedback. Ambient visual feedback from SSP holds promise as an acceptable means for facilitating empathic patient-centered nonverbal communication.

社会信号处理与分析的基本框架: 从社会传感网络到计算辩证解析方法

The availability of real-time and reliable social signals is a necessity for implementing closed-loop and feedback based dynamic social management. However, the concepts and methods for social signals are far behind those for physical signal processing and analysis, and need to be developed in a systematic approach. As social and new media and social networking reach to all corners of societies rapidly, the problem of social signal processing and analysis becomes significant and the demand for effective solutions is great and urgent. This paper investigates issues related to the modeling and management of social systems, characterization of social signals and channels, social sensors and construction of social sensing networks, artificial societies, computational experiments, and parallel execution (the ACP approach) for reasoning and synthesis in computational social systems, computational dialectics, and computational analytics. Our objective is to establish a framework and methodology for acquisition, processing, analysis, analytics, and application of social signals.

Assessing language acquisition from parent-child interaction: An event-related potential study on perception of intonation contours in infancy

The aim of this paper is to present our multidisciplinary project to study parent-child interaction. The goal of the project is to identify, test, and simulate components of child and adult speech and gestures and the consequences they might have on child language acquisition. Since typical parent-child interaction is built upon both interlocutors’ intention-reading, responsiveness to joint-attention, and imitation of speech/gestures, we make video recordings along with recordings of speech data to grasp the integration of semantic and pragmatic aspects of language acquisition. The understanding of parent-child interaction benefits further from information on brain activation involved in speech processing. As a first step to achieve the project goals, an electroencephalography/event-related potential (EEG/ERP) study exploring children’s early perception of intonation contours involved in human interactions was performed. This paper discusses the characteristics of integration of multimodal social-emotional (speech, prosody, faces, posture) signals as part of the dynamics of communication in typically developing children. Possible application fields are social signal processing (SSP; an emerging research domain that aims to provide computers ability to understand human social signals), and improvement of diagnosis of late or atypical language development in pathologies that affect the dynamics of social interaction (such as autism spectrum disorders).

Socially Aware Interactive Playgrounds

Interactive playgrounds are technology-enhanced installations that aim to provide rich game experiences for children by combining the benefits of traditional playgrounds with those of digital games. These game experiences could be attained by addressing three design considerations: context-awareness, adaptability, and personalization. The authors propose using social signal processing (SSP) to enhance current interactive playgrounds to meet these criteria. This article surveys how SSP techniques can help playgrounds automatically sense and interpret children's social interactions, adapt game mechanics to induce targeted social behavior, and learn from the sensed behavior to meet players expectations and desires.

Assessment of the communicative and coordination skills of children with Autism Spectrum Disorders and typically developing children using social signal processing

To cooperate with a partner, it is essential to communicate by sharing information through all available avenues, including hand gestures, gazes, head gestures and naturally, speech. In this paper, we compare the communicative and coordination skills of children with typical development to those of children with Autism Spectrum Disorders (ASDs) in cooperative joint action tasks. Communicative skills were assessed using a pragmatic annotation grid. Coordination skills were assessed based on automatically extracted features that characterize interactive behavior (turn-taking, synchronized gestures). First, we tested the performance of the interactive features when discriminating between the two groups of children (typical vs. ASD). Features characterizing the gestural rhythms of the therapist and the duration of his gestural pauses were particularly accurate at discriminating between the two groups. Second, we tested the ability of these features for the continuous classification problem of predicting the developmental age of the child. The duration of the verbal interventions of the therapist were predictive of the age of the child in all tasks. Furthermore, more features were predictive of the age of the child when the child had to lead the task. We conclude that social signal processing is a promising tool for the study of communication and interaction in children with ASD because we showed that therapists adapt differentially in three different tasks according to age and clinical status.

Advanced Human Behavioral Cues Detection: A Method in Social Signal Processing

Social Signal Processing (SSP) is the new research domain that aims at understanding social interactions and social intelligence. The essence of social intelligence is the ability to recognize human social signals and behaviors cues (verbal and non-verbal). We present a unified model for non-verbal behavioral cues SSP under the categories of face detection like eyes-mouth-lips-ears-nose, emotional assessments like anger-happy-sadness-neutral and physical appearance like height (short-average-tall), attractiveness (smartness) and body shape(slim-normal-flat) in real-world cluttered images which is one of the method in social signal processing. It is rather difficult task to detect and interpret temporal patterns of nonverbal behavioral cues in a given context. Our main objective is to detect non-verbal behavioral cues by introducing advanced human behavioral cues detection method. Our proposed method shown a better results and practically applicable in the real world environment. We implemented our proposed method in MATLAB R2007b simulator and designed a GUI for our proposed method .

Social Multimedia Signals: Sense, Process, and Put Them to Work

Social media gives ordinary people the power to be content creators and information disseminators. This information is embedded in multimedia shared across social networks, containing valuable indications about various facets of human life-about what captures our attention, our sharing biases, and the digital traces we abdicate. Social multimedia signal processing aims to transform the noise-like phenomena in social media into signals useful for building novel, socially aware multimedia applications and targeted advertising techniques as well as exploring new marketing methods. With a fresh way to look at the existence of multimedia in online social networks, we can also explore new marketing methods and targeted advertising techniques.

Ontological Representation of Light Wave Camera Data to Support Vision-Based AmI

Recent advances in technologies for capturing video data have opened a vast amount of new application areas in visual sensor networks. Among them, the incorporation of light wave cameras on Ambient Intelligence (AmI) environments provides more accurate tracking capabilities for activity recognition. Although the performance of tracking algorithms has quickly improved, symbolic models used to represent the resulting knowledge have not yet been adapted to smart environments. This lack of representation does not allow to take advantage of the semantic quality of the information provided by new sensors. This paper advocates for the introduction of a part-based representational level in cognitive-based systems in order to accurately represent the novel sensors' knowledge. The paper also reviews the theoretical and practical issues in part-whole relationships proposing a specific taxonomy for computer vision approaches. General part-based patterns for human body and transitive part-based representation and inference are incorporated to an ontology-based previous framework to enhance scene interpretation in the area of video-based AmI. The advantages and new features of the model are demonstrated in a Social Signal Processing (SSP) application for the elaboration of live market researches.

Social signals: from theory to applications

The Special Issue Editorial introduces the research milieu in which Social Signal Processing originates, by merging computer scientists and social scientists and giving rise to this field in parallel with Human-Computer Interaction, Affective Computing, and Embodied Conversational Agents, all similarly characterized by high interdisciplinarity, stress on multimodality of communication, and the continuous loop from theory to simulation and application. Some frameworks of the cognitive and social processes underlying social signals are identified as reference points (Theory of Mind and Intersubjectivity, mirror neurons, and the ontogenesis and phylogenesis of communication), while three dichotomies (automatic vs. controlled, individualistic vs. intersubjective, and meaning vs. influence) are singled out as leads to navigate within the theoretical and applicative studies presented in the Special Issue.

Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines

Synchrony refers to individuals' temporal coordination during social interactions. The analysis of this phenomenon is complex, requiring the perception and integration of multimodal communicative signals. The evaluation of synchrony has received multidisciplinary attention because of its role in early development, language learning, and social connection. Originally studied by developmental psychologists, synchrony has now captured the interest of researchers in such fields as social signal processing, robotics, and machine learning. This paper emphasizes the current questions asked by synchrony evaluation and the state-of-the-art related methods. First, we present definitions and functions of synchrony in youth and adulthood. Next, we review the noncomputational and computational approaches of annotating, evaluating, and modeling interactional synchrony. Finally, the current limitations and future research directions in the fields of developmental robotics, social robotics, and clinical studies are discussed.

Social signal processing and personal robotics for psychopathology: Signals, communicative acts and behaviours

Social signal processing and personal robotics for psychopathology: Signals, communicative acts and behaviours

A Social Signal Processing View on Leadership: Specific Behaviors Characterize Considerate Leaders

Combining leadership research with findings from small group research and clinical psychology, we propose that three specific observable interaction behaviors of leaders are partially responsible f...

Conceptual frameworks for multimodal social signal processing

This special issue is about a research area which is developing rapidly. Pentland gave it a name which has become widely used, ‘Social Signal Processing’ (SSP for short), and his phrase provides the title of a European project, SSPnet, which has a brief to consolidate the area. The challenge that Pentland highlighted was understanding the nonlinguistic signals that serve as the basis for “subconscious discussions between humans about relationships, resources, risks, and rewards”. He identified it as an area where computational research had made interesting progress, and could usefully make more.

Human behavior analysis in video surveillance: A Social Signal Processing perspective

The analysis of human activities is one of the most intriguing and important open issues for the automated video surveillance community. Since few years ago, it has been handled following a mere Computer Vision and Pattern Recognition perspective, where an activity corresponded to a temporal sequence of explicit actions (run, stop, sit, walk, etc.). Even under this simplistic assumption, the issue is hard, due to the strong diversity of the people appearance, the number of individuals considered (we may monitor single individuals, groups, crowd), the variability of the environmental conditions (indoor/outdoor, different weather conditions), and the kinds of sensors employed. More recently, the automated surveillance of human activities has been faced considering a new perspective, that brings in notions and principles from the social, affective, and psychological literature, and that is called Social Signal Processing (SSP). SSP employs primarily nonverbal cues, most of them are outside of conscious awareness, like face expressions and gazing, body posture and gestures, vocal characteristics, relative distances in the space and the like. This paper is the first review analyzing this new trend, proposing a structured snapshot of the state of the art and envisaging novel challenges in the surveillance domain where the cross-pollination of Computer Science technologies and Sociology theories may offer valid investigation strategies.

A three-component framework for empathic technologies to augment human interaction

Empathy can be considered one of our most important social processes. In that light, empathic technologies are the class of technologies that can augment empathy between two or more individuals. To provide a basis for such technologies, a three component framework is presented based on psychology and neuroscience, consisting of cognitive empathy, emotional convergence, and empathic responding. These three components can be situated in affective computing and social signal processing and pose different opportunities for empathic technologies. To leverage these opportunities, automated measurement possibilities for each component are identified using (combinations of) facial expressions, speech, and physiological signals. Thereafter, methodological challenges are discussed, including ground truth measurements and empathy induction. Finally, a research agenda is presented for social signal processing. This framework can help to further research on empathic technologies and ultimately bring it to fruition in meaningful innovations. In turn, this could enhance empathic behavior, thereby increasing altruism, trust, cooperation, and bonding.