Social Signal Processing for Automatic Role Recognition

Abstract

Introduction According to the Oxford Dictionary of Sociology, “Role is a key concept in sociological theory. It highlights the social expectations attached to particular social positions and analyses the workings of such expectations” (Scott & Marshall, 2005). Furthermore, “Role theory concerns one of the most important features of social life, characteristic behaviour patterns or roles” (Biddle, 1986). Besides stating that the notion of role is crucial in sociological inquiry, the definitions introduce the two main elements of role theory, namely expectations and characteristic behaviour patterns . In particular, the definitions suggest that the expectations of others – typically associated to the position someone holds in a given social context – shape roles in terms of stable and recognizable behavioural patterns. Social signal processing (SSP) relies on the similar key idea that social and psychological phenomena leave physical, machine detectable traces in terms of both verbal (e.g., lexical choices) and nonverbal (prosody, postures, facial expressions, etc.) behavioural cues (Vinciarelli, Pantic, & Bourlard, 2009; Vinciarelli et al., 2012). In particular, most SSP works aim at automatically inferring phenomena like conflict, personality, mimicry, effectiveness of delivery, etc. from verbal and nonverbal behaviour. Hence, given the tight relationship between roles and behavioural patterns, SSP methodologies appear to be particularly suitable to map observable behaviour into roles, i.e. to perform automatic role recognition (ARR). Not surprisingly, ARR was one of the earliest problems addressed in the SSP community and the proposed approaches typically include three main steps, namely person detection (segmentation of raw data streams into segments corresponding to a given individual), behavioural cues extraction (detection and representation of relevant behavioural cues), and role recognition (mapping of detected cues into roles). Most of the works presented in the literature propose experiments over two main types of data, i.e. meeting recordings and broadcast material. The probable reason is that these contexts are naturalistic, but sufficiently constrained to allow effective automatic analysis. The rest of this chapter is organized as follows: role recognition technology, which introduces the main technological components of an ARR system; previous work, which surveys the most important ARR approaches proposed in the literature; open issues, which outlines the main open issues and challenges of the field; and the last section, which draws some conclusions.