Form and Function of Hand Gestures for Interpretation and Generation

Abstract

This paper investigates the relation between the form and function of hand gestures in audio and video recordings of American and English political iscourse of different type. Gestures have an important function in face-to-face communication contributing to the successful delivery of the message by reinforcing what is expressed by speech or by adding new information to what is uttered. The relation between form and function of gestures has been described by some of the pioneers of gestural studies. However, since gestures are multifunctional and they must be interpreted in context, it is important to investigate to what extent the form of gestures can be used to interpret their function automatically. Individuating the relation between form and function of gestures is also important for generating appropriate gestures in various communicative situations and this knowledge is vital for the integration of machine-human communicative and cognitive functions. In this paper we show that the automatic classification of the semiotic types of hand gestures using their shape description is quite successful and this is an important step towards their interpretation in face-to-face communication as well as their interpretation and generation in advanced multimodal interactive systems. More specifically in the present work we annotated the semiotic types of hand gestures produced by five politicians in different contexts, adding this information to existing multimodal annotations. Then, we trained machine learning algorithms to identify the semiotic type of the hand gestures. The F1 score obtained by the best performing algorithms on the classification of four semiotic types is 0.7 outperforming the F1 score of 0.59 obtained in a preceding pilot study which addressed the identification of three semiotic types of hand gestures produced by a speaker in a small dataset.