Human-centered AI and robotics

This study examines interaction patterns in a series of game activities for learning social skills by autistic youth in a 3D game‐based collaborative virtual learning environment (CVLE). Researchers studying collaborative learning have indicated the importance of social interactions and social presence. However, few studies have examined the relationship of avatar social interactions with embodied social presence in a 3D game‐based CVLE. Specifically, we examined avatar‐mediated verbal and nonverbal interactions by autistic youth in iSocial, a game‐based 3D CVLE for developing social competencies. How are avatar‐mediated verbal and nonverbal social interactions related to the extent of embodied social presence? Building on prior studies on embodied social presence (ESP) theory and the verbal and nonverbal social interaction framework, this paper aims to explore the link between the combination of verbal (appropriate and inappropriate) and nonverbal (avatar proximity, orientation, joint attention and gesture) interaction patterns and experienced ESP level. We report on the results of 3 cohorts of a total of 11 youth aged from 11 to 14 who were diagnosed with Autism Spectrum Disorder learning in 13 game activities in iSocial. The video data of the participants’ screen recordings were analyzed and coded based on the ESP theory and verbal and nonverbal social interaction framework. Through cluster analysis, the results identify distinct patterns of verbal and nonverbal interaction that are associated with different levels of embodied social presence. The findings of this study (1) shed light on the link between social interactions and embodied social presence and (2) provide a deeper understanding of how the unique spatial and visual characteristics of 3D CVLE and the design of game activities in 3D CVLE may transform collaborative learning, especially for autistic youth.

As technology becomes ubiquitous in the homes of many families, children learn to incorporate touchscreen devices into their everyday activities. The present study explored young children’s role in shaping their gaming experience to maintain social interactivity. Twenty-nine children at 2.5 years old in the U.S. played a non-educational game on a touchscreen in the presence of a researcher. We examined the temporal distribution and functions of toddlers’ active initiation of verbal and nonverbal social interactions. The results indicated modality differences. The toddlers tended to initiate verbal interactions in the middle segment of gaming and nonverbal interactions at the beginning. Moreover, toddlers who initiated a greater proportion of verbal (as opposed to nonverbal) interactions tended to amass more social interactions overall. Finally, the child-initiated interactions served functions of sharing success, seeking assistance, requesting a change, and inviting others to participate. Together, these findings provide insight into the social aspects of touchscreen gaming as toddlers navigate this new form of play.

Social interactions arise from patterns of communicative signs, whose perception and interpretation require a multitude of cognitive functions. The semiotic framework of Peirce’s Universal Categories (UCs) laid ground for a novel cognitive-semiotic typology of social interactions. During functional magnetic resonance imaging (fMRI), 16 volunteers watched a movie narrative encompassing verbal and non-verbal social interactions. Three types of non-verbal interactions were coded (“unresolved,” “non-habitual,” and “habitual”) based on a typology reflecting Peirce’s UCs. As expected, the auditory cortex responded to verbal interactions, but non-verbal interactions modulated temporal areas as well. Conceivably, when speech was lacking, ambiguous visual information (unresolved interactions) primed auditory processing in contrast to learned behavioral patterns (habitual interactions). The latter recruited a parahippocampal-occipital network supporting conceptual processing and associative memory retrieval. Requesting semiotic contextualization, non-habitual interactions activated visuo-spatial and contextual rule-learning areas such as the temporo-parietal junction and right lateral prefrontal cortex. In summary, the cognitive-semiotic typology reflected distinct sensory and association networks underlying the interpretation of observed non-verbal social interactions.

Interpersonal communication and relationship building promote successful collaborations. This study investigated the effect of conversational nonverbal and verbal interactions of a robot on bonding and relationship building with a human partner. Participants interacted with two robots that differed in their nonverbal and verbal expressiveness. The interactive robot actively engaged the participant in a conversation before, during and after a collaborative task whereas the non-interactive robot remained passive. The robots' nonverbal and verbal interactions increased participants' perception of the robot as a social actor and strengthened bonding and relationship building between human and robot. The results of our study indicate that the evaluation of the collaboration improves when the robot maintains eye contact, the robot is attributed a certain personality, and the robot is perceived as being alive. Our study could not show that an interactive robot receives more help by the collaboration partner. Future research should investigate additional factors that facilitate helpful behavior among humans, such as similarity, attributional judgement and empathy.

Social interaction between residents and staff is an important factor influencing sense of well-being. This study examined the relationship between staff-resident interactions and psychological well-being of persons with dementia. A total of 831 observations of 110 persons with dementia in 17 nursing homes and 6 assisted living facilities were included. Psychological well-being was measured by observed displays of positive and negative emotional expressions. Social interaction was determined by the type of social interaction (ie, verbal interaction, nonverbal interaction, and both verbal and nonverbal interactions) and the quality of interaction (ie, positive, negative, and neutral). Verbal or both verbal and nonverbal interactions showed significant relationship with positive and negative emotional expressions. Positive interaction was significantly associated with more positive emotional expression, whereas negative interaction was not. Staff-resident interactions are important to promote the psychological well-being of persons with dementia in residential care.

A common approach when simulating face-to-face interpersonal scenarios with virtual humans is to afford users only verbal interaction while providing rich verbal and non-verbal interaction from the virtual human. This is due to the difficulty in providing robust recognition of user non-verbal behavior and interpretation of these behaviors within the context of the verbal interaction between user and virtual human. To afford robust hand and tool-based non-verbal interaction with life-sized virtual humans, we propose virtual multi-tools. A single hand-held, tracked interaction device acts as a surrogate for the virtual multi-tools: the user's hand, multiple tools, and other objects. By combining six degree-of-freedom, high update rate tracking with extra degrees of freedom provided by buttons and triggers, a commodity device, the Nintendo Wii Remote, provides the kinesthetic and haptic feedback necessary to provide a high-fidelity estimation of the natural, unencumbered interaction provided by one's hands and physical hand-held tools. These qualities allow virtual multi-tools to be a less error-prone interface to social and task-oriented non-verbal interaction with a life-sized virtual human. This paper discusses the implementation of virtual multi-tools for hand and tool-based interaction with life-sized virtual humans, and provides an initial evaluation of the usability of virtual multi-tools in the medical education scenario of conducting a neurological exam of a virtual human.

Research on interaction in speaking assessment suggests that both verbal and nonverbal interaction are integral parts of the construct of interactional competence (Galaczi & Taylor, 2018; Plough et al., 2018; Young, 2011). However, little has been done to investigate which features significantly contribute to interactional competence scores. This study, therefore, examined which interaction features that raters noticed in individual scripted interview and paired discussion tasks to gain an insight into the interactional competence construct, providing validity evidence for an inclusion of interactional competence in speaking assessment. Sixty-eight student performances were rated based on interaction rating scales. Exploratory factor analysis revealed four factors: nonverbal communication, topic management, interactional management, and interactive listening. Logistic regressions showed that while raters attended to more topic management features in the individual scripted interview task, they noticed more interactional management features in the paired discussion task. Simple regressions showed that nonverbal communication and topic management features predicted interactional competence scores in the individual scripted interview task, whereas nonverbal communication, topic management, interactional management, and interactive listening features were predictors of scores in the paired discussion task. The findings suggest that both nonverbal and verbal interaction features are important in the interactional competence construct with the paired task providing test-takers with more opportunities to demonstrate their interactional ability.

Human language is inherently embodied and grounded in sensorimotor representations of the self and the world around it. This suggests that the body schema and ideomotor action-effect associations play an important role in language understanding, language generation, and verbal/physical interaction with others. There are computational models that focus purely on non-verbal interaction between humans and robots, and there are computational models for dialog systems that focus only on verbal interaction. However, there is a lack of research that integrates these approaches. We hypothesize that the development of computational models of the self is very appropriate for considering joint verbal and physical interaction. Therefore, they provide the substantial potential to foster the psychological and cognitive understanding of language grounding, and they have significant potential to improve human-robot interaction methods and applications. This review is a first step toward developing models of the self that integrate verbal and non-verbal communication. To this end, we first analyze the relevant findings and mechanisms for language grounding in the psychological and cognitive literature on ideomotor theory. Second, we identify the existing computational methods that implement physical decision-making and verbal interaction. As a result, we outline how the current computational methods can be used to create advanced computational interaction models that integrate language grounding with body schemas and self-representations.

Regressive Augmentation: Investigating Ubicomp’s Romantic Promises

Maternal Mobile Device Use During a Structured Parent–Child Interaction Task

Maze exploration behaviors using an integrated evolutionary robotics environment

This narrative account describes a collaborative qualitative video data analysis process between a bilingual Deaf female researcher and a bilingual Puerto Rican female researcher. Via three processing points, we examine our journeys to co-construct meanings from a single video data source which was part of a larger ethnographic study of an urban community change initiative. We highlight how our respective epistemologies informed the process of watching, analyzing, and interpreting nonverbal and verbal interactions from a video segment. The video watching process included a hunch and discovery of a critical incident. While engaging independently and collaboratively in analysis, we confirmed how the critical incident revealed concepts of access and participation. This article is distinctive in that it highlights Deaf epistemology and qualitative inquiry processes through video data analysis of nonverbal interactions. Our work contributes to the growing body of methodology literature emphasizing collaborative social practices for video data analysis.

This study analyzed the communication among construction workers at Mabapi Estate. Its primary objective was to establish the relationship between languages and work in construction industry through the description and explication of communication behaviors and competences that construction workers relied on to participate in intelligible socially organized verbal and non-verbal interactions. To gather data for the study, an ethnographic participant observation and semi-structured interview methods were used. The study established that construction workers: created a common language that comprised coinages and vocabulary derived from different language communities, had the propensity to talk about issues that may appear uncouth to the civilized society and that they employed varied non-verbal cues to enhance their verbal interactions.

The purpose of this study was to investigate gender differences in problem-solving and conflict-resolution skills in preschool children. Children between 4 and 5 years of age completed 3 problem-solving tasks with either a same-sex or a different-sex peer. Children's verbal and nonverbal interactions were analyzed. Girls used mitigation more often than did boys. Mixed-sex dyads engaged in controlling verbal interactions more often than same-sex dyads. There were relationships between verbal and nonverbal behaviors and task success; these relationships also differed across pair types. The results of the study demonstrate that the gender differences in types of verbal interactions previously observed in preschool children's free play are also present in their problem-solving interactions and that children are able to alter the types of behaviors they use depending upon both partner gender and the type of task involved.

This study involves quality of interaction between Day Centre staff and Service-Users with Learning Disabilities in two Day Centres (Dc1 n=50, Dc2 n=247). The Quality of Interactions Schedule (QUIS) was applied within an eight month prospective design, involving 32 visits which constructed a ‘typical day’ composed of 20-30 minute observational sessions within each Day Centre. The largest proportions of interactions were of a positive nature (87%) across both Day Centres. More positive care interactions were seen in Dc1 and significantly higher rates of positive interactions in Dc1. The greatest use of Verbal and Non-verbal interaction was observed in Dc1. Service users initiated more of the interactions in Dc2. Lengthier verbal interactions were seen in Dc1 and conversely greater amounts of short verbal interactions were seen in Dc2. Within both centres time-tables that were inspected indicated similar proportions spent in: Work, Leisure, Education, Community and Social skills sessions. In comparison to previous studies a relatively low proportion of activities were community based and social skills orientated. Results are framed within comparable observational studies in Day Centres; differences in measurement characteristics employed (Cummins, 2000, 2002) and service evaluation of quality of life in Day centres for adults with Learning Disabilities. Reliability and validity of results were also examined.