Non-verbal Actions Research Articles

The multimodal EchoBorg: not as smart as it looks

In this paper we present a Multimodal Echoborg interface to explore the effect of different embodiments of an Embodied Conversational Agent (ECA) in an interaction. We compared an interaction where the ECA was embodied as a virtual human (VH) with one where it was embodied as an Echoborg, i.e, a person whose actions are covertly controlled by a dialogue system. The Echoborg in our study not only shadowed the speech output of the dialogue system but also its non-verbal actions. The interactions were structured as a debate between three participants on an ethical dilemma. First, we collected a corpus of debate sessions with three humans debaters. This we used as baseline to design and implement our ECAs. For the experiment, we designed two debate conditions. In one the participant interacted with two ECAs both embodied by virtual humans). In the other the participant interacted with one ECA embodied by a VH and the other by an Echoborg. Our results show that a human embodiment of the ECA overall scores better on perceived social attributes of the ECA. In many other respects the Echoborg scores as poorly as the VH except copresence.

THE ANALYSIS AND INTERPRETATION OF GREETING AND SMILING GESTURES IN COMMUNICATION

Effective communication is the combined harmony of verbal and nonverbal actions. Gestures and spoken language often go hand in hand. Usually a part of the meaning is expressed by a certain gesture. The meaning of a gesture has some variations depending on the strength of the handshake, its duration, a smile accompanied by other facial expressions such as eye movements, jaw-drop, the tense mouth which are accurately reflecting what’s going on inside the speaker.

Language teachers and multimodal instructional reflections during video-based online learning tasks

ABSTRACT This study explores the multiple ways in which a group of five in-service language teachers reflected in an online video-embedded learning environment. The findings suggest that when engaging in video-based reflective tasks, teachers evaluated and interpreted instructional practices presented to them based on multimodal classroom interactions (i.e. both verbal and non-verbal actions such as pause, gaze or body language). Furthermore, through reflecting on the multimodal classroom interactions, teachers also developed new awareness about their instructional practices and zoomed in on teaching techniques essential for language teaching. The findings stressed the importance of approaching teacher reflection from the multimodal perspective afforded by technology such as videos.

168 A Novel Approach to Interactive, Online History-Taking in Medical Education During the COVID-19 Pandemic

The COVID-19 pandemic resulted in an unprecedented shift from face-to-face to online teaching with a subsequent deleterious impact on the quality of teaching delivery within medical education Our aim was to demonstrate the feasibility of utilizing a novel approach to interactive, online simulated history-taking.The session was designed for students attending a virtual work-experience programme and is founded upon gamification principles. History-taking is a humanistic social process requiring effective communication skills and recognition of verbal and non-verbal cues. The session enabled students to directly instruct a passive clinician to take a history from a simulated patient actor, incorporating both verbal and non-verbal actions. This novel teaching method is analogous to Freire’s work, highlighting the importance of the learner being an active participant, thus enabling experiential learning This interactive approach required a simulated patient actor, a passive clinician and a verbal instructor to be present. Students were presented with a case scenario and viewed the interaction between the simulated patient actor and passive clinician online via a video conferencing service (in this case, Zoom). Students provided instructions to the passive clinician using the typed chat function. These typed instructions were relayed by the verbal instructor to the passive clinician via an earpiece and the passive clinician would then embody these instructions. This allowed students to work as a group to directly elicit a history and witness their instructions being actioned. Examples of instructions included questions to verbalize, adjustments to body language and alteration of the tone of voice. Following good student instruction, the passive clinician gradually grew in competence and confidence throughout the scenario, thus allowing real-time, interactive feedback of their history-taking approach. The session concluded with a reflection of the communication traits that helped to gain an effective history from the patient using a debrief model.

The Empathic Communication Analytical Framework (ECAF): A multimodal perspective on emotional communication in interpreter-mediated consultations

Empathic communication (EC) in healthcare occurs when patients express empathic opportunities, such as emotions, to which doctors respond empathically. This interactional process during which participants try to achieve specific communicative goals (e.g., seeking and displaying empathy) serves as a context in which doctors and patients perform verbal and nonverbal actions and collaboratively co-construct meaning. This applies to interpreter-mediated consultations (IMCs) too, where interpreters perform additional actions of a similar kind. However, there is a dearth of research on the ways in which participants perform these actions in the context of EC, and how these actions in turn help (re)shape the context of EC in IMCs (Theys et al., 2020). To date, any tools for studying EC investigate participants’ actions in isolation, without studying them in the context of EC or in relation to the participants’ awareness of their own and others’ ongoing interactions. In this article, we present the Empathic Communication Analytical Framework (ECAF). The tool draws on valid, complementary analytical tools that allow for a fine-grained, three-level multimodal analysis of interactions. The first level of analysis allows for instances of EC in spoken language IMCs to be identified and for participants’ verbal actions in the context of EC to be studied. The second level allows analysts to investigate participants’ verbal and nonverbal actions in the previously identified context of EC. The third level of analysis links the participants’ concurrent verbal and nonverbal (inter)actions to their levels of attention and awareness and shows how participants’ actions are shaped and in turn help to reshape the context of EC in IMCs. In this article, we present the various levels of the ECAF framework, discuss its application to real-life data, and adopt a critical stance towards its affordances and limitations by looking into one excerpt of EC in IMCs. It is shown that the three distinct yet interconnected levels of analysis in the ECAF framework allow participants’ concurrent multimodal interactions in the context of EC to be studied.

Lying Without Saying Something False? A Cross-Cultural Investigation of the Folk Concept of Lying in Russian and English Speakers

AbstractThe present study examines cross-cultural differences in people’s concept of lying with regard to the question of whether lying requires an agent tosaysomething they believe to be false. While prominent philosophical views maintain that lying entails that a person explicitly expresses a believed-false claim, recent research suggests that people’s concept of lying might also include certain kinds of deception that are communicated more indirectly. An important drawback of previous empirical work on this topic is that only few studies have investigated people’s concept of lying in non-Western samples. In the present study, we compare people’s intuitions about lying with indirect deceptions (i.e., presuppositions, conversational implicatures, and non-verbal actions) in a sample ofN = 255 participants from Russia andN = 300 participants from the United Kingdom. Our findings show a strong degree of similarity between lie ratings of participants from Russia and the United Kingdom, with both samples holding it possible for agents to lie with deceptive statements and actions that do not involve the agentsayingsomething they believe to be false.

Tactics and Technology of Non-Verbal Investigative Actions Production in Computer Crimes Cases: Theory and Practice

The paper considers the doctrine of information and computer forensic support of investigative and judicial actions tactics, which is part of the system of the private theory of information and computer support of forensic activity. The subject of the teaching is the laws of gathering, scrutinizing and applying computer information in the production of investigative and judicial actions. The objects are the tactics and technology of investigative and judicial actions. Based on the theoretical aspects of the teaching, the authors have developed the tactics and technology of non-verbal investigative actions in cases of computer crimes. Among these are inspection of the scene of the crime, search, seizure, investigative experiment, given the choice of tactical impact and making a tactical decision depending on the specifics of investigative situations in the conditions of tactical risk associated with possible counteraction to the investigation. The tactical and technological support for the production of the above-mentioned non-verbal investigative actions is developed taking into account the features of digital traces, which are characterized by a high speed of transformation, are easily destroyed and modified, can be represented by an almost infinite number of copies, are characterized by the impossibility of perception directly by the senses, but only with the use of special devices and programs for detection, fixation and preservation, are confirmed by control numbers (hash sums) or other data indicating their integrity. The basic principles of working with digital traces in the production of non-verbal investigative actions are determined. They are the preservation of digital traces unchanged at all stages of working with them; full reflection of all manipulations in the protocols of investigative actions; the exceptional importance of preparatory measures, including the selection of a specialist and the determination of his competence, the availability of the necessary equipment and software for working with digital traces. For each of the above-mentioned investigative actions in the investigation of computer crimes, tactical techniques and technological support for the most effective obtaining of criminally significant evidentiary and investigative information have been developed.

IDEATION DISCOURSE: MEANS OF COMMUNICATIVE EFFECT ACHIEVEMENT (ON THE MATERIAL OF TED-PLATFORM)

Interactive aspects of communication include the following categories embodied in the metacommunicative elements of discourse, one of which is speech mode. The article focuses on pattern identification of communicative effect on the recipient by varying serious and humorous modes of the speech. The attention is paid to the identification of the initial strategies of communicative effect in both serious and humorous mode of English ideative speech. Communicative mode is an emotionally-stylistic communication format that occurs during the interaction between communicants and determines their changing attitudes and the choice of all means of communication. Serious mode orients the subjects of discourse to perceive communicative action as the one corresponding to norms. Humorous mode orients the subjects of discourse to perceive communicative action as the one inconsistent with norms. The paper gives results of inferential analysis of responsive actions on how successful the ideators’ speeches are according to the recipients’ instant and delayed responsive actions. The evaluation of communicative success has been conducted with the help of inferencial analysis of verbal and non-verbal actions of the audience taking into account instant and delayed responsive actions, in particular total number of the recording views, average monthly number of views; laughter, applause, cheers, murmur.

The Dialogue of Creativity: Teaching the Creative Process by Animating Student Work as a Collaborating Creative Agent

Material artifacts play an important role in many learning environments. Such artifacts can include sketches, manipulatives, 3D models, toys and games, or the scrap materials found in makerspaces. Some theorists have argued that material artifacts, even though they do not move or talk, should be considered to have autonomous agency and to interact as equals with human participants. But there have been few empirical studies that explore whether or how material artifacts are attributed agency by human participants. This paper contributes to this issue by analyzing interactions between professor and student in design studio classrooms, where the student’s created work is the central focus. I analyze the close coordination of talk—including syntactic constructions and verb aspect—with nonverbal action, including eye gaze, gesture, body orientation, and body position. In the first set of findings, I identify six interactional mechanisms that attribute agency to the work, in both verbal and nonverbal modalities. I demonstrate that through the use of these six laminated multimodal resources, the student’s creative work is socially constructed as an agentive participant. These findings contribute to our understanding of the role of artifact agency in social practices. In the second set of findings, I show how professors enlist these six practices in discursive patterns that scaffold students in mastering the dialogue of creativity: a process that distributes creative agency between the student and their unfolding work. These dialogues model for students a creative process characterized by iteration, ambiguity, exploration, and emergence. I conclude by discussing the implications for our understanding of teaching and learning for creativity.

THE ETIOLOGY OF PERPETRATOR OF CHILDREN SEXUAL VIOLENCE: A CASE STUDY IN THE PPA UNIT IN THE POLICE STATION OF SUMBAWA REGENCY

Sexual violence is any verbal and nonverbal action to manipulate another person to make the person engage in unwanted sexual activity. Recently, sexual violence against children in Sumbawa Besar District raised the concerns of various groups. This is due to the increasing number of cases of sexual violence against children. The purpose of this research is to find out causes (the etiology) of the perpetrators in comitting sexual violence. The method used in this research is qualitative research with the case study approach. The data were collected using interview and observation methods. The subjects of this study were perpetrators of sexual violence in Sumbawa district. The data credibility was examined with triangulation of source, technique, and theory. The four stages of data analysis were data collection, data sortation, data conclusions, and data verification. The results showed that there were various reasons of the perpetrators to commit sexual violence towards children. Some of the factors are problems in their families and the need and opportunity to fulfill their sexual desires.

A two-lab direct replication attempt of Southgate, Senju and Csibra (2007).

The study by Southgate et al. (2007 Psychol. Sci. 18, 587–592. (doi:10.1111/j.1467-9280.2007.01944.x)) has been widely cited as evidence for false-belief attribution in young children. Recent replication attempts of this paradigm have yielded mixed results: several studies did not replicate the original findings, raising doubts about the suitability of the paradigm to assess non-verbal action prediction and Theory of Mind. In a preregistered collaborative study including two of the original authors, we tested one hundred and sixty 24- to 26-month-olds across two locations using the original stimuli, procedure and analyses as closely as possible. We found no evidence for action anticipation: only roughly half of the infants looked to the location of an agent's impending action when action prediction did not require taking into account the agent's beliefs and a similar number when the agent held a false-belief. These results and other non-replications suggest that this paradigm does not reliably elicit action prediction and thus cannot assess false-belief understanding in 2-year-olds. While the present results do not support any claim regarding the presence or absence of Theory of Mind in infants, we conclude that an important piece of evidence that has to date supported arguments for the existence of this competence can no longer serve that function.

Рец. на кн.: RAY WILKINSON, JOHN P. RAE, GITTE RASMUSSEN (eds.). ATYPICAL INTERACTION: THE IMPACT OF COMMUNICATIVE IMPAIRMENTS WITHIN EVERYDAY TALK. Cham: Palgrave Macmillan, 2020. XXIII+470 p.

The collection of articles under review includes conversation analytic studies of interactions involving the participation of at least one person with communicative impairments (aphasia, dementia, dysarthria, etc.). The authors concentrate on how these impairments influence interaction—the organization of repair in particular—as well as on issues of participants’ adaption to impairments, collaboration, the agency of people with impairments, and practices of face maintenance. Three more general issues connected to this field of study are discussed in the review. The first issue is a choice of analytic categories and the application of the category of repair. This category seems justified since participants frequently have to clarify the meaning of their partners’ actions. However, this choice may appear problematic if one does not take into account that interactions with people with impairments have their own progressivity, which differs from the progressivity of ordinary conversation. The second issue is the role of nonverbal actions, which is crucial in circumstances where some of the participants lack verbal resources. The third issue concerns the problem of the understanding which participants try to achieve in the course of interaction and which researchers try to achieve in the course of analysis. This task becomes more challenging in comparison to ordinary conversations. On the one hand, actions of persons with impairments are sometimes ambiguous and require special interpretative efforts from their partners. On the other hand, there is always a risk that the partner will interpret actions of impaired person inadequately.

Multimodality in the classroom presentation genre: Findings from a study of Turkish psychology undergraduate talks

Despite its obvious importance to learning and assessment across the academy, the undergraduate classroom presentation has received less research attention than other academic genres, and little is known about how multiple modes of communication are deployed within it. To explore how the use of different modes varied between sections, and how these actions affected the speech of presenters, this research into student presentations given at a university in Turkey combined a move-step analysis of speech with a mixed-methods study of multimodality. The study's main results were as follows: first, that presentation sections were distinctively configured by arrays of multimodal action; second, that the effectiveness of speech in performing specific moves in the genre was moderated in several specific ways by actions in other modes; and third, that some moves were performed in part by non-verbal actions. These findings are briefly discussed with reference to their theoretical and pedagogical implications.

I saw him first: Competitive nonverbal flirting among women, the tactics used and their perceived effectiveness

Here we explored nonverbal actions women use to flirt competitively against each other for the purposes of accessing a mate. We also investigated the perceived effectiveness of these competitive flirting actions. Using act nomination, Study 1 (n = 91) yielded 11 actions (eye contact with the man, dancing in his line of sight, smiling at him, touching him, giggling at his jokes, butting in between the other woman and the man, showing distaste for her, brushing against him, hugging him, flirting with other men, waving to him) for competitive flirtation against other women. Actions that signal possession (e.g., tie-signs) were predicted to be perceived as the most effective. While other actions were included in Study 2 (n = 139), results showed the most effective actions were tie-signs: touching him, initiating eye contact, hugging him, giggling at his jokes, and butting in between him and the rival. These findings are discussed in terms of prior research.

Non-Verbal Signs and Secret Communication as Universal Signs of Intercultural Communication

The article is devoted to the study of the national cognitive nature, the hidden essence of non-verbal techniques. The main purpose of the study is to prove that, based on materials on nonverbal uses in language, they are used to convey a confidential, hidden message, to show that in international communication there is a change in the semantics of nonverbal actions depending on the culture and individual knowledge of each nation. Methods In human life, in parallel with verbal communication, an experiment was conducted on nonverbal techniques, sometimes used in pure form and used to convey a secret and hidden message. The article presents an analysis based on a statistical method compared with some kinemes used in a common language. In the studies on non-verbal techniques, it is noted that they perform a main function in the relations between people. In this article, in the course of analyzing the nature of nonverbal techniques, we characterized some kinemes used depending on the worldview of the Kazakh people, and concluded that their background is directly related to the mystery. Application of the study: the results, materials, and conclusions of the work can be used in the history of language, sociolinguistics, and paralinguistic studies, and allow researchers of gesture semantics to explore nonverbal techniques from a new perspective. Scientists-linguists, psychologists have created many works related to the definition of human nature, character, and the place of nonverbal techniques in the speech act. However, it was not said that auxiliary means located outside the language will be used to transmit confidential, non-public information. In this work, we stopped at the place of nonverbal methods of transmitting confidential, hidden information and analyzed the functions in confidential communication.

Perception of students regarding sexual harassment

Background: Sexual harassment means unwanted sexually determined behavior such as physical contacts and advances, making sexual derogative comments, and sexual pressure whether by verbal or -nonverbal actions. Young female adults are most vulnerable to harassment and violence towards them. Despite of high literacy levels young female adults show tendency to tolerate harassment and delay in early identification of warning signs, leading to sexual harassment. Therefore, it becomes very important to understand how the students perceive different types of sexual harassment. Objective of the study was to assess the perception of undergraduate girl students regarding Sexual Harassment.Methods: A descriptive, cross sectional survey design on 100 female undergraduate students from the King George’s Medical University of Lucknow, India was undertaken using convenient sampling technique. The data were collected from 15th May to 15th June 2018. A self structured perception questionnaire developed by researchers was used.Results: The data gathered from the study suggests that majority (81%) of the participants responded that sexual harassment is related to all unwelcome sexual behaviour, 72% of the participants feel most afraid of sexual harassment during night, and 83% of the participants responded that the most probable place for sexual harassment is public transport. 91% of the participants believed that the incidences of the sexual harassment has increased in the last 10 years and majority (81%) of the participants didn’t had knowledge of any legal provisions regarding sexual harassment.Conclusions: The data shows that the definition of sexual harassment can differ according to individual’s perception. To understand how people perceive and define sexual harassment is crucial for explaining and understanding how they react to sexual harassment and why they often do not stand up against sexual harassment.

Is Lying Bound to Commitment? Empirically Investigating Deceptive Presuppositions, Implicatures, and Actions.

Lying is an important moral phenomenon that most people are affected by on a daily basis-be it in personal relationships, in political debates, or in the form of fake news. Nevertheless, surprisingly little is known about what actually constitutes a lie. According to the traditional definition of lying, a person lies if they explicitly express something they believe to be false. Consequently, it is often assumed that people cannot lie by more indirectly communicating believed-false claims, for instance by merely conversationally implicating them. In this paper, we subject this claim to an empirical test. In a preregistered study of 300 participants, we investigate how people judge cases of implicit deceptions that would usually be excluded by the traditional definition of lying (i.e., conversational implicatures, presuppositions, and nonverbal actions). Our results show that people do in fact consider it possible to lie by indirect means, suggesting that people have a broader concept of lying than is usually assumed. Moreover, our findings indicate that lie judgments are closely tied to the extent to which agents are perceived as having committed themselves to the believed-false claims they have communicated. We discuss the implications of our results for the traditional definition of lying and propose a new commitment-based definition of lying that can account for the findings of our experiment.

Don’t Turn a Deaf Ear: A Case for Assessing Interactive Listening

Abstract The reciprocal nature of spoken interaction means that participants constantly alternate between speaker and listener roles. However, listener or recipient actions—also known as interactive listening (IL)—are somewhat underrepresented in language tests. In conventional listening tests, they are not directly assessed. In speaking tests, they have often been overshadowed by an emphasis on production features or subsumed under broader constructs such as interactional competence. This article is an effort to represent the rich IL phenomena that can be found in peer interactive speaking assessments, where the candidate–candidate format and discussion task offer opportunities to elicit and assess IL. Taking a close look at candidate discourse and nonverbal actions through a conversation analytic approach, the analysis focuses on three IL features (i) listenership displays, (ii) contingent responses, and (iii) collaborative completions, and unpacks their relative strength in evidencing listener understanding. This article concludes by making a case for revisiting the role of IL, calling for more explicit inclusion of IL in L2 assessment as well as pedagogy.

Multimodally Enhanced Opportunities for Language Learning: Gestures Used in Word Search Sequences in ESL Tutoring

Focusing on word search sequences initiated by ESL learners in conversation tutoring, this study examines how the participants use gestures in order to facilitate language learning as well as mutual understanding. Adopting the methodological framework of Conversation Analysis, it analyzes two particular sequential contexts: (a) when a tutee uses gestures without a candidate solution to her/his word search, directly soliciting the tutor’s co-participation; and (b) when a tutee uses gestures with a candidate solution to her/his word search but there is mismatch between the candidate solution and the accompanying gesture. A fine-grained analysis of the participants’ moment-by-moment verbal and nonverbal actions reveals that gestures create multimodally enhanced opportunities for language learning by allowing the tutor to offer lexical items that are new or unfamiliar to the tutee and/or to provide corrective feedback on the lexical errors. The findings from this study offer implications for the role of gesture in L2 learning and for some of the key concepts in second language acquisition such as output, corrective feedback, and communication strategies.

Objects and materiality in pragmatic development: Here-and-now to then and-there

Abstract Language acquisition involves more than learning how to produce words in complex strings. It involves a diversity of aptitudes about how, when, with whom and in what way to use language abilities. While it is acknowledged that these skills are learned through social interaction (Blum-Kulka, S. (1997). Dinner talk: cultural patterns of sociability and socialization in family discourse. Lawrence Erlbaum Associates, Inc, Mahwah, NJ; Rogoff, B. (2003). The cultural nature of human development. Oxford University Press, Oxford), our understanding about precisely how they emerge and how they are taught and learned remains preliminary at best. Additionally, much of our understanding is strictly limited to spoken language. The analysis and arguments herein detail the consequentiality of child directed interaction strategies (CDIS) which facilitate non-verbal actions and motivate episodic retrospection, making a tangible link between the current interaction and past experiences. Through a multimodal interaction analysis (Author and Pirini, J. (2020). Multimodal (Inter)action Analysis. In McKinley, J. and Rose, H. (Eds.) The Routledge handbook of research methods in applied linguistics. Rouledge, London, pp. 488–499; Norris, S. (2004). Analyzing multimodal interaction: a methodological framework. Routledge, London. https://doi.org/10.4324/9780203379493; Norris, S. (2011). Identity in (inter)action: introducing Multimodal (Inter)action Analysis. de Gruyter Mouton, Berlin & New York. https://doi.org/10.1515/9781934078280; Norris, S. (2019). Systematically working with multimodal data: research methods in multimodal discourse analysis. Wiley Blackwell, Hoboken, NJ; Pirini, J. (2014). Introduction to Multimodal (Inter)action Analysis. In: Norris, S. and Maier, C. (Eds.). Interactions, texts and images: a reader in multimodality. Mouton de Gruyter, New York. https://doi.org/10.1515/9781614511175.77) of the practice of showing material objects during interaction, I show that non-verbal action, material culture and the physical world are crucial to developing a certain socio-cognitive pragmatic aptitude. CDIS motivating ‘showing’ of tangible objects of personal significance may be the non-verbal antecedent of selecting and introducing new topics during interaction. These CDIS defer interactional agency and motivate non-verbal communicative actions more comfortably within the zone of proximal development. Importantly, the materiality of the objects themselves are of fleeting interactional priority. Instead, the objects provide a bridge between materiality in the here-and-now to past experiences in the there-and-then. Facilitating non-verbal actions of showing help motivate explorations of episodic memory by creating a tangible and immediate link within the unfolding interaction.