Verbal Cues Research Articles

Children's reliance on pointing and mutual exclusivity in word-referent mapping: The role of vocabulary and language exposure.

This study explored monolingual and multilingual two- to five-year-olds' reliance on a non-verbal and a verbal cue during word-referent mapping, in relation to vocabulary knowledge and, for the multilinguals, Dutch language exposure. Ninety monolingual and sixty-seven multilingual children performed a referential conflict experiment that pitted a non-verbal (pointing) cue and a verbal (mutual exclusivity) cue. Mixed-effect regressions showed no main effects of vocabulary and language exposure. An interaction between vocabulary and group showed that lower vocabulary scores were associated with a stronger reliance on pointing over mutual exclusivity for multilinguals (but not monolinguals). Furthermore, an interaction between vocabulary, language exposure, and cue word (novel vs. familiar label) indicated that multilinguals with lower exposure and lower vocabulary showed a stronger reliance on pointing over mutual exclusivity when a novel rather than familiar word was used. These findings suggest that multilingual and monolingual children go through different trajectories when learning to map words to referents.

Cross- and multimodal anaphoric references in mystery movies: A cognitive perspective

Abstract The article presents findings from both quantitative and qualitative research regarding the resolution of cross- and multi-modal anaphoric references to objects in two Sherlock Holmes movies directed by Guy Ritchie. The goal is to illustrate how words referring to objects are linked to images depicting the objects as well as sounds produced by the objects to constitute coreferential chains with respect to the viewers’ perception. The theoretical assumptions draw concepts primarily from cognitive linguistics (such as mental spaces, compression, frame-metonymic relations), with additional notions adopted from (verbal) semantics, pragmatics, visual semantics, and film studies. A cognitive account of regular and associative cross- and multimodal anaphora is suggested, along with a model for the representation of cross- and multi-modal coreferential chains for future manual and automated annotation tools. Overall, the work emphasizes the importance of considering both verbal and nonverbal cues in understanding anaphoric references in cinematic contexts by highlighting the cognitive processes involved while viewers integrate information from different modes.

Sign Language to Speech Conversion

Human communication relies heavily on verbal and non-verbal cues, with sign language serving as a crucial method of interaction for individuals with hearing impairments. However, the communication barrier between sign language users and those unfamiliar with it remains a significant challenge. This paper presents an innovative approach to bridging this gap through an Arduino-based system that converts sign language gestures directly into speech without the use of a camera. By utilizing flex sensors and an accelerometer, this system detects hand gestures and movements, processes this data, and produces corresponding audio output. This approach offers a more accessible and portable solution compared to traditional camera-based systems, potentially revolutionizing real-time communication for the deaf community. The experimental results demonstrate a gesture recognition accuracy of 92% and a response time of under 500 milliseconds, indicating the system's viability for practical applications

Accomplishing lesson ending: bringing lesson to an end

ABSTRACT This ethnomethodological study examines the dynamics of lesson closure in diverse classroom settings, drawing on the sequential analysis of five video-recorded lessons in history, literature, chemistry, and geography classes. The detailed analysis of teacher-student interactions during the lesson endings in a tenth-grade high school classroom in Ankara sheds light on the temporal, social, and interactional processes involved in concluding lessons. The ringing of the bell emerges as a key temporal marker signalling the end of the lesson, while teachers play a central role in orchestrating closure through verbal and non-verbal cues. The interactional dynamics between teachers and students also reveal the negotiation of roles and responsibilities, with students demonstrating orientation to the lesson closure through preparatory actions. Furthermore, the structured nature of lesson closure is evident in sequential patterns and the deployment of terminal components. The study highlights the dynamic nature of classroom interactions during lesson endings, underscoring the formal interactional dynamics at play. Overall, the findings contribute to the understanding of lesson closure processes and offer insights for educators, researchers, and policymakers aiming to enhance teaching and learning experiences in diverse educational contexts.

Effect of integrated manual and verbal cueing on functional transfers in chronic stroke survivors: a randomized controlled study

Purpose The profession of physical therapy has historically relied on manual facilitation to improve motor control strategies and performance in persons rehabilitating from a stroke, yet there is insufficient evidence to support its use during functional task training. The purpose of this study was to determine the effects of integrated cueing (verbal and manual) and verbal cueing approaches during sit-to-stand training on midline alignment & muscle activation in chronic stroke survivors. Methods Twenty-one chronic right-brained stroke survivors with hemiplegia were randomly assigned to the Integrated Cueing or Verbal Only group and outcome measures were recorded using an 18-Camera Motion Capture System, force plates, and surface electromyography (EMG). Results Both groups demonstrated a significant improvement in symmetry toward the midline after thirty training repetitions. Significant improvements in muscle activation were found in two muscle groups on the affected side of the body in the Integrated Cueing group, gastrocnemius and rectus femoris. Conclusion Both the Verbal Only and Integrated Cueing groups made significant progress toward more symmetrical movement, yet more significant changes in the activation of hemiparetic extensor muscles were seen in the Integrated Cueing group. These findings support the use of manual cueing in movement activation and performance during the training of functional tasks. IMPLICATIONS FOR REHAB Physical therapists commonly use manual tactile cues to facilitate movement performance but there is currently insufficient evidence to support this use with stroke survivors. In a relatively small sample size, both verbal cues and integrated cues (verbal plus manual) improved symmetry during sit to stand in chronic right brain stroke survivors. Integrated cueing enhanced motor activation and performance during sit to stand better than verbal cues alone. Manual tactile cueing should be considered during functional task retraining.

Group Rapport Building in an Institutionaized Face-To-Face Interaction: a Case Study

When people are engaged in any interaction, they are likely to respond to each other's verbal and nonverbal cues. This responsiveness is a critical factor in the notion of rapport. In this paper, the author argues that the same human dynamics that govern everyday conversations are at play between participants in institutionalized conversations such as classroom discourse. The purpose of this study is to develop critical aspects of "conversational involvement" (Duranti, 1986; Goodwin, 1986; Gumperz, 1982; Tannen, 1985) as rapport-building devices and to provide observable phenomena that characterize rapport with participants' verbal and nonverbal cues in informal and formal conversation. In this study, nine people participated: a Korean professor of an elementary Korean language class at a major US university and eight undergraduate students whose first language is English. Three fifty-minute classroom sessions were videotaped, and segments that illustrated the teacher's and students' behaviors that facilitated rapport were transcribed and qualitatively analyzed. The author recorded the frequency of the teacher's verbal and nonverbal cues that appeared to foster rapport. This study found that the teacher shared small talk in informal before-class conversations by bringing up insignificant details about daily life. The teacher played the role of teacher during class by directing a lesson. However, she continued to utilize various rapport-building devices through selected verbal and nonverbal cues to engage her students in the conversation. In this classroom, as in everyday conversation, the participants demonstrated their desire to listen to, be responsive to, and be influenced by individuals in a certain way. The participants were part and parcel of achieving their goals in the classroom. The participants created a certain alignment to achieve their goals and signal that they were involved in the interaction. Then, the students would be able to learn the target language more through the involvement of the classroom activity, which is one of the outcomes of good rapport.

If instruments could talk: Trace-based communication and its implications in the curriculum for trainee-supervisor interactions

Contrary to common belief, effective communication is not only achieved via verbal exchanges of information. It requires a thoughtful combination of verbal and non-verbal cues since, in some circumstances, verbal communication in healthcare may be impossible, unfavourable or ineffective. The semiotics literature suggests that effective non-verbal communication goes beyond body language and gestures. It also includes the use of objects (i.e. traces) to transmit a message. We call this trace-based communication (TBC). In this perspective paper, we offer some reflections from our experience researching and using trace-based communication for training purposes. Our intention was to bring further awareness to the opportunities that trace-based communication might enable, including opportunities to enhance trainee-supervisor interactions, and consequently inform curriculum design.

Sentiment Analysis and Facial Expression Recognition in Customer Service Interactions

In the evolving landscape of digital customer service, the need for advanced methods to accurately understand and respond to customer emotions has become critical. Traditional systems often rely solely on textual data, missing non-verbal cues that significantly contribute to the customer's emotional state. This study proposes a combined approach integrating Facial Expression Recognition (FER) and Natural Language Processing (NLP) to enhance emotion detection accuracy in customer service interactions. The FER component employs Convolutional Neural Networks (CNNs) to analyze facial expressions, while the NLP component uses Long Short-Term Memory (LSTM) networks to process textual data. This multimodal system aims to provide a comprehensive understanding of customer emotions by capturing both verbal and non-verbal cues. Experiments demonstrate that the integrated FER and NLP model significantly outperforms standalone models, achieving an accuracy of 92.3%, compared to 85.2% for FER-only and 87.4% for NLP-only models. The results highlight the benefits of a multimodal approach, showing substantial improvements in both training and validation performance. This study also compares the proposed model with other state-of-the-art models such as the Deep Learning Assisted Semantic Text Analysis (DLSTA) and Multimodal Emotion Recognition using Deep Belief Networks (DBN). While DLSTA achieves higher accuracy in text-based emotion detection, and DBNs provide robust emotion classification by integrating various modalities, our model effectively balances the strengths of both visual and textual data. The findings suggest that integrating FER and NLP can significantly enhance the quality of customer service by enabling more empathetic and effective interactions. Future work will focus on optimizing computational efficiency, addressing data variability, and ensuring adaptability across diverse customer service scenarios.

From unimodal to multimodal dynamics of verbal and nonverbal cues during unstructured conversation.

Conversations encompass continuous exchanges of verbal and nonverbal information. Previous research has demonstrated that gestures dynamically entrain each other and that speakers tend to align their vocal properties. While gesture and speech are known to synchronize at the intrapersonal level, few studies have investigated the multimodal dynamics of gesture/speech between individuals. The present study aims to extend our comprehension of unimodal dynamics of speech and gesture to multimodal speech/gesture dynamics. We used an online dataset of 14 dyads engaged in unstructured conversation. Speech and gesture synchronization was measured with cross-wavelets at different timescales. Results supported previous research on intrapersonal speech/gesture coordination, finding synchronization at all timescales of the conversation. Extending the literature, we also found interpersonal synchronization between speech and gesture. Given that the unimodal and multimodal synchronization occurred at similar timescales, we suggest that synchronization likely depends on the vocal channel, particularly on the turn-taking dynamics of the conversation.

Arabic within culture forensic interviews: Arabic native speaking lay-observer truth and lie accuracy, confidence, and verbal cue selection.

Cross cultural differences in behavioral and verbal norms and expectations can undermine credibility, often triggering a lie bias which can result in false convictions. However, current understanding is heavily North American and Western European centric, hence how individuals from non-western cultures infer veracity is not well understood. We report novel research investigating native Arabic speakers' truth and lie judgments having observed a matched native language forensic interview with a mock person of interest. 217 observers viewed a truthful or a deceptive interview and were directed to attend to detailedness as a veracity cue or given no direction. Overall, a truth bias (66% accuracy) emerged, but observers were more accurate (79%) in the truth condition with the truthful interviewee rated as more plausible and more believable than the deceptive interviewee. However, observer accuracy dropped to just 23% when instructed to use the detailedness cue when judging veracity. Verbal veracity cues attended too were constant across veracity conditions with 'corrections' emerging as an important veracity cue. Some results deviate from the findings of research with English speaking western participants in cross- and matched-culture forensic interview contexts, but others are constant. Nonetheless, this research raises questions for research to practice in forensic contexts centred on the robustness of western centric psychological understanding for non-western within culture interviews centred on interview protocols for amplifying veracity cues and the instruction to note detailedness of verbal accounts which significantly hindered Arabic speaker's performance. Findings again highlight the challenges of pancultural assumptions for real-world practices.

Personality Expression using Co-speech Gesture

We express our personality through verbal and nonverbal behavior. While verbal cues are mostly related to the semantics of what we say, nonverbal cues include our posture, gestures, and facial expressions. Appropriate expression of these behavioral elements improves conversational virtual agents’ communication capabilities and realism. Although previous studies focus on co-speech gesture generation, they do not consider the personality aspect of the synthesized animations. We show that automatically generated co-speech gestures naturally express personality traits, and heuristics-based adjustments for such animations can further improve personality expression. To this end, we present a framework for enhancing co-speech gestures with the different personalities of the Five-Factor model. Our experiments suggest that users perceive increased realism and improved personality expression when combining heuristics-based motion adjustments with co-speech gestures.

AI-Enhanced Cross-Cultural Competence in STEM Education

Cross-cultural competence (CCC) is leaders' most critical skill set in science, technology, engineering, and math (STEM) education because it can help them effectively manage cultural diversity. The research provides insights into the potential of AI to foster more inclusive and culturally aware educational environments for STEM students using a theory that combines Constructivism and Cultural-Historical Activity Theory (CHAT). The study discusses two research questions: (1) How can AI-driven tools be used in STEM education to enhance cross-cultural sensitivity and communication? (2) What is the impact of AI-facilitated reflective practices on students' self-awareness of cultural biases? This research shows how AI-based language translators and cultural simulations enable learners to adapt to different cultures. According to Constructivism, learning occurs when learners actively construct their knowledge from experiences towards cultural sensitivity, understanding, and respecting other cultures while reducing stereotypes. CHAT, on its part, views AI as aligning with AI as one of the artifacts or tools that are at the disposal of human beings in order to facilitate social interaction; thus, it has the potential to enhance intercultural communication where verbal cues are not only linguistic but also non-verbal. A total of 228 respondents were reached through surveys, resulting in preliminary results showing improved CCC due to accommodative devices such as culturally-specific software simulations or language translators employing AI models. This study adds value to the existing literature by presenting empirical evidence of how AI can foster CCC in STEM education. It highlights the imperative of AI's theoretical and practical applications in educational curricula.

Multimodal Sarcasm Detection: A Comprehensive Review

Sarcasm, a form of verbal irony, involves using remarks that intentionally convey the opposite of their literal meaning. It serves to criticize or humorously undermine a situation. Verbal and non-verbal cues—such as changes in tone, incongruence across modalities, and word emphasis—often convey sarcasm. As technology advances, more people express their opinions online, necessitating the development of efficient models for detecting speech nature. Machine Learning algorithms, Linguistic Models, Ensemble Learning, and Multi-modal approaches play crucial roles in this endeavor. While most research has focused on unimodal approaches using textual data, the field now embraces multimodal sarcasm detection. This approach integrates diverse data types, including images, text, and audio, to enhance accuracy and automate sarcasm classification. In our review paper, we delve into the captivating world of multimodal sarcasm detection, tracing its evolution from early unimodal methods to the current state of multimodal techniques. We aim to provide a comprehensive overview, shedding light on challenges, methodologies, and prospects. Keywords: Sarcasm Detection, Multimodal Approaches, Machine Learning Algorithms, Linguistic Models, Ensemble Learning

Exploring Levels of Interspecies Interaction: Expectations, Knowledge, and Empathy in Human-Dog Relationships.

This exploratory study examines the complex dynamics of human-dog relationships and their impact on interspecies communication. Twelve human-dog dyads were studied using narrative interviews to explore how people perceive their relationships with their dogs. In addition, the dyads engaged in a cooperative task to observe interaction dynamics during everyday activities. This study shows that individual expectations frame interactions and that traditional notions of dog ownership are evolving into more family-like relationships. Effective communication relies on a nuanced mix of verbal and non-verbal cues, with empathy emerging as a fundamental element guiding these interactions. Our findings underline the profound influence of human expectations, knowledge and empathy on communication with dogs. They also highlight the critical role of compatibility between human and dog dyads, and emphasize that such compatibility is a key determinant of satisfaction in interspecies relationships. These findings contribute to a deeper understanding of how human factors modulate communication and satisfaction in human-animal interactions.

Presenting manuscript tables and diagrams to the Middle English reader

Abstract In this paper, we examine information-organising graphic devices such as tables and diagrams in Middle English manuscripts. Our focus is on text producers’ metadiscourse describing these devices and instructing the reader in their use. We also pay attention to the visual and spatial relationship between the graphic device and the surrounding text. Our findings indicate that the metadiscourse associated with graphic devices serves similar functions to image captions (verbal cueing) in modern textbooks: identifying and describing the devices and instructing the reader in using them.

You talkin’ to me? Functional breed selection may have fundamentally influenced dogs’ sensitivity to human verbal communicative cues

BackgroundThe ability to learn from humans via observation was considered to be equally present across properly socialized dogs. We showed recently that cooperative working breeds learned from a human demonstrator more effectively. We hypothesized that functional breed selection could affect sensitivity to human attention-eliciting behavior. Accordingly, we ran the first ever study on dogs that compared the effect of ostensive and neutral verbal communication in a social learning scenario. We used the detour paradigm around a transparent V-shaped fence with either ostensive (addressing the receiver both with words and specific, attention-eliciting prosody) or neutral speech (monotonous reciting of a short poem) demonstration. The other features (gestures, movement) of the demonstration sequence were kept identical between the two conditions. We tested (N = 70) companion dogs from 17 cooperative and 16 independent breeds in three 1-min trials. Subjects had to obtain the reward by detouring around the fence.ResultsDetour latencies of the cooperative dogs improved after both ostensive and neutral speech demonstrations. The independent dogs did not improve their detour latency in either of the conditions. Remarkably, ostensive verbal utterances elicited longer relative looking time towards the demonstrator, cooperative dogs looked longer at the demonstrator, and longer looking time resulted in more successful detours.ConclusionsOur study provides the first indication that functional breed selection had a significant impact on dogs’ sensitivity to ostensive human communication, which, apart from being crucially important for social learning from humans, until now was considered as a uniformly present heritage of domestication in dogs.Graphical

Investigating the role of verbal cues on learning of tool-use actions in 18- and 24-month-olds in an online looking time experiment

Investigating the role of verbal cues on learning of tool-use actions in 18- and 24-month-olds in an online looking time experiment

Contributions of Speaking, Listening, and Semantic Depth to Word Learning in Typical 3- and 4-Year-Olds.

This study examined learning via perception, learning via production, and semantic depth as contributors to word learning in preschool-aged children. There is broad evidence that semantic depth is an important contributor to word learning, especially when semantic cues are repeated and spaced out over time. Perceptual learning and production learning each support word learning sometimes, but not in all cases. The purpose of this study was to examine all three learning mechanisms within a single experimental paradigm. Thirty-six typically developing preschool children completed the experiment. They were familiarized with 16 novel words that were contextualized as alien names. These aliens came in four sets, each set comprising one base alien and three modified aliens marked by suffixes. Children completed four familiarizations: two in which they simply listened to the alien names (perceptual learning) and two where they repeated the alien names (production learning). Those conditions were crossed with a semantic depth manipulation (aliens with and without verbal semantic cues). Following each familiarization, referent identification and confrontation naming tasks were completed to assess learning. Children were able to identify more alien referents following familiarizations with semantic depth. There were no significant effects of either perceptual learning or production learning. This study confirms and expands on the benefits of semantic depth, but the results are unclear about the relative importance of perception and production to word learning. Nevertheless, the study suggests benefits to simultaneously studying multiple factors related to word learning.

Multimodal Affective Communication Analysis: Fusing Speech Emotion and Text Sentiment Using Machine Learning

Affective communication, encompassing verbal and non-verbal cues, is crucial for understanding human interactions. This study introduces a novel framework for enhancing emotional understanding by fusing speech emotion recognition (SER) and sentiment analysis (SA). We leverage diverse features and both classical and deep learning models, including Gaussian naive Bayes (GNB), support vector machines (SVMs), random forests (RFs), multilayer perceptron (MLP), and a 1D convolutional neural network (1D-CNN), to accurately discern and categorize emotions in speech. We further extract text sentiment from speech-to-text conversion, analyzing it using pre-trained models like bidirectional encoder representations from transformers (BERT), generative pre-trained transformer 2 (GPT-2), and logistic regression (LR). To improve individual model performance for both SER and SA, we employ an extended dynamic Bayesian mixture model (DBMM) ensemble classifier. Our most significant contribution is the development of a novel two-layered DBMM (2L-DBMM) for multimodal fusion. This model effectively integrates speech emotion and text sentiment, enabling the classification of more nuanced, second-level emotional states. Evaluating our framework on the EmoUERJ (Portuguese) and ESD (English) datasets, the extended DBMM achieves accuracy rates of 96% and 98% for SER, 85% and 95% for SA, and 96% and 98% for combined emotion classification using the 2L-DBMM, respectively. Our findings demonstrate the superior performance of the extended DBMM for individual modalities compared to individual classifiers and the 2L-DBMM for merging different modalities, highlighting the value of ensemble methods and multimodal fusion in affective communication analysis. The results underscore the potential of our approach in enhancing emotional understanding with broad applications in fields like mental health assessment, human–robot interaction, and cross-cultural communication.

Words matter: Effects of instructional cues on pressure pain threshold values in healthy people

BackgroundPressure pain threshold (PPT) measurements require standardised verbal instructional cues to ensure that the increasing pressure is stopped at the correct time consistently. This study aimed to compare how PPT values and their test-retest reliability were affected by different instructional cues. MethodsAt two separate sessions, two PPT measurements were taken at the anterior knee for each of four different instructional cues: the cue of the German Neuropathic Research Network instructions (‘DFNS’), the point where pressure first feels uncomfortable (‘Uncomfortable’), 3/10 on the numerical pain rating scale (‘3NPRS’), and where pain relates to an image from the pictorial-enhanced NPRS scale (‘Pictorial’). Linear mixed modeling was used to quantify differences between pairs of instructional cues. Test-retest reliability was estimated using intraclass correlation coefficients (ICC[2,1] and ICC[2,k]). ResultsTwenty participants were recruited. The cue resulting in greatest PPT value was DFNS (394.32 kPa, 95%CI [286.32 to 543.06]), followed by Pictorial (342.49 kPa, 95%CI [248.68 to 471.68]), then Uncomfortable (311.85 kPa, 95%CI [226.43 to 429.48]), and lastly 3NPRS (289.78 kPa, 95%CI [210.41 to 399.09]). Five of six pairwise contrasts were statistically significant. Regardless of the cues, the point estimates of ICC (2,1) ranged from 0.80 to 0.86, and the ICC (2,k) values ranged from 0.89 to 0.93. No statistically significant differences were found between any pairwise contrasts of reliability indices. ConclusionWords matter when instructing people when to stop testing in pressure algometry. Clinicians should use the same instructional cue when assessing pain thresholds to ensure reliability.