Robot Interaction Research Articles

Remote-Access Marine Robotics Infrastructure and Experiments at LABUST

Marine robotics is a complex field with a potentially high demanding logistics and high deployment cost. This is a barrier to many research groups. On the other hand, some research groups have substantial equipment and infrastructure that are typically underutilized. The push for sharing infrastructures was recently accentuated due to the COVID-19 pandemic. In particular, remote access experiments became the norm during the lockdown periods. LABUST completed new infrastructure during the COVID-19 pandemic and prepared it for state-of-the-art remote access from anywhere in the globe. This is important both for research and educational purposes. This article describes the infrastructure, equipment, and methods used for implementing the remote access including the calibration procedures. It also reports on real remote access trials for different applications such as Autonomous ships, diver–robot interaction, and Remotely Operated Vehicle (ROV) pilots. These examples include both pool and at-sea remote trials, proving the capabilities of the infrastructure.

Evaluating speech-in-speech perception via a humanoid robot.

Underlying mechanisms of speech perception masked by background speakers, a common daily listening condition, are often investigated using various and lengthy psychophysical tests. The presence of a social agent, such as an interactive humanoid NAO robot, may help maintain engagement and attention. However, such robots potentially have limited sound quality or processing speed. As a first step toward the use of NAO in psychophysical testing of speech- in-speech perception, we compared normal-hearing young adults' performance when using the standard computer interface to that when using a NAO robot to introduce the test and present all corresponding stimuli. Target sentences were presented with colour and number keywords in the presence of competing masker speech at varying target-to-masker ratios. Sentences were produced by the same speaker, but voice differences between the target and masker were introduced using speech synthesis methods. To assess test performance, speech intelligibility and data collection duration were compared between the computer and NAO setups. Human-robot interaction was assessed using the Negative Attitude Toward Robot Scale (NARS) and quantification of behavioural cues (backchannels). Speech intelligibility results showed functional similarity between the computer and NAO setups. Data collection durations were longer when using NAO. NARS results showed participants had a relatively positive attitude toward "situations of interactions" with robots prior to the experiment, but otherwise showed neutral attitudes toward the "social influence" of and "emotions in interaction" with robots. The presence of more positive backchannels when using NAO suggest higher engagement with the robot in comparison to the computer. Overall, the study presents the potential of the NAO for presenting speech materials and collecting psychophysical measurements for speech-in-speech perception.

What Affects Human Decision Making in Human–Robot Collaboration?: A Scoping Review

The advent of Industry 4.0 has heralded advancements in Human–robot Collaboration (HRC), necessitating a deeper understanding of the factors influencing human decision making within this domain. This scoping review examines the breadth of research conducted on HRC, with a particular focus on identifying factors that affect human decision making during collaborative tasks and finding potential solutions to improve human decision making. We conducted a comprehensive search across databases including Scopus, IEEE Xplore and ACM Digital Library, employing a snowballing technique to ensure the inclusion of all pertinent studies, and adopting the PRISMA Extension for Scoping Reviews (PRISMA-ScR) for the reviewing process. Some of the important aspects were identified: (i) studies’ design and setting; (ii) types of human–robot interaction, types of cobots and types of tasks; (iii) factors related to human decision making; and (iv) types of user interfaces for human–robot interaction. Results indicate that cognitive workload and user interface are key in influencing decision making in HRC. Future research should consider social dynamics and psychological safety, use mixed methods for deeper insights and consider diverse cobots and tasks to expand decision-making studies. Emerging XR technologies offer the potential to enhance interaction and thus improve decision making, underscoring the need for intuitive communication and human-centred design.

HoloGCS: mixed reality-based ground control station for unmanned aerial vehicle

Human–robot interaction (HRI), which studies the interaction between robots and humans, appears as a promising research idea for the future of smart factories. In this study, HoloLens as ground control station (HoloGCS) is implemented, and its performance is discussed. HoloGCS is a mixed reality-based system for controlling and monitoring unmanned aerial vehicles (UAV). The system incorporates HRI through speech commands and video streaming, enabling UAV teleoperation. HoloGCS provides a user interface that allows operators to monitor and control the UAV easily. To demonstrate the feasibility of the proposed systems, a user case study (user testing and SUS-based questionnaire) was performed to gather qualitative results. In addition, throughput, RTT, latency, and speech accuracy were also gathered and analyzed to evaluate quantitative results.

“I See What You Feel”: An Exploratory Study to Investigate the Understanding of Robot Emotions in Deaf Children

Research in the field of human–robot interactions (HRIs) has advanced significantly in recent years. Social humanoid robots have undergone severe testing and have been implemented in a variety of settings, for example, in educational institutions, healthcare facilities, and senior care centers. Humanoid robots have also been assessed across different population groups. However, research on various children groups is still scarce, especially among deaf children. This feasibility study explores the ability of both hearing and deaf children to interact with and recognize emotions expressed by NAO, the humanoid robot, without relying on sounds or speech. Initially, the children watched three video clips portraying emotions of happiness, sadness, and anger. Depending on the experimental condition, the children observed the humanoid robot respond to the emotions in the video clips in a congruent or incongruent manner before they were asked to recall which emotion the robot exhibited. The influence of empathy on the ability to recognize emotions was also investigated. The results revealed that there was no difference in the ability to recognize emotions between the two conditions (i.e., congruent and incongruent). Indeed, NAO responding with congruent emotions to video clips did not contribute to the children recognizing the emotion in NAO. Specifically, the ability to predict emotions in the video clips and gender (females) were identified as significant predictors to identify emotions in NAO. While no significant difference was identified between hearing and deaf children, this feasibility study aims to establish a foundation for future research on this important topic.

Compliant-Control-Based Assisted Walking with Mobile Manipulator.

In this paper, a new approach involving the use of a mobile manipulator to assist humans with mobility impairments to walk is proposed. First, in order to achieve flexible interaction between humans and mobile manipulators, we propose a variable admittance controller that can adaptively regulate the virtual mass and damping parameters based on the interaction forces and the human motion intention predicted using the fuzzy theory. Moreover, a feedforward velocity compensator based on a designed state observer is proposed to decrease the inertia resistance of the manipulator, effectively enhancing the compliance of the human-robot interaction. Then, the configuration of the mobile manipulator is optimized based on a null-space approach by considering the singularity, force capacity, and deformation induced by gravity. Finally, the proposed assisted walking approach for the mobile manipulator is implemented using the human-robot interaction controller and the null-space controller. The validity of the proposed controllers and the feasibility of assisted human walking are verified by conducting a set of tests involving different human volunteers.

Integrating Egocentric and Robotic Vision for Object Identification Using Siamese Networks and Superquadric Estimations in Partial Occlusion Scenarios.

This paper introduces a novel method that enables robots to identify objects based on user gaze, tracked via eye-tracking glasses. This is achieved without prior knowledge of the objects' categories or their locations and without external markers. The method integrates a two-part system: a category-agnostic object shape and pose estimator using superquadrics and Siamese networks. The superquadrics-based component estimates the shapes and poses of all objects, while the Siamese network matches the object targeted by the user's gaze with the robot's viewpoint. Both components are effectively designed to function in scenarios with partial occlusions. A key feature of the system is the user's ability to move freely around the scenario, allowing dynamic object selection via gaze from any position. The system is capable of handling significant viewpoint differences between the user and the robot and adapts easily to new objects. In tests under partial occlusion conditions, the Siamese networks demonstrated an 85.2% accuracy in aligning the user-selected object with the robot's viewpoint. This gaze-based Human-Robot Interaction approach demonstrates its practicality and adaptability in real-world scenarios.

Construction 4.0: A comparative analysis of research and practice

This paper presents an overview of the existing literature on Construction 4.0 technologies over the past decade and their most common applications in both research and practice, aimed at achieving three objectives. First, the search for the most relevant articles on Construction 4.0, published in the scientific literature, and small firms that are developing and delivering 4.0 technologies in the construction industry allows to identify the numerous applications associated with Construction 4.0. Second, the applications found in the scientific literature and those identified in practice are classified and compared based on a framework consisting of three distinct axes. Third, the classification framework highlights current research trends and potential areas for future research, which can be summarized as follows: (i) development of hybrid digital solutions; (ii) alignment with effective collection of more structured data, smart interactive web technologies, robotics, autonomous systems, and intelligent built assets; and (iii) strengthen the capacities of artificial intelligence and machine learning.

Exo skeleton pertinence and control techniques: A state-of-the-art review

Exo-Skeleton is a wearable robotic device which was emerged in later 1960s that has a multitude of applications ranging from weightlifting to wearer’s stability improvement. This paper makes a novel approach in reviewing the various exo-skeleton models that are available at the present. The idea of this paper is to study and compare the models in terms of medical applicability like gait rehabilitation, physiotherapy, human strength augmentation, various control strategies of exo-skeleton, ergonomic study, and need for exo-skeleton system for enhancing the life of humans. Since exo-skeletons are wearable devices, it requires precise controlling of the actuators and repeatability, and accuracy plays a vital role. The paper elucidated with a detailed analysis, reviewed and summarized the core essence of 300 research papers and patents in the field of exo-skeleton with the aforesaid aspects of application and drive system with their control methodologies from 2007 to 2022 and shown a trend with year-wise data from which it is clear that the future of industrial work will become a collaborative activity involving the exo-skeleton system with human labours. Human – robot interaction is vital and must for enabling an integration which can be achieved with the amelioration in the control and actuation techniques for precise control. With the furtherance in the technology the exo-skeletons can be purpose built that can be of rigid or flexible structures with active, passive, or quasi-passive controls based on the user needs. Joint handed operation of Exo-skeleton with the alternative treatment methodologies will yield a plethora of benefits in the near future in comparison with the present conventional treatment modes that could result in reduction of surgeries.

Design and control of a compliant robotic actuator with parallel spring-damping transmission

AbstractPhysically compliant actuator brings significant benefits to robots in terms of environmental adaptability, human–robot interaction, and energy efficiency as the introduction of the inherent compliance. However, this inherent compliance also limits the force and position control performance of the actuator system due to the induced oscillations and decreased mechanical bandwidth. To solve this problem, we first investigate the dynamic effects of implementing variable physical damping into a compliant actuator. Following this, we propose a structural scheme that integrates a variable damping element in parallel to a conventional series elastic actuator. A damping regulation algorithm is then developed for the parallel spring-damping actuator (PSDA) to tune the dynamic performance of the system while remaining sufficient compliance. Experimental results show that the PSDA offers better stability and dynamic capability in the force and position control by generating appropriate damping levels.

Dynamic Motion Primitives-Based Trajectory Learning for Physical Human–Robot Interaction Force Control

One promising function of interactive robots is to provide a specific interaction force to human users. For example, rehabilitation robots are expected to promote patients' recovery by interacting with them with a prescribed force. However, motion uncertainties of different individuals, which are hard to predict due to the varying motion speed and noises during motion, degrade the performance of existing control methods. This paper proposes a method to learn a desired reference trajectory for a robot based on dynamic motion primitives (DMPs) and iterative learning (IL). By controlling the robot to follow the generated desired reference trajectory, the interaction force can achieve a desired value. In our proposed approach, DMPs are first employed to parameterize the demonstration trajectories of the human user. Then a recursive least square (RLS)-based estimator is developed and combined with the Adam optimization method to update the trajectory parameters so that the desired reference trajectory of the robot is iteratively obtained by resolving the DMPs. Since the proposed method parameterizes the trajectories depending on the phrase variable, it removes the essential assumption of traditional IL methods where the iteration period should be invariant, and thus has improved robustness compared with the existing methods. Experiments are performed using an interactive robot to validate the effectiveness of our proposed scheme.

A Variable Impedance Scheme Based on Power-Shaping Signals and Partial Knowledge of Link-Side Dynamics for Flexible-Joint Robot Interaction and Tracking Control

A Variable Impedance Scheme Based on Power-Shaping Signals and Partial Knowledge of Link-Side Dynamics for Flexible-Joint Robot Interaction and Tracking Control

Scarecrows in Oz: The Use of Large Language Models in HRI

The proliferation of Large Language Models (LLMs) presents both a critical design challenge and a remarkable opportunity for the field of Human–Robot Interaction (HRI). While the direct deployment of LLMs on interactive robots may be unsuitable for reasons of ethics, safety, and control, LLMs might nevertheless provide a promising baseline technique for many elements of HRI. Specifically, in this article, we argue for the use of LLMs as Scarecrows : “brainless,” straw-man black-box modules integrated into robot architectures for the purpose of quickly enabling full-pipeline solutions, much like the use of “Wizard of Oz” (WoZ) and other human-in-the-loop approaches. We explicitly acknowledge that these Scarecrows, rather than providing a satisfying or scientifically complete solution, incorporate a form of the wisdom of the crowd and, in at least some cases, will ultimately need to be replaced or supplemented by a robust and theoretically motivated solution. We provide examples of how Scarecrows could be used in language-capable robot architectures as useful placeholders and suggest initial reporting guidelines for authors, mirroring existing guidelines for the use and reporting of WoZ techniques.

Data-driven Communicative Behaviour Generation: A Survey

The development of data-driven behaviour generating systems has recently become the focus of considerable attention in the fields of human–agent interaction and human–robot interaction. Although rule-based approaches were dominant for years, these proved inflexible and expensive to develop. The difficulty of developing production rules, as well as the need for manual configuration to generate artificial behaviours, places a limit on how complex and diverse rule-based behaviours can be. In contrast, actual human–human interaction data collected using tracking and recording devices makes humanlike multimodal co-speech behaviour generation possible using machine learning and specifically, in recent years, deep learning. This survey provides an overview of the state of the art of deep learning-based co-speech behaviour generation models and offers an outlook for future research in this area.

Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception.

The field of multimodal robotic musical performing arts has garnered significant interest due to its innovative potential. Conventional robots face limitations in understanding emotions and artistic expression in musical performances. Therefore, this paper explores the application of multimodal robots that integrate visual and auditory perception to enhance the quality and artistic expression in music performance. Our approach involves integrating GRU (Gated Recurrent Unit) and GoogLeNet models for sentiment analysis. The GRU model processes audio data and captures the temporal dynamics of musical elements, including long-term dependencies, to extract emotional information. The GoogLeNet model excels in image processing, extracting complex visual details and aesthetic features. This synergy deepens the understanding of musical and visual elements, aiming to produce more emotionally resonant and interactive robot performances. Experimental results demonstrate the effectiveness of our approach, showing significant improvements in music performance by multimodal robots. These robots, equipped with our method, deliver high-quality, artistic performances that effectively evoke emotional engagement from the audience. Multimodal robots that merge audio-visual perception in music performance enrich the art form and offer diverse human-machine interactions. This research demonstrates the potential of multimodal robots in music performance, promoting the integration of technology and art. It opens new realms in performing arts and human-robot interactions, offering a unique and innovative experience. Our findings provide valuable insights for the development of multimodal robots in the performing arts sector.

Understanding the consequences of robotic interaction quality and outcome quality: A three-phased affordance theory-based approach

ABSTRACT Robotic interaction quality and outcome quality in restaurants are crucial in engaging customers and facilitating post-purchase behavior while few papers have examined their impact on psychological engagement and value co-creation by experienced customers. Drawing upon the affordance theory and interlinking the service-dominant logic, this study has investigated robotic interaction and outcome quality in the restaurant context using the empirical data collected from 487 respondents in China by adopting structural equation modeling to investigate the relevant impacts. The findings show that robotic interaction and outcome quality positively affect psychological engagement, whereas robotic outcome quality only positively influences value co-creation. There are positive relationships between customers’ psychological engagement, value co-creation, and intention to recommend the restaurant. We also discussed the theoretical and managerial implications.

AMHGCN: Adaptive multi-level hypergraph convolution network for human motion prediction

Human motion prediction is the key technology for many real-life applications, e.g., self-driving and human–robot interaction. The recent approaches adopt the unrestricted full-connection graph representation to capture the relationships inside the human skeleton. However, there are two issues to be solved: (i) these unrestricted full-connection graph representation methods neglect the inherent dependencies across the joints of the human body; (ii) these methods represent human motions using the features extracted from a single level and thus can neither fully exploit the various connection relationships among the human body nor guarantee the human motion prediction results to be reasonable. To tackle the above issues, we propose an adaptive multi-level hypergraph convolution network (AMHGCN), which uses the adaptive multi-level hypergraph representation to capture various dependencies among the human body. Our method has four different levels of hypergraph representations, including (i) the joint-level hypergraph representation to capture inherent kinetic dependencies in the human body, (ii) the part-level hypergraph representation to exploit the kinetic characteristics at a higher level (in comparison to the joint-level) by viewing some part of the human body as an entirety, (iii) the component-level hypergraph representation to model the semantic information, and (iv) the global-level hypergraph representation to extract long-distance dependencies in the human body. In addition, to take full advantage of the knowledge carried in the training data, we propose a reverse loss (i.e., adopting the future human poses to predict the historical poses reversely) to realize data augmentation. Extensive experiments show that our proposed AMHGCN can achieve state-of-the-art performance on three benchmarks, i.e., Human3.6M, CMU-Mocap, and 3DPW.

Three‐dimensional mapping and immersive human–robot interfacing utilize Kinect‐style depth cameras and virtual reality for agricultural mobile robots

AbstractNavigation and autonomous operation in farmland or greenhouse environments present significant challenges for agricultural mobile robots on account of unstructured scene conditions and complex task requirements. The immersive interfaces based on human–robot interaction have been extensively developed, which bridge the high‐level decision of human beings and the precise sensing and high motion ability of robots. In this study, we propose a human–robot interaction framework based on three‐dimensional mapping and virtual reality (VR) visualization. On the construction client, dense three‐dimensional point‐cloud maps are created based on the simultaneous localization and mapping real‐time appearance‐based mapping algorithm with a Kinect‐style depth camera. Subsequently, point‐cloud filtering and mesh creation methods are introduced for postprocessing the map models. A data communication network is employed to transmit the optimized models to the VR‐based exploration client. Utilizing VR visualization, the operator can gain an intuitive and comprehensive understanding of the environments to be explored. The three‐dimensional agricultural scenes are mapped into VR models through the system, which can combine the physical worlds with the VR spaces in a better manner. Compared with the video streaming‐based method for three‐dimensional mapping, the map models with colors and textures are entirely displayed in the VR headset, thus providing robust and global human–robot interaction interfaces and enhancing the immersive exploration for human beings to a considerable extent. Experimental results indicate that the three‐dimensional map models exhibit relatively sufficient construction integrity and yield favorable optimization effects. Additionally, according to a user questionnaire survey, immersive VR‐based interfaces can be more effectively utilized for the three‐dimensional mapping of mobile robots. Research evidence and results demonstrate that our proposed system framework offers positive assistance and support for enhancing human–robot interaction and executing specific robotic tasks in unstructured field agricultural environments. Code implementation and related data sets are shared at the link https://github.com/WangDongBUAA/Mapping_Interfacing.

A Mood Semantic Awareness Model for Emotional Interactive Robots.

The rapid development of natural language processing technology and improvements in computer performance in recent years have resulted in the wide-scale development and adoption of human-machine dialogue systems. In this study, the Icc_dialogue model is proposed to enhance the semantic awareness of moods for emotional interactive robots. Equipped with a voice interaction module, emotion calculation is conducted based on model responses, and rules for calculating users' degree of interest are formulated. By evaluating the degree of interest, the system can determine whether it should transition to a new topic to maintain the user's interest. This model can also address issues such as overly purposeful responses and rigid emotional expressions in generated replies. Simultaneously, this study explores topic continuation after answering a question, the construction of dialogue rounds, keyword counting, and the creation of a target text similarity matrix for each text in the dialogue dataset. The matrix is normalized, weights are assigned, and the final text score is calculated. In the text with the highest score, the content of dialogue continuation is determined by calculating a subsequent sentence with the highest similarity. This resolves the issue in which the conversational bot fails to continue dialogue on a topic after answering a question, instead waiting for the user to voluntarily provide more information, resulting in topic interruption. As described in the experimental section, both automatic and manual evaluations were conducted to validate the significant improvement in the mood semantic awareness model's performance in terms of dialogue quality and user experience.

Human Robot Interaction with Social Humanoid Robots

The widespread use of robots in social areas has revealed the necessity of having social features. Seamless integration of robots into human life should be achieved through a natural interaction. In order to establish this interaction efficiently, robot applications should be user-friendly. In the pertinent literature, it has been observed that users experienced serious difficulties in this regard, and it has not been recognized widely. In addition to this, it is seen that human-robot interaction is not efficient in several cases. In this study, the problems encountered during utilization of Pepper Robot, which is defined as the first social humanoid robot, in the process of human-robot interaction were revealed and an exemplary application was developed to resolve these problems. In this application, it was seen that it is necessary for social humanoid robots to be the initiator of the interaction in order to socialize with humans in a natural way.