Human-robot Interaction Performance Research Articles

Effective Acoustic Model-Based Beamforming Training for Static and Dynamic Hri Applications.

Human-robot collaboration will play an important role in the fourth industrial revolution in applications related to hostile environments, mining, industry, forestry, education, natural disaster and defense. Effective collaboration requires robots to understand human intentions and tasks, which involves advanced user profiling. Voice-based communication, rich in complex information, is key to this. Beamforming, a technology that enhances speech signals, can help robots extract semantic, emotional, or health-related information from speech. This paper describes the implementation of a system that provides substantially improved signal-to-noise ratio (SNR) and speech recognition accuracy to a moving robotic platform for use in human-robot interaction (HRI) applications in static and dynamic contexts. This study focuses on training deep learning-based beamformers using acoustic model-based multi-style training with measured room impulse responses (RIRs). The results show that this approach outperforms training with simulated RIRs or matched measured RIRs, especially in dynamic conditions involving robot motion. The findings suggest that training with a broad range of measured RIRs is sufficient for effective HRI in various environments, making additional data recording or augmentation unnecessary. This research demonstrates that deep learning-based beamforming can significantly improve HRI performance, particularly in challenging acoustic environments, surpassing traditional beamforming methods.

Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions

There is an increasingly urgent need for humans to interactively control robotic systems to perform increasingly precise remote operations, concomitant with the rapid development of space exploration, deep-sea discovery, nuclear rehabilitation and management, and robotic-assisted medical devices. The potential high value of medical telerobotic applications was also evident during the recent coronavirus pandemic and will grow in future. Robotic teleoperation satisfies the demands of the scenarios in which human access carries measurable risk, but human intelligence is required. An effective teleoperation system not only enables intuitive human-robot interaction (HRI) but ensures the robot can also be operated in a way that allows the operator to experience the “feel” of the robot working on the remote side, gaining a “sense of presence”. Extended reality (XR) technology integrates real-world information with computer-generated graphics and has the potential to enhance the effectiveness and performance of HRI by providing depth perception and enabling judgment and decision making while operating the robot in a dynamic environment. This review examines novel approaches to the development and evaluation of an XR-enhanced telerobotic platform for intuitive remote teleoperation applications in dangerous and difficult working conditions. It presents a strong review of XR-enhanced telerobotics for remote robotic applications; a particular focus of the review includes the use of integrated 2D/3D mixed reality with haptic interfaces to perform intuitive remote operations to remove humans from dangerous conditions. This review also covers primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can better control or interact with real robotic platforms using these devices and systems to extend the user’s reality and provide a more intuitive interface. The objective of this article is to present recent, relevant, common, and accessible frameworks implemented in research articles published on XR-enhanced telerobotics for industrial applications. Finally, we present and classify the application context of the reviewed articles in two groups: mixed reality–enhanced robotic telemanipulation and mixed reality–enhanced robotic tele-welding. The review thus addresses all elements in the state of the art for these systems and ends with recommended research areas and targets. The application range of these systems and the resulting recommendations is readily extensible to other application areas, such as remote robotic surgery in telemedicine, where surgeons are scarce and need is high, and other potentially high-risk/high-need scenarios.

Inspiring Real-Time Evaluation and Optimization of Human–Robot Interaction with Psychological Findings from Human–Human Interaction

The increasingly central role of robotic agents in daily life requires effective human–robot interaction (HRI). For roboticists to optimize interaction design, it is crucial to understand the potential effects of robotic agents on human performance. Yet a systematic specification of contributing factors is lacking, and objective measures of HRI performance are still limited. In these regards, the findings of research on human–human interaction can provide valuable insights. In this review, we break down the complex effects of robotic agents on interacting humans into some basic building blocks based on human–human interaction findings, i.e., the potential effects of physical presence, motor actions, and task co-representation in HRI. For each effect, we advise on future directions regarding its implication. Furthermore, we propose that the neural correlates of these effects could support real-time evaluation and optimization of HRI with electroencephalograph (EEG)-based brain–computer interface (BCI).

DMS-SK/BLSTM-CTC Hybrid Network for Gesture/Speech Fusion and Its Application in Lunar Robot–Astronauts Interaction

In the future manned lunar exploration mission, astronauts would work with the lunar robots, which has a high requirement for human–robot interaction (HRI). As the accuracy of gesture recognition interaction does not fulfill the requirement for human–robot joint exploration missions, we propose the DMS-SK/BLSTM-CTC hybrid network to improve the performance of HRI. For gesture recognition, considering VGG-SK has low accuracy and complex architecture, we delete the fourth convolution module, optimize the last global pooling layer, introduce dilated convolution block and multiscale convolution block in VGG-SK, and get the DMS-SK-based gesture recognition sub-network. Compared with the traditional recognition methods, the accuracy and performance of DMS-SK improve. For speech recognition, considering that Bidirectional long–short-term memory unit (BLSTM) has the advantages of processing temporal information, and the Connectionist Temporal Classification (CTC) algorithm can simplify speech data preprocessing, we use BLSTM based on CTC as the speech recognition sub-network. Finally, we combine DMS-SK with BLSTM-CTC, and propose the DMS-SK/BLSTM-CTC hybrid network as the gesture/speech hybrid network. In addition, we use 10 gestures in the American Sign Language (ASL) dataset and 10 speech commands to construct the gesture/speech hybrid dataset. Experimental results show that compared with the pure gesture or pure speech networks, the recognition accuracy of the gesture-speech hybrid network improves by 2% and 12%, respectively, its accuracy reaches 97.38%, which fulfills the requirement of astronauts for HRI.

Optimization of criterion for objective evaluation of HRI performance that approximates subjective evaluation: a case study in robot competition

The standard approach to evaluate the performance of human-robot interaction (HRI) is subjective evaluation, for example, using questionnaires. Because such subjective evaluation is time-consuming, an alternative evaluation method based on only objective factors (i.e. human reaction behavior) is required for autonomous learning by robots and for scoring in robot competitions, etc. To this end, we aimed to investigate the extent to which subjective evaluation results can be approximated using objective factors. We observed and stored HRI history data through a robot-competition task in which the robot was required to generate comprehensible and unambiguous natural language expressions and gestures to guide inexpert users in virtual everyday environments. In addition, to acquire subjective evaluation results, we asked third-parties to evaluate the HRI performance by reviewing the stored interaction histories. From the results of a case study of robot-competition, we demonstrate the necessity for an objective method for the evaluation of HRI performance and a determination process for an effective evaluation criterion. The results of a multiple linear regression analysis to estimate the subjective evaluation results reveal that the subjective evaluation of HRI can indeed be reasonably approximated base on objective factors.