Approach For Human-robot Interaction Research Articles

Integrating Egocentric and Robotic Vision for Object Identification Using Siamese Networks and Superquadric Estimations in Partial Occlusion Scenarios.

This paper introduces a novel method that enables robots to identify objects based on user gaze, tracked via eye-tracking glasses. This is achieved without prior knowledge of the objects' categories or their locations and without external markers. The method integrates a two-part system: a category-agnostic object shape and pose estimator using superquadrics and Siamese networks. The superquadrics-based component estimates the shapes and poses of all objects, while the Siamese network matches the object targeted by the user's gaze with the robot's viewpoint. Both components are effectively designed to function in scenarios with partial occlusions. A key feature of the system is the user's ability to move freely around the scenario, allowing dynamic object selection via gaze from any position. The system is capable of handling significant viewpoint differences between the user and the robot and adapts easily to new objects. In tests under partial occlusion conditions, the Siamese networks demonstrated an 85.2% accuracy in aligning the user-selected object with the robot's viewpoint. This gaze-based Human-Robot Interaction approach demonstrates its practicality and adaptability in real-world scenarios.

A Robotic Teleoperation System Enhanced by Augmented Reality for Natural Human–Robot Interaction

Telekinesis, as commonly portrayed in science fiction literature and cinema, is a super power wherein users control and manipulate objects absent in physical interaction. In real world, enhancing human–robot interaction needs the synthesis of human intuitive processes with robotic arm. This paper introduces a robotic teleoperation system achieving the essence of telekinetic operations, combining the profound capabilities of augmented reality (AR) with the robotic arm operations. Utilizing AR, the proposed methodology offers operators with a visual feedback, facilitating a level of control surpassing the capacities of natural interfaces. By using AR-driven visual recognition, this system achieves operations in a virtual environment, subsequently actualized in the real world through the robotic arm. Through multiple experiments, we found that the system has a small margin of error in telekinesis operations, meeting the needs of remote operation. Furthermore, our system can operate on objects in the real world. These experiments underscore the capability of the remote control system to assist humans in accomplishing a wider range of tasks through the integration of AR and robotic arms, providing a natural human–robot interaction approach.

Proposal and Evaluation of a Robot to Improve the Cognitive Abilities of Novice Baseball Spectators Using a Method for Selecting Utterances Based on the Game Situation

Herein, an engineering method is developed to improve spectators’ sports-watching abilities. We present an interactive robot that supports the cognitive abilities of baseball novices in sports spectating. The robot watches a game with a person and utters words applicable to the game situation. We propose two methods to support cognitive watching: five categories of utterances (confirmation, information, prediction, emotion, and evaluation) and utterance rules for player participation and game scenes. We also propose a method for generating utterances for each category. To evaluate the generated utterances, we conducted an experiment whereby spectators watched baseball footage with the robot. The results of the experiment showed that the robot’s utterances could support the cognitive ability sub-factor of individual game intelligence, specifically “Spectating while knowing the player’s strengths.” In addition, the feeling of heaviness that occurs when watching alone could potentially be reduced by watching with a robot. This study is the first to attempt to support and improve spectators’ abilities to watch sports using the human–robot interaction approach. We anticipate that this approach will be used in the future to build a relationship (increase intimacy) with an agent (robot) and to support and improve cognitive abilities regardless of the type of sport.

An Integrated in Situ Image Acquisition and Annotation Scheme for Instance Segmentation Models in Open Scenes With a Human–Robot Interaction Approach

An Integrated in Situ Image Acquisition and Annotation Scheme for Instance Segmentation Models in Open Scenes With a Human–Robot Interaction Approach

A novel multifunctional intelligent bed integrated with multimodal human–robot interaction approach and safe nursing methods

AbstractThe authors propose a multifunctional intelligent bed (MIB) that integrates multiple modes of interaction to improve the welfare of mobility‐impaired users and reduce the workload of medical personnel. The MIB features independent autonomous omnidirectional movement, position adjustment, multi‐degree‐of‐freedom (DOF) movement regulation and posture memory functions to facilitate comfortable and convenient interaction for mobility‐impaired users. In particular, an integrated “MIB‐state perception‐interaction interfaces” system is established, and a bed fall risk detection algorithm and assisted get‐up‐transfer algorithm is proposed. By recognising and sharing human body state characteristics, nursing collaboration can be achieved with caregivers or other nursing robots. Comprehensive experiments demonstrate that the MIB is a novel MIB that is highly adaptable to the environment, convenient to interact with and safe. By integrating the proposed algorithms, daily safety monitoring, assisted get‐up and defecation tasks can be effectively accomplished. This technology demonstrates excellent applicability and promising prospects for implementation in hospitals, nursing centres and homes catering to elderly and disabled individuals with mobility impairments.

Recent advancements in multimodal human-robot interaction.

Robotics have advanced significantly over the years, and human-robot interaction (HRI) is now playing an important role in delivering the best user experience, cutting down on laborious tasks, and raising public acceptance of robots. New HRI approaches are necessary to promote the evolution of robots, with a more natural and flexible interaction manner clearly the most crucial. As a newly emerging approach to HRI, multimodal HRI is a method for individuals to communicate with a robot using various modalities, including voice, image, text, eye movement, and touch, as well as bio-signals like EEG and ECG. It is a broad field closely related to cognitive science, ergonomics, multimedia technology, and virtual reality, with numerous applications springing up each year. However, little research has been done to summarize the current development and future trend of HRI. To this end, this paper systematically reviews the state of the art of multimodal HRI on its applications by summing up the latest research articles relevant to this field. Moreover, the research development in terms of the input signal and the output signal is also covered in this manuscript.

An AR-assisted Deep Reinforcement Learning-based approach towards mutual-cognitive safe human-robot interaction

With the emergence of Industry 5.0, the human-centric manufacturing paradigm requires manufacturing equipment (robots, etc.) interactively assist human workers to deal with dynamic and complex production tasks. To achieve symbiotic human–robot interaction (HRI), the safety issue serves as a prerequisite foundation. Regarding the growing individualized demand of manufacturing tasks, the conventional rule-based safe HRI measures could not well address the safety requirements due to inflexibility and lacking synergy. To fill the gap, this work proposes a mutual-cognitive safe HRI approach including worker visual augmentation, robot velocity control, Digital Twin-enabled motion preview and collision detection , and Deep Reinforcement Learning-based robot collision avoidance motion planning in the Augmented Reality-assisted manner. Finally, the feasibility of the system design and the performance of the proposed approach are validated by establishing and executing the prototype HRI system in a practical scene. • Introduced an AR-assisted system architecture for mutual-cognitive safe HRI. • Implemented distance-based robot velocity control and area-based workers’ visual aids functions. • Developed the robot DT-enabled motion preview for workers to enhance safe cognition and for robots to detect collision. • Proposed a curriculum learning-based DRL motion planning policy for collision avoidance.

Human-Robot Interaction with Sliding Mode Control for Rehabilitation

The paper presents a Human-Robot Interaction approach together with a sliding mode position controller for robot-assisted arm rehabilitation. The controller is designed such that the chattering effect is low, while the tracking errors are kept in an acceptable range for rehabilitation applications. The Human-Robot Interaction logic is implemented outside the position control loop, and takes into account the human's effort and intention of movement. The human's contribution to the movement can be tuned iteratively, and the robot guides the movement by making it harder to move in a wrong direction, and easier in the direction towards the rehabilitation trajectory. Thus, our approach increases the percentage of time spent on practicing correct rehabilitation movements. Finally, experimental results illustrate its effectiveness.

Perspective-Corrected Spatial Referring Expression Generation for Human–Robot Interaction

Intelligent robots designed to interact with humans in real scenarios need to be able to refer to entities actively by natural language. In spatial referring expression generation, the ambiguity is unavoidable due to the diversity of reference frames, which will lead to an understanding gap between humans and robots. To narrow this gap, in this paper, we propose a novel perspective-corrected spatial referring expression generation (PcSREG) approach for human-robot interaction by considering the selection of reference frames. The task of referring expression generation is simplified into the process of generating diverse spatial relation units. First, we pick out all landmarks in these spatial relation units according to the entropy of preference and allow its updating through a stack model. Then all possible referring expressions are generated according to different reference frame strategies. Finally, we evaluate every expression using a probabilistic referring expression resolution model and find the best expression that satisfies both of the appropriateness and effectiveness. We implement the proposed approach on a robot system and empirical experiments show that our approach can generate more effective spatial referring expressions for practical applications.

Face mediated human–robot interaction for remote medical examination

Realtime visual feedback from consequences of actions is useful for future safety-critical human–robot interaction applications such as remote physical examination of patients. Given multiple formats to present visual feedback, using face as feedback for mediating human–robot interaction in remote examination remains understudied. Here we describe a face mediated human–robot interaction approach for remote palpation. It builds upon a robodoctor–robopatient platform where user can palpate on the robopatient to remotely control the robodoctor to diagnose a patient. A tactile sensor array mounted on the end effector of the robodoctor measures the haptic response of the patient under diagnosis and transfers it to the robopatient to render pain facial expressions in response to palpation forces. We compare this approach against a direct presentation of tactile sensor data in a visual tactile map. As feedback, the former has the advantage of recruiting advanced human capabilities to decode expressions on a human face whereas the later has the advantage of being able to present details such as intensity and spatial information of palpation. In a user study, we compare these two approaches in a teleoperated palpation task to find the hard nodule embedded in the remote abdominal phantom. We show that the face mediated human–robot interaction approach leads to statistically significant improvements in localizing the hard nodule without compromising the nodule position estimation time. We highlight the inherent power of facial expressions as communicative signals to enhance the utility and effectiveness of human–robot interaction in remote medical examinations.

Identifying Mild Cognitive Impairment by Using Human–Robot Interactions

Background: Mild cognitive impairment (MCI), which is common in older adults, is a risk factor for dementia. Rapidly growing health care demand associated with global population aging has spurred the development of new digital tools for the assessment of cognitive performance in older adults. Objective: To overcome methodological drawbacks of previous studies (e.g., use of potentially imprecise screening tools that fail to include patients with MCI), this study investigated the feasibility of assessing multiple cognitive functions in older adults with and without MCI by using a social robot. Methods: This study included 33 older adults with or without MCI and 33 healthy young adults. We examined the utility of five robotic cognitive tests focused on language, episodic memory, prospective memory, and aspects of executive function to classify age-associated cognitive changes versus MCI. Standardized neuropsychological tests were collected to validate robotic test performance. Results: The assessment was well received by all participants. Robotic tests assessing delayed episodic memory, prospective memory, and aspects of executive function were optimal for differentiating between older adults with and without MCI, whereas the global cognitive test (i.e., Mini-Mental State Examination) failed to capture such subtle cognitive differences among older adults. Furthermore, robot-administered tests demonstrated sound ability to predict the results of standardized cognitive tests, even after adjustment for demographic variables and global cognitive status. Conclusion: Overall, our results suggest the human–robot interaction approach is feasible for MCI identification. Incorporating additional cognitive test measures might improve the stability and reliability of such robot-assisted MCI diagnoses.

Interactive Industrial Robot Programming based on Mixed Reality and Full Hand Tracking

The paper show a state-of-the-art approach for programming industrial robotics manipulator via mixed reality holographic interface. In mixed reality, we implement basic functionality for robot programming: setting up the base and tool frames, setting the sequence of Cartesian poses and Joint positions for robot motion, robot movements simulation. The software developed is based on Unity, ROS, and MoveIt frameworks. We compared cursor-based interface (Microsoft HoloLens 1) and full hand tracking (Microsoft HoloLens 2 and Oculus Quest 2) functionalities for interactive robot programming. We tested our approach on the UR10e collaborative robot in the virtual scene and real environment. The full hand tracking human-robot interaction approach improves the setup of Cartesian and Joints robot goals in respect to cursor based interaction.

A three-dimensional mapping and virtual reality-based human–robot interaction for collaborative space exploration

Human–robot interaction is a vital part of human–robot collaborative space exploration, which bridges the high-level decision and path planning intelligence of human and the accurate sensing and modelling ability of the robot. However, most conventional human–robot interaction approaches rely on video streams for the operator to understand the robot’s surrounding, which lacks situational awareness and force the operator to be stressed and fatigued. This research aims to improve efficiency and promote the natural level of interaction for human–robot collaboration. We present a human–robot interaction method based on real-time mapping and online virtual reality visualization, which is implemented and verified for rescue robotics. At the robot side, a dense point cloud map is built in real-time by LiDAR-IMU tightly fusion; the resulting map is further transformed into three-dimensional normal distributions transform representation. Wireless communication is employed to transmit the three-dimensional normal distributions transform map to the remote control station in an incremental manner. At the remote control station, the received map is rendered in virtual reality using parameterized ellipsoid cells. The operator controls the robot with three modes. In complex areas, the operator can use interactive devices to give low-level motion commands. In the less unstructured region, the operator can specify a path or even a target point. Afterwards, the robot follows the path or navigates to the target point autonomously. In other words, these two modes rely more on the robot’s autonomy. By virtue of virtual reality visualization, the operator can have a more comprehensive understanding of the space to be explored. In this case, the high-level decision and path planning intelligence of human and the accurate sensing and modelling ability of the robot can be well integrated as a whole. Although the method is proposed for rescue robots, it can also be used in other out-of-sight teleoperation-based human–robot collaboration systems, including but not limited to manufacturing, space, undersea, surgery, agriculture and military operations.

Physical Human–Robot Interaction of a Robotic Exoskeleton By Admittance Control

In this paper, a physical human–robot interaction approach is presented for the developed robotic exoskeleton using admittance control to deal with a human subject's intention as well as the unknown inertia masses and moments in the robotic dynamics. The human subject's intention is represented by the reference trajectory when the robotic exoskeleton is complying with the external interaction force. Online estimation of the stiffness is employed to deal with the variable impedance property of the robotic exoskeleton. Admittance control is first presented based on the measured force in order to generate a reference trajectory in interaction tasks. Then, adaptive control is proposed to deal with the uncertain robotic dynamics and a stability criterion can be obtained. Bounded errors are shown in the motion tracking while the robustness of the variable stiffness control is guaranteed. The experimental results indicate that the proposed control enables the human subjects to execute an admittance control task on the exoskeleton robot effectively.

Survey on Human-Robot Interaction for Robot Programming in Industrial Applications

The recent trends in modern industry highlight an increasing use of robots for a wide range of applications, which span from established manufacturing operations to novel tasks characterized by a close collaboration with the operators. Although human-robot collaboration allows to relieve operators of exhausting works, an effective collaboration requires a straightforward interaction to foster the use of robot assistants. This paper provides a comprehensive survey on human-robot interaction approaches and related interfaces addressed to robot programming. An overview of on-line and off-line robot programming techniques is first presented. Then, novel intuitive interaction means, such as those based on multi-modal interaction, virtual and augmented reality, are considered. The paper aims at pointing out that collaborative robotics can effectively reduce operator’s physical workload if easy to use interfaces for robot programming are provided.

Natural Gesture Modeling and Recognition Approach Based on Joint Movements and Arm Orientations

Gesture recognition, as one of the most promising human–robot interaction approaches, has attracted research interests for a long time. Though many feasible methods have been proposed in this field, the current gesture modeling and recognition methods are deficient in expressiveness, naturalness, and efficiency. In this paper, we present a novel method of modeling and recognizing natural gestures in 3-D space. Inspired by the joint space modeling, which is frequently used in robotics, we decompose the gestures into information of joint movements, joint angles, and arm orientations. We calculate joint angles and detect joint movements with the orientation data measured by inertial measurement units. Numbers of gestures can be modeled by combining three types of criteria, namely, joint angle criteria, arm orientation criteria, and joint movement criteria. Our system can cope with the repetitive movements, which are common in gestures while rarely considered before. Different from the previous statistical learning or template matching approaches, our approach does not need training. User can add new gesture to the gesture set or edit existing gesture definitions conveniently. To evaluate our system, we conducted real-time gesture recognition experiments with ten subjects. Twelve gestures, including three static gestures and nine dynamic gestures, were modeled for the experiments. The average recognition accuracy of totaling 1560 gesture samples was 91.86%.

Interactive multiple object learning with scanty human supervision

We present a fast and online human-robot interaction approach that progressively learns multiple object classifiers using scanty human supervision. Given an input video stream recorded during the human-robot interaction, the user just needs to annotate a small fraction of frames to compute object specific classifiers based on random ferns which share the same features. The resulting methodology is fast (in a few seconds, complex object appearances can be learned), versatile (it can be applied to unconstrained scenarios), scalable (real experiments show we can model up to 30 different object classes), and minimizes the amount of human intervention by leveraging the uncertainty measures associated to each classifier.We thoroughly validate the approach on synthetic data and on real sequences acquired with a mobile platform in indoor and outdoor scenarios containing a multitude of different objects. We show that with little human assistance, we are able to build object classifiers robust to viewpoint changes, partial occlusions, varying lighting and cluttered backgrounds.

Supporting Human–Robot Interaction Based on the Level of Visual Focus of Attention

We propose a human–robot interaction approach for social robots that attracts and controls the attention of a target person depending on her/his current visual focus of attention. The system detects the person's current task (attention) and estimates the level by using the “task-related contextual cues” and “gaze pattern.” The attention level is used to determine the suitable time to attract the target person's attention toward the robot. The robot detects the interest or willingness of the target person to interact with it. Then, depending on the level of interest of the target person, the robot generates awareness and establishes a communication channel with her/him. To evaluate the performance, we conducted an experiment using our static robot to attract the target human's attention when she/he is involved in four different tasks: reading, writing, browsing, and viewing paintings. The proposed robot determines the level of attention of the current task and considers the situation of the target person. Questionnaire measures confirmed that the proposed robot outperforms a simple attention control robot in attracting participants’ attention in an acceptable way. It also causes less disturbance and establishes effective eye contact. We implemented the system into a commercial robotic platform (Robovie-R3) to initiate interaction between visitors and the robot in a museum scenario. The robot determined the visitors’ gaze points and established a successful interaction with a success rate of 91.7%.

Interactive indoor environment mapping through visual tracking of human skeleton

This paper presents a novel human-robot interaction approach to grid mapping of an indoor environment based on a 3D kinect sensor and the grid-based mapping algorithm. It mainly includes three modules: skeleton tracking, robot control and GMapping. Firstly, the skeleton tracking module builds a human skeleton model, extracts the skeleton joints’ position information from 3D visual data and generates digital signals through identifying some simple motions and events. Then according to different digital signals and joints’ position information, the robot control module enables the robot to take different actions such as following the person, stop and so on. Finally, the grid map of the environment is built through GMapping algorithm based on odometry and laser data, which is improved by Rao-Blackwellised particle filters. The proposed approach has been implemented successfully in several different buildings and can be applied to service robots. Compared with traditional roaming for mapping, human guiding the robot for mapping is more efficient and takes less time in a complicated environment. Meanwhile, compared with wearable motion sensors attached to the human body, this approach is more convenient and make the user more comfortable.

Two-Stage Hidden Markov Model in Gesture Recognition for Human Robot Interaction

Hidden Markov Model (HMM) is very rich in mathematical structure and hence can form the theoretical basis for use in a wide range of applications including gesture representation. Most research in this field, however, uses only HMM for recognizing simple gestures, while HMM can definitely be applied for whole gesture meaning recognition. This is very effectively applicable in Human-Robot Interaction (HRI). In this paper, we introduce an approach for HRI in which not only the human can naturally control the robot by hand gesture, but also the robot can recognize what kind of task it is executing. The main idea behind this method is the 2-stages Hidden Markov Model. The 1st HMM is to recognize the prime command-like gestures. Based on the sequence of prime gestures that are recognized from the 1st stage and which represent the whole action, the 2nd HMM plays a role in task recognition. Another contribution of this paper is that we use the output Mixed Gaussian distribution in HMM to improve the recognition rate. In the experiment, we also complete a comparison of the different number of hidden states and mixture components to obtain the optimal one, and compare to other methods to evaluate this performance.