Human-robot Interaction Interface Research Articles

A Comprehensive Study on State-Of-Art Learning Algorithms in Emotion Recognition

The potential uses of emotion recognition in domains like human-robot interaction, marketing, emotional gaming, and human-computer interface have made it a prominent research subject. Better user experiences can result from the development of technologies that can accurately interpret and respond to human emotions thanks to a better understanding of emotions. The use of several sensors and computational algorithms is the main emphasis of this paper's thorough analysis of the developments in emotion recognition techniques. Our results show that using more than one modality improves the performance of emotion recognition when a variety of metrics and computational techniques are used. This paper adds to the body of knowledge by thoroughly examining and contrasting several state-of-art computational techniques and measurements for emotion recognition. The study emphasizes how crucial it is to use a variety of modalities along with cutting-edge machine learning algorithms in order to attain more precise and trustworthy emotion assessment. Additionally, we pinpoint prospective avenues for additional investigation and advancement, including the incorporation of multimodal data and the investigation of innovative features and fusion methodologies. This study contributes to the development of technology that can better comprehend and react to human emotions by offering practitioners and academics in the field of emotion recognition insightful advice.

Teleoperated Surgical Robot with Adaptive Interactive Control Architecture for Tissue Identification.

The remote perception of teleoperated surgical robotics has been a critical issue for surgeons in fulfilling their remote manipulation tasks. In this article, an adaptive teleoperation control framework is proposed. It provides a physical human-robot interaction interface to enhance the ability of the operator to intuitively perceive the material properties of remote objects. The recursive least square (RLS) is adopted to estimate the required human hand stiffness that the operator can achieve to compensate for the contact force. Based on the estimated stiffness, a force feedback controller is designed to avoid the induced motion and to convey the haptic information of the slave side. The passivity of the proposed teleoperation system is ensured by the virtual energy tank. A stable contact test validated that the proposed method achieved stable contact between the slave robot and the hard environment while ensuring the transparency of the force feedback. A series of human subject experiments was conducted to empirically verify that the proposed teleoperation framework can provide a more smooth, dexterous, and intuitive user experience with a more accurate perception of the mechanical property of the interacted material on the slave side, compared to the baseline method. After the experiment, the design idea about the force feedback controller of the bilateral teleoperation is discussed.

Development of an Active Physical Interface for Physical Human-Robot Interaction: Investigation of Soft Pneumatic Actuator Straps for Automatic Enclosure System

Wearable robots have become increasingly prevalent in various applications, including rehabilitation, power augmentation, and assistance. However, one of the challenges in designing wearable robots is how to attach them to the human body. The attachment method should be secure, reliable, comfortable, effective, and controlled for the user. Moreover, the attachment points should not interfere with the user’s daily activities, and the attachment process should not be time-consuming or complicated. Typical straps nowadays require a time-consuming and cumbersome donning and doffing procedure from therapists for users needing rehabilitation therapy. Therefore, we propose a novel pneumatically actuated soft strap to enclose the limb and automate part of the strapping procedure. This paper proposes a preliminary design utilizing soft bending actuators for attaching physical interfaces to humans, with integrated active elements for facilitating and automating the strapping process. Finite element analysis was conducted to assess pressure requirements, bending curvature, and geometry, with simulation results demonstrating a promising agreement, with a root mean square error (RMSE) of 3.4° in bending angle. In the future, an additional locking mechanism would be required to provide the necessary holding force and fully constrain the limb.

A touch-free human-robot collaborative surgical navigation robotic system based on hand gesture recognition.

Robot-assisted minimally invasive surgery (RAMIS) has gained significant traction in clinical practice in recent years. However, most surgical robots rely on touch-based human-robot interaction (HRI), which increases the risk of bacterial diffusion. This risk is particularly concerning when surgeons must operate various equipment with their bare hands, necessitating repeated sterilization. Thus, achieving touch-free and precise manipulation with a surgical robot is challenging. To address this challenge, we propose a novel HRI interface based on gesture recognition, leveraging hand-keypoint regression and hand-shape reconstruction methods. By encoding the 21 keypoints from the recognized hand gesture, the robot can successfully perform the corresponding action according to predefined rules, which enables the robot to perform fine-tuning of surgical instruments without the need for physical contact with the surgeon. We evaluated the surgical applicability of the proposed system through both phantom and cadaver studies. In the phantom experiment, the average needle tip location error was 0.51 mm, and the mean angle error was 0.34 degrees. In the simulated nasopharyngeal carcinoma biopsy experiment, the needle insertion error was 0.16 mm, and the angle error was 0.10 degrees. These results indicate that the proposed system achieves clinically acceptable accuracy and can assist surgeons in performing contactless surgery with hand gesture interaction.

A Mobile Natural Human–Robot Interaction Method for Virtual Chinese Acupuncture

Most of the existing virtual acupuncture methods cannot allow operators to interact naturally in a large area. To address this problem, this article proposes a mobile natural human–robot interaction interface for virtual Chinese acupuncture teaching, including an automatic hand tracking method and an acupuncture interaction method combining vision and force feedback. The automatic hand tracking method can automatically track and capture the operator’s hand movements, allowing the operator to perform acupuncture in a large area. The acupuncture interaction method combining vision and force reproduces the real acupuncture operation mode. The proposed interaction method provides the operator with visual feedback and force feedback without limitation, which improves the naturalness and authenticity of the operation. Through a series of experiments, the proposed method was proven to be more natural and efficient in the virtual acupuncture teaching scene from the point view of interaction. The operators achieve a stronger sense of immersion using this method.

In-Hand Admittance Controller for a Robotic Assistive Walker Based on Tactile Grasping Feedback

An intelligent assistive robotic walker can be a tremendous help to patients with mobility disorders, e.g., hemiplegia patients. However, at the same time, a full-size assistive walker could be difficult to steer due to the reduced balancing ability of the patient. An intuitive interface can provide a promising solution to the low operability of such robotized assistive devices. Commonly developed gait assistance methodologies perform real-time adjustment of admittance model parameters in a forward direction. However, such a variable admittance model often accidentally affects the rotation behavior as well, which can lead to reducing the operability of the robotic device. To resolve such operational issues, we propose an in-Hand Admittance Controller (i-HAC) as a novel physical human-robot interaction interface for a robotic assistive walker. i-HAC is composed of multiple admittance models based on grasping feedback from sensitive robotic skin placed on both handlebars of a robotic assistive walker. This controller is formulated as a multiple-point-mass-damper system and can generate intuitive movement in both single and dual grasping situations. We conduct two experiments to demonstrate the benefits of i-HAC on a full-sized robotic walker in going forward with single grasping and rotating with dual grasping in comparison to the conventional admittance controller. Our experimental studies show that i-HAC potentially reduces the operation effort by 85% in the single grasping case and improves rotation consistency by 93% in the dual grasping case.

Virtual Health Care Robot Using Kinect Sensor

Abstract: Kinect cameras and therefore the other sensor combination framework is used to obtain real-time tracking of a person’s action skeleton. Various gestures, performed by different persons, Quaternion's of joint angles are first used as robust and significant features. we present a gesture human activity recognition system for the event of human-robot interaction (HRI) interface. Next, classifiers are used to recognize the different gestures. This work deals with different challenging tasks, like the real-time implementation of a gesture act recognition system and also the temporal resolution of gestures. The HRI interface developed during this work includes three Kinect and robots which will be remotely controlled by one operator standing before of one of the Kinect sensors. Moreover, the system is provided with a nation action for control of the robot.

Low-intensity near-infrared light-tunable gold nanorod-incorporated hydrogel actuator system for remotely controlled human–robot interface applications

Light-responsive soft actuators have recently drawn attention as the need for lightweight remotely controlled actuator systems with high environmental adaptability and safe human–robot interaction interface characteristics has increased. In this study, we focused on a near-infrared (NIR) light-tunable hydrogel soft actuator system capable of fast and precisely controllable mechanical actuation even under the irradiation of NIR light with low intensity. We designed a bilayer type hydrogel soft actuator consisting of a poly(N-isopropylacrylamide) (PNIPAAm) active layer containing gold nanorods (AuNRs) as a photothermal agent and poly(acrylamide) (PAAm) passive layer. AuNRs with an aspect ratio of 4.12 showed the largest longitudinal plasmon resonance absorption peak in the NIR 880 nm range, which was used as a photothermal agent. The AuNR-incorporated PNIPAAm/PAAm hydrogel with a bilayer structure (PNIPAAm-AuNR/PAAm) exhibited more than 70% gel fraction and temperature-sensitive swelling behaviors. In particular, rod-shaped PNIPAAm-AuNR/PAAm hydrogels showed rapid bending deformation by a photothermally induced phase transition of PNIPAAm upon 880 nm laser irradiation, but a PNIPAAm/PAAm bilayer hydrogel without AuNRs did not show a light-induced bending movement. The NIR-responsive bending actuation of the hydrogel can be precisely controlled by cross-linking density, content of photothermal agent, NIR laser intensity, and thickness ratio of the active/passive layer. It was observed that the full bending deformations of the hydrogel into a ring shape through an arc occurred within a minute under NIR irradiation with a lower intensity of less than 1.0 W/cm2. Hydrogels showed a cell viability of more than 95% in the biocompatibility test, indicating no significant cytotoxicity. Therefore, these PNIPAAm-AuNR/PAAm hydrogels are promising soft actuator materials that can be controlled by low-intensity NIR irradiation for remotely controlled human–robot interaction interface applications.

Designing Physical Human-Robot Interaction Interfaces: A Scalable Method for Simulation Based Design.

Designing the physical coupling between the human body and the wearable robot is a challenging endeavor. The typical approach of tightening the wearable robot against the body, and softening the interface materials does not work well. It makes the task of simultaneously improving comfort, and anchoring the robot to the body at the physical human robot interaction interface (PHRII), difficult. Characterizing this behavior experimentally with sensors at the interface is challenging due to the soft-soft interactions between the PHRII materials and the human tissue. Therefore, modeling the interaction between the wearable robot and the hand is a necessary step to improve design. In this paper, we introduce a methodology to systematically improve the design of the PHRII by combining experimentally measured characteristics of the biological tissue with a novel dynamic modeling tool. Using a novel and scalable simulation framework, HuRoSim, we quantified the interaction between the human hand and an exoskeleton. In the first of our experiments, we use HuRoSim to predict complex interactions between the hand and the coupled exoskeleton. In our second experiment, we then demonstrate how HuRoSim can be coupled with experimental measurements of the stiffness of the dorsal surface of the hand to optimize the design of the PHRII. This approach of data-driven modeling of the interaction between the body and a wearable robot, such as a hand exoskeleton, can be generalized to other forms of wearable devices as well, demonstrating a scalable and systematic method for improving the design of the PHRII for future devices coupled to the body.

A Gesture-Based Natural Human–Robot Interaction Interface With Unrestricted Force Feedback

This article presents a novel gesture-based natural human–robot interaction interface, which integrates a markerless gesture tracking system and an unrestricted electromagnetic force feedback mechanism. In this proposed interface, a markerless gesture tracking system is developed to relate the motion of the operator’s hand to the robot manipulator so that the operator can naturally and friendly control the robot without any markers. More importantly, this interface uses a new unrestricted electromagnetic force feedback mechanism to avoid the friction, hysteresis, and other nonlinear influence in the conventional actuation dynamics. The interface makes the operator obtain the effective force immersion of the robot. Therefore, the proposed interface can provide the promising operation accuracy for the operator. To effectively regulate the electric currents of coils and provide accurate force feedback, the broad learning system (BLS) is introduced in this unrestricted electromagnetic force feedback mechanism. In addition, two interval Kalman filters (IKFs) are applied to estimate the position and orientation of the operator’s hand, respectively, improving the measurement accuracy of the proposed interface. Experimental results show that the proposed interface is suitable for high-precision human–robot interactive tasks and enables the operator to focus on the tasks and operate dual robot manipulator, which provides natural and efficient human–robot interaction.

Collaborative Human–Robot Interaction Interface: Development for a Spinal Surgery Robotic Assistant

The growing introduction of robotics in non-industrial applications where the environment is unstructured and changing, has led to the need of development of safer and more intuitive, human–robot interfaces. In such environments, the use of collaborative robots has potential benefits, due to the combination of user experience, knowledge and flexibility with the robot’s accuracy, stiffness and repeatability. Nevertheless, in order to guarantee a functional collaboration in these environments, the interaction between user and robot must be intuitive, natural, fast and easy to use. On one hand, commercial collaborative robots are less accurate and less stiff than the traditional industrial ones, on the other hand, the later have not intuitive interaction interfaces. There are tasks in which the stiffness of industrial robots and the intuitive interaction interfaces of collaborative commercial robots, are desirable. This is the case of some robotic assisted surgical procedures, such as robotic assisted spine surgery, with high accuracy demands and with the need of intuitive surgeon–robot interaction. This paper presents a hand guiding methodology for functional human–robot collaboration and the introduction of novel algorithms to enhance its behavior. Also its implementation on a robotic surgical assistant for spine procedures is presented. It is emphasized how a traditional industrial robot can be used as a collaborative one when the available commercial collaborative robots do not have the required accuracy and stiffness for the task.

A Review on Voice-based Interface for Human-Robot Interaction

With the recent developments of technology and the advances in artificial intelligence and machine learning techniques, it has become possible for the robot to understand and respond to voice as part of Human-Robot Interaction (HRI). The voice-based interface robot can recognize the speech information from humans so that it will be able to interact more naturally with its human counterpart in different environments. In this work, a review of the voice-based interface for HRI systems has been presented. The review focuses on voice-based perception in HRI systems from three facets, which are: feature extraction, dimensionality reduction, and semantic understanding. For feature extraction, numerous types of features have been reviewed in various domains, such as time, frequency, cepstral (i.e. implementing the inverse Fourier transform for the signal spectrum logarithm), and deep domains. For dimensionality reduction, subspace learning can be used to eliminate the redundancies of high-dimensional features by further processing extracted features to reflect their semantic information better. For semantic understanding, the aim is to infer from the extracted features the objects or human behaviors. Numerous types of semantic understanding have been reviewed, such as speech recognition, speaker recognition, speaker gender detection, speaker gender and age estimation, and speaker localization. Finally, some of the existing voice-based interface issues and recommendations for future works have been outlined.

Human-robot interaction control of a haptic master manipulator used in laparoscopic minimally invasive surgical robot system

As an interface for physical human-robot interaction (pHRI), the master manipulator of robot-assisted surgery system is critical to the success rate of surgical procedures. In this paper, a pHRI control scheme of the haptic master manipulator used in laparoscopic surgical robots is presented for enhancing the precision and comfort of operations. Firstly, the inverse dynamic equations of the master manipulator are derived and a modified friction model is proposed to improve the accuracy of friction torque calculation. Then a torque observer based on generalized momentum is designed. The proposed observer can precisely track the joint driving torque needed by the motion state and comply with the operator's intentions. Thirdly, the compensator based on time-delay neural network (TDNN) is proposed to improve the performance of pHRI. Finally, the pHRI control strategy which can adjust the degree of operation compliance actively is introduced. The results of theoretical analysis and comparative experiments indicate that the proposed strategy can substantially reduce the force and torque applied to surgeons by the master manipulator during the operation.

Audio-Driven Robot Upper-Body Motion Synthesis

Body language is an important aspect of human communication, which an effective human-robot interaction interface should mimic well. Human beings exchange information and convey their thoughts and feelings through gaze, facial expressions, body language, and tone of voice along with spoken words, and infer 65% of the meaning of the communicated messages from these nonverbal cues. Modern robotic platforms are, however, limited in their ability to automatically generate behaviors that align with their speech. In this article, we develop a neural-network-based system that takes audio from a user as an input and generates upper-body gestures, including head, hand, and torso movements of the user on a humanoid robot, namely, Softbank Robotics' Pepper. Our system was evaluated quantitatively as well as qualitatively using Web surveys when driven by natural speech and synthetic speech. We compare the impact of generic and person-specific neural-network models on the quality of synthesized movements. We further investigate the relationships between quantitative and qualitative evaluations and examine how the speaker's personality traits affect the synthesized movements.

Development of pass-through augmented reality interface for human robot interaction

In this technology driven world, human machine interaction is indispensable. Conventional human machine interface techniques pose the challenges of extensive training and increased cognitive load. Usage of hand gestures, face and speech recognition, eye ball tracking, etc. are in the upward trend to overcome these limitations and prove to be more intuitive and user friendly. In this paper, a vision-based pass-through augmented reality system is developed that enables the user to command the mobile robot for various mobility and navigation tasks. A pass-through augmented reality setup is achieved by a combination of a typical virtual reality headset and a stereo camera with depth perception. The stereo camera and a Leap Motion sensor mounted on a wearable VR headset acts as the vision system for the user and the gesture recognition system respectively. Necessary graphical user interface with the list of functions and information will appear on the VR headset display from which the user can select the option/command to be given to the mobile robot. The gesture made by the user in the GUI is detected and communicated to the robot through wireless means to perform the corresponding task.

Development of pass-through augmented reality interface for human robot interaction

In this technology driven world, human machine interaction is indispensable. Conventional human machine interface techniques pose the challenges of extensive training and increased cognitive load. Usage of hand gestures, face and speech recognition, eye ball tracking, etc. are in the upward trend to overcome these limitations and prove to be more intuitive and user friendly. In this paper, a vision-based pass-through augmented reality system is developed that enables the user to command the mobile robot for various mobility and navigation tasks. A pass-through augmented reality setup is achieved by a combination of a typical virtual reality headset and a stereo camera with depth perception. The stereo camera and a Leap Motion sensor mounted on a wearable VR headset acts as the vision system for the user and the gesture recognition system respectively. Necessary graphical user interface with the list of functions and information will appear on the VR headset display from which the user can select the option/command to be given to the mobile robot. The gesture made by the user in the GUI is detected and communicated to the robot through wireless means to perform the corresponding task.

Haptics Electromyography Perception and Learning Enhanced Intelligence for Teleoperated Robot

Due to the lack of transparent and friendly human-robot interaction (HRI) interface, as well as various uncertainties, it is usually a challenge to remotely manipulate a robot to accomplish a complicated task. To improve the teleoperation performance, we propose a new perception mechanism by integrating a novel learning method to operate the robots in the distance. In order to enhance the perception of the teleoperation system, we utilize a surface electromyogram signal to extract the human operator's muscle activation. As a response to the changes in the external environment, as sensed through haptic and visual feedback, a human operator naturally reacts with various muscle activations. By imitating the human behaviors in task execution, not only motion trajectory but also arm stiffness adjusted by muscle activation, it is expected that the robot would be able to carry out the repetitive tasks autonomously or uncertain tasks with improved intelligence. To this end, we develop a robot learning algorithm based on probability statistics under an integrated framework of the hidden semi-Markov model (HSMM) and the Gaussian mixture method. This method is employed to obtain a generative task model based on the robot's trajectory. Then, Gaussian mixture regression based on HSMM is applied to correct the robot trajectory with the reproduced results from the learned task model. The execution procedures consist of a learning phase and a reproduction phase. To guarantee the stability, immersion, and maneuverability of the teleoperation system, a variable gain control method that involves electromyography (EMG) is introduced. Experimental results have demonstrated the effectiveness of the proposed method.

Position Based Impedance Control Strategy for a Lower Limb Rehabilitation Robot.

Active rehabilitation training can improve patients' neural engagement and the rehabilitation effects. A position based impedance control strategy is proposed for implement of active training in this paper. Firstly, it is necessary to accurately estimate patients' active torques during the training process, which can be calculated by using the human-robot dynamic model. A novel friction model is designed, where the influence of the joint coupling factor and viscosity is considered to improve the model accuracy. In order to prevent sudden changes of the interaction force, a virtual tunnel around the reference trajectory is established to limit the motion range to ensure patient safety in case of emergency, such as muscle spasm. Then, a position based impedance control strategy designed in the joint space is used to convert human torques into the required movement. The strategy is implemented by a double closed loop control structure, the outer loop of which converts the interaction force into angular, angular velocity, and angular acceleration deviations by the inverse impedance equation. The regulated trajectory will be used as reference for the inner position control loop. The experimental results show that the proposed system dynamic model can be used to accurately estimate the patients' active torques, and the proposed control strategy can be used to design a compliant human-robot interaction interface, so that patients can complete the training task comfortably and naturally, and also safety of the training can be ensured.

CGAN Based Facial Expression Recognition for Human-Robot Interaction

As an emerging research topic for proximity service (ProSe), automatic emotion recognition enables the machines to understand the emotional changes of human beings which can not only facilitate natural, effective, seamless, and advanced human-robot interaction or human-computer interface but also promote emotional health. Facial expression recognition (FER) is a vital task for emotion recognition. However, significant gap between human and machine exists in FER task. In this paper, we present a conditional generative adversarial network-based approach to alleviate the intra-class variations by individually controlling the facial expressions and learning the generative and discriminative representations simultaneously. The proposed framework consists of a generator G and three discriminators (Di, Da, and Dexp). The generator G transforms any query face image into another prototypic facial expression image with other factors preserved. Armed with action units condition, the generator G pays more attention to information relevant to facial expression. Three loss functions ($L_{I}$ , La, and Lexp) corresponding to the three discriminators (Di, Da, and Dexp) were designed to learn generative and discriminative representations. Moreover, after rendering the generated expression back to its original facial expression, cycle consistency loss is also applied to guarantee the identity and produce more constrained visual representations. Optimized by combining both synthesis and classification loss functions, the learnt representation is explicitly disentangled from other variations such as identity, head pose, and illumination. Qualitative and quantitative experimental results demonstrate the proposed FER system is effective for expression recognition.

Interactive Robots Control Using Mixed Reality

Abstract The paper presents a mixed reality-based approach for interactive control of robotic manipulators and mobile platforms. In particular, we designed an interactive and understandable interface for human-robot interaction. The interface provides tools for robot path programming and visualizes it. Path visualization helps workers understand robot behavior, it is important for safety human-robot interaction. The paper presents an architecture of that system and the implementation for an industrial robot KUKA iiwa and mobile robot platform Plato. The main issue of a multi-platform system is related to the synchronization of coordinate frames for all elements. To deal with this problem we realized 3 setting options: manually, by a camera with markers, point clouds processing. We implemented our interface on Microsoft HoloLens and evaluated it on users.