Human-robot Interaction System Research Articles

Adaptive Position Constrained Assist-as-Needed Control for Rehabilitation Robots

In rehabilitation practice, motivating patients with neurological injuries to actively increase muscle activity and ensure their safety are important. Therefore, this study proposed a position-constrained assist-as-needed (AAN) control method for rehabilitation robots. A human-robot interaction system with position constraints was first established based on prescribed performance. Aiming at implementing the AAN strategy, the robot assistance level metric (RALM), a constructed global continuous differentiable function incorporating dead-zone and saturation characteristics, was introduced to quantify the robotic assistance and facilitate seamless operation. To bridge the gap between the position constraints and the AAN strategy, a sliding manifold was constructed for the constrained human-robot dynamic system, where RALM was regarded as a weight factor to achieve a human-dominated mode, a robot-dominated mode, and their smooth transition, regarded as a human-robot shared mode. The stability of the closed-loop system was guaranteed by using the Lyapunov theory, and the proposed controller was verified by several physical experiments on a knee exoskeleton driven by pneumatic muscles (PMs).

A Two-Stage Selective Fusion Framework for Joint Intent Detection and Slot Filling.

Spoken language understanding (SLU) is the core of the speech-centric human-robot interaction system, which mainly involves intent detection and slot filling. The recent SLU research focuses on the joint modeling of the two tasks due to their correlation. Furthermore, the slot information consists of slot position and slot type. Although the slot types are semantically related to the intent, the slot positions of the same intent may vary a lot in different utterances due to the diversity of spoken language. Thus, the conventional one-stage slot filling task may introduce unrelated information for slot position prediction in the slot-intent interaction of the joint modeling. Therefore, we propose a novel two-stage selective fusion framework for joint intent detection and slot filling. Unlike the previous one-stage framework, the proposed framework decomposes the slot filling into two stages, i.e., the slot proposal and slot classification. The slot proposal network consisting of BERT and bidirectional long short-term memory (Bi-LSTM)-conditional random field (CRF) predicts the slot positions. Instead of the tokenwise fusion in the existing methods, the slot-intent feature fusion is only performed in the slot classification. A selective fusion mechanism is designed to facilitate the slot-intent interaction within each slot candidate for more accurate slot-type classification. Experiments on five standard benchmarks (i.e., ATIS, SNIPS, MixATIS, MixSNIPS, and DSTC4) show that the proposed framework achieves the best performance in comparison with several state-of-the-art methods.

Kinematic-driven human-robot interaction system with deep learning for flexible acupuncture needling manipulations

Acupuncture is an important therapeutic method in traditional Chinese medicine and essentially characterized by flexible needling manipulations that are complicated for repetitions. In this work, we designed a bio-inspired acupuncture robot combined with human–machine interaction system that can implement flexible and precise needle manipulations as manual acupuncture. In the proposed framework, the manual needling process was recorded with online imaging techniques. With a deep learning-based acupuncture estimator, kinetic parameters of lift insertion (LI) and twist rotation (TR) manipulations were extracted in three-dimensional space and the precent of correct key points for operator’s hand pose estimation achieved 94.03%. It is founded that the variation of operator’s finger vector angle for LI manipulation was much weaker than TR manipulation. Furthermore, inspired by kinematic characteristics of manual acupuncture, a six-degree-of-freedom robot was used to simulate the coordinated motion of hand joints in needling manipulations through robot kinematics analysis and trajectory planning with polynomial interpolation method. Numerical and experiential results showed that the acupuncture robot can mimic various needling manipulations as manual acupuncture. The flexibility and stability of the robotic acupuncture framework were evaluated for a wide range of needling parameters. It was demonstrated that the robot can follow the periodic dynamics of LI and TR manipulations with various frequencies and amplitudes, and perform accurate acupuncture operations with smooth needling trajectories as joint jitters during needle manipulations were completely eliminated. The presented results provide a new strategy for modern applications of acupuncture treatment with robotic and flexible needling manipulations to improve the therapeutic efficacy.

Mechanically Robust and Linearly Sensitive Soft Piezoresistive Pressure Sensor for a Wearable Human-Robot Interaction System.

Soft piezoresistive pressure sensors play an underpinning role in enabling a plethora of future Internet of Things (IoT) applications such as human-robot interaction (HRI) technologies, wearable devices, and metaverse ecosystems. Despite significant attempts to enhance the performance of these sensors, existing sensors still fall short of achieving high strain tolerance and linearity simultaneously. Herein, we present a low-cost, facile, and scalable approach to fabricating a highly strain-tolerant and linearly sensitive soft piezoresistive pressure sensor. Our design utilizes thin nanocracked gold films (NC-GFs) deposited on poly(dimethylsiloxane) (PDMS) as electrodes of the sensor. The large mismatch stress between gold (Au) and PDMS induces the formation of secondary wrinkles along the pyramidal-structured electrode under pressure; these wrinkles function as protuberances on the electrode and enable exceptional linear sensitivity of 4.2 kPa-1 over a wide pressure range. Additionally, our pressure sensor can maintain its performance even after severe mechanical deformations, including repeated stretching up to 30% strain, due to the outstanding strain tolerance of NC-GF. Our sensor's impressive sensing performance and mechanical robustness make it suitable for diverse IoT applications, as demonstrated by its use in wearable pulse monitoring devices and human-robot interaction systems.

A Continuously Variable Transmission-Based Variable Stiffness Actuator for pHRI: Design Optimization and Performance Verification

Abstract Physical human–robot interfaces (pHRIs) enabled the robots to work alongside the human workers complying with the regulations set for physical human–robot interaction systems. A variety of actuation systems named variable stiffness/impedance actuators (VSAs) are configured to be used in these systems’ design. Recently, we introduced a new continuously variable transmission (CVT) mechanism as an alternative solution in configuring VSAs for pHRI. The optimization of this CVT has significant importance to enhance its application area and to detect the limitations of the system. Thus, in this paper, we present a design optimization approach (an adjustment strategy) for this system based on the design goals, desired force, and minimization of the size of the system. To implement such design goals, the static force analysis of the CVT is performed and validated. Furthermore, the fabrication of the optimized prototype is presented, and the experimental verification is performed considering the requirements of VSAs: independent position and stiffness variation, and shock absorbing. Finally, the system is calibrated to display 6 N continuous output force throughout its transmission variation range.

Application of hedge algebras algorithm for mobile robot controller via hand gestures and wireless protocol

With the superior development of technology, mobile robots have become an essential part of humans’ daily life. Consequently, interacting and dealing with them pushes us to develop and propose different suitable Human-Robot Interaction (HRI) systems that can detect the interacted user’s actions and achieve the desired output in real-time. In this paper, we propose a closed-loop smart mechanism for two main agents: the hand gloves’ controller and the mobile robot. To be more specific, the developed model employs flex sensors to measure the curve of the finger. The sensor signals are then processed by aiding the Hedge Algebras Algorithm to control the movement direction and customize the speed of the mobile robot via wireless communication. Numerical simulation and experiments demonstrate that the mobile robot could operate reliably, respond rapidly to control signals, and vary its speed continually based on the different finger gestures. Besides, the control results are also compared with those obtained from the traditional fuzzy controller to prove the superiority and efficiency of the proposed method.

Brain–Computer Interface Integrated With Augmented Reality for Human–Robot Interaction

Brain-computer interface (BCI) has been gradually used in human-robot interaction systems. Steady-state visual evoked potential (SSVEP) as a paradigm of electroencephalography (EEG) has attracted more attention in the BCI system research due to its stability and efficiency. However, an independent monitor is needed in the traditional SSVEP-BCI system to display stimulus targets, and the stimulus targets map fixedly to some pre-set commands. These limit the development of the SSVEP-BCI application system in complex and changeable scenarios. In this study, the SSVEP-BCI system integrated with augmented reality (AR) is proposed. Furthermore, a stimulation interface is made by merging the visual information of the objects with stimulus targets, which can update the mapping relationship between stimulus targets and objects automatically to adapt to the change of the objects in the workspace. During the online experiment of the AR-based SSVEP-BCI cue-guided task with the robotic arm, the success rate of grasping is 87.50 ± 3.10% with the SSVEP-EEG data recognition time of 0.5 s based on FB-tCNN. The proposed AR-based SSVEP-BCI system enables the users to select intention targets more ecologically and can grasp more kinds of different objects with a limited number of stimulus targets, resulting in the potential to be used in complex and changeable scenarios.

Towards a more anthropomorphic interaction with robots in museum settings: An experimental study

In recent years, social robots have been increasingly used in the cultural sector to make artworks and exhibits more appealing to a larger audience. On the other hand, visitors, more accustomed to using novel technologies, are increasingly attracted to the possibility of exploiting new ways of interaction. However, more research needs to be done on how social robots successfully interact verbally and non-verbally with visitors. There are few studies on the impact of these technologies and, in particular, a greater level of involvement in using social robots instead of the more usual voice apps to access interactively and acquire information about a work of art or a museum exhibition. This paper proposes a human–robot interaction system based on a three-level architecture able to process and merge heterogeneous sensory information and obtain a set of anthropomorphic mechanisms and metaphors to make the interaction with visitors in museum settings as natural as possible. Moreover, an experimental study has been proposed to assess the impact of this proposed solution, embedded in the social robot Pepper, on museum visitors in the context of an exhibit on Leonardo da Vinci. In particular, the study first aimed to find a potential difference in the interactions between visitors grouped by age and interfacing with the robot equipped with the proposed system or a voice app. Secondly, the study focused on evaluating the effectiveness and usefulness of the robot equipped with the proposed interaction system. A comparative User Experience was conducted between the robot and voice app and a User Experience only for the robot using a specific questionnaire for robotic applications. The results showed a good ability of the proposed system integrated into the robot to make interaction with visitors more pleasant, attractive, and friendly. Engagement and dialogue with them are more natural, stable, and lasting. The cultural information is conveyed more easily and clearly.

Design and Modeling of a Smart Torque-Adjustable Rotary Electroadhesive Clutch for Application in Human–Robot Interaction

The increasing need for sharing workspace and interactive physical tasks between robots and humans has raised concerns regarding the safety of such operations. In this regard, controllable clutches have shown great potential for addressing important safety concerns at the hardware level by separating the high-impedance actuator from the end-effector by providing the power transfer from electromagnetic source to the human. However, the existing clutches suffer from high power consumption and large weight, which make them undesirable from the design point of view. In this article, for the first time, the design and development of a novel, lightweight, and low-power torque-adjustable rotary clutch using electroadhesive materials are presented. The performance of three different pairs of clutch plates is investigated in the context of the smoothness and quality of output torque. The performance degradation issue due to the polarization of the insulator is addressed through the utilization of an alternating current waveform activation signal. Moreover, the effect of the activation frequency on the output torque and power consumption of the clutch is investigated. Finally, a time-dependent model for the output torque of the clutch is presented, and the performance of the clutch was evaluated through experiments, including physical human–robot interaction. The proposed clutch offers a torque-to-power consumption ratio that is six times better than commercial magnetic particle clutches. The proposed clutch presents great potential for developing safe, lightweight, and low-power physical human–robot interaction systems, such as exoskeletons and robotic walkers.

Human–robot collaborative interaction with human perception and action recognition

This paper presents a human–robot interaction system (HRIS) that utilizes human perception and action recognition to enable the robot to understand human intentions and flexibly interact with humans. A monocular multi-person three-dimensional (3D) pose estimation method is first proposed to perceive multi-person two-dimensional (2D) and 3D poses in interaction scenarios. Furthermore, a 3D skeleton poses tracking approach is adopted to locate the identity of each person in consecutive frames and enhance interactive stability. Then, an action recognition model is developed, which exploits tracked pose features to recognize the intentions of humans. An action-controlled interaction system is built with a modular approach to ensure flexibility in meeting multiple task requirements and facilitating flexible interaction. In the system, a distance-based safety solution is designed to avoid collisions between humans and robots. Finally, experimental results are presented to demonstrate the feasibility and effectiveness of the proposed methods and system.

Recent Advances in Sensor-Actuator Hybrid Soft Systems: Core Advantages, Intelligent Applications, and Future Perspectives.

The growing demand for soft intelligent systems, which have the potential to be used in a variety of fields such as wearable technology and human-robot interaction systems, has spurred the development of advanced soft transducers. Among soft systems, sensor-actuator hybrid systems are considered the most promising due to their effective and efficient performance, resulting from the synergistic and complementary interaction between their sensor and actuator components. Recent research on integrated sensor and actuator systems has resulted in a range of conceptual and practical soft systems. This review article provides a comprehensive analysis of recent advances in sensor and actuator integrated systems, which are grouped into three categories based on their primary functions: i) actuator-assisted sensors for intelligent detection, ii) sensor-assisted actuators for intelligent movement, and iii) sensor-actuator interactive devices for a hybrid of intelligent detection and movement. In addition, several bottlenecks in current studies are discussed, and prospective outlooks, including potential applications, are presented. This categorization and analysis will pave the way for the advancement and commercialization of sensor and actuator-integrated systems.

Development of a Human-Robot Interaction System for Industrial Applications

The use of robots is increasing day by day. In this study, it was aimed to develop manufacturing-assistant robot software for small production plants involving non-mass production. The main purpose of this study is to eliminate the difficulty of recruiting an expert robot operator thanks to the developed software and to facilitate the use of robots for non-experts. The developed software consists of three parts: the convolutional neural network (CNN), process selection-trajectory generation, and trajectory regulation modules. Before the operations in these modules are executed, operators record the desired process and the trajectory of the process in the video by hand gestures and index finger. Then recorded video is separated into images. The separated images are classified by the CNN module and the positions of landmarks (joint and fingernail of index finger) were calculated by the same module and using the images. Eight different pre-trained CNN structures were tested in the CNN module, and the best result Xception structure (test loss = 0.0051) was used. The desired process was determined and the trajectory of the process was created with the CNN output data. The connection of the generated trajectory with the object was detected by the trajectory regulation module, and unnecessary trajectory parts were cleaned. Regulated trajectory and desired tasks such as welding or sealing were simulated via an industrial robot in a simulation environment. As a result, an industrial robot could be programmed by non-expert operators for companies whose production line is not standard by using the developed software.

Fielded Human-Robot Interaction for a Heterogeneous Team in the DARPA Subterranean Challenge

Human supervision of multiple fielded robots is a challenging task which requires a thoughtful design and implementation of both the underlying infrastructure and the human interface. It also requires a skilled human able to manage the workload and understand when to trust the autonomy, or manually intervene. We present an end-to-end system for human-robot interaction with a heterogeneous team of robots in complex, communication-limited environments. The system includes the communication infrastructure, autonomy interaction, and human interface elements. Results of the DARPA Subterranean Challenge Final Systems Competition are presented as a case study of the design and analyze the shortcomings of the system.

Special Issue on Humancentric Robotic Technology and its Applications for Coexistence with Humans

In today’s super-aging society, it is essential to maintain and improve the quality of life of the elderly and to counter the shortage of young workers. Such problems are becoming evident in several areas, not just in the manufacturing sector but also in the fields of transport/mobility and healthcare/welfare. In these fields, there is an abundance of situations that cannot be solved simply by replacing human workers with robots. In the manufacturing sector, for instance, the transmission of skills and knowledge is threatened as the members of the baby boomer generation, who have both knowledge and experience, have reached retirement age at a time when the younger population is declining. Another example is the provision of care with human warmth at nursing sites. As we evolve toward Society 5.0, which is represented by a digital transformation and which encompasses elements such as digital twins, it will be critical to pioneer robotic technologies that allow robots to work with humans to achieve a prosperous coexistence and create an affluent society. For this special issue, which features humancentric robotic systems coexisting and working together with humans, we invited researchers to submit papers covering a wide range of topics, including element technologies as well as operating and evaluation methods. This special issue on Humancentric Robotic Technology and its Applications for Coexistence with Humans includes two review papers, 29 regular papers, and two development reports that cover the following topics. • Research on rehabilitation equipment and training systems • Research on power assist and support actuators • Research on human-robot interaction systems • Research on motor characteristics and the musculoskeletal system • Research on the utilization and development of vital signs and sensors • Research on IoT technology and smart technologies We thank all authors and reviewers of the papers, as well as the Editorial Board of the Journal of Robotics and Mechatronics, for their help with this special issue.

Adaptive fuzzy sliding mode control of an actuator powered by two opposing pneumatic artificial muscles

Pneumatic artificial muscle (PAM) is a potential actuator in human–robot interaction systems, especially rehabilitation systems. However, PAM is a nonlinear actuator with uncertainty and a considerable delay in characteristics, making control challenging. This study presents a discrete-time sliding mode control approach combined with the adaptive fuzzy algorithm (AFSMC) to deal with the unknown disturbance of the PAM-based actuator. The developed fuzzy logic system has parameter vectors of the component rules that are automatically updated by an adaptive law. Consequently, the developed fuzzy logic system can reasonably approximate the system disturbance. When operating the PAM-based system in multi-scenario studies, experimental results confirm the efficiency of the proposed strategy.

Skin‐Inspired Hierarchical Structure Sensor for Ultrafast Active Human–Robot Interaction

AbstractBiomimetic sensors that mimic human skin perception have received enormous attention in human motion detection and human–robot interaction. However, the narrow working range limits the response linearity of mechanical stimuli. Especially the ultrafast active response is needed to detect and react to the potential collision. Inspired by the skin structure, a hierarchical structure with poly(ethylene oxide)‐based ionic conductive polymer electrolytes and polyurethane is designed. The observable synergistic sensing mechanism works throughout the compression process, providing the device with a continuous change in electrical response with excellent linearity (R2 = 0.998) over a broad range (100 Pa–500 kPa). The compressive deformation and corresponding mechanical behavior of hierarchical structures are dynamically and simultaneously observed by using in situ electron microscopy at the atomistic scale. Furthermore, a biomimetic skin is established in the human–robot interaction system, which can provide active and efficient collision avoidance.

Wearable sensor network for lower limb angle estimation in robotics applications

In human-robot interaction, sensors are relevant in guaranteeing stability and high performance in real-time applications. Nonetheless, accuracy and portable sensors for robots usually have high costs and little flexibility to process signals with free software. Therefore, we propose a wearable sensor network to measure lower limb angular position in human-robot interaction systems. The methodology employed to achieve the aim consisted in implementing a wireless network using low-cost devices, verifying design requirements, and making a validation via a proof of concept. The requirements to design the network include low loss of information, real-time communication, and sensor fusion to estimate the angular position using a gyroscope and accelerometer. Hence, the sensor network developed has a client-server architecture based on ESP8266 microcontrollers. In addition, this network uses the standard 802.11 b/g/n to transmit angular velocity and acceleration measures. Furthermore, we implement the user datagram protocol (UDP) protocol to operate in real-time with a sample time of 10 ms. Finally, we implement a proof of concept to show the system’s effectiveness. Thus, we use the Kalman filter to estimate the angular position of the foot, shin, thigh, and hip. Results indicate that the implemented sensor network is suitable for real-time robotic applications.

An AR-assisted Deep Reinforcement Learning-based approach towards mutual-cognitive safe human-robot interaction

With the emergence of Industry 5.0, the human-centric manufacturing paradigm requires manufacturing equipment (robots, etc.) interactively assist human workers to deal with dynamic and complex production tasks. To achieve symbiotic human–robot interaction (HRI), the safety issue serves as a prerequisite foundation. Regarding the growing individualized demand of manufacturing tasks, the conventional rule-based safe HRI measures could not well address the safety requirements due to inflexibility and lacking synergy. To fill the gap, this work proposes a mutual-cognitive safe HRI approach including worker visual augmentation, robot velocity control, Digital Twin-enabled motion preview and collision detection , and Deep Reinforcement Learning-based robot collision avoidance motion planning in the Augmented Reality-assisted manner. Finally, the feasibility of the system design and the performance of the proposed approach are validated by establishing and executing the prototype HRI system in a practical scene. • Introduced an AR-assisted system architecture for mutual-cognitive safe HRI. • Implemented distance-based robot velocity control and area-based workers’ visual aids functions. • Developed the robot DT-enabled motion preview for workers to enhance safe cognition and for robots to detect collision. • Proposed a curriculum learning-based DRL motion planning policy for collision avoidance.

HAbot: a human-centered augmented reality robot programming method with the awareness of cognitive load.

In the era of Industry 4.0, manufacturing enterprises are actively adopting collaborative robots (Cobots) in their productions. Current online and offline robot programming methods are difficult to use and require extensive experience or skills. On the other hand, the manufacturing industries are experiencing a labor shortage. An essential question, therefore, is: how would a new robot programming method help novice users complete complex tasks effectively, efficiently, and intuitively? To answer this question, we proposed HAbot, a novel human-centered augmented reality programming interface with awareness of cognitive load. Using NASA's system design theory and the cognitive load theory, a set of guidelines for designing an AR-based human-robot interaction system is obtained through a human-centered design process. Based on these guidelines, we designed and implemented a human-in-the-loop workflow with features for cognitive load management. The effectiveness and efficiency of HAbot are verified in two complex tasks compared with existing online programming methods. We also evaluated HAbot quantitatively and qualitatively through a user study with 16 participants. According to the user study, compared with existing methods, HAbot has higher efficiency, a lower overall cognitive load, lower cognitive loads for each type, and higher safety.

An Intelligent Rehabilitation Robot With Passive and Active Direct Switching Training: Improving Intelligence and Security of Human–Robot Interaction Systems

An Intelligent Rehabilitation Robot With Passive and Active Direct Switching Training: Improving Intelligence and Security of Human–Robot Interaction Systems