Human-robot Interaction Environment Research Articles

Temporal Fusion of Time-Distributed VGG-16 and LSTM for Precise Action Recognition in Video Sequences

Autonomous systems play a vital role in creating an ambient environment, but for comprehensive human-robot interaction, the system must interpret and identify specific human behaviours. The demand for recognizing human activity from videos has significantly increased across diverse domains, such as medicine, intelligent video surveillance, human-robot interaction, and ambient-assisted environments. Deep learning has emerged as a pioneering technique for human activity recognition from videos due to its ability to learn features from input videos automatically. This research proposes a novel approach for recognizing activities in videos by leveraging the combined power of the VGG-16 convolutional neural network (CNN) and the Long Short-Term Memory (LSTM). To assess the efficiency of our suggested methodology, experiments are conducted on the widely used benchmark dataset UCF-50. The experimental results demonstrate that our VGG-16 and LSTM-based model outperforms state-of-the-art methods for activity recognition, achieving higher accuracy and robustness across various activity categories.

Equivalent-input-disturbance rejection-based adaptive motion control for pneumatic artificial muscle arms via hysteresis compensation models

With outstanding flexibility, pneumatic artificial muscles (PAMs) have shown unique advantages in human–robot interaction environments. However, asymmetric hysteresis nonlinearities, unknown transient or persistent time-varying disturbances, etc., seriously degrade control performance/robustness of PAM arms, and increase actuation failure risks. Meanwhile, existing disturbance rejection methods either rarely handle matched/unmatched disturbances, or require known disturbance dynamics. These defects limit the disturbance rejection ability in practical applications. To this end, a new adaptive output feedback control method is proposed in this article, which achieves efficient positioning and tracking control of PAM arms. Without requiring a priori knowledge of inverse plant models and disturbance upper bounds, the proposed method further enhances robustness against uncertainties and disturbances. Specifically, a generalized Bouc–Wen hysteresis loop is used to describe the asymmetric dynamic force of PAMs. Also, a new adaptive law is designed to compensate for the linearized parameters online. In particular, by applying a modified state observer to derive the equivalent-input-disturbance estimation solution, this article firstly realizes state observation, parameter estimation, and unknown transient/persistent external disturbance rejection simultaneously. The rigorous stability analysis demonstrates that the tracking errors converge to small neighborhoods of the origins. Finally, hardware experimental results with comparisons validate the effectiveness and robustness superiority of the proposed method.

A Gaussian Mixture Model Based Fast Motion Planning Method Through Online Environmental Feature Learning

Rapidly-exploring Random Tree (RRT) and its applications for mobile robot navigation have attracted extensive interest in human-robot interaction environments. However, most current sampling-based motion planning algorithms are easy to fall into the trap space in complex and dynamic environments. In this paper, a fast motion planning algorithm based on Gaussian Mixture Model is proposed to allow robotic systems to efficiently extract feature nodes and generate cheuristic paths considering the pedestrian density and the environmental structure. Firstly, the preliminary feature nodes are obtained by training GMM, in which the covariance matrix of each feature node contains the environmental structure information. Secondly, an online feature learning method is presented to extract features during the movement of the robot by the dual-channel scale filter and the secondary distance fusion. Once the free grids can be connected to the feature nodes without any collision, a heuristic path is generated directly to guide the robot to avoid the crowds according to the proposed pedestrian matrix. For the same environment, the learned information can be reused as prior knowledge in the next path planning to further improve the efficiency of re-planning. A series of experimental studies demonstrate that our proposed method can remarkably reduce the time and enhance the success rate of navigation in trapped and narrow environments.

DL-DARE: Deep learning-based different activity recognition for the human–robot interaction environment

DL-DARE: Deep learning-based different activity recognition for the human–robot interaction environment

Enhanced extended state observer-based model-free force control for a Series Elastic Actuator

With inherent mechanical compliance and programmable force controllability, the Series Elastic Actuator (SEA) has been a popular choice for modern mechanical systems. It is crucial to realize accurate and robust force control for a SEA, especially in a physical human–robot interaction condition. In this paper, a model-free force controller (MFFC) for the SEA is developed, which relies only on the input–output measurement. Firstly, based on the ultra-local model concept, the SEA force dynamics is reformulated in a simpler way, and all unknown terms are grouped as one term called lumped total disturbance. Secondly, an enhanced extended stated observer (EESO) is designed to simultaneously estimate the lumped total disturbance, the force velocity, and the control input gain. Thirdly, a disturbance rejective intelligent PD (iPD) control scheme is established. In addition, the effects of the control input gain are analyzed and the close-loop stability is demonstrated with Lyapunov method. The control accuracy and robustness of the MFFC are verified in a physical human–robot interaction environment.

A Soft Self-Stable Actuator and Its Energy-Efficient Grasping

Due to high compliance and adaptiveness, soft robots show more advantages than traditional rigid robots in grasping irregularly shaped or fragile objects. Moreover, soft robots attract increasing attention as more and more robots are adopted in unstructured or human–robot interaction environments. However, during the grasping process, most soft robots need constant input stimuli to maintain the grasping configurations, which directly induces low energy efficiency. This article demonstrates the self-stable property of the dual-chamber structure, which can maintain bending deformation without input stimuli based on the differential pressure of the two chambers. Based on the self-stable property, a soft self-stable actuator (SSSA) driven by the tendon is proposed, and it can maintain the bending deformation without energy consumption, which can improve the energy efficiency of the actuator significantly. The self-stability of the actuator during the restoring period is analyzed, and the energy consumption model of the actuator is developed. Finally, the experiments are conducted to verify the validity of the models. The results show the actuator can achieve self-stability without any energy input, and the energy consumption can be reduced by about 45%. Hence, the proposed actuator can be adopted in applications where energy efficiency is sensitive.

Feature-similarity network via soft-label training for infrared facial emotional classification in human-robot interaction

Facial emotional classification (FEC) is a challenging issue in human-robot interaction. In this study, we proposed a novel FEC network with feature similarity-based soft label training. The proposed model aims at addressing the problems of occlusion and label ambiguity. It argues that the mouth region is very important for FEC task. To this end, all the facial emotional images are firstly used to extract the infrared facial features by the trained ResNet networks. The facial features similarities are computed by the cosine function. The emotional labels can be constructed by the similarities values. Then, Kullback-Leibler distance is employed to measure the difference between the groundtruth label and predicted label. Extension experimental results on two public datasets demonstrate that the proposed model can obtain state-of-the-art performance compared with the existing studies. Furthermore, we have applied proposed model on the assisted driving scene and human robot interaction environment.

Research on Real-Time Joint Stiffness Configuration of a Series Parallel Hybrid 7-DOF Humanoid Manipulator in Continuous Motion

In this paper, the real-time joint stiffness configuration strategy of a series parallel hybrid 7-DOF (degree of freedom) humanoid manipulator with flexible joints in continuous motion is studied. Firstly, considering the potential human robot accidental collision, combined with the manipulator safety index (MSI) and human body injury thresholds, the motion speed and joint stiffness of the robot are optimized in advance. Secondly, using hyperbolic tangent function for reference, the relationship between joint torques and passive joint deflection angles of the robot is given, which is beneficial for the real-time calculation of joint stiffness and obtain reasonable joint stiffness. Then, the structural model of the selected humanoid manipulator is described, on this basis, the relationship between the joint space stiffness and the Cartesian space stiffness of the humanoid manipulator is analyzed through Jacobian matrix, and the results show that the posture and joint space stiffness of the humanoid manipulator directly affect the Cartesian space stiffness of the humanoid manipulator. Finally, according to whether the humanoid manipulator works in the human-robot interaction environment, the real-time joint stiffness configuration of the humanoid manipulator in continuous motion is simulated and analyzed. The research shows that the humanoid manipulator with flexible joints can adjust the joint stiffness in real-time during continuous motion, and the joint stiffness configuration strategy can effectively improve the safety of human body in human-robot collision. In addition, in application, when the joint space stiffness of the robot is lower, the position accuracy can be improved by trajectory compensation.

Design of Muscle Reflex Control for Upright Standing Push-Recovery Based on a Series Elastic Robot Ankle Joint.

In physical human–robot interaction environment, ankle joint muscle reflex control remains significant and promising in human bipedal stance. The reflex control mechanism contains rich information of human joint dynamic behavior, which is valuable in the application of real-time decoding motion intention. Thus, investigating the human muscle reflex mechanism is not only meaningful in human physiology study but also useful for the robotic system design in the field of human–robot physical interaction. In this paper, a specialized ankle joint muscle reflex control algorithm for human upright standing push-recovery is proposed. The proposed control algorithm is composed of a proportional-derivative (PD)-like controller and a positive force controller, which are employed to mimic the human muscle stretch reflex and muscle tendon force reflex, respectively. Reflex gains are regulated by muscle activation levels of contralateral ankle muscles. The proposed method was implemented on a self-designed series elastic robot ankle joint (SERAJ), where the series elastic actuator (SEA) has the potential to mimic human muscle–tendon unit (MTU). During the push-recovery experimental study, the surface electromyography (sEMG), ankle torque, body sway angle, and velocity of each subject were recorded in the case where the SERAJ was unilaterally kneed on each subject. The experimental results indicate that the proposed muscle reflex control method can easily realize upright standing push-recovery behavior, which is analogous to the original human behavior.

Manipulator residual estimation and its application in collision detection

PurposeThis paper aims to present a method to improve the sensitive and low probabilities of false alarm of a manipulator in a human–robot interaction environment, which can improve the performance of the system owing to non-linear uncertainty in the model of the robot controller.Design/methodology/approachA novel collision detection method based on adaptive residual estimation is proposed, promoting the detection accuracy of the collision of the manipulator during operation. First, a general momentum residual estimator is designed to incorporate the non-linear factors of the manipulator (e.g. joint friction, speed and acceleration) into the residual-related uncertainty of the model. Second, model parameters are estimated through gradient correction. The residual filter is used to determine the dynamic threshold, resulting in higher detection accuracy. Finally, the performance of the residual estimation scheme is evaluated by comparing the dynamic threshold with residual in real-time experiments where a single Universal Robot 5 robot end–effector collides with the obstacle.FindingsExperimental results demonstrate that the collision detection system can improve sensitivity and lead to low probabilities of false alarm of non-linear uncertainty in the model.Practical implicationsThe method proposed in this article can be applied to industry and human–robot interaction area.Originality/valueAn adaptive collision detection method is proposed in this paper to address non-linear uncertainties of the model in industrial application.

Active Learning Environments with Robotic Tangibles: Children's Physical and Virtual Spatial Programming Experiences

As computational thinking becomes increasingly important for children to learn, we must develop interfaces that leverage the ways that young children learn to provide opportunities for them to develop these skills. Active Learning Environments with Robotic Tangibles (ALERT) and Robopad, an analogous on-screen virtual spatial programming environment for educational Human Robot Interaction (HRI), have been developed. Evaluations of these in the context of free play and open-ended learning activities show that both systems afford opportunities for young children to engage in spatial programming, creating improvisational and sequential programs that mediate interactions between the environment, robots, and humans in responsive and creative ways. These systems demonstrate innovative opportunities for advancing mixed reality spatial programming activities as a form of HRI that fosters engaging seamless cyberlearning experiences, across formal and informal environments.

Robust Action Selection by the Robot in Human-Robot Interaction (HRI) Environment

Robust Action Selection by the Robot in Human-Robot Interaction (HRI) Environment

Detection of movement intention using EEG in a human-robot interaction environment

Introduction : This paper presents a detection method for upper limb movement intention as part of a brain-machine interface using EEG signals, whose final goal is to assist disabled or vulnerable people with activities of daily living. Methods EEG signals were recorded from six naive healthy volunteers while performing a motor task. Every volunteer remained in an acoustically isolated recording room. The robot was placed in front of the volunteers such that it seemed to be a mirror of their right arm, emulating a Brain Machine Interface environment. The volunteers were seated in an armchair throughout the experiment, outside the reaching area of the robot to guarantee safety. Three conditions are studied: observation, execution, and imagery of right arm’s flexion and extension movements paced by an anthropomorphic manipulator robot. The detector of movement intention uses the spectral F test for discrimination of conditions and uses as feature the desynchronization patterns found on the volunteers. Using a detector provides an objective method to acknowledge for the occurrence of movement intention. Results When using four realizations of the task, detection rates ranging from 53 to 97% were found in five of the volunteers when the movement was executed, in three of them when the movement was imagined, and in two of them when the movement was observed. Conclusions Detection rates for movement observation raises the question of how the visual feedback may affect the performance of a working brain-machine interface, posing another challenge for the upcoming interface implementation. Future developments will focus on the improvement of feature extraction and detection accuracy for movement intention using EEG data.

Risked-based design for the physical human-robot interaction (pHRI): an overview

Collaboration and sharing workspace between human and robots has turned into a challenge that has to be taken into account in many industrial domains, including the process industry. In this study, literature related to risk-based safety system design is reviewed as well as relevant research of control design for safety issues. The paper mainly presents two key contributions. 1) planning for safety: risk-based safety analysis methodologies are reviewed to analyze current methods for identifying the potential risks caused by abnormal operation and failures in the human-robot interaction environment. The risk assessment methodologies are including not only functional risk analysis, but also human reliability analysis. The results from risk evaluation will help making HRI safety strategies which are series decision makings to support the further engineering design of a pHRI safety system. 2) Control design system for achieving safety: to reach a comprehensive safety system design and limit accidents in the human-robot interaction environment, four scenarios are taken into consideration: proximity detection, collision avoidance, docking and compliance control. Current control techniques in each domain to guarantee the safety of system are reviewed. Finally, most common used methods in the above-mentioned areas are introduced and their performance is discussed.

목적성 행동 모방학습을 통한 의도 인식을 위한 거울뉴런 시스템 계산 모델

The understanding of another's behavior is a fundamental cognitive ability for primates including humans. Recent neuro-physiological studies suggested that there is a direct matching algorithm from visual observation onto an individual's own motor repertories for interpreting cognitive ability. The mirror neurons are known as core regions and are handled as a functionality of intent recognition on the basis of imitative learning of an observed action which is acquired from visual-information of a goal-directed action. In this paper, we addressed previous works used to model the function and mechanisms of mirror neurons and proposed a computational model of a mirror neuron system which can be used in human-robot interaction environments. The major focus of the computation model is the reproduction of an individual's motor repertory with different embodiments. The model's aim is the design of a continuous process which combines sensory evidence, prior task knowledge and a goal-directed matching of action observation and execution. We also propose a biologically inspired plausible equation model.

Development of a Simulated Environment for Human-Robot Interaction

Skenario interaksi manusia-robot adalah sangat rumit dan memerlukan definisi yang tepat dari variabel lingkungan untuk pengujian yang ketat aspek-aspek berbeda perilaku robot. Kondisi lingkungan mempengaruhi perilaku manusia dan robot, sehingga manusia dan robot merespon berbeda pada kondisi akustik atau pencahayaan berbeda. Pada penelitian ini dilakukan eksperimen berulang-ulang dengan manusia sebagai subjek pengujian yang menyebabkan perubahan perilaku pada mereka, sehingga responnya tidak ada yang mirip dengan percobaan yang sudah dilakukan. Karenanya, membuat tidak mungkin untuk melakukan skenario interaksi secara berulang. Pengembangan dan penggunaan simulasi 3D, di mana parameter yang berbeda dapat disesuaikan, adalah solusi yang paling menguntungkan dalam kasus tersebut. Ini tidak hanya menuntut pengembangan robot simulasi yang berbeda tetapi juga simulasi dari lingkungan yang dinamis termasuk mitra interaksi. Pada makalah ini dihadirkan kerangka simulasi yang memungkinkan simulasi interaksi manusia dengan robot termasuk mitra interaksi tersimulasi dan dinamikanya.

Recognition of Affect Based on Gait Patterns

To provide a means for recognition of affect from a distance, this paper analyzes the capability of gait to reveal a person's affective state. We address interindividual versus person-dependent recognition, recognition based on discrete affective states versus recognition based on affective dimensions, and efficient feature extraction with respect to affect. Principal component analysis (PCA), kernel PCA, linear discriminant analysis, and general discriminant analysis are compared to either reduce temporal information in gait or extract relevant features for classification. Although expression of affect in gait is covered by the primary task of locomotion, person-dependent recognition of motion capture data reaches 95% accuracy based on the observation of a single stride. In particular, different levels of arousal and dominance are suitable for being recognized in gait. It is concluded that gait can be used as an additional modality for the recognition of affect. Application scenarios include monitoring in high-security areas, human-robot interaction, and cognitive home environments.

Following Controller for Autonomous Mobile Robots Using Behavioral Cues

This paper proposes an autonomous-robot following controller that can integrate information provided from behavioral cues of the leader to increase the reliability and the performance of following. The controller continuously estimates the future predicted position of the leader as it moves, and then directs the follower robot to this position. A Kalman filter is employed for an estimation that uses vision-based measurements of leader position, a dynamic model of the leader, and a behavioral-cue model of the leader. The behavioral-cue model serves to either tune the dynamic model and/or create pseudomeasurements to further help the Kalman filter estimate the leader's future position. Once the leader's future position is estimated, a trajectory planner plans a path to the future position, and a motor controller implements the required control signals to the robot wheels. It is suggested that this controller may have particular importance for human following by autonomous robots in future human-robot interaction environments.