Humanoid Locomotion Research Articles

Real-world humanoid locomotion with reinforcement learning.

Humanoid robots that can autonomously operate in diverse environments have the potential to help address labor shortages in factories, assist elderly at home, and colonize new planets. Although classical controllers for humanoid robots have shown impressive results in a number of settings, they are challenging to generalize and adapt to new environments. Here, we present a fully learning-based approach for real-world humanoid locomotion. Our controller is a causal transformer that takes the history of proprioceptive observations and actions as input and predicts the next action. We hypothesized that the observation-action history contains useful information about the world that a powerful transformer model can use to adapt its behavior in context, without updating its weights. We trained our model with large-scale model-free reinforcement learning on an ensemble of randomized environments in simulation and deployed it to the real-world zero-shot. Our controller could walk over various outdoor terrains, was robust to external disturbances, and could adapt in context.

Enhancement of humanoid robot locomotion on slippery floors using an adaptive controller

AbstractThis paper presents a comprehensive strategy to improve the locomotion performance of humanoid robots on various slippery floors. The strategy involves the implementation and adaptation of a divergent component of motion (DCM) based control architecture for the humanoid NAO, and the introduction of an embedded yaw controller (EYC), which is based on a proportional-integral-derivative (PID) control algorithm. The EYC is designed not only to address the slip behavior of the robot on low-friction floors but also to tackle the issue of non-straight walking patterns that we observed in this humanoid, even on non-slippery floors. To fine-tune the PID gains for the EYC, a systematic trial-and-error approach is employed. We iteratively adjusted the P (Proportional), I (Integral), and D (Derivative) parameters while keeping the others fixed. This process allowed us to optimize the PID controller’s response to different walking conditions and floor types. A series of locomotion experiments are conducted in a simulated environment, where the humanoid step frequency and PID gains are varied for each type of floor. The effectiveness of the strategy is evaluated using metrics such as robot stability, energy consumption, and task duration. The results of the study demonstrate that the proposed approach significantly improves humanoid locomotion on different slippery floors, by enhancing stability and reducing energy consumption. The study has practical implications for designing more versatile and effective solutions for humanoid locomotion on challenging surfaces and highlights the adaptability of the existing controller for different humanoid robots.

Data-Efficient and Safe Learning for Humanoid Locomotion Aided by a Dynamic Balancing Model

In this letter, we formulate a novel Markov Decision Process (MDP) for safe and data-efficient learning for humanoid locomotion aided by a dynamic balancing model. In our previous studies of biped locomotion, we relied on a low-dimensional robot model, commonly used in high-level Walking Pattern Generators (WPGs). However, a low-level feedback controller cannot precisely track desired footstep locations due to the discrepancies between the full order model and the simplified model. In this study, we propose mitigating this problem by complementing a WPG with reinforcement learning. More specifically, we propose a structured footstep control method consisting of a WPG, a neural network, and a safety controller. The WPG provides an analytical method that promotes efficient learning while the neural network maximizes long-term rewards, and the safety controller encourages safe exploration based on step capturability and the use of control-barrier functions. Our contributions include the following (1) a structured learning control method for locomotion, (2) a data-efficient and safe learning process to improve walking using a physics-based model, and (3) the scalability of the procedure to various types of humanoid robots and walking.

Dynamic model for using casterboard by a humanoid robot

ABSTRACTThe efficacy of traveling and movement is substantially enhanced by the use of wheel mechanisms, particularly when compared to bipedal locomotion. Humanoid locomotion can also be extended using appropriate tools. In this study, we focus on the casterboard as a tool that facilitates traveling, because it has comparatively high portability, and the rider can move forward without pushing off the ground. We present a dynamic model for the use of the casterboard in a humanoid robot. Given that the ankles and waist of the rider play a key role in traveling using the casterboard, the angles of those joints are utilized as the input of the proposed model. A simplified small-scale robot is used to determine essential elements, and a modified casterboard is employed to eliminate the risk of falling without loss of its unique properties, for verification of the dynamic model. Using simulations and experiments, it is demonstrated that the dynamic model effectively emulates the behavior of the actual setup with respect to forward movement and turning. The model serves as the basis for the planning of a reference trajectory using the casterboard.

Active Balance Control of Humanoid Locomotion Based on Foot Position Compensation

A foot positioning compensator is developed in this paper for a full-body humanoid to retrieve its balance during continuous walking. An online Foot Position Compensator (FPC) is designed to improve the robustness of biped walking, which can modify predefined step position and step duration online with sensory feedback. Foot placement parameters are learned by the FPC based on the Policy Gradient Reinforcement Learning (PGRL) method. Moreover, the FPC assists the humanoid robot in rejecting external disturbances and recovering the walking position by re-planning the trajectories of walking pattern and the Center of Mass (CoM). An upper body pose control strategy is also presented to further enhance the performance of humanoid robots to overcome strong external disturbances. The advantages of this proposed method are that it neither requires prior information about the walking terrain conditions, nor relies on range sensor information for surface topology measurement. The effectiveness of the proposed method is verified via Webots simulation and real experiments on a full-body humanoid NAO robot.

Perfect tracking of ZMP trajectory for humanoid locomotion using repetitive control

In this paper, the tracking of the zero moment point (ZMP) trajectory for humanoid locomotion is addressed. For that, a combination of a repetitive controller (RC) and a linear quadratic regulator (LQR) is proposed. The RC controller achieves a zero steady state tracking error for any periodic desired output with a fixed period. Here, the robot model is simplified as a cart table model. In the first stage, a walking pattern with constant gait is considered to allow the use of a repetitive controller. The results are then compared to a classical PD control. In the second part, the proposed controller is applied to track a desired ZMP profile generated using a central pattern generator (CPG). Here, the speed of the oscillator is adjusted via a proper selection of the parameters of the CPG and the RC design is simultaneously updated to assure the perfect tracking of the ZMP.

Humanoid Locomotion Control and Generation Based on Contact Wrench Cones

This paper presents a general framework for locomotion control and generation of humanoid robots. Different from most of the existing work which uses the zero-moment point (2mp) to determine the feasibility of robot’s motion, we use the so-called contact wrench cone to derive motion feasibility conditions, whole-body motion controllers, and locomotion generators. The contact wrench cone consists of all feasible wrenches that can be applied to the robot through contacts, which provide allowable external forces and moments for realizing the robot’s motion. Algorithms are proposed to compute quantities defined on linear representations of a general convex cone, which can be various contact wrench cones as needed in developing motion generators and controllers. Based on the contact wrench cone for contact links and the proposed algorithms as well as a decomposition of the whole-body dynamics of a floating-base humanoid robot, we derive two motion tracking controllers. One controller contains a single quadratic program with linear inequality constraints, while the other consists of two quadratic programs which can be quickly solved by one of the proposed algorithms and in a closed form, respectively. Both controllers can be applied in real-time and achieve similar motion tracking performance in simulation. Based on contact wrench cones, furthermore, we derive two motion generation methods for humanoid robots. The first method adapts a reference motion, most often infeasible, to the robot by warping the motion’s time line so that the motion trajectory will remain the same but the velocity and acceleration profiles will be changed. The second method generates bipedal locomotion for given footsteps. All the proposed motion controllers and generators are applicable to general scenarios including uneven terrains and motions with the support of other links besides feet.

Comparative Study of NeuroEvolution Algorithms in Reinforcement Learning for Self-Driving Cars

Neuroevolution has been used to train neural networks for challenging deep Reinforcement Learning (RL) problems like Atari, image hard maze, and humanoid locomotion. The performance is comparable to the performance of neural networks trained by algorithms like Q-learning and policy gradients. This work conducts a detailed comparative study of using neuroevolution algorithms in solving the self-driving car problem. Different neuroevolution algorithms are used to train deep neural networks to predict the steering angle of a car in a simulated environment. Neuroevolution algorithms are compared to the Double Deep Q-Learning (DDQN) algorithm. Based on the experimental results, the neuroevolution algorithms show better performance than DDQN algorithm. The Evolutionary Strategies (ES) algorithm outperforms the rest in accuracy in driving in the middle of the lane, with the best average result of 97.13%. Moreover, the Random Search (RS) algorithm outperforms the rest in terms of driving the longest while keeping close to the middle of the lane, with the best average result of 403.54m. These results confirm that the entire family of genetic and evolutionary algorithms with all their performance optimization techniques, are available to train and develop self driving cars.

When joggers meet robots: the past, present, and future of research on humanoid robots

Spawned by fast-paced progress in new materials and integrate circuit technology, the past two decades have witnessed tremendous development of humanoid robots for both scientific and commercial purposes, e.g. emergency response and daily life assistant. At the root of this trend are the increasing research interests and cooperation opportunities across different laboratories and countries. The application-driven requirements of high effectiveness and reliability of humanoid robots led intensive research and development in humanoid locomotion and control theories. In spite of the progress in the area, challenges such as unnatural locomotion control, inefficient multi-motion planning, and relatively slow disturbances recovery set further requirements for the next generation of humanoid robots. Therefore, the purpose of this work is to review the current development of highly representative bipedal humanoid robots and discuss the potential to move the ideas and models forward from laboratory settings into the real world. To this end, we also review the current clinical understanding of the walking and running dynamics to make the robot more human-like.

Bifurcation analysis of a two-dimensional discrete Hindmarsh\u2013Rose type model

In this paper, bifurcation analysis of a discrete Hindmarsh–Rose model is carried out in the plane. This paper shows that the model undergoes a flip bifurcation, a Neimark–Sacker bifurcation, and 1:2 resonance which includes a pitchfork bifurcation, a Neimark–Sacker bifurcation, and a heteroclinic bifurcation. The sufficient conditions of existence of the fixed points and their stability are first derived. The flip bifurcation and Neimark–Sacker bifurcation are analyzed by using the inner product method and normal form theory. The conditions for the occurrence of 1:2 resonance are also presented. Furthermore, the sufficient conditions of pitchfork, Neimark–Sacker, and heteroclinic bifurcations are derived and expressed by implicit functions. The numerical analysis shows us consistence with the theoretical results and exhibits interesting dynamics, especially symmetric and invariant closed orbits. The dynamics observed in this paper can be used to mimic the dynamical behaviors of one single neuron and design a humanoid locomotion model for applications in bio-engineering and so on.

Continuous Control Monte Carlo Tree Search Informed by Multiple Experts.

Efficient algorithms for 3D character control in continuous control setting remain an open problem in spite of the remarkable recent advances in the field. We present a sampling-based model-predictive controller that comes in the form of a Monte Carlo tree search (MCTS). The tree search utilizes information from multiple sources including two machine learning models. This allows rapid development of complex skills such as 3D humanoid locomotion with less than a million simulation steps, in less than a minute of computing on a modest personal computer. We demonstrate locomotion of 3D characters with varying topologies under disturbances such as heavy projectile hits and abruptly changing target direction. In this paper we also present a new way to combine information from the various sources such that minimal amount of information is lost. We furthermore extend the neural network, involved in the algorithm, to represent stochastic policies. Our approach yields a robust control algorithm that is easy to use. While learning, the algorithm runs in near real-time, and after learning the sampling budget can be reduced for real-time operation.

Humanoid gait generation in complex environments based on template models and optimality principles learned from human beings

In this paper, we present an inverse optimal control-based transfer of motions from human experiments to humanoid robots and apply it to walking in constrained environments. To this end, we introduce a 3D template model, which describes motion on the basis of center-of-mass trajectory, foot trajectories, upper-body orientation, and phase duration. Despite its abstract architecture, with prismatic joints combined with damped series elastic actuators instead of knees, the model (including dynamics and constraints) is suitable for describing both human and humanoid locomotion with appropriate parameters. We present and apply an inverse optimal control approach to identify optimality criteria based on human motion capture experiments. The identified optimal strategy is then transferred to a humanoid robot template model for gait generation by solving an optimal control problem, which takes into account the properties of the robot and differences in the environment. The results of this step are the center-of-mass trajectory, the foot trajectories, the torso orientation, and the single and double support phase durations for a sequence of steps, allowing the humanoid robot to walk within a new environment. In a previous paper, we have already presented one computational cycle (from motion capture data to an optimized robot template motion) for the example of walking over irregular stepping stones with the aim of transferring the motion to two very different humanoid robots (iCub@Heidelberg and HRP-2@LAAS). This study represents an extension, containing an entirely new part on the transfer of the optimized template motion to the iCub robot by means of inverse kinematics in a dynamic simulation environment and also on the real robot.

Dynamic Humanoid Locomotion: A Scalable Formulation for HZD Gait Optimization

Hybrid zero dynamics (HZD) has emerged as a popular framework for dynamic walking but has significant implementation difficulties when applied to the high degrees of freedom humanoids. The primary impediment is the process of gait design—it is difficult for optimizers to converge on a viable set of virtual constraints defining a gait. This paper presents a methodology that allows for fast and reliable generation of dynamic robotic walking gaits through the HZD framework, even in the presence of underactuation. Specifically, we describe an optimization formulation that builds upon the novel combination of HZD and direct collocation methods. Furthermore, achieving a scalable implementation required developing a defect-variable substitution formulation to simplify expressions, which ultimately allows us to generate compact analytic Jacobians of the constraints. We experimentally validate our methodology on an underactuated humanoid, DURUS, a spring-legged machine designed to facilitate energy-economical walking. We show that the optimization approach, in concert with the HZD framework, yields dynamic and stable walking gaits in hardware with a total electrical cost of transport of 1.33.

Stable and fast model-free walk with arms movement for humanoid robots

Controlling a humanoid robot with its typical many degrees of freedom is a complex task, and many methods have been proposed to solve the problem of humanoid locomotion. In this work, we generate a...

Robust trajectory optimization under frictional contact with iterative learning

Optimization is often difficult to apply to robots due to the presence of errors in model parameters, which can cause constraints to be violated during execution on the robot. This paper presents a method to optimize trajectories with large modeling errors using a combination of robust optimization and parameter learning. In particular it considers the context of contact modeling, which is highly susceptible to errors due to uncertain friction estimates, contact point estimates, and sensitivity to noise in actuator effort. A robust time-scaling method is presented that computes a dynamically-feasible, minimum-cost trajectory along a fixed path under frictional contact. The robust optimization model accepts confidence intervals on uncertain parameters, and uses a convex parameterization that computes dynamically-feasible motions in seconds. Optimization is combined with an iterative learning method that uses feedback from execution to learn confidence bounds on modeling parameters. It is applicable to general problems with multiple uncertain parameters that satisfy a monotonicity condition that requires parameters to have conservative and optimistic settings. The method is applied to manipulator performing a “waiter” task, on which an object is moved on a carried tray as quickly as possible, and to a simulated humanoid locomotion task. Experiments demonstrate this method can compensate for large modeling errors within a handful of iterations.

Team THOR's Entry in the DARPA Robotics Challenge Finals 2015

This paper describes Team THOR's approach to human-in-the-loop disaster response robotics for the 2015 DARPA Robotics Challenge (DRC) Finals. Under the duress of unpredictable networking and terrain, fluid operator interactions and dynamic disturbance rejection become major concerns for effective teleoperation. We present a humanoid robot designed to effectively traverse a disaster environment while allowing for a wide range of manipulation abilities. To complement the robot hardware, a hierarchical software foundation implements network strategies that provide real-time feedback to an operator under restricted bandwidth using layered user interfaces. Our strategy for humanoid locomotion includes a backward-facing knee configuration paired with specialized toe and heel lifting strategies that allow the robot to traverse difficult surfaces while rejecting external perturbations. With an upper body planner that encodes operator preferences, predictable motion plans are executed in unforeseen circumstances. These plans are critical for manipulation in unknown environments. Our approach was validated during the DRC Finals competition, where Team THOR scored three points in 18 min of operation time, and the results are presented along with an analysis of each task.

Bidirectional association of joint angle trajectories for humanoid locomotion: the restricted Boltzmann machine approach

Humans walk is the combination of different discrete subphases. The joint trajectories (ankle, hip, and knee) are different in every instance for human walking. So the association of these trajectories is difficult for stable walking. This research presents a novel approach to associate the joint trajectories of humanoid walking for every instance and different speeds. It is the first attempt to extract the features from the joint angle trajectories with the help of a restricted Boltzmann machine. The extracted features of joint trajectories are associated with bidirectional associative memory. The proposed approach has several advantages, since it is possible to associate the most appropriate trajectory when the input trajectory is not present in the supervised data. The BAM provides the most associative significant trajectories nearest to the input trajectory. This novel approach is well suited for real-time scenarios of stable walking of humanoid robots. By providing only one sensory data (trajectory) at any of joints among hip, knee, and ankle, the another trajectories can be associated that is most important application of this research. The indigenous data set has been generated for the experiments, and the results are verified with real-time and simulation environment of NaO and HOAP2 biped robots.

An optimized feature selection technique based on incremental feature analysis for bio-metric gait data classification

The classification of humanoid locomotion is a troublesome exercise because of non-linearity associate with gait signals. The classification using the different machine learning technique leads for over fitting and under fitting. To select the optimized feature is a difficult task. The high dimension feature vector requires a high computational cost. The hand craft feature selection machine learning techniques performed poor. We have used the incremental feature selection strategy for feature selection. In this paper we first selected the feature and identify the principle feature then we classify gait data using different machine learning technique (KNN, ANN, SVM, DNN and classifier fusion) and shown the performance comparison. Experimental result on real time datasets propose method is better than previous method as far as humanoid locomotion classification is concerned and the generalization accuracy provided by new feature selection method i.e. incremental feature selection (IFS) with analysis of variance (ANOVA) (Zhang et al., 20). During the feature extraction, 17 features were selected from the existing literatures (Wang et al. in IEEE Trans Circ Syst Video Technol 14(2):149–158, 15). Using all the features could lead to over fitting, information redundancy and dimension disaster. Thus, a system with optimal features was selected using ANOVA combined with IFA. These selected features were then fed as an input to the ANN, SVM, KNN and DNN model. These individual classifiers were combined to produce classifier fusion model. The 5-fold cross-validation was used to evaluate the performance of the proposed model. Based on the empirical results it may be concluded that classifier fusion provides satisfactory results (92.23 %) compared to other individual classifiers. One-way analysis of variance test, Friedman’s test and Kruskal-Wallis test has also been conducted to validate the statistical significance of the results. The proposed system can be used as recommender system based on behavioral gait pattern about the performance of player of Indian cricket team, Biometric and help to diagnosis Parkinson disease.

Component based computational model for bipedal locomotion

Bipedal locomotion has been an active area of research for many decades, it has wide ranging applications in the field of humanoid locomotion, as well as in the understanding of the biomechanics of normal human gait. Inherently human gait is a complex non-linear dynamic system, which is usually modeled by a set of differential equations satisfying a given set of constraints. In this paper an attempt has been made to view gait from the perspective of software engineering. In doing so, the entire gait cycle has been discretized into phases and sub-phases and modeled using a hybrid automaton, subsequently the automaton has been integrated with the BIP(Behavior, Interaction, Priority) framework, thereby creating a component based computational framework for modeling biped locomotion. The correctness of the developed model has been validated and verified through simulation runs in OpenSim.

Visual servo walking control for humanoids with finite-time convergence and smooth robot velocities

ABSTRACTIn this paper, we address the problem of humanoid locomotion guided from information of a monocular camera. The goal of the robot is to reach a desired location defined in terms of a target image, i.e., a positioning task. The proposed approach allows us to introduce a desired time to complete the positioning task, which is advantageous in contrast to the classical exponential convergence. In particular, finite-time convergence is achieved while generating smooth robot velocities and considering the omnidirectional waking capability of the robot. In addition, we propose a hierarchical task-based control scheme, which can simultaneously handle the visual positioning and the obstacle avoidance tasks without affecting the desired time of convergence. The controller is able to activate or inactivate the obstacle avoidance task without generating discontinuous velocity references while the humanoid is walking. Stability of the closed loop for the two task-based control is demonstrated theoretically even during the transitions between the tasks. The proposed approach is generic in the sense that different visual control schemes are supported. We evaluate a homography-based visual servoing for position-based and image-based modalities, as well as for eye-in-hand and eye-to-hand configurations. The experimental evaluation is performed with the humanoid robot NAO.