Real Robot Control Research Articles

Safe Multiagent Learning With Soft Constrained Policy Optimization in Real Robot Control

Safe Multiagent Learning With Soft Constrained Policy Optimization in Real Robot Control

Simplified Physical Model‐based Balance‐preserving Motion Re‐targeting for Physical Simulation

AbstractIn this study, we propose a novel motion re‐targeting framework that provides natural motions of target robot character models similar to the given source motions of a different skeletal structure. The natural target motion requires satisfying kinematic constraints to show a similar motion to the source motion although the kinematical structure between the source and the target character models differ from each other. Simultaneously, the target motion should maintain physically plausible features such as keeping the balance of the target character model. To handle the issue, we utilize a simple physics model (an inverted‐pendulum‐on‐a‐cart model) during the motion re‐targeting process. By interpreting the source motion's balancing property via the pendulum model, the target motion inherits the balancing property of the source motion. The inheritance is derived by performing the motion analysis to extract the necessary parameters for re‐targeting the pendulum model's motion pattern and parameter learning to estimate the suitable parameters for the target character model. Based on the simple physics inheritance, the proposed framework provides balance‐preserving target motions, even applicable to the full‐body physics simulation or a real robot control. We validate the framework by experimenting with motion re‐targeting from animal character and human character source models to the quadruped‐ and humanoid‐type target models with Muaythai punching, kicking and walking motions. We also implement comparisons with the existing methods to clarify the enhancement.

Collaborative robot assisted smart needle placement

Abstract Needles are key tools to realize minimally invasive interventions. Physicians commonly rely on subjectively perceived insertion forces at the distal end of the needle when advancing the needle tip to the desired target. However, detecting tissue transitions at the distal end of the needle is difficult since the sensed forces are dominated by shaft forces. Disentangling insertion forces has the potential to substantially improve needle placement accuracy.We propose a collaborative system for robotic needle insertion, relaying haptic information sensed directly at the needle tip to the physician by haptic feedback through a light weight robot. We integrate optical fibers into medical needles and use optical coherence tomography to image a moving surface at the tip of the needle. Using a convolutional neural network, we estimate forces acting on the needle tip from the optical coherence tomography data. We feed back forces estimated at the needle tip for real time haptic feedback and robot control. When inserting the needle at constant velocity, the force change estimated at the tip when penetrating tissue layers is up to 94% between deep tissue layers compared to the force change at the needle handle of 2.36 %. Collaborative needle insertion results in more sensible force change at tissue transitions with haptic feedback from the tip (49.79 ± 25.51)% compared to the conventional shaft feedback (15.17 ± 15.92) %. Tissue transitions are more prominent when utilizing forces estimated at the needle tip compared to the forces at the needle shaft, indicating that a more informed advancement of the needle is possible with our system.

Deep Q Learning in Stabilization of Inverted Pendulum

Reinforcement Learning is a concept in which an agent takes actions in an environment to maximize its aggregate reward. Agents in Reinforcement Learning algorithms are penalized for bad actions and rewarded for good ones. Deep Reinforcement learning is used to overcome some shortcomings in Reinforcement learning like lack of variety in the open source collection and change in task’s difficulty based on reward or set of actions. In a larger environment, inference of new states from already explored states is a difficult task due to its time & space complexity. Hence the Q-value function which represents the quality value is approximated by Deep Q-learning that uses a neural network for the same. The results of implementing Deep Q Learning (DQN), Double DQN, Dueling DQN, Noisy DQN & DQN Prioritized Experience Replay techniques & their performance in stabilization of inverted pendulum are highlighted in this paper. Deep Reinforcement learning can be applied in various platform like for recognition, perception in computer vision, for simulation to real robot control in robotics, for sequence generation, translation in Natural Language processing, for Poker, Bridge StarCraft in games, for pricing, trading risk management in finance, for e- commerce, customer management in business management, for diagnosis using Electronic Medical Records in healthcare and in adaptive decision control.

Niching genetic network programming with rule accumulation for decision making: An evolutionary rule-based approach

As one of the most important research branches of evolutionary computation (EC), learning classifier system (LCS) is dedicated to discover decision making classifiers (“IF-THEN” type rules) via evolution and learning. Recent advances in LCS have shown distinguished generalization property over traditional approaches. In this paper, a novel LCS named niching genetic network programming with rule accumulation (nGNP-RA) is proposed. The unique features of the proposal arise from the following three points: First, it utilizes an advanced graph-based EC named GNP as the rule generator, resulting higher knowledge representation ability than traditional genetic algorithm (GA)-based LCSs; Second, a novel niching mechanism is developed in GNP to encourage the discovery of high-quality diverse rules; Third, a novel reinforcement learning (RL)-based mechanism is embedded to assign accurate credits to the discovered rules. To verify the effectiveness and robustness of nGNP-RA over traditional systems, two decision making testbeds are applied, including the benchmark tileworld problem and the real mobile robot control application.

A Single-Channel SSVEP-Based BCI with a Fuzzy Feature Threshold Algorithm in a Maze Game

A mobile robot control in a maze game using steady-state visually evoked potential (SSVEP)-based brain–computer interface (BCI) is developed to help the severely disabled. In order to correctively induce the SSVEP of subjects, four visual stimuli including “counterclockwise,” “clockwise,” “forward,” and “backward” are displayed on monitor and flickering at different frequencies. The spectral features of EEG are extracted by using fast Fourier transform to accurately represent the characteristics of SSVEP. A fuzzy feature threshold algorithm is proposed to track the power spectrum of EEG and automatically adjust the threshold of EEG spectrum to achieve a suitable performance and stability of a BCI system. In this study, the system accuracies were 86.58 and 85.54% for robot movement simulation test and real robot control, respectively. Then, it would be suitable for the severely disabled to control the mobile robot in a maze game.

Model-based reinforcement learning with dimension reduction

The goal of reinforcement learning is to learn an optimal policy which controls an agent to acquire the maximum cumulative reward. The model-based reinforcement learning approach learns a transition model of the environment from data, and then derives the optimal policy using the transition model. However, learning an accurate transition model in high-dimensional environments requires a large amount of data which is difficult to obtain. To overcome this difficulty, in this paper, we propose to combine model-based reinforcement learning with the recently developed least-squares conditional entropy (LSCE) method, which simultaneously performs transition model estimation and dimension reduction. We also further extend the proposed method to imitation learning scenarios. The experimental results show that policy search combined with LSCE performs well for high-dimensional control tasks including real humanoid robot control.

Cloud based Real-time Multi-robot Collision Avoidance for Swarm Robotics

Nowadays, Cloud Computing has brought many new and efficient approaches for computation-intensive application areas. One typical area is Cloud based real-time device control system, such as the IoT Cloud Platform. This kind of platform shifts computation load from devices to the Cloud and provides powerful processing capabilities to a simple device. In Swarm robotics, robots are supposed to be small, energy efficient and low-cost, but still smart enough to carry out individual and swarm intelligence. These two goals are normally contradictory to each other. Besides, in real world robot control, real-time on-line data processing is required, but most of the current Cloud Robotic Systems are focusing on off-line batch processing. However, Cloud based real-time device control system may provide a way that leads this research area out of its dilemma. This paper explores the availability of Cloud based real-time control of massive complex robots by implementing a relatively complicated but better performed local collision avoidance algorithm. The Cloud based application and corresponding Cloud driver, which connects the robot and the Cloud, are developed and deployed in Cloud environment. Simulation tests are carried out and the results show that, when the number of robots increases, by simply scaling computation resources for the application, the algorithm can still maintain the preset control frequency. Such characteristics verify that the Cloud Computing environment is a new platform for studying massive complex robots in swarm robotics.

Special issue on whole-body control of contacts and dynamics for humanoid robots

Whether you are walking on a concrete floor, standing on a carpet, or sitting on a soft chair, your entire body continuously controls the posture and the contact forces that are produced by acting on rigid and compliant surfaces. Sometimes, like when reaching for a distant object or standing inside a moving bus that suddenly brakes, humans plan intentionally new contacts to preserve their balance and avoid falling. For humanoid robots to act in unstructured natural environments as humans do, contacts and physical interactions are necessary and unavoidable. This special issue of the Autonomous Robots journal aims at presenting the advances in whole-body control of real robots, focusing on control and learning techniques applied to estimation, control and adaptation of whole-body dynamics movement and contact forces that go beyond basic balancing abilities.

Open Issues in Evolutionary Robotics.

One of the long-term goals in evolutionary robotics is to be able to automatically synthesize controllers for real autonomous robots based only on a task specification. While a number of studies have shown the applicability of evolutionary robotics techniques for the synthesis of behavioral control, researchers have consistently been faced with a number of issues preventing the widespread adoption of evolutionary robotics for engineering purposes. In this article, we review and discuss the open issues in evolutionary robotics. First, we analyze the benefits and challenges of simulation-based evolution and subsequent deployment of controllers versusevolution on real robotic hardware. Second, we discuss specific evolutionary computation issues that have plagued evolutionary robotics: (1)the bootstrap problem, (2)deception, and (3)the role of genomic encoding and genotype-phenotype mapping in the evolution of controllers for complex tasks. Finally, we address the absence of standard research practices in the field. We also discuss promising avenues of research. Our underlying motivation is the reduction of the current gap between evolutionary robotics and mainstream robotics, and the establishment of evolutionary robotics as a canonical approach for the engineering of autonomous robots.

Approximate postdictive reasoning with answer set programming

We present an answer set programming realization of the h-approximation (HPX) theory [8] as an efficient and provably sound reasoning method for epistemic planning and projection problems that involve postdictive reasoning. The efficiency of HPX stems from an approximate knowledge state representation that involves only a linear number of state variables, as compared to an exponential number for theories that utilize a possible-worlds based semantics. This causes a relatively low computational complexity, i.e, the planning problem is in NP under reasonable restrictions, at the cost that HPX is incomplete. In this paper, we use the implementation of HPX to investigate the incompleteness issue and present an empirical evaluation of the solvable fragment and its performance. We find that the solvable fragment of HPX is indeed reasonable and fairly large: in average about 85% of the considered projection problem instances can be solved, compared to a PWS-based approach with exponential complexity as baseline. In addition to the empirical results, we demonstrate the manner in which HPX can be applied in a real robotic control task within a smart home, where our scenario illustrates the usefulness of postdictive reasoning to achieve error-tolerance by abnormality detection in a high-level decision-making task.

Continuous probabilistic model building genetic network programming using reinforcement learning

Recently, a novel probabilistic model-building evolutionary algorithm (so called estimation of distribution algorithm, or EDA), named probabilistic model building genetic network programming (PMBGNP), has been proposed. PMBGNP uses graph structures for its individual representation, which shows higher expression ability than the classical EDAs. Hence, it extends EDAs to solve a range of problems, such as data mining and agent control. This paper is dedicated to propose a continuous version of PMBGNP for continuous optimization in agent control problems. Different from the other continuous EDAs, the proposed algorithm evolves the continuous variables by reinforcement learning (RL). We compare the performance with several state-of-the-art algorithms on a real mobile robot control problem. The results show that the proposed algorithm outperforms the others with statistically significant differences.

Development of PC-Based Simulation and Control Platform for a 6-DOF Robotic Arm

This paper developed a 6-DOF (degree of freedom) PC-Based robotic arm system. The system mainly include in software platform and servo control card, servo control card based on microcontroller STC12C5A60S2 was designed to drive the servomotor connected with each joint of robot. The software was implemented by combining MFC with OpenGL. By using the OpenGL functions, the software is able to draw and simulate the 3D kinematic scheme of the robot, it also provides 3D motion planning simulation feature. With the help of simulation in the GUI, users can visualize the manipulator motion planning. Furthermore, user also can control the real robotic arm through this software. Finally, point-to-point motion and continuous path motion are all tested in simulation and real robot control. The entire system has been successfully implemented.

A Novel Graph-Based Estimation of the Distribution Algorithm and its Extension Using Reinforcement Learning

In recent years, numerous studies have drawn the success of estimation of distribution algorithms (EDAs) to avoid the frequent breakage of building blocks of the conventional stochastic genetic operators-based evolutionary algorithms (EAs). In this paper, a novel graph-based EDA called probabilistic model building genetic network programming (PMBGNP) is proposed. Using the distinguished graph (network) structure of a graph-based EA called genetic network programming (GNP), PMBGNP ensures higher expression ability than the conventional EDAs to solve some specific problems. Furthermore, an extended algorithm called reinforced PMBGNP is proposed to combine PMBGNP and reinforcement learning to enhance the performance in terms of fitness values, search speed, and reliability. The proposed algorithms are applied to solve the problems of controlling the agents' behavior. Two problems are selected to demonstrate the effectiveness of the proposed algorithms, including the benchmark one, i.e., the Tileworld system, and a real mobile robot control.

Evaluation of linearly solvable Markov decision process with dynamic model learning in a mobile robot navigation task

Linearly solvable Markov Decision Process (LMDP) is a class of optimal control problem in which the Bellman's equation can be converted into a linear equation by an exponential transformation of the state value function (Todorov, 2009b). In an LMDP, the optimal value function and the corresponding control policy are obtained by solving an eigenvalue problem in a discrete state space or an eigenfunction problem in a continuous state using the knowledge of the system dynamics and the action, state, and terminal cost functions. In this study, we evaluate the effectiveness of the LMDP framework in real robot control, in which the dynamics of the body and the environment have to be learned from experience. We first perform a simulation study of a pole swing-up task to evaluate the effect of the accuracy of the learned dynamics model on the derived the action policy. The result shows that a crude linear approximation of the non-linear dynamics can still allow solution of the task, despite with a higher total cost. We then perform real robot experiments of a battery-catching task using our Spring Dog mobile robot platform. The state is given by the position and the size of a battery in its camera view and two neck joint angles. The action is the velocities of two wheels, while the neck joints were controlled by a visual servo controller. We test linear and bilinear dynamic models in tasks with quadratic and Guassian state cost functions. In the quadratic cost task, the LMDP controller derived from a learned linear dynamics model performed equivalently with the optimal linear quadratic regulator (LQR). In the non-quadratic task, the LMDP controller with a linear dynamics model showed the best performance. The results demonstrate the usefulness of the LMDP framework in real robot control even when simple linear models are used for dynamics learning.

A Differential Game Theoretic Approach for Two-Agent Collision Avoidance with Travel Limitations

This work presents a collision avoidance control strategy that solves the Hamilton-Jacobi-Isaacs (HJI) equation for an agent to quickly take action assuming a worst-case scenario. By doing so, the agent can develop a control strategy that is robust to the strategies of other agents with whom collision is possible. Consequently, if the governing dynamics of the agent are sufficient, then a collision can be avoided. We build on the idea of finding control solutions by using a differential game theoretic approach (Mettenheim and Breitner, 2009). This is beneficial because the opposing agent’s strategy is incorporated into the control by assuming it plays the worst-case actions. The approach in this work solves the zero-sum aspects of the control on-line using a fast solution method that operates over partitions in the state space (Goode et al., ASME J Dyn Syst Meas Control, 2011). We form the solution to the Homicidal Chauffeur game which is used to provide the control for an evader attempting to avoid a pursuer, an agent that deviates from its normal path and into that of the evader. Furthermore, the evader is constrained to remain within defined boundaries of its assigned travel area, such as a road lane, water channel, etc. The control strategy consists of three parts: (1) a zero-sum approximation of collision avoidance, (2) high-level path planning, and (3) low-level vehicle control. Each component is explained, and an example is given using a real robotic vehicle control system. Here, we show how the control can be implemented using a simple processor located on a vehicle that seeks to avoid a collision with another oncoming vehicle, making a left turn.

Network-Based Humanoid Robot Remote Interaction with the Actual Situation Fusion Technology

With the rapid development of network technology, network-based humanoid robot technology will also be open to the development of gradual and orderly progress. This article is based on the C / S architecture, the server responsible for controlling the record of news and information network transit between paragraphs; through remote interaction, real-time client to complete the real humanoid robot control functions. Interoperability between the client, first to sign the server. Server information of all registered users to return to the client process, then the client users will be able to get online users to select the remote robot interaction. .When a user operation, the client program as a virtual robot through the virtual robot laboratory will be displayed in real-time robot control results.

Autonomous stair climbing with multisensor feedback

AbstractThis article presents an autonomous stair climbing procedure with multisensor feedback. The task is performed by the six-legged robot Messor. The control procedure is performed in a closed-loop, and requires appropriate measurements. The sensory system of the robot consists of a structural light system and a laser range finder. The first one provides information about the dimensions of stairs and the second one measures current pose of the robot on stairs. The execution of the procedure is fully automatic. Once the stairs are found no other external input to the system is required till the top of the stairs. The climbing strategy was developed in simulation and then transferred to the real robot controller. The Messor robot is able to climb stairs of different sizes, and thanks to the sensory feedback copes well with the imperfectness of the environment.

Learning model-free robot control by a Monte Carlo EM algorithm

We address the problem of learning robot control by model-free reinforcement learning (RL). We adopt the probabilistic model for model-free RL of Vlassis and Toussaint (Proceedings of the international conference on machine learning, Montreal, Canada, 2009), and we propose a Monte Carlo EM algorithm (MCEM) for control learning that searches directly in the space of controller parameters using information obtained from randomly generated robot trajectories. MCEM is related to, and generalizes, the PoWER algorithm of Kober and Peters (Proceedings of the neural information processing systems, 2009). In the finite-horizon case MCEM reduces precisely to PoWER, but MCEM can also handle the discounted infinite-horizon case. An interesting result is that the infinite-horizon case can be viewed as a ‘randomized’ version of the finite-horizon case, in the sense that the length of each sampled trajectory is a random draw from an appropriately constructed geometric distribution. We provide some preliminary experiments demonstrating the effects of fixed (PoWER) vs randomized (MCEM) horizon length in two simulated and one real robot control tasks.

A CORBA-based simulation and control framework for mobile robots

SUMMARYThis paper presents a distributed multiple mobile robots framework which allows programming and control of virtual and real mobile robots. The system provides the map building, path planning, robot task planning, simulation, and actual robot control functions in an indoor environment. Users can program the virtual robots in a customized simulation environment and check the performance of execution, i.e., if the simulation result is satisfying, users can download the code to a real robot. The paper focuses on the distributed architecture and key technologies of virtual robots simulation and control of real robots. A method for construction and transfer of a key index value (which stores the robot configuration) is proposed. Using this method, only the robot key configuration index is needed to build the robot in the virtual environment. This results in reduced network load and improved real time performance of the distributed system. Experiments were conducted to compare the performance of the proposed system with the performance of a centralized system. The results show that the distributed system uses less system resources and has better real time performance. What is more, this framework has been applied to Yaskawa's robot “SmartPal.” The simulation and experiment results show that our robotic framework can simulate and control the robot to perform complex tasks.