Reinforcement Learning Policy Research Articles

A versatile door opening system with mobile manipulator through adaptive position-force control and reinforcement learning

The ability of robots to navigate through doors is crucial for their effective operation in indoor environments. Consequently, extensive research has been conducted to develop robots capable of opening specific doors. However, the diverse combinations of door handles and opening directions necessitate a more versatile door opening system for robots to successfully operate in real-world environments. In this paper, we propose a mobile manipulator system that can autonomously open various doors without prior knowledge. By using convolutional neural networks, point cloud extraction techniques, and external force measurements during exploratory motion, we obtained information regarding handle types, poses, and door characteristics. Through two different approaches, adaptive position-force control and deep reinforcement learning, we successfully opened doors without precise trajectory or excessive external force. The adaptive position-force control method involves moving the end-effector in the direction of the door opening while responding compliantly to external forces, ensuring safety and manipulator workspace. Meanwhile, the deep reinforcement learning policy minimizes applied forces and eliminates unnecessary movements, enabling stable operation across doors with different poses and widths. The RL-based approach outperforms the adaptive position-force control method in terms of compensating for external forces, ensuring smooth motion, and achieving efficient speed. It reduces the maximum force required by 3.27 times and improves motion smoothness by 1.82 times. However, the non-learning-based adaptive position-force control method demonstrates more versatility in opening a wider range of doors, encompassing revolute doors with four distinct opening directions and varying widths.

Entry guidance for spatial no-fly zones avoidance via model-based reinforcement learning

This paper proposes a novel guidance law for hypersonic entry vehicles, considering no-fly zones with height limits. Traditional planar assumptions restrict the flexibility of trajectory design for scenarios like radar avoidance. Besides, the numerical integration proves inefficient for long-term prediction and avoidance. Therefore, a model-based reinforcement learning policy is designed. It offline learns entry dynamics in advance and onboard plans a feasible trajectory. The planner's state includes flight status and no-fly zones; action presents waypoints; and reward ensures constraint while maximizing terminal precision. Then, analytical prediction converts spatial no-fly zone constraints to flight-path angle constraints, improving precision compared to traditional one-step estimates. Finally, the two parts are assembled into the predictor-corrector framework, which gives the augmented guidance commands. While retaining its robustness to bias, our method reduces online optimization calculations and outperforms constraint satisfaction. Experiments show that the model-based method reduces 60% training in offline training compared with proximal policy optimization. Besides, our method is 80% faster than conventional predictor-corrector guidance regarding online computation speed.

Fill-and-Spill: Deep Reinforcement Learning Policy Gradient Methods for Reservoir Operation Decision and Control

Fill-and-Spill: Deep Reinforcement Learning Policy Gradient Methods for Reservoir Operation Decision and Control

Towards Robust Decision-Making for Autonomous Highway Driving Based on Safe Reinforcement Learning.

Reinforcement Learning (RL) methods are regarded as effective for designing autonomous driving policies. However, even when RL policies are trained to convergence, ensuring their robust safety remains a challenge, particularly in long-tail data. Therefore, decision-making based on RL must adequately consider potential variations in data distribution. This paper presents a framework for highway autonomous driving decisions that prioritizes both safety and robustness. Utilizing the proposed Replay Buffer Constrained Policy Optimization (RECPO) method, this framework updates RL strategies to maximize rewards while ensuring that the policies always remain within safety constraints. We incorporate importance sampling techniques to collect and store data in a Replay buffer during agent operation, allowing the reutilization of data from old policies for training new policy models, thus mitigating potential catastrophic forgetting. Additionally, we transform the highway autonomous driving decision problem into a Constrained Markov Decision Process (CMDP) and apply our proposed RECPO for training, optimizing highway driving policies. Finally, we deploy our method in the CARLA simulation environment and compare its performance in typical highway scenarios against traditional CPO, current advanced strategies based on Deep Deterministic Policy Gradient (DDPG), and IDM + MOBIL (Intelligent Driver Model and the model for minimizing overall braking induced by lane changes). The results show that our framework significantly enhances model convergence speed, safety, and decision-making stability, achieving a zero-collision rate in highway autonomous driving.

Continuous control of structural vibrations using hybrid deep reinforcement learning policy

An efficient active structural control scheme by utilizing the collective concepts of data-driven and physics-inspired reinforcement learning (RL) approaches is being introduced here. The controller is designed to deliver optimal feedback forces based on the full-state information, wherein the training process involves the use of deep neural networks (NNs) and a specially designed gradient descent-based sequence within the RL framework. This integration of algorithms results in a unique active control policy that accelerates the learning process, thereby demands considerably less computational resources to develop an optimal and stable controller as compared to existing data-driven approaches. Most importantly, the mentioned data-driven and physics-inspired approaches encompass deep deterministic policy gradient (DDPG) and an iterative gradient-based state feedback control (SFSC) algorithms, respectively, to establish the architecture of the hybrid control policy, referred to as hybrid RL-controller. Additional advantage of the hybrid RL-controller is its ability to operate in continuous state–action spaces, which allows the designed controller to address diverse structural control problems. The staggered performance of the hybrid RL-controller, both in continuous and discrete time, is evaluated in three case studies that involve structures in linear and nonlinear regimes. The outcomes include a detailed comparison of the hybrid RL-controller with the individually designed DDPG and SFSC RL control strategies, as well as the uncontrolled scenario. Furthermore, this study thoroughly investigates the real-life implementation concerns of control strategies, such as perturbations in model parameters and input forces, and time delays in the feedback loop. Finally, the results from this study corroborate that the designed RL-controller showcases superior performance and faster execution time in the feedback loop, making it suitable for the vibration control of multidimensional complex structures.

Control of heap leach piles using deep reinforcement learning

In this study, we propose a novel methodology for the automatic control of heap leaching by means of policies obtained using Reinforcement Learning (RL). This methodology models the leaching dynamics as a Markov Decision Process (MDP) whose reward function captures the economic profit of the heap leaching operation. As a case study, the leaching process of copper oxide heaps is simulated and controlled under various conditions. Results show that controlling this process using the proposed approach outperforms a heuristic control strategy that emulates real mining operations by increasing recovery rates by 2.25 times, reducing water consumption by 32.4% and acid consumption by 19.9%, and enhancing economic returns by 17.5%. This approach highlights the robustness of a Deep Reinforcement Learning (DRL) policy in heap leaching operations under significant out-of-distribution (OOD) conditions, demonstrating its adaptability and effectiveness under diverse and unpredictable conditions. Furthermore, this research highlights the potential for this methodology to be applied to other leachable ores, as it could reduce the overall environmental impact of this operation by using fewer resources.

Deep reinforcement learning sensor scheduling for effective monitoring of dynamical systems

Advances in technology have enabled the use of sensors with varied modalities to monitor different parts of systems, each providing diverse levels of information about the underlying system. However, resource limitations and computational power restrict the number of sensors/data that can be processed in real-time in most complex systems. These challenges necessitate the need for selecting/scheduling a subset of sensors to obtain measurements that guarantee the best monitoring objectives. This paper focuses on sensor scheduling for systems modeled by hidden Markov models. Despite the development of several sensor selection and scheduling methods, existing methods tend to be greedy and do not take into account the long-term impact of selected sensors on monitoring objectives. This paper formulates optimal sensor scheduling as a reinforcement learning problem defined over the posterior distribution of system states. Further, the paper derives a deep reinforcement learning policy for offline learning of the sensor scheduling policy, which can then be executed in real-time as new information unfolds. The proposed method applies to any monitoring objective that can be expressed in terms of the posterior distribution of the states (e.g. state estimation, information gain, etc.). The performance of the proposed method in terms of accuracy and robustness is investigated for monitoring the security of networked systems and the health monitoring of gene regulatory networks.

Unleashing mixed-reality capability in Deep Reinforcement Learning-based robot motion generation towards safe human–robot collaboration

The integration of human–robot collaboration yields substantial benefits, particularly in terms of enhancing flexibility and efficiency within a range of mass-personalized manufacturing tasks, for example, small-batch customized product inspection and assembly/disassembly. Meanwhile, as human–robot collaboration lands broader in manufacturing, the unstructured scene and operator uncertainties are increasingly involved and considered. Consequently, it becomes imperative for robots to execute in a safe and adaptive manner rather than solely relying on pre-programmed instructions. To tackle it, a systematic solution for safe robot motion generation in human–robot collaborative activities is proposed, leveraging mixed-reality technologies and Deep Reinforcement Learning. This solution covers the entire process of collaboration starting with an intuitive interface that facilitates bare-hand task command transmission and scene coordinate transformation before the collaboration begins. In particular, mixed-reality devices are employed as effective tools for representing the state of humans, robots, and scenes. This enables the learning of an end-to-end Deep Reinforcement Learning policy that addresses both the uncertainties in robot perception and decision-making in an integrated manner. The proposed solution also implements policy simulation-to-reality deployment, along with motion preview and collision detection mechanisms, to ensure safe robot motion execution. It is hoped that this work could inspire further research in human–robot collaboration to unleash and exploit the powerful capabilities of mixed reality.

Analyzing Generalization in Policy Networks: A Case Study with the Double-Integrator System

Extensive utilization of deep reinforcement learning (DRL) policy networks in diverse continuous control tasks has raised questions regarding performance degradation in expansive state spaces where the input state norm is larger than that in the training environment. This paper aims to uncover the underlying factors contributing to such performance deterioration when dealing with expanded state spaces, using a novel analysis technique known as state division. In contrast to prior approaches that employ state division merely as a post-hoc explanatory tool, our methodology delves into the intrinsic characteristics of DRL policy networks. Specifically, we demonstrate that the expansion of state space induces the activation function $\tanh$ to exhibit saturability, resulting in the transformation of the state division boundary from nonlinear to linear. Our analysis centers on the paradigm of the double-integrator system, revealing that this gradual shift towards linearity imparts a control behavior reminiscent of bang-bang control. However, the inherent linearity of the division boundary prevents the attainment of an ideal bang-bang control, thereby introducing unavoidable overshooting. Our experimental investigations, employing diverse RL algorithms, establish that this performance phenomenon stems from inherent attributes of the DRL policy network, remaining consistent across various optimization algorithms.

Online Reinforcement Learning-Based Pedagogical Planning for Narrative-Centered Learning Environments

Pedagogical planners can provide adaptive support to students in narrative-centered learning environments by dynamically scaffolding student learning and tailoring problem scenarios. Reinforcement learning (RL) is frequently used for pedagogical planning in narrative-centered learning environments. However, RL-based pedagogical planning raises significant challenges due to the scarcity of data for training RL policies. Most prior work has relied on limited-size datasets and offline RL techniques for policy learning. Unfortunately, offline RL techniques do not support on-demand exploration and evaluation, which can adversely impact the quality of induced policies. To address the limitation of data scarcity and offline RL, we propose INSIGHT, an online RL framework for training data-driven pedagogical policies that optimize student learning in narrative-centered learning environments. The INSIGHT framework consists of three components: a narrative-centered learning environment simulator, a simulated student agent, and an RL-based pedagogical planner agent, which uses a reward metric that is associated with effective student learning processes. The framework enables the generation of synthetic data for on-demand exploration and evaluation of RL-based pedagogical planning. We have implemented INSIGHT with OpenAI Gym for a narrative-centered learning environment testbed with rule-based simulated student agents and a deep Q-learning-based pedagogical planner. Our results show that online deep RL algorithms can induce near-optimal pedagogical policies in the INSIGHT framework, while offline deep RL algorithms only find suboptimal policies even with large amounts of data.

Faster Stochastic Variance Reduction Methods for Compositional MiniMax Optimization

This paper delves into the realm of stochastic optimization for compositional minimax optimization—a pivotal challenge across various machine learning domains, including deep AUC and reinforcement learning policy evaluation. Despite its significance, the problem of compositional minimax optimization is still under-explored. Adding to the complexity, current methods of compositional minimax optimization are plagued by sub-optimal complexities or heavy reliance on sizable batch sizes. To respond to these constraints, this paper introduces a novel method, called Nested STOchastic Recursive Momentum (NSTORM), which can achieve the optimal sample complexity and obtain the nearly accuracy solution, matching the existing minimax methods. We also demonstrate that NSTORM can achieve the same sample complexity under the Polyak-Lojasiewicz (PL)-condition—an insightful extension of its capabilities. Yet, NSTORM encounters an issue with its requirement for low learning rates, potentially constraining its real-world applicability in machine learning. To overcome this hurdle, we present ADAptive NSTORM (ADA-NSTORM) with adaptive learning rates. We demonstrate that ADA-NSTORM can achieve the same sample complexity but the experimental results show its more effectiveness. All the proposed complexities indicate that our proposed methods can match lower bounds to existing minimax optimizations, without requiring a large batch size in each iteration. Extensive experiments support the efficiency of our proposed methods.

A novel reinforcement learning policy optimization based adaptive VSG control technique for improved frequency stabilization in AC microgrids

In this paper, an improved Reinforcement Learning (RL) based approach for adaptive Virtual Synchronous Generator (VSG) control of grid-forming inverters is proposed. High penetration of inverter-based resources causes recurrent frequency stabilization issues. For this matter, a novel model-free Actor–Critic policy optimization technique is proposed to improve frequency transients metrics such as frequency nadir and Rate of Change of Frequency (RoCoF). Stability analysis of the VSG controlled system is conducted through small signal modeling to determine the stability margins of both the virtual inertia parameter J and damping coefficient Dp. The control problem is formulated in RL terms using a new technique for state, action and reward construction that improves the learning process of the agents. The proposed approach’s efficiency is tested for three RL algorithms, namely, Deep Deterministic Policy Gradient (DDPG), Soft Actor–Critic (SAC), and Twin-Delayed DDPG (TD3) with special attention given to the latter. Various performance metrics such as training time and maximum cumulative episodic reward are used to compare the three algorithms. The resulting agent based VSG adaptive controllers are simulated and then compared for power quality and frequency stabilization capabilities. Finally, a comparison between non-adaptive VSG and TD3 based VSG is presented to conclude on the efficiency improvements achieved by the proposed approach.

Prescribed intelligent elliptical pursuing by UAVs: A reinforcement learning policy

This paper proposes a prescribed intelligent elliptical pursuing algorithm driven by deep reinforcement learning (DRL) for Unmanned Aerial Vehicles (UAVs), aiming to achieve an optimal performance without violating appointed-time constraints. Specifically, an unknown system dynamics estimator (USDE) with merely one filtering coefficient is employed to counteract uncertainties effectively. Subsequently, an elliptical enclosing controller is devised for UAVs to achieve asymptotic convergence towards the target ellipse orbit. Particularly, a proximal policy optimization (PPO)-based optimal learning term is contrived to achieve control optimality related to consumption and efficiency, where a refined reward function is established to characterize the appointed-time constraints. The salient merit is that an efficient learning paradigm is formulated to fulfill a near optimal appointed-time enclosing free from prescribed performance control. Finally, a plenty of simulations substantiate the feasibility of developed algorithm.

Inverse-Reinforcement-Learning-Based Robotic Ultrasound Active Compliance Control in Uncertain Environments

Robotic ultrasound systems (RUS) have gained increasing attention because they can automate repetitive procedures and relieve operators' workloads. However, the complexity and uncertainty of the human surface pose a challenge for stable scanning control. This paper proposes a general active compliance control strategy based on inverse reinforcement learning (IRL) to perform adaptable scanning for uncertain and unstructured environments. We analyze the manual scanning process pattern and propose a velocity-and-force-related control strategy to achieve variable force control and handle unpredictable deformation. Then, a hybrid policy optimization framework is proposed to improve transferability. In this framework, a reinforcement learning (RL) policy with a predefined reward is built to establish the relationship between contact force and posture. Furthermore, the policy is re-optimized using IRL and generated demonstrations for IRL training. The policy is trained on simple standard phantoms and further evaluated for stability and transferability in unseen and complex environments. Quantitative results show that the difference between the proposed method and the 3-D reconstructed model in terms of posture is (2.3±1.3°, 1.9±1.2°) in continuous scans. Overall, our method provides a solution for improving the usability of robotic ultrasound systems in real-world environments.

A Reinforcement Learning and Prediction-Based Lookahead Policy for Vehicle Repositioning in Online Ride-Hailing Systems

A Reinforcement Learning and Prediction-Based Lookahead Policy for Vehicle Repositioning in Online Ride-Hailing Systems

Data-Driven Inverse Reinforcement Learning Control for Linear Multiplayer Games.

This article proposes a data-driven inverse reinforcement learning (RL) control algorithm for nonzero-sum multiplayer games in linear continuous-time differential dynamical systems. The inverse RL problem in the games is solved by a learner reconstructing the unknown expert players' cost functions from demonstrated expert's optimal state and control input trajectories. The learner, thus, obtains the same control feedback gains and trajectories as the expert, only using data along system trajectories without knowing system dynamics. This article first proposes a model-based inverse RL policy iteration framework that has: 1) policy evaluation step for reconstructing cost matrices using Lyapunov functions; 2) state-reward weight improvement step using inverse optimal control (IOC); and 3) policy improvement step using optimal control. Based on the model-based policy iteration algorithm, this article further develops an online data-driven off-policy inverse RL algorithm without knowing any knowledge of system dynamics or expert control gains. Rigorous convergence and stability analysis of the algorithms are provided. It shows that the off-policy inverse RL algorithm guarantees unbiased solutions while probing noises are added to satisfy the persistence of excitation (PE) condition. Finally, two different simulation examples validate the effectiveness of the proposed algorithms.

Modifying RL Policies with Imagined Actions: How Predictable Policies Can Enable Users to Perform Novel Tasks

It is crucial that users are empowered to use the functionalities of a robot to creatively solve problems on the fly. A user who has access to a Reinforcement Learning (RL) based robot may want to use the robot's autonomy and their knowledge of its behavior to complete new tasks. One way is for the user to take control of some of the robot's action space through teleoperation while the RL policy simultaneously controls the rest. However, an out-of-the-box RL policy may not readily facilitate this. For example, a user's control may bring the robot into a failure state from the policy's perspective, causing it to act in a way the user is not familiar with, hindering the success of the user's desired task. In this work, we formalize this problem and present Imaginary Out-of-Distribution Actions, IODA, an initial algorithm for addressing that problem and empowering user's to leverage their expectation of a robot's behavior to accomplish new tasks.

The stochastic share-a-ride problem with electric vehicles and customer priorities

We introduce a stochastic share-a-ride problem in which a fleet of electric vehicles (EV) in a ride-hailing system are dynamically dispatched to serve passenger and parcel orders in a shared manner. We assume uncertain demands of both passenger and parcel orders and consider that passenger orders have priority over parcel orders. Passengers must be transported directly from their origins to destinations, while parcels can share a vehicle with other orders. The operator of the ride-hailing platform needs to decide whether to accept a newly arrived service request, how to assign orders to vehicles, and how to route and charge the EVs. To develop dynamic policies for the problem, we formulate the problem as a Markov decision process (MDP) and propose a reinforcement learning (RL) approach to solve the problem. We develop action-space restriction and state-space aggregation schemes to facilitate the implementation of the RL algorithm. We also present two rolling horizon heuristic methods to develop dynamic policies for our problem. We conduct computational experiments based on real-world taxi data from New York City. The computational results show that our RL policies perform better than the three benchmark policies in terms of serving more orders and collecting more rewards. Our RL policies are able to make high-quality decisions more efficiently when compared with the rolling horizon policies.

Redesign of Leg Assembly and Implementation of Reinforcement Learning for a Multi-Purpose Rehabilitation Robotic Device (RoboREHAB)

Patients who are suffering from neuromuscular disorders or injuries that impair motor control need to undergo rehabilitation to regain mobility. Gait training is commonly prescribed to patients to regain muscle memory. Automated-walking training devices were created to aid this process; while these devices establish accurate ankle-path trajectories, the knee and hip movements are inaccurate. In this work, a redesign of the leg assembly in a multi-purpose rehabilitation robotic device (RoboREHAB) was explored to improve hip- and knee-movement accuracy by adding an extra link and rollers to the assembly. Motion analysis was employed to test feasibility, reinforcement learning was utilized to train the new leg assembly to walk, and the joint motions achieved with the redesign were compared to those achieved by motion-capture (mocap) data. As a key result, the motion analysis showed an improvement in the knee- and hip-path trajectories due to the added roller/joint segment. The redesigned leg assembly, under the reinforcement-learning policy, showed a 5% deviation from the motion-capture joint trajectories with a maximum deviation of 51.177 mm but maintained a similar profile to the mocap trajectory data. This is an improvement over the original two-segment design, which achieved a maximum deviation of 72.084 mm. These results in the knee- and hip-joint movements more closely reflect the mocap and motion-analysis results, validating the redesign and opening it up to further experimentation and technical improvement.

A Novel Learning-Based Trajectory Generation Strategy for a Quadrotor.

In this article, a learning-based trajectory generation framework is proposed for quadrotors, which guarantees real-time, efficient, and practice-reliable navigation by online making human-like decisions via reinforcement learning (RL) and imitation learning (IL). Specifically, inspired by human driving behavior and the perception range of sensors, a real-time local planner is designed by combining learning and optimization techniques, where the smooth and flexible trajectories are online planned efficiently in the observable area. In particular, the key problems in the framework, temporal optimality (time allocation), and spatial optimality (trajectory distribution) are solved by designing an RL policy, which provides human-like commands in real-time (e.g., slower or faster) to achieve better navigation, instead of generating traditional low-level motions. In this manner, real-time trajectories are calculated using convex optimization according to the efficient and accurate decisions of the RL policy. In addition, to improve generalization performance and to accelerate the training, an expert policy and IL are employed in the framework. Compared with existing works, the kernel contribution is to design a real-time practice-oriented intelligent trajectory generation framework for quadrotors, where human-like decision-making and model-based optimization are integrated to plan high-quality trajectories. The results of comparative experiments in known and unknown environments illustrate the superior performance of the proposed trajectory generation strategy in terms of efficiency, smoothness, and flexibility.