Reinforcement Learning Policy Research Articles

Inverse-Reinforcement-Learning-Based Robotic Ultrasound Active Compliance Control in Uncertain Environments

Robotic ultrasound systems (RUS) have gained increasing attention because they can automate repetitive procedures and relieve operators' workloads. However, the complexity and uncertainty of the human surface pose a challenge for stable scanning control. This paper proposes a general active compliance control strategy based on inverse reinforcement learning (IRL) to perform adaptable scanning for uncertain and unstructured environments. We analyze the manual scanning process pattern and propose a velocity-and-force-related control strategy to achieve variable force control and handle unpredictable deformation. Then, a hybrid policy optimization framework is proposed to improve transferability. In this framework, a reinforcement learning (RL) policy with a predefined reward is built to establish the relationship between contact force and posture. Furthermore, the policy is re-optimized using IRL and generated demonstrations for IRL training. The policy is trained on simple standard phantoms and further evaluated for stability and transferability in unseen and complex environments. Quantitative results show that the difference between the proposed method and the 3-D reconstructed model in terms of posture is (2.3±1.3°, 1.9±1.2°) in continuous scans. Overall, our method provides a solution for improving the usability of robotic ultrasound systems in real-world environments.

A Reinforcement Learning and Prediction-Based Lookahead Policy for Vehicle Repositioning in Online Ride-Hailing Systems

A Reinforcement Learning and Prediction-Based Lookahead Policy for Vehicle Repositioning in Online Ride-Hailing Systems

Data-Driven Inverse Reinforcement Learning Control for Linear Multiplayer Games.

This article proposes a data-driven inverse reinforcement learning (RL) control algorithm for nonzero-sum multiplayer games in linear continuous-time differential dynamical systems. The inverse RL problem in the games is solved by a learner reconstructing the unknown expert players' cost functions from demonstrated expert's optimal state and control input trajectories. The learner, thus, obtains the same control feedback gains and trajectories as the expert, only using data along system trajectories without knowing system dynamics. This article first proposes a model-based inverse RL policy iteration framework that has: 1) policy evaluation step for reconstructing cost matrices using Lyapunov functions; 2) state-reward weight improvement step using inverse optimal control (IOC); and 3) policy improvement step using optimal control. Based on the model-based policy iteration algorithm, this article further develops an online data-driven off-policy inverse RL algorithm without knowing any knowledge of system dynamics or expert control gains. Rigorous convergence and stability analysis of the algorithms are provided. It shows that the off-policy inverse RL algorithm guarantees unbiased solutions while probing noises are added to satisfy the persistence of excitation (PE) condition. Finally, two different simulation examples validate the effectiveness of the proposed algorithms.

Modifying RL Policies with Imagined Actions: How Predictable Policies Can Enable Users to Perform Novel Tasks

It is crucial that users are empowered to use the functionalities of a robot to creatively solve problems on the fly. A user who has access to a Reinforcement Learning (RL) based robot may want to use the robot's autonomy and their knowledge of its behavior to complete new tasks. One way is for the user to take control of some of the robot's action space through teleoperation while the RL policy simultaneously controls the rest. However, an out-of-the-box RL policy may not readily facilitate this. For example, a user's control may bring the robot into a failure state from the policy's perspective, causing it to act in a way the user is not familiar with, hindering the success of the user's desired task. In this work, we formalize this problem and present Imaginary Out-of-Distribution Actions, IODA, an initial algorithm for addressing that problem and empowering user's to leverage their expectation of a robot's behavior to accomplish new tasks.

The stochastic share-a-ride problem with electric vehicles and customer priorities

We introduce a stochastic share-a-ride problem in which a fleet of electric vehicles (EV) in a ride-hailing system are dynamically dispatched to serve passenger and parcel orders in a shared manner. We assume uncertain demands of both passenger and parcel orders and consider that passenger orders have priority over parcel orders. Passengers must be transported directly from their origins to destinations, while parcels can share a vehicle with other orders. The operator of the ride-hailing platform needs to decide whether to accept a newly arrived service request, how to assign orders to vehicles, and how to route and charge the EVs. To develop dynamic policies for the problem, we formulate the problem as a Markov decision process (MDP) and propose a reinforcement learning (RL) approach to solve the problem. We develop action-space restriction and state-space aggregation schemes to facilitate the implementation of the RL algorithm. We also present two rolling horizon heuristic methods to develop dynamic policies for our problem. We conduct computational experiments based on real-world taxi data from New York City. The computational results show that our RL policies perform better than the three benchmark policies in terms of serving more orders and collecting more rewards. Our RL policies are able to make high-quality decisions more efficiently when compared with the rolling horizon policies.

Redesign of Leg Assembly and Implementation of Reinforcement Learning for a Multi-Purpose Rehabilitation Robotic Device (RoboREHAB)

Patients who are suffering from neuromuscular disorders or injuries that impair motor control need to undergo rehabilitation to regain mobility. Gait training is commonly prescribed to patients to regain muscle memory. Automated-walking training devices were created to aid this process; while these devices establish accurate ankle-path trajectories, the knee and hip movements are inaccurate. In this work, a redesign of the leg assembly in a multi-purpose rehabilitation robotic device (RoboREHAB) was explored to improve hip- and knee-movement accuracy by adding an extra link and rollers to the assembly. Motion analysis was employed to test feasibility, reinforcement learning was utilized to train the new leg assembly to walk, and the joint motions achieved with the redesign were compared to those achieved by motion-capture (mocap) data. As a key result, the motion analysis showed an improvement in the knee- and hip-path trajectories due to the added roller/joint segment. The redesigned leg assembly, under the reinforcement-learning policy, showed a 5% deviation from the motion-capture joint trajectories with a maximum deviation of 51.177 mm but maintained a similar profile to the mocap trajectory data. This is an improvement over the original two-segment design, which achieved a maximum deviation of 72.084 mm. These results in the knee- and hip-joint movements more closely reflect the mocap and motion-analysis results, validating the redesign and opening it up to further experimentation and technical improvement.

Learn Zero-Constraint-Violation Safe Policy in Model-Free Constrained Reinforcement Learning.

We focus on learning the zero-constraint-violation safe policy in model-free reinforcement learning (RL). Existing model-free RL studies mostly use the posterior penalty to penalize dangerous actions, which means they must experience the danger to learn from the danger. Therefore, they cannot learn a zero-violation safe policy even after convergence. To handle this problem, we leverage the safety-oriented energy functions to learn zero-constraint-violation safe policies and propose the safe set actor-critic (SSAC) algorithm. The energy function is designed to increase rapidly for potentially dangerous actions, locating the safe set on the action space. Therefore, we can identify the dangerous actions prior to taking them and achieve zero-constraint violation. Our major contributions are twofold. First, we use the data-driven methods to learn the energy function, which releases the requirement of known dynamics. Second, we formulate a constrained RL problem to solve the zero-violation policies. We prove that our Lagrangian-based constrained RL solutions converge to the constrained optimal zero-violation policies theoretically. The proposed algorithm is evaluated on the complex simulation environments and a hardware-in-loop (HIL) experiment with a real autonomous vehicle controller. Experimental results suggest that the converged policies in all environments achieve zero-constraint violation and comparable performance with model-based baseline.

Concurrent Learning of Control Policy and Unknown Safety Specifications in Reinforcement Learning

Concurrent Learning of Control Policy and Unknown Safety Specifications in Reinforcement Learning

Task-Driven Autonomous Driving: Balanced Strategies Integrating Curriculum Reinforcement Learning and Residual Policy

Task-Driven Autonomous Driving: Balanced Strategies Integrating Curriculum Reinforcement Learning and Residual Policy

Integrating Inverse Reinforcement Learning and Direct Policy Search for Modeling Multipurpose Water Reservoir Systems

Integrating Inverse Reinforcement Learning and Direct Policy Search for Modeling Multipurpose Water Reservoir Systems

Model-Based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian.

Safety is essential for reinforcement learning (RL) applied in the real world. Adding chance constraints (or probabilistic constraints) is a suitable way to enhance RL safety under uncertainty. Existing chance-constrained RL methods, such as the penalty methods and the Lagrangian methods, either exhibit periodic oscillations or learn an overconservative or unsafe policy. In this article, we address these shortcomings by proposing a separated proportional-integral Lagrangian (SPIL) algorithm. We first review the constrained policy optimization process from a feedback control perspective, which regards the penalty weight as the control input and the safe probability as the control output. Based on this, the penalty method is formulated as a proportional controller, and the Lagrangian method is formulated as an integral controller. We then unify them and present a proportional-integral Lagrangian method to get both their merits with an integral separation technique to limit the integral value to a reasonable range. To accelerate training, the gradient of safe probability is computed in a model-based manner. The convergence of the overall algorithm is analyzed. We demonstrate that our method can reduce the oscillations and conservatism of RL policy in a car-following simulation. To prove its practicality, we also apply our method to a real-world mobile robot navigation task, where our robot successfully avoids a moving obstacle with highly uncertain or even aggressive behaviors.

Historical Decision-Making Regularized Maximum Entropy Reinforcement Learning.

The challenge of the exploration-exploitation dilemma persists in off-policy reinforcement learning (RL) algorithms, impeding the improvement of policy performance and sample efficiency. To tackle this challenge, a novel historical decision-making regularized maximum entropy (HDMRME) RL algorithm is developed to strike the balance between exploration and exploitation. Built upon the maximum entropy RL framework, the historical decision-making regularization method is proposed to enhance the exploitation capability of RL policies. The theoretical analysis involves proving the convergence of HDMRME, investigating the tradeoff between exploration and exploitation of HDMRME, examining the disparity between the Q-function learned through HDMRME and the classic one, and analyzing the suboptimality of the trained policy. The performance of HDMRME is evaluated across various continuous-action control tasks from Mujoco and OpenAI Gym platforms. Comparative experiments demonstrate that HDMRME exhibits superior sample efficiency and achieves more competitive performance compared with other state-of-the-art RL algorithms.

Prediction of Bitcoin Prices Based on Blockchain Information: A Deep Reinforcement Learning Approach

Bitcoin, the first decentralized cryptocurrency, has attracted significant attention from investors and researchers alike due to its volatile and unpredictable price movements. However, predicting the price of Bitcoin remains a challenging task. This paper presents a detailed literature review on previous studies that have attempted to predict the price of Bitcoin. It discusses the main drivers of Bitcoin prices, including its attractiveness, macroeconomic and financial factors with a particular focus on the use of Blockchain information. We apply time series to daily data for the period from 28/04/2013 to 28/01/2023. We used Python and TensorFlow library version 2.11.0 and propose a deep multimodal reinforcement learning policy combining Convolutional Neural Network (CNN) and Long ShortTerm Memory (LSTM) neural network for cryptocurrencies’ prices prediction. Also, this study attempts to predict the price of Bitcoin using a special type of deep neural networks, a Deep Autoencoders. Two results are worth noting: Autoencoders turns out to be the best method of predicting Bitcoin prices, and Bitcoin-specific Blockchain information is the most important variable in predicting Bitcoin prices. This study highlights the potential utility of incorporating Blockchain factors in price prediction models. Also, our findings show that sentiment indicator, Ethereum, XRP and Doge Coin prices, global currency ratio, macroeconomic factors, and Blockchain information of Ethereum did not contribute significantly toward predicting Bitcoin prices. These conclusions provide decision support for investors and a reference for the governments to design better regulatory policies.

Cooperative Deep Reinforcement Learning Policies for Autonomous Navigation in Complex Environments

Cooperative Deep Reinforcement Learning Policies for Autonomous Navigation in Complex Environments

Evolving interpretable decision trees for reinforcement learning

In recent years, reinforcement learning (RL) techniques have achieved great success in many different applications. However, their heavy reliance on complex deep neural networks makes most RL models uninterpretable, limiting their application in domains where trust and security are important. To address this challenge, we propose MENS-DT-RL, an algorithm capable of constructing interpretable models for RL via the evolution of decision tree (DT) models. MENS-DT-RL uses a multi-method ensemble algorithm to evolve univariate DTs, guiding the process with a fitness metric that prioritizes interpretability and consistent high performance. Three different initializations for the MENS-DT-RL are proposed, including the use of Imitation Learning (IL) techniques, and a novel pruning approach that reduces solution size without compromising performance. To evaluate the proposed approach, we compare it with other models from the literature on three benchmark tasks from the OpenAI Gym library, as well as on a fertilization problem inspired by real-world crop management. To the best of our knowledge, the proposed scheme is the first to solve the Lunar Lander benchmark with both interpretability and a high confidence rate (90% of episodes are successful), as well as the first to solve the Mountain Car environment with a tree of only 7 nodes. On the real-world task, the proposed MENS-DT-RL is able to produce solutions with the same quality as deep RL policies, with the added bonus of interpretability. We also analyze the best solutions found by the algorithm and show that they are not only interpretable but also diverse in their behavior, empowering the end user with the choice of which model to apply. Overall, the findings show that the proposed approach is capable of producing high-quality transparent models for RL, achieving interpretability without losing performance.

A residual reinforcement learning method for robotic assembly using visual and force information

Robotic autonomous assembly is critical in intelligent manufacturing and has always been a research hotspot. Most previous approaches rely on prior knowledge, such as geometric parameters and pose information of the assembled parts, which are hard to estimate in unstructured environments. This paper proposes a residual reinforcement learning (RL) policy for robotic assembly via combining visual and force information. The residual RL policy, which consists of a visual-based policy and a force-based policy, is trained and tested in an end-to-end manner. In the assembly procedure, the visual-based policy focuses on spatial search, while the force-based policy handles the interactive behaviors. The experimental results reveal the high sample efficiency of our approach, which exhibits the ability to generalize across diverse assembly tasks involving variations in geometries, clearances, and configurations. The validation experiments are conducted both in simulation and on a real robot.

Plausibility of BBR as CUBIC’S Replacement and proposed improvement to BBR using GENET

Recent advancements in deep reinforcement learning (RL) have opened new avenues for enhancing network congestion control algorithms. Our research builds upon these developments, particularly focusing on the BBR (Bottleneck Bandwidth and Round-trip propagation time) congestion control algorithm. We propose integrating GENET's reinforcement learning framework, a novel training paradigm that has demonstrated success in various network adaptation algorithms, including adaptive video streaming, congestion control, and load balancing. GENET leverages curriculum learning to effectively train RL models by progressively introducing more challenging network environments. This method counters the common pitfalls in RL training, such as suboptimal performance in a wide range of environments and poor generalization in narrowly defined training scenarios. Our approach exploits the strengths of GENET in identifying and emphasizing network conditions where the current RL model underperforms compared to traditional rule-based baselines, thereby facilitating significant improvements. This research aims to demonstrate that applying GENET's methodology to the BBR congestion control algorithm can yield RL policies that surpass both regularly trained RL policies and conventional baselines, thereby advancing the efficiency and reliability of network congestion control.

Aircraft fleet availability optimisation: a reinforcement learning approach

AbstractA fleet of aircraft can be seen as a set of degrading systems that undergo variable loads as they fly missions and require maintenance throughout their lifetime. Optimal fleet management aims to maximise fleet availability while minimising overall maintenance costs. To achieve this goal, individual aircraft, with variable age and degradation paths, need to operate cooperatively to maintain high fleet availability while avoiding mechanical failure by scheduling preventive maintenance actions. In recent years, reinforcement learning (RL) has emerged as an effective method to optimise complex sequential decision-making problems. In this paper, an RL framework to optimise the operation and maintenance of a fleet of aircraft is developed. Three cases studies, with varying number of aircraft in the fleet, are used to demonstrate the ability of the RL policies to outperform traditional operation/maintenance strategies. As more aircraft are added to the fleet, the combinatorial explosion of the number of possible actions is identified as a main computational limitation. We conclude that the RL policy has potential to support fleet management operators and call for greater research on the application of multi-agent RL for fleet availability optimisation.

Learning to bid and rank together in recommendation systems

AbstractMany Internet applications adopt real-time bidding mechanisms to ensure different services (types of content) are shown to the users through fair competitions. The service offering the highest bid price gets the content slot to present a list of items in its candidate pool. Through user interactions with the recommended items, the service obtains the desired engagement activities. We propose a contextual-bandit framework to jointly optimize the price to bid for the slot and the order to rank its candidates for a given service in this type of recommendation systems. Our method can take as input any feature that describes the user and the candidates, including the outputs of other machine learning models. We train reinforcement learning policies using deep neural networks, and compute top-K Gaussian propensity scores to exclude the variance in the gradients caused by randomness unrelated to the reward. This setup further facilitates us to automatically find accurate reward functions that trade off between budget spending and user engagements. In online A/B experiments on two major services of Facebook Home Feed, Groups You Should Join and Friend Requests, our method statistically significantly boosted the number of groups joined by 14.7%, the number of friend requests accepted by 7.0%, and the number of daily active Facebook users by about 1 million, against strong hand-tuned baselines that have been iterated in production over years.

NPU-Accelerated Imitation Learning for Thermal Optimization of QoS-Constrained Heterogeneous Multi-Cores

Thermal optimization of a heterogeneous clustered multi-core processor under user-defined QoS targets requires application migration and DVFS. However, selecting the core to execute each application and the VF levels of each cluster is a complex problem because (1) the diverse characteristics and QoS targets of applications require different optimizations, and (2) per-cluster DVFS requires a global optimization considering all running applications. State-of-the-art resource management for power or temperature minimization either relies on measurements that are commonly not available (such as power) or fails to consider all the dimensions of the optimization (e.g., by using simplified analytical models). To solve this, ML methods can be employed. In particular, IL leverages the optimality of an oracle policy, yet at low run-time overhead, by training a model from oracle demonstrations. We are the first to employ IL for temperature minimization under QoS targets. We tackle the complexity by training NN at design time and accelerate the run-time NN inference using NPU. While such NN accelerators are becoming increasingly widespread, they are so far only used to accelerate user applications. In contrast, we use for the first time an existing accelerator on a real platform to accelerate NN-based resource management. To show the superiority of IL compared to RL in our targeted problem, we also develop multi-agent RL-based management. Our evaluation on a HiKey 970 board with an Arm big.LITTLE CPU and NPU shows that IL achieves significant temperature reductions at a negligible run-time overhead. We compare TOP-IL against several techniques. Compared to ondemand Linux governor, TOP-IL reduces the average temperature by up to 17 ˆC at minimal QoS violations for both techniques. Compared to the RL policy, our TOP-IL achieves 63 % to 89 % fewer QoS violations while resulting similar average temperatures. Moreover, TOP-IL outperforms the RL policy in terms of stability. We additionally show that our IL-based technique also generalizes to different software (unseen applications) and even hardware (different cooling) than used for training.