Optimal Reward Function Research Articles

Compliance control of a rehabilitation massage robot in dynamic scenes

Abstract Medical robots encounter challenges when interacting with people or operating in complex and dynamic environments due to the variability of human morphology and the unpredictability of environmental changes. Compliance of human-robot interaction is the primary goal of medical robots when in contact with the human body. Therefore, robots must be able to adaptively adjust their forces and actions to ensure safety and comfort during the contact process. This paper focuses on the compliance control of rehabilitation massage robots in dynamic scenes. We propose a mechanical arm compliance control method based on the Soft Actor-Critic (SAC) algorithm. We construct a simulated massage environment in a dynamic scene according to the task requirements and design a massage path covering the entire back. Under the framework of deep reinforcement learning, the optimal reward function is designed to achieve constant force control under dynamic scenes. Through numerous simulation experiments, we have verified that the robotic arm can move along the predetermined path under the massage while maintaining a constant contact force with the body simulation module. The actual contact force and target contact force control are realized within 0.1 N.

ELRL-MD: a deep learning approach for myocarditis diagnosis using cardiac magnetic resonance images with ensemble and reinforcement learning integration

Objective. Myocarditis poses a significant health risk, often precipitated by viral infections like coronavirus disease, and can lead to fatal cardiac complications. As a less invasive alternative to the standard diagnostic practice of endomyocardial biopsy, which is highly invasive and thus limited to severe cases, cardiac magnetic resonance (CMR) imaging offers a promising solution for detecting myocardial abnormalities. Approach. This study introduces a deep model called ELRL-MD that combines ensemble learning and reinforcement learning (RL) for effective myocarditis diagnosis from CMR images. The model begins with pre-training via the artificial bee colony (ABC) algorithm to enhance the starting point for learning. An array of convolutional neural networks (CNNs) then works in concert to extract and integrate features from CMR images for accurate diagnosis. Leveraging the Z-Alizadeh Sani myocarditis CMR dataset, the model employs RL to navigate the dataset’s imbalance by conceptualizing diagnosis as a decision-making process. Main results. ELRL-DM demonstrates remarkable efficacy, surpassing other deep learning, conventional machine learning, and transfer learning models, achieving an F-measure of 88.2% and a geometric mean of 90.6%. Extensive experimentation helped pinpoint the optimal reward function settings and the perfect count of CNNs. Significance. The study addresses the primary technical challenge of inherent data imbalance in CMR imaging datasets and the risk of models converging on local optima due to suboptimal initial weight settings. Further analysis, leaving out ABC and RL components, confirmed their contributions to the model’s overall performance, underscoring the effectiveness of addressing these critical technical challenges.

Performance Analysis of Different Reward Functions in Reinforcement Learning for the Scheduling of Modular Automotive Production Systems

Conventional, linear production lines struggle with the new flexibility requirements of the automotive market. Modular production has the potential to radically improve the production flexibility. However, scheduling modular production systems is still an open research question. Reinforcement learning (RL) is a form of artificial intelligence that shows great potential in scheduling complex modular production systems. Nonetheless, the performance of RL agents heavily depends on their reward function. Designing an optimal reward function is highly complex. This paper addresses this research gap and systematically compares six different reward functions for a variety of different modular production scenarios. In addition, a new learning method using resets is proposed and its performance is compared with the standard learning approach. The results suggest that dense reward functions perform better than a sparse one. However, there are major case-to-case discrepancies. The proposed learning method outperforms the standard learning method by 7 % on average. Nonetheless, the performance difference between different scenarios is larger than with the standard approach.

Energy Efficiency Resource Management for D2D-NOMA Enabled Network: A Dinkelbach Combined Twin Delayed Deterministic Policy Gradient Approach

In this paper, a joint cluster association and power allocation resource management scheme is proposed in a D2D-NOMA-enabled heterogeneous network. The target is to maximize users' long-term cumulative energy efficiency. A novel <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$K$</tex-math></inline-formula> -times iteration method based on matching theory is constructed to handle the NOMA cluster association, which is formulated as the perfect matching of a weighted bipartite graph. Considering it is difficult to obtain the complete instantaneous channel state information, we offer a deep reinforcement learning (DRL) framework, which guarantees the feasibility of DRL methods for the NP-hard power allocation. Meanwhile, an optimal reward function is designed to enhance the training and interacting efficiency. Based on the proposed DRL framework, the Dinkelbach-TD3 power allocation approach is proposed to shrink the action range of twin delayed deep deterministic policy gradient (TD3) with the fractional programming method Dinkelbach, which elaborately increases the training efficiency and the precision of the DRL methods. Simulation results demonstrate that the proposed DRL framework has significant energy efficiency performance improvement.

Vibration and Position Control of a Two-Link Flexible Manipulator Using Reinforcement Learning

In recent years, industries have increasingly emphasized the need for high-speed, energy-efficient, and cost-effective solutions. As a result, there has been growing interest in developing flexible link manipulator robots to meet these requirements. However, reducing the weight of the manipulator leads to increased flexibility which, in turn, causes vibrations. This research paper introduces a novel approach for controlling the vibration and motion of a two-link flexible manipulator using reinforcement learning. The proposed system utilizes trust region policy optimization to train the manipulator’s end effector to reach a desired target position, while minimizing vibration and strain at the root of the link. To achieve the research objectives, a 3D model of the flexible-link manipulator is designed, and an optimal reward function is identified to guide the learning process. The results demonstrate that the proposed approach successfully suppresses vibration and strain when moving the end effector to the target position. Furthermore, the trained model is applied to a physical flexible manipulator for real-world control verification. However, it is observed that the performance of the trained model does not meet expectations, due to simulation-to-real challenges. These challenges may include unanticipated differences in dynamics, calibration issues, actuator limitations, or other factors that affect the performance and behavior of the system in the real world. Therefore, further investigations and improvements are recommended to bridge this gap and enhance the applicability of the proposed approach.

Constant Force-Tracking Control Based on Deep Reinforcement Learning in Dynamic Auscultation Environment.

Intelligent medical robots can effectively help doctors carry out a series of medical diagnoses and auxiliary treatments and alleviate the current shortage of social personnel. Therefore, this paper investigates how to use deep reinforcement learning to solve dynamic medical auscultation tasks. We propose a constant force-tracking control method for dynamic environments and a modeling method that satisfies physical characteristics to simulate the dynamic breathing process and design an optimal reward function for the task of achieving efficient learning of the control strategy. We have carried out a large number of simulation experiments, and the error between the tracking of normal force and expected force is basically within ±0.5 N. The control strategy is tested in a real environment. The preliminary results show that the control strategy performs well in the constant force-tracking of medical auscultation tasks. The contact force is always within a safe and stable range, and the average contact force is about 5.2 N.

Reward criteria impact on the performance of reinforcement learning agent for autonomous navigation

In reinforcement learning, an agent takes action at every time step (follows a policy) in an environment to maximize the expected cumulative reward. Therefore, the shaping of a reward function plays a crucial role in an agent’s learning. Designing an optimal reward function is not a trivial task. In this article, we propose a reward criterion using which we develop different reward functions. The reward criterion chosen is based on the percentage of positive and negative rewards received by an agent. This reward criteria further gives rise to three different classes, ‘Balanced Class,’ ‘Skewed Positive Class,’ and ‘Skewed Negative Class.’ We train a Deep Q-Network agent on a point-goal based navigation task using the different reward classes. We also compare the performance of the proposed classes with a benchmark class. Based on the experiments, the skewed negative class outperforms the benchmark class by achieving very less variance. On the other hand, the benchmark class converges relatively faster than the skewed negative class.

Option compatible reward inverse reinforcement learning

• New method of assigning reward functions for a hierarchical IRL problem is introduced. • The recovered reward functions can be used to transfer knowledge across related tasks. • It also shows better robustness to noise included in expert demonstrations. Reinforcement learning in complex environments is a challenging problem. In particular, the success of reinforcement learning algorithms depends on a well-designed reward function. Inverse reinforcement learning (IRL) solves the problem of recovering reward functions from expert demonstrations. In this paper, we solve a hierarchical inverse reinforcement learning problem within the options framework, which allows us to utilize intrinsic motivation of the expert demonstrations. A gradient method for parametrized options is used to deduce a defining equation for the Q-feature space, which leads to a reward feature space. Using a second-order optimality condition for option parameters, an optimal reward function is selected. Experimental results in both discrete and continuous domains confirm that our recovered rewards provide a solution to the IRL problem using temporal abstraction, which in turn are effective in accelerating transfer learning tasks. We also show that our method is robust to noises contained in expert demonstrations.

An Efficient Algorithm for Deep Stochastic Contextual Bandits

In stochastic contextual bandit (SCB) problems, an agent selects an action based on certain observed context to maximize the cumulative reward over iterations. Recently there have been a few studies using a deep neural network (DNN) to predict the expected reward for an action, and the DNN is trained by a stochastic gradient based method. However, convergence analysis has been greatly ignored to examine whether and where these methods converge. In this work, we formulate the SCB that uses a DNN reward function as a non-convex stochastic optimization problem, and design a stage-wised stochastic gradient descent algorithm to optimize the problem and determine the action policy. We prove that with high probability, the action sequence chosen by our algorithm converges to a greedy action policy respecting a local optimal reward function. Extensive experiments have been performed to demonstrate the effectiveness and efficiency of the proposed algorithm on multiple real-world datasets.

Scalable Inverse Reinforcement Learning Through Multifidelity Bayesian Optimization.

Data in many practical problems are acquired according to decisions or actions made by users or experts to achieve specific goals. For instance, policies in the mind of biologists during the intervention process in genomics and metagenomics are often reflected in available data in these domains, or data in cyber-physical systems are often acquired according to actions/decisions made by experts/engineers for purposes, such as control or stabilization. Quantification of experts' policies through available data, which is also known as reward function learning, has been discussed extensively in the literature in the context of inverse reinforcement learning (IRL). However, most of the available techniques come short to deal with practical problems due to the following main reasons: 1) lack of scalability: arising from incapability or poor performance of existing techniques in dealing with large systems and 2) lack of reliability: coming from the incapability of the existing techniques to properly learn the optimal reward function during the learning process. Toward this, in this brief, we propose a multifidelity Bayesian optimization (MFBO) framework that significantly scales the learning process of a wide range of existing IRL techniques. The proposed framework enables the incorporation of multiple approximators and efficiently takes their uncertainty and computational costs into account to balance exploration and exploitation during the learning process. The proposed framework's high performance is demonstrated through genomics, metagenomics, and sets of random simulated problems.

Limitations of Recursive Logit for Inverse Reinforcement Learning of Bicycle Route Choice Behavior in Amsterdam

Used for route choice modelling by the transportation research community, recursive logit is a form of inverse reinforcement learning. By solving a large-scale system of linear equations recursive logit allows estimation of an optimal (negative) reward function in a computationally efficient way that performs for large networks and a large number of observations. In this paper we review examples of recursive logit and inverse reinforcement learning models applied to real world GPS travel trajectories and explore some of the challenges in modeling bicycle route choice in the city of Amsterdam using recursive logit as compared to a simple baseline multinomial logit model with environmental variables. We discuss conceptual, computational, numerical and statistical issues that we encountered and conclude with recommendation for further research.

Inverse reinforcement learning control for trajectory tracking of a multirotor UAV

The main purpose of this paper is to learn the control performance of an expert by imitating the demonstrations of a multirotor UAV (unmanned aerial vehicle) operated by an expert pilot. First, we collect a set of several demonstrations by an expert for a certain task which we want to learn. We extract a representative trajectory from the dataset. Here, the representative trajectory includes a sequence of state and input. The trajectory is obtained using hidden Markov model (HMM) and dynamic time warping (DTW). In the next step, the multirotor learns to track the trajectory for imitation. Although we have data of feed-forward input for each time sequence, using this input directly can deteriorate the stability of the multirotor due to insufficient data for generalization and numerical issues. For that reason, a controller is needed which generates the input command for the suitable flight maneuver. To design such a controller, we learn the hidden reward function of a quadratic form from the demonstrated flights using inverse reinforcement learning. After we find the optimal reward function that minimizes the trajectory tracking error, we design a reinforcement learning based controller using this reward function. The simulation and experiment applied to a multirotor UAV show successful imitation results.

A continuity question of Dubins and Savage

Abstract Lester Dubins and Leonard Savage posed the question as to what extent the optimal reward function U of a leavable gambling problem varies continuously in the gambling house Γ, which specifies the stochastic processes available to a player, and the utility function u, which determines the payoff for each process. Here a distance is defined for measurable houses with a Borel state space and a bounded Borel measurable utility. A trivial example shows that the mapping Γ ↦ U is not always continuous for fixed u. However, it is lower semicontinuous in the sense that, if Γn converges to Γ, then lim inf Un ≥ U. The mapping u ↦ U is continuous in the supnorm topology for fixed Γ, but is not always continuous in the topology of uniform convergence on compact sets. Dubins and Savage observed that a failure of continuity occurs when a sequence of superfair casinos converges to a fair casino, and queried whether this is the only source of discontinuity for the special gambling problems called casinos. For the distance used here, an example shows that there can be discontinuity even when all the casinos are subfair.

Optimal oil production under mean-reverting Lévy models with regime switching

This paper is concerned with the problem of finding optimal extraction policies for an oil field in light of various financial and economical restrictions and constraints. Taking into account the fact that the oil price in worldwide commodity markets fluctuates randomly following global and seasonal macroeconomic parameters, we model the evolution of the oil price as a mean-reverting regime-switching jump-diffusion process. We formulate this problem as a finite-time horizon optimal control problem, which we solve using the method of viscosity solutions. Moreover, we construct and prove the convergence of a numerical scheme for approximating the optimal reward function and the optimal extraction policy. A numerical example illustrating these results is presented.

Exercising control when confronted by a (Brownian) spider

We consider the Brownian “spider,” a construct introduced in Dubins and Schwarz (1988) and in Barlow and Pitman (1989). In this note, the author proves the “spider” bounds by using the dynamic programming strategy of guessing the optimal reward function and subsequently establishing its optimality by proving its excessiveness.

Optimal Decision for Selling an Illiquid Stock

This paper is concerned with liquidation of an illiquid stock. The stock price follows a fluid model which is dictated by the rates of selling and buying over time. The objective is to maximize the expected overall return. The method of constrained viscosity solution is used to characterize the dynamics governing the optimal reward function and the associated boundary conditions. Numerical examples are given to illustrate the results.

Big Vee: The story of a function, an algorithm, and three mathematical worlds

The Dubins-Savage theory of gambling takes place in a world of finitely additive probability measures defined on all subsets of a set of arbitrary cardinality. When the theory is specialized to a more conventional world of countably additive measures defined on the Borel subsets of a standard Borel space, the question arises whether the gambler is harmed when restricted to measurable strategies; that is, whether the optimal reward function V remains the same. The answer to this question uses methods from the world of descriptive set theory. Ashok Maitra was perhaps the unique person who was completely at home in all three of these mathematical worlds.

On Finite-Difference Approximations for Normalized Bellman Equations

A class of stochastic optimal control problems involving optimal stopping is considered. Methods of Krylov are adapted to investigate the numerical solutions of the corresponding normalized Bellman equations and to estimate the rate of convergence of finite difference approximations for the optimal reward functions.

LIQUIDATION OF A LARGE BLOCK OF STOCK WITH REGIME SWITCHING

In the literature, stock-selling rules are mainly concerned with liquidation of the security within a short period of time. In practice, this is feasible only when a relatively smaller number of shares of a stock is involved. Selling a large position in a market place normally depresses the market if sold in a short period of time, which would result in poor filling prices. Comparing to the existing results in the literature, this work has two distinct features. First, the underlying stock price is modeled using a geometric Brownian motion formulation with regime switching in which the jump rate depends on the selling intensity. A larger selling intensity makes the regime more likely to change from a higher return mode to a lower one or forces the return mode to stay in the lower one longer. Secondly, we consider the liquidation strategy for selling a large block of stock by selling much smaller number of shares over a longer period of time. By using a fluid model, in which the number of shares is treated as fluid (continuous), we treat the selling rule problem where the corresponding liquidation is dictated by the rate of selling over time. Our objective is to maximize the expected overall return. Thus it may be formulated as a stochastic control problem with state constraints. Method viscosity solution is used to characterize the dynamics governing the optimal reward function and the associated boundary conditions. Numerical examples are reported to illustrate the results.

Liquidation of a large block of stock

In the financial engineering literature, stock-selling rules are mainly concerned with liquidation of the security within a short period of time. This is practically feasible only when a relative smaller number of shares of a stock is involved. Selling a large position in a market place normally depresses the market if sold in a short period of time, which would result in poor filling prices. In this paper, we consider the liquidation strategy for selling a large block of stock by selling much smaller number of shares over a longer period of time. In particular, we treat the selling rule problem by using a fluid model in the sense that the number of shares are treated as fluid (continuous) and the corresponding liquidation is dictated by the rate of selling over time. The objective is to maximize the expected overall return. The underlying problem may be formulated as a stochastic control problem with state constraints. Method of constrained viscosity solution is used to characterize the dynamics governing the optimal reward function and the associated boundary conditions. Numerical examples are reported to illustrate the results.