Actor-critic Reinforcement Learning Method Research Articles

An effective deep actor-critic reinforcement learning method for solving the flexible job shop scheduling problem

An effective deep actor-critic reinforcement learning method for solving the flexible job shop scheduling problem

A Novel Adaptive Undersampling Framework for Class-Imbalance Fault Detection

Class-imbalance is a prevalent and challenging problem in the field of fault detection. The undersampling ensemble framework is an effective method to deal with imbalance problems. However, designing a suitable sampling strategy to generate effective and divergent subsets is a major difficulty of this type of method. Hence, we propose a novel adaptive undersampling framework. It models the entire training process as a Markov decision process (MDP), thus, enabling dynamic decision-making for subsequent sampling strategies based on the current training performance of the ensemble framework. The sampler is optimized by the soft actor-critic reinforcement learning method. Considering the imbalance dataset's nature and the need for state definition, the clustering method is applied to the original training dataset. The state (training performance) and the action (sampling strategy) are determined according to the clustering results. The unique state definition and sampling decision mechanism are designed to ensure the convergence speed of MDP and improve the divergence of the subsets. We validate the effectiveness of the proposed framework on the real-world wind turbine blade cracking datasets and the high-speed train braking system dataset. The experimental results show that the classification performance and robustness of the proposed framework are significantly better than the 16 benchmark methods.

A novel priority dispatch rule generation method based on graph neural network and reinforcement learning for distributed job-shop scheduling

With the development of a global economy, distributed manufacturing becomes common in the industrial field. The Distributed Job-shop Scheduling Problem (DJSP), which is widespread in real-life production, is a hotspot in the academic field. The existing Priority Dispatch Rules (PDRs), which are used to assign a value to each waiting job according to some method and select the job with minimum or maximum “value” for next processing, are all relatively simple but lack self-learning ability, while the metaheuristics are all complex and with fixed evolutionary trajectory and cannot change with the manufacturing environment. This paper proposes a novel PDR generation method based on Graph Neural Network (GNN) and Reinforcement Learning (RL), which can self-learn and self-evolute by interacting with the scheduling environment. To combine DJSP with GNN closely, a new solution representation based on disjunctive graph is designed. DJSP is formulated as a Markov decision process, and the problem features and inner connections among the vertices of the disjunctive graph are fully explored by the GNN. An Actor-Critic RL method is applied to automatically train the network parameters to optimize the policy, so that it can be used to schedule the best action at each step. Comprehensive experiments on 240 test instances are conducted to evaluate the performance of the proposed method, and the results indicate that the proposed method shows greater effectiveness, generalizability and stability than other 8 classical PDRs, 5 metaheuristics and 3 RL-based methods.

Scheduling single-satellite observation and transmission tasks by using hybrid Actor-Critic reinforcement learning

Earth observation satellites(EOS) generate a large amount of observation data in intensive observation scenarios, while the ability of data storage on the EOS is limited. It makes integrating satellite observation and data transmission tasks for EOS imperative. This paper establishes a time-continuous model for the single EOS integrated scheduling problem, which considers data transmission and observation simultaneously. A hybrid Actor-Critic reinforcement learning method is adopted to solve the EOS integrated scheduling problem for a more efficient solution in intensive observation scenarios. Furthermore, the algorithm can flexibly determine the start and end time of the data transmission task. Experimental results show that the hybrid Actor-Critic reinforcement learning method deals with a large scale of problems with high efficiency and good results.

Integrated eco-driving automation of intelligent vehicles in multi-lane scenario via model-accelerated reinforcement learning

The development of intelligent driving technologies is expected to have the potential in energy economics. Some reported studies mainly focused on the economical driving performance in cruising, following, or ramping scenarios, where longitudinal control is primarily considered. The impact of lateral decisions on economical performance is rarely discussed, especially in traffic flows. In the multi-lane scenario, the upper decision-making module could output reasonable behavior selections to avoid the limitation of single longitudinal control and further enhance the energy-saving potential in traffic flows, such as the appropriate lane-keeping or lane-changing proposal. Furthermore, designing comprehensive rules to coordinate diverse driving goals with separated decision-making and control modules is challenging. Therefore, this paper proposes an integrated decision and control framework for economical driving in the multi-lane scenario, based on the actor–critic reinforcement learning method. The proposed integrated framework contains two function layers: a static-evaluating layer and a dynamic-tracking layer. The former, i.e., the critic network, considers static information, evaluates potentially feasible lanes, and selects an advantage lane as the lane-changing proposal. The latter, i.e., the actor network, obtains dynamic traffic information and solves a constrained control problem. Finally, the solution aims to achieve obstacle avoidance and economical and stable tracking to the proposed advantage lane as far as possible. Furthermore, a model-accelerated soft actor–critic (MSAC) algorithm is developed to simultaneously solve the integrated decision and control problem. Simulation results show that the proposed learning-based integrated method could achieve economical driving and significantly outperform baselines in accumulated performance, energy efficiency, and driving comfort.

Reinforcement Learning for Active Noise Control in a Hydraulic System

Abstract Hydraulic pressure ripple in a pump, as a result of converting rotational power to fluid power, continues to be a problem faced when developing hydraulic systems due to the resulting noise generated. In this paper, we present simulation results from leveraging an actor-critic reinforcement learning method as the control method for active noise control in a hydraulic system. The results demonstrate greater than 96%, 81%, and 61% pressure ripple reduction for the first, second, and third harmonics, respectively, in a single operating point test, along with the advantage of feed forward like control for high bandwidth response during dynamic changes in the operating point. It also demonstrates the disadvantage of long convergence times while the controller is effectively learning the optimal control policy. Additionally, this work demonstrates the ancillary benefit of the elimination of the injection of white noise for the purpose of system identification in the current state of the art.

Learning from demonstration: Teaching a myoelectric prosthesis with an intact limb via reinforcement learning.

Prosthetic arms should restore and extend the capabilities of someone with an amputation. They should move naturally and be able to perform elegant, coordinated movements that approximate those of a biological arm. Despite these objectives, the control of modern-day prostheses is often nonintuitive and taxing. Existing devices and control approaches do not yet give users the ability to effect highly synergistic movements during their daily-life control of a prosthetic device. As a step towards improving the control of prosthetic arms and hands, we introduce an intuitive approach to training a prosthetic control system that helps a user achieve hard-to-engineer control behaviours. Specifically, we present an actor-critic reinforcement learning method that for the first time promises to allow someone with an amputation to use their non-amputated arm to teach their prosthetic arm how to move through a wide range of coordinated motions and grasp patterns. We evaluate our method during the myoelectric control of a multi-joint robot arm by non-amputee users, and demonstrate that by using our approach a user can train their arm to perform simultaneous gestures and movements in all three degrees of freedom in the robot's hand and wrist based only on information sampled from the robot and the user's above-elbow myoelectric signals. Our results indicate that this learning-from-demonstration paradigm may be well suited to use by both patients and clinicians with minimal technical knowledge, as it allows a user to personalize the control of his or her prosthesis without having to know the underlying mechanics of the prosthetic limb. These preliminary results also suggest that our approach may extend in a straightforward way to next-generation prostheses with precise finger and wrist control, such that these devices may someday allow users to perform fluid and intuitive movements like playing the piano, catching a ball, and comfortably shaking hands.

Neuromuscular control of the point to point and oscillatory movements of a sagittal arm with the actor–critic reinforcement learning method

In this study, we have used a single link system with a pair of muscles that are excited with alpha and gamma signals to achieve both point to point and oscillatory movements with variable amplitude and frequency. The system is highly nonlinear in all its physical and physiological attributes. The major physiological characteristics of this system are simultaneous activation of a pair of nonlinear muscle-like-actuators for control purposes, existence of nonlinear spindle-like sensors and Golgi tendon organ-like sensor, actions of gravity and external loading. Transmission delays are included in the afferent and efferent neural paths to account for a more accurate representation of the reflex loops. A reinforcement learning method with an actor–critic (AC) architecture instead of middle and low level of central nervous system (CNS), is used to track a desired trajectory. The actor in this structure is a two layer feedforward neural network and the critic is a model of the cerebellum. The critic is trained by state-action-reward-state-action (SARSA) method. The critic will train the actor by supervisory learning based on the prior experiences. Simulation studies of oscillatory movements based on the proposed algorithm demonstrate excellent tracking capability and after 280 epochs the RMS error for position and velocity profiles were 0.02, 0.04 rad and rad/s, respectively.

The role of multisensor data fusion in neuromuscular control of a sagittal arm with a pair of muscles using actor-critic reinforcement learning method

In this study, we consider the role of multisensor data fusion in neuromuscular control using an actor-critic reinforcement learning method. The model we use is a single link system actuated by a pair of muscles that are excited with alpha and gamma signals. Various physiological sensor information such as proprioception, spindle sensors, and Golgi tendon organs have been integrated to achieve an oscillatory movement with variable amplitude and frequency, while achieving a stable movement with minimum metabolic cost and coactivation. The system is highly nonlinear in all its physical and physiological attributes. Transmission delays are included in the afferent and efferent neural paths to account for a more accurate representation of the reflex loops. This paper proposes a reinforcement learning method with an Actor-Critic architecture instead of middle and low level of central nervous system (CNS). The Actor in this structure is a two layer feedforward neural network and the Critic is a model of the cerebellum. The Critic is trained by the State-Action-Reward-State-Action (SARSA) method. The Critic will train the Actor by supervisory learning based on previous experiences. The reinforcement signal in SARSA is evaluated based on available alternatives concerning the concept of multisensor data fusion. The effectiveness and the biological plausibility of the present model are demonstrated by several simulations. The system showed excellent tracking capability when we integrated the available sensor information. Addition of a penalty for activation of muscles resulted in much lower muscle coactivation while keeping the movement stable.