State-action Space Research Articles

A general motion controller based on deep reinforcement learning for an autonomous underwater vehicle with unknown disturbances

This paper studies the application of deep Reinforcement Learning (RL) in the motion control of an underactuated autonomous underwater vehicle (AUV) with unknown disturbances. Firstly, a general state space, action space and reward function are designed for motion control problems rather than each specific motion control task, which ensures the generality of our method. Furthermore, a virtual AUV model with partial random disturbances is established, and on this basis, a simulation training method is developed to solve the problems of extremely high risk and extremely low efficiency caused by training in actual experiments. Then, in order to directly deploy the optimal control policy obtained through simulation training to an actual AUV, we employ Extended State Observers (ESOs) to estimate the unknown disturbances in five degrees of freedom, and give a deployment method using the estimated values as the disturbance state vector and compensation vector. Combining the above training method and deployment method, a novel general motion controller is proposed. Finally, four different AUV motion control simulations are carried out, and the results confirm the generality and effectiveness of our proposed controller.

Adaptive Discretization in Online Reinforcement Learning

Adaptive Discretization in Reinforcement Learning Performance guarantees for RL algorithms are typically for worst case instances, which are pathological by design and not observed in meaningful applications. Moreover, many domains (such as computer systems and networking applications) have large state-action spaces and require algorithms to execute with low latency. This phenomenon highlights a trifecta of goals for practical RL algorithms: low sample, storage, and computational complexity. In this work, we develop an algorithmic framework for nonparametric RL with data-driven adaptive discretization. Our framework has provably better sample, storage, and computational complexity than uniform discretization or kernel regression methods. Moreover, we highlight how the performance guarantees are min-max optimal with respect to a novel instance-specific complexity measure that captures structure in facility location and newsvendor models.

A Vision-Based Bio-Inspired Reinforcement Learning Algorithms for Manipulator Obstacle Avoidance

Path planning for robotic manipulators has proven to be a challenging issue in industrial applications. Despite providing precise waypoints, the traditional path planning algorithm requires a predefined map and is ineffective in complex, unknown environments. Reinforcement learning techniques can be used in cases where there is a no environmental map. For vision-based path planning and obstacle avoidance in assembly line operations, this study introduces various Reinforcement Learning (RL) algorithms based on discrete state-action space, such as Q-Learning, Deep Q Network (DQN), State-Action-Reward- State-Action (SARSA), and Double Deep Q Network (DDQN). By positioning the camera in an eye-to-hand position, this work used color-based segmentation to identify the locations of obstacles, start, and goal points. The homogeneous transformation technique was used to further convert the pixel values into robot coordinates. Furthermore, by adjusting the number of episodes, steps per episode, learning rate, and discount factor, a performance study of several RL algorithms was carried out. To further tune the training hyperparameters, genetic algorithms (GA) and particle swarm optimization (PSO) were employed. The length of the path travelled, the average reward, the average number of steps, and the time required to reach the objective point were all measured and compared for each of the test cases. Finally, the suggested methodology was evaluated using a live camera that recorded the robot workspace in real-time. The ideal path was then drawn using a TAL BRABO 5 DOF manipulator. It was concluded that waypoints obtained via Double DQN showed an improved performance and were able to avoid the obstacles and reach the goal point smoothly and efficiently.

Deep reinforcement learning based energy storage management strategy considering prediction intervals of wind power

Wind power generation combined with energy storage is able to maintain energy balance and realize stable operation. This article proposes a data-driven energy storage management strategy considering the prediction intervals of wind power. Firstly, a power interval prediction model is established based on long-short term memory and lower and upper bound estimation (LUBE) to quantify the uncertainty of wind power, which solves the issue that traditional LUBE cannot adopt gradient descent method. Secondly, the energy storage management is transformed into Markov decision process and solved by deep reinforcement learning. The state space, action space and reward function of the interaction between agent and environment are established, and the value function is approximated through the deep Q network. Then, according to the real-time state, such as wind power, power prediction intervals, local load, dynamic electricity price and state of charge, the proposed strategy can make the charge/discharge schedule automatically. Finally, the effectiveness and superiority of the proposed energy storage management strategy are verified based on real wind farm dataset. The proportion of wrong decisions is zero, and daily transaction cost and wear cost of energy storage management system decrease significantly.

COLREGs-Compliant Multi-Ship Collision Avoidance Based on Multi-Agent Reinforcement Learning Technique

The congestion of waterways can easily lead to traffic hazards. Moreover, according to the data, the majority of sea collisions are caused by human error and the failure to comply with the Convention on the International Regulation for the preventing Collision at Sea (COLREGs). To avoid this situation, ship automatic collision avoidance has become one of the most important research issues in the field of marine engineering. In this study, an efficient method is proposed to solve multi-ship collision avoidance problems based on the multi-agent reinforcement learning (MARL) algorithm. Firstly, the COLREGs and ship maneuverability are considered for achieving multi-ship collision avoidance. Subsequently, the Optimal Reciprocal Collision Avoidance (ORCA) algorithm is utilized to detect and reduce the risk of collision. Ships can operate at the safe velocity computed by the ORCA algorithm to avoid collisions. Finally, the Nomoto three-degrees-of-freedom (3-DOF) model is used to simulate the maneuvers of ships. According to the above information and algorithms, this study designs and improves the state space, action space and reward function. For validating the effectiveness of the method, this study designs various simulation scenarios with thorough performance evaluations. The simulation results indicate that the proposed method is flexible and scalable in solving multi-ship collision avoidance, complying with COLREGs in various scenarios.

Intelligent Distributed Temperature and Humidity Control Mechanism for Uniformity and Precision in the Indoor Environment

The temperature and relative humidity (hereafter called humidity) in the indoor environment is closely related to the operation of its control system. The centralized control system, which has identical air inlets not only leads to uneven indoor temperatures and humidity but also highly controlled latency when interference occurs. To address this challenge and improve the precision and uniformity of temperatures and humidity in the indoor environment, we propose a distributed temperature and humidity control (DTHC) framework based on deep reinforcement learning (DRL). In this work, we use a constant temperature and humidity air-conditioning (CTHA) system for a museum as a case study to validate the optimization performance of the proposed controller. The state–action space, reward function, and DRL network structure are proposed. The air flow rate of multiple air inlets of CTHA is adjusted according to the feedback from the distributed temperature and humidity sensors. We develop a DRL-based computational fluid dynamics (CFD) experiment platform to evaluate the proposed mechanism. The experiment results show that our approach can improve the precision and uniformity of the temperature and humidity while enhancing the anti-interference capability of the control system. The adjustment time and energy consumed to reach the desired indoor air temperatures and humidity are reduced compared with rule-based methods.

Independent double DQN-based multi-agent reinforcement learning approach for online two-stage hybrid flow shop scheduling with batch machines

Two-stage hybrid flow shop scheduling with batch machines and jobs arriving over time is complex and challenging in various real-world production scenarios. For the online scheduling problem, traditional heuristic rules can quickly respond to dynamically arrived jobs, while with poor and unstable performance. To close the research gap in the problem, this paper proposes an independent double deep-q-network-based multi-agent reinforcement learning (MA-IDDQN) approach to produce an adaptive rule for batch forming and scheduling. Specifically, the online scheduling problem is transformed into a cooperative Markov decision process by defining state space, action space, and reward function for different agents. Then, two agents are constructed and trained via double DQN to address the batch forming task and scheduling task respectively. Meanwhile, multi-agent cooperates through the behavior analysis mechanism among agents. Moreover, we designed a ε-greedy policy considering waiting in batch forming to make a reasonable decision through historical data. To validate the proposed approach, 27 instances with different scales are settled and contrasted. By comparing with frequently-used heuristic rules and other deep reinforcement learning methods, the experimental results demonstrate that the MA-IDDQN can integrate online batch forming and scheduling to minimize the total tardiness time effectively.

Dynamic Task Software Caching-Assisted Computation Offloading for Multi-Access Edge Computing

In multi-access edge computing (MEC), most existing task software caching works focus on statically caching data at the network edge, which may hardly preserve high reusability due to the time-varying user requests in practice. To this end, this work considers dynamic task software caching at the MEC server to assist users’ task execution. Specifically, we formulate a joint task software caching update (TSCU) and computation offloading (COMO) problem to minimize users’ energy consumption while guaranteeing delay constraints, where the limited cache size and computation capability of the MEC server, as well as the time-varying task demand of users are investigated. This problem is proved to be non-deterministic polynomial-time hard, so we transform it into two sub-problems according to their temporal correlations, i.e., the real-time COMO problem and the Markov decision process-based TSCU problem. We first model the COMO problem as a multi-user game and propose a decentralized algorithm to address its Nash equilibrium solution. We then propose a double deep Q-network (DDQN)-based method to solve the TSCU policy. To reduce the computation complexity and convergence time, we provide a new design for the deep neural network (DNN) in DDQN, named state coding and action aggregation (SCAA). In SCAA-DNN, we introduce a dropout mechanism in the input layer to code users’ activity states. Additionally, at the output layer, we devise a two-layer architecture to dynamically aggregate caching actions, which is able to solve the huge state-action space problem. Simulation results show that the proposed solution outperforms existing schemes, saving over 12% energy, and converges with fewer training episodes.

Deep Reinforcement Learning Approach for Material Scheduling Considering High-Dimensional Environment of Hybrid Flow-Shop Problem

Manufacturing sites encounter various scheduling problems, which must be dealt with to efficiently manufacture products and reduce costs. With the development of smart factory technology, many elements at manufacturing sites have become unmanned and more complex. Moreover, owing to the mixing of several processes in one production line, the need for efficient scheduling of materials has emerged. The aim of this study is to solve the material scheduling problem of many machines in a hybrid flow-shop environment using deep reinforcement learning. Most previous work has ignored some conditions, which were critical for solving practical problems. Such critical conditions make the scheduling more complex and difficult to solve. They expand the size of the state and large action space and make learning in an environment with many machines problematic. In this study, a reinforcement learning approach was developed considering practical factors such as the processing time and material transfer to solve realistic manufacturing scheduling problems. Additionally, a method to simplify the high-dimensional environmental space at manufacturing sites for efficient learning was established to solve the problem of learning in a high-dimensional space. Through experiments, we showed that our approach could optimally schedule material scheduling in multi-process lines, which contributes to realistic manufacturing intelligence.

A general motion control architecture for an autonomous underwater vehicle with actuator faults and unknown disturbances through deep reinforcement learning

This paper studies the application of deep reinforcement learning (RL) in the motion control tasks of an underactuated AUV with actuator faults and unknown disturbances. Firstly, a general state space and action space that can be applied to a variety of motion control tasks are customized. Furthermore, a universal reward function is designed to minimize the energy consumption of the AUV under the condition that the learned optimal control policy can achieve fast and high-precision control. Then, a simulation method with random reference values and simulated random disturbances is given to train the deep RL algorithm. In order to deploy the optimal control policy obtained from the simulation training directly to an actual AUV, we design five extended state observers (ESOs) to estimate the unknown disturbances in different directions, and take the estimated values as the disturbance states required to obtain the optimal control action. Combined with these ideas, a general AUV motion control architecture with simulation training and deployment process is developed. Finally, four different AUV motion control experiments are carried out, and the results confirm the generality and effectiveness of our architecture.

A Load-Balanced and Energy-Efficient Navigation Scheme for UAV-Mounted Mobile Edge Computing

With the development of Unmanned Aerial Vehicles (UAVs) in recent years, UAV-mounted Mobile Edge Computing (MEC) systems are widely used for broadband connectivity and emergency communications in areas without internet. However, the onboard energy of UAVs is limited, and how to provide long-term computing services for ground User Equipments (UEs) still faces significant challenges. In this paper, UAVs are used as mobile base stations to offer MEC services for UEs. Unlike existing solutions that mainly solve UAV path planning from convex optimization methods, we propose a Multi-Agent Path planning (MAP) scheme based on deep reinforcement learning. We aim to minimize the total energy consumption of UAVs while completing the offloading tasks of UEs. At the same time, fairness is taken into account to ensure UE offloading balance and UAV load balance. To this end, we establish the optimization problem and design the state space, action space, and reward function for modeling each UAV through Deep Neural Networks (DNNs). Extensive simulation experiments show that the proposed scheme outperforms other benchmark schemes.

Day‐ahead optimal dispatching of hybrid power system based on deep reinforcement learning

The problem of optimal dispatching of a power system containing a high proportion of renewable energy is of great significance for the realisation of new energy consumption and the economic and reliable operation of the power system. For the solution of non-linear, non-convex, multi-objective problems for the optimal operation design of a power system with wind and photovoltaic access, traditional methods have difficulties in terms of computational real-time and iterative convergence. To address this issue, a deep reinforcement learning-based optimal scheduling method for the hybrid power system is proposed, which enables continuous action control to obtain an optimal scheduling strategy through the interaction between the agent and the hybrid power system. Firstly, a mathematical description of the optimal scheduling problem containing wind power and photovoltaic power system is presented, and the state space, action space, and reward function of the agent are designed. Secondly, the basic framework of the deep reinforcement learning optimal scheduling model is constructed, and the basic principles of the twin delayed deep deterministic policy gradient algorithm are introduced. Finally, the effectiveness of the deep reinforcement learning model for day-ahead optimal scheduling of the hybrid power system is verified by means of an arithmetic analysis of the modified New England 39-bus system.

An Optimal Scheduling Strategy of a Microgrid with V2G Based on Deep Q-Learning

In recent years, the access of various distributed power sources and electric vehicles (EVs) has brought more and more randomness and uncertainty to the operation and regulation of microgrids. Therefore, an optimal scheduling strategy for microgrids with EVs based on Deep Q-learning is proposed in this paper. Firstly, a vehicle-to-grid (V2G) model considering the mobility of EVs and the randomness of user charging behavior is proposed. The charging time distribution model, charging demand model, state-of-charge (SOC) dynamic model and the model of travel location are comprehensively established, thereby realizing the construction of the mathematical model of the microgrid with EVs: it can obtain the charging/discharging situation in the EV station, so as to obtain the overall output power of the EV station. Secondly, based on Deep Q-learning, the state space and action space are set up according to the actual microgrid system, and the design of the optimal scheduling reward function is completed with the goal of economy. Finally, the calculation example results show that compared with the traditional optimization algorithm, the strategy proposed in this paper has the ability of online learning and can cope with the randomness of renewable resources better. Meanwhile, the agent with experience replay ability can be trained to complete the evolution process, so as to adapt to the nonlinear influence caused by the mobility of EVs and the periodicity of user behavior, which is feasible and superior in the field of optimal scheduling of microgrids with renewable resources and EVs.

Blockchain-Based Multi-Access Edge Computing for Future Vehicular Networks: A Deep Compressed Neural Network Approach

Vehicular ad hoc networks (VANETs) have become an important branch of future 6G smart wireless communications. As an emerging key technology, multi-access edge computing (MEC) provides low-latency, high-speed, and high-capacity network services for the VANETs. In this paper, we propose a novel framework for blockchain-based, hierarchical multi-access edge computing for the future VANET ecosystem (BMEC-FV). In the underlying VANET environment, we propose a trust model to ensure the security of the communication link between vehicles. Multiple MEC servers calculate the trust between vehicles through computing offloading. Meanwhile, the blockchain system plays an important role to manage the entire BMEC-FV architecture. We aim to optimize the throughput and the quality of services (QoS) for MEC users in the lower layer of the system architecture. In this framework, the main challenge is how to effectively reach consensus among blockchain nodes while ensuring the performance of MEC systems and blockchains. The blocksize of blockchain nodes, the number of consensus nodes, reliable features of each vehicle, and the number of producing blocks for each block producer are considered in a joint optimization problem, which is modeled as a Markov decision process with state space, action space, and reward function. Since it is difficult for this to be solved by traditional methods, we propose a novel deep compressed neural network scheme. Simulation results illustrate the superiority of the BMEC-FV ecosystem.

Utilizing Human Feedback in Autonomous Driving: Discrete vs. Continuous

Deep reinforcement learning (Deep RL) algorithms are defined with fully continuous or discrete action spaces. Among DRL algorithms, soft actor–critic (SAC) is a powerful method capable of handling complex and continuous state–action spaces. However, a long training time and data efficiency are the main drawbacks of this algorithm, even though SAC is robust for complex and dynamic environments. One of the proposed solutions to overcome this issue is to utilize human feedback. In this paper, we investigate different forms of human feedback: head direction vs. steering and discrete vs. continuous feedback. To this end, a real-time human demonstration from steer and human head direction with discrete or continuous actions were employed as human feedback in an autonomous driving task in the CARLA simulator. We used alternating actions from a human expert and SAC to have a real-time human demonstration. Furthermore, to test the method without potential individual differences in human performance, we tested the discrete vs. continuous feedback in an inverted pendulum task, with an ideal controller to stand in for the human expert. The results for both the CARLA and the inverted pendulum tasks showed a significant reduction in the training time and a significant increase in gained rewards with discrete feedback, as opposed to continuous feedback, while the action space remained continuous. It was also shown that head direction feedback can be almost as good as steering feedback. We expect our findings to provide a simple yet efficient training method for Deep RL for autonomous driving, utilizing multiple sources of human feedback.

A policy iteration method for improving robot assembly trajectory efficiency

Bolt assembly by robots is a vital and difficult task for replacing astronauts in extra-vehicular activities (EVA), but the trajectory efficiency still needs to be improved during the wrench insertion into hex hole of bolt. In this paper, a policy iteration method based on reinforcement learning (RL) is proposed, by which the problem of trajectory efficiency improvement is constructed as an issue of RL-based objective optimization. Firstly, the projection relation between raw data and state-action space is established, and then a policy iteration initialization method is designed based on the projection to provide the initialization policy for iteration. Policy iteration based on the protective policy is applied to continuously evaluating and optimizing the action-value function of all state-action pairs till the convergence is obtained. To verify the feasibility and effectiveness of the proposed method, a noncontact demonstration experiment with human supervision is performed. Experimental results show that the initialization policy and the generated policy can be obtained by the policy iteration method in a limited number of demonstrations. A comparison between the experiments with two different assembly tolerances shows that the convergent generated policy possesses higher trajectory efficiency than the conservative one. In addition, this method can ensure safety during the training process and improve utilization efficiency of demonstration data.

A Deep Deterministic Policy Gradient Approach for Vehicle Speed Tracking Control With a Robotic Driver

In performance tests, replacing humans with robotic drivers has many advantages, such as high efficiency and high security. To realize the vehicle speed tracking control with a robotic driver, this article proposes a novel deep reinforcement learning (DRL) approach based on deep deterministic policy gradient (DDPG). Specifically, the design of the approach includes state space, action space, reward function, and control algorithm. Then, to shorten the training time, the proposed approach utilizes the basic fundamental relationship between vehicle speed and pedal opening to intervene in network exploration. Furthermore, to solve speed fluctuations in low-speed sections, the replay buffer is optimized by adding weighted training samples. Experiments are conducted on fifteen cars, and results show that the proposed algorithm can effectively control the vehicle speed. Generally, it only needs three or four episodes of training to meet the requirements. Compared with the Segment-PID method, the proposed method has a smoother speed and fewer overbound times. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article was motivated by the algorithm of deep reinforcement learning (DRL), which can be applied to automatic control including vehicle speed tracking control with a robotic driver. With the complexity, the delay of the vehicle model, and the dead zone of the pedal, it is difficult for existing classic control theories to achieve satisfactory vehicle speed tracking. This article analyzes the mathematical model based on the DRL theory and the vehicle dynamic model, then builds the speed tracking controller based on the deep deterministic policy gradient (DDPG) algorithm, and makes corresponding optimizations for the problems in the experiment. Through the control software integrated with the DDPG algorithm, the operator can easily realize the training and testing of the neural network, which greatly reduces the time and cost of vehicle testing and is of great significance to the emission certification of new vehicles.

Discrete particle swarm algorithm with Q-Learning for solving flexible job shop scheduling problem with parallel batch processing machine

For the flexible job shop scheduling problem with parallel batch processing machines (FJSP-PBPM), the mathematical model of the problem is defined, and an improved enhanced discrete particle swarm algorithm is proposed to solve this problem. In terms of global search, an improved discrete particle swarm algorithm is adopted, and effective particle velocity and position update formulas are designed; in the local search, in order to speed up the convergence speed and avoid local convergence, it is integrated into the q-learning method in reinforcement learning. The state space, action space, reward function, and Q-table update methods that meet the scheduling problem are designed, and the proposed hybrid algorithm achieves an effective balance between global search and local search capabilities. At the same time, according to the characteristics of FJSP-PBPM, effective process sequence scheduling rules and batch machine selection rules are designed respectively to improve the search ability of the algorithm. Through the standard FJSP benchmark problem and the production data of a transformer manufacturer that meets the characteristics of the FJSP-PBPM problem, the validity and feasibility of the proposed algorithm are verified.

Optimizing Constrained Guidance Policy With Minimum Overload Regularization

Using reinforcement learning (RL) algorithm to optimize guidance law can address non-idealities in complex environment. However, the optimization is difficult due to huge state-action space, unstable training, and high requirements on expertise. In this paper, the constrained guidance policy of a neural guidance system is optimized using improved RL algorithm, which is motivated by the idea of traditional model-based guidance method. A novel optimization objective with minimum overload regularization is developed to restrain the guidance policy directly from generating redundant missile maneuver. Moreover, a bi-level curriculum learning is designed to facilitate the policy optimization. Experiment results show that the proposed minimum overload regularization can reduce the vertical overloads of missile significantly, and the bi-level curriculum learning can further accelerate the optimization of guidance policy.

Optimistic Initialization for Exploration in Continuous Control

Optimistic initialization underpins many theoretically sound exploration schemes in tabular domains; however, in the deep function approximation setting, optimism can quickly disappear if initialized naively. We propose a framework for more effectively incorporating optimistic initialization into reinforcement learning for continuous control. Our approach uses metric information about the state-action space to estimate which transitions are still unexplored, and explicitly maintains the initial Q-value optimism for the corresponding state-action pairs. We also develop methods for efficiently approximating these training objectives, and for incorporating domain knowledge into the optimistic envelope to improve sample efficiency. We empirically evaluate these approaches on a variety of hard exploration problems in continuous control, where our method outperforms existing exploration techniques.