Deep reinforcement learning for long-horizon reservoir operation: Temporal horizon, state representation, and hydrological data synthesis

The arrival of intelligent, general-purpose robots that can learn to perform new tasks autonomously has been promised for a long time now. Deep reinforcement learning, which combines reinforcement learning with deep neural network function approximation, has the potential to enable robots to learn to perform a wide range of new tasks while requiring very little prior knowledge or human help. This framework might therefore help to finally make general purpose robots a reality. However, the biggest successes of deep reinforcement learning have so far been in simulated game settings. To translate these successes to the real world, significant improvements are needed in the ability of these methods to learn quickly and safely. This thesis investigates what is needed to make this possible and makes contributions towards this goal. <br/><br/>Before deep reinforcement learning methods can be successfully applied in the robotics domain, an understanding is needed of how, when, and why deep learning and reinforcement learning work well together. This thesis therefore starts with a literature review, which is presented in Chapter 2. While the field is still in some regards in its infancy, it can already be noted that there are important components that are shared by successful algorithms. These components help to reconcile the differences between classical reinforcement learning methods and the training procedures used to successfully train deep neural networks. The main challenges in combining deep learning with reinforcement learning center around the interdependencies of the policy, the training data, and the training targets. Commonly used tools for managing the detrimental effects caused by these interdependencies include target networks, trust region updates, and experience replay buffers. Besides reviewing these components, a number of the more popular and historically relevant deep reinforcement learning methods are discussed.<br/><br/>Reinforcement learning involves learning through trial and error. However, robots (and their surroundings) are fragile, which makes these trials---and especially errors---very costly. Therefore, the amount of exploration that is performed will often need to be drastically reduced over time, especially once a reasonable behavior has already been found. We demonstrate how, using common experience replay techniques, this can quickly lead to forgetting previously learned successful behaviors. This problem is investigated in Chapter 3. Experiments are conducted to investigate what distribution of the experiences over the state-action space leads to desirable learning behavior and what distributions can cause problems. It is shown how actor-critic algorithms are especially sensitive to the lack of diversity in the action space that can result form reducing the amount of exploration over time. Further relations between the properties of the control problem at hand and the required data distributions are also shown. These include a larger need for diversity in the action space when control frequencies are high and a reduced importance of data diversity for problems where generalizing the control strategy across the state-space is more difficult.<br/><br/>While Chapter 3 investigates what data distributions are most beneficial, Chapter 4 instead proposes practical algorithms to {select} useful experiences from a stream of experiences. We do not assume to have any control over the stream of experiences, which makes it possible to learn from additional sources of experience like other robots, experiences obtained while learning different tasks, and experiences obtained using predefined controllers. We make two separate judgments on the utility of individual experiences. The first judgment is on the long term utility of experiences, which is used to determine which experiences to keep in memory once the experience buffer is full. The second judgment is on the instantaneous utility of the experience to the learning agent. This judgment is used to determine which experiences should be sampled from the buffer to be learned from. To estimate the short and long term utility of the experiences we propose proxies based on the age, surprise, and the exploration intensity associated with the experiences. It is shown how prior knowledge of the control problem at hand can be used to decide which proxies to use. We additionally show how the knowledge of the control problem can be used to estimate the optimal size of the experience buffer and whether or not to use importance sampling to compensate for the bias introduced by the selection procedure. Together, these choices can lead to a more stable learning procedure and better performing controllers. <br/><br/>In Chapter 5 we look at what to learn form the collected data. The high price of data in the robotics domain makes it crucial to extract as much knowledge as possible from each and every datum. Reinforcement learning, by default, does not do so. We therefore supplement reinforcement learning with explicit state representation learning objectives. These objectives are based on the assumption that the neural network controller that is to be learned can be seen as consisting of two consecutive parts. The first part (referred to as the state encoder) maps the observed sensor data to a compact and concise representation of the state of the robot and its environment. The second part determines which actions to take based on this state representation. As the representation of the state of the world is useful for more than just completing the task at hand, it can also be trained with more general (state representation learning) objectives than just the reinforcement learning objective associated with the current task. We show how including these additional training objectives allows for learning a much more general state representation, which in turn makes it possible to learn broadly applicable control strategies more quickly. We also introduce a training method that ensures that the added learning objectives further the goal of reinforcement learning, without destabilizing the learning process through their changes to the state encoder. <br/><br/>The final contribution of this thesis, presented in Chapter 6, focuses on the optimization procedure used to train the second part of the policy; the mapping from the state representation to the actions. While we show that the state encoder can be efficiently trained with standard gradient-based optimization techniques, perfecting this second mapping is more difficult. Obtaining high quality estimates of the gradients of the policy performance with respect to the parameters of this part of the neural network is usually not feasible. This means that while a reasonable policy can be obtained relatively quickly using gradient-based optimization approaches, this speed comes at the cost of the stability of the learning process as well as the final performance of the controller. Additionally, the unstable nature of this learning process brings with it an extreme sensitivity to the values of the hyper-parameters of the training method. This places an unfortunate emphasis on hyper-parameter tuning for getting deep reinforcement learning algorithms to work well. Gradient-free optimization algorithms can be more simple and stable, but tend to be much less sample efficient. We show how the desirable aspects of both methods can be combined by first training the entire network through gradient-based optimization and subsequently fine-tuning the final part of the network in a gradient-free manner. We demonstrate how this enables the policy to improve in a stable manner to a performance level not obtained by gradient-based optimization alone, using many fewer trials than methods using only gradient-free optimization.<br/>

Computer vision has advanced so far that machines now can think and see as we humans do. Especially deep learning has raised the bar of excellence in computer vision. However, the recent emergence of deep reinforcement learning is threatening to soar even greater heights as it combines deep neural networks with reinforcement learning along with numerous added advantages over both. This, being a relatively recent technique, has not yet seen many works, and so its true potential is yet to be unveiled. Thus, this chapter focuses on shedding light on the fundamentals of deep reinforcement learning, starting with the preliminaries followed by the theory and basic algorithms and some of its variations, namely, attention aware deep reinforcement learning, deep progressive reinforcement learning, and multi-agent deep reinforcement learning. This chapter also discusses some existing deep reinforcement learning works regarding computer vision such as image processing and understanding, video captioning and summarization, visual search and tracking, action detection, recognition and prediction, and robotics. This work further aims to elucidate the existing challenges and research prospects of deep reinforcement learning in computer vision. This chapter might be considered a starting point for aspiring researchers looking to apply deep reinforcement learning in computer vision to reach the pinnacle of performance in the field by tapping into the immense potential that deep reinforcement learning is showing.

Most deep reinforcement learning techniques are unsuitable for robotics, as they require too much interaction time to learn useful, general control policies. This problem can be largely attributed to the fact that a state representation needs to be learned as a part of learning control policies, which can only be done through fitting expected returns based on observed rewards. While the reward function provides information on the desirability of the state of the world, it does not necessarily provide information on how to distill a good, general representation of that state from the sensory observations. State representation learning objectives can be used to help learn such a representation. While many of these objectives have been proposed, they are typically not directly combined with reinforcement learning algorithms. We investigate several methods for integrating state representation learning into reinforcement learning. In these methods, the state representation learning objectives help regularize the state representation during the reinforcement learning, and the reinforcement learning itself is viewed as a crucial state representation learning objective and allowed to help shape the representation. Using autonomous racing tests in the TORCS simulator, we show how the integrated methods quickly learn policies that generalize to new environments much better than deep reinforcement learning without state representation learning.

State representation modeling for deep reinforcement learning based recommendation

We design an effective and scalable Deep Reinforcement Learning (DRL) approach for the Routing, Modulation and Spectrum Assignment (RMSA) problem in elastic optical networks. We use Convolutional Neural Networks (CNN) to embed the state and Deep Neural Networks (DNN) to learn the policy. We propose a novel state representation and reward function that interestingly guide the agent on assigning appropriate routes and spectrum by incorporating information on the spectrum utilisation and spectrum fragmentation. This gives the agent information about the consequence or cost of each action across the network, reducing the level of knowledge abstraction required for the agent. To show the effectiveness of the reward function and the importance of well-designed state representations, we have designed two state representations: the first with aggregation of spectrum occupancy information and the second without aggregation. The Proximal Policy Optimization (PPO) algorithm is investigated with an actor critic model where an entropy bonus is added to the loss function to ensure sufficient exploration. The proposed solution is compared with a greedy heuristic and a PPO with standard reward and state representation. Numerical results show that the proposed model provides very good solutions and works well on dataset instances with large topologies (up to 75 nodes). The proposed PPO outperformed the baseline algorithms by obtaining the largest throughput on all test instances. In addition, its spectrum usage has the lowest fragmentation.

Reinforcement learning takes sequential decision-making approaches by learning the policy through trial and error based on interaction with the environment. Combining deep learning and reinforcement learning can empower the agent to learn the interactions and the distribution of rewards from state-action pairs to achieve effective and efficient solutions in more complex and dynamic environments. Deep reinforcement learning (DRL) has demonstrated astonishing performance in surpassing the human-level performance in the game domain and many other simulated environments. This paper introduces the basics of reinforcement learning and reviews various categories of DRL algorithms and DRL models developed for medical image analysis and radiation treatment planning optimization. We will also discuss the current challenges of DRL and approaches proposed to make DRL more generalizable and robust in a real-world environment. DRL algorithms, by fostering the designs of the reward function, agents interactions and environment models, can resolve the challenges from scarce and heterogeneous annotated medical image data, which has been a major obstacle to implementing deep learning models in the clinic. DRL is an active research area with enormous potential to improve deep learning applications in medical imaging and radiation therapy planning.

Sequence-wise recommendation, where recommend exercises to each student step by step, is one of the most exciting tasks in the field of intelligent tutoring systems (ITS). It is important to develop a personalized sequence-wise recommendation framework that immerses students in learning and helps them acquire as much necessary knowledge as possible, rather than merely focusing on providing non-mastered exercises, which is referred to optimize a single objective. However, due to the different knowledge levels of students and the large scale of exercise banks, it is difficult to generate a personalized exercise recommendation for each student. To fully exploit the multifaceted beneficial information collected from e-learning platforms, we design a dynamic multi-objective sequence-wise recommendation framework via deep reinforcement learning, i.e., DMoSwR-DRL, which automatically select the most suitable exercises for each student based on the well-designed domain-objective rewards. Within this framework, the interaction between students and exercises can be explicitly modeled by integrating the actor–critic network and the state representation component, which can greatly help the agent perform effective reinforcement learning. Specifically, we carefully design a state representation module with dynamic recurrent mechanism, which integrates concept information and exercise difficulty level, thus generating a continuous state representation of the student. Subsequently, a flexible reward function is designed to simultaneously optimize the four domain-specific objectives of difficulty, novelty, coverage, and diversity, providing the students with a trade-off sequence-wise recommendation. To set up the online evaluation, we test DMoSwR-DRL on a simulated environment which can model qualitative development of knowledge level and predicts their performance for a given exercise. Comprehensive experiments are conducted on four classical exercise-answer datasets, and the results show the effectiveness and advantages of DMoSwR-DRL in terms of recommendation quality.

STACoRe: Spatio-temporal and action-based contrastive representations for reinforcement learning in Atari

Our project focuses on enhancing collision avoidance between autonomous vehicles using deep reinforcement learning (DRL), and it also aims to improve the collision avoidance between autonomous vehicles using deep reinforcement learning. By leveraging advanced DRL algorithms and ensuring robust, realtime decision-making, the project seeks to create a safer and more efficient autonomous driving system. The approach integrates various state representations, actions, and reward functions to optimize driving behaviors. The system's state representation includes vehicle positions, velocities, distances to obstacles, and relative locations of other vehicles. Actions encompass accelerating, decelerating, and turning maneuvers. The reward function is designed to promote safe driving behaviors, such as maintaining speed and lane discipline, while penalizing collisions and erratic movements. The project also incorporates Multi-Agent Reinforcement Learning (MARL) to enable vehicle coordination, where each vehicle learns to maximize its own reward while considering the actions of others. Communication protocols between vehicles enhance decision- making and collision avoidance here vehicles share information about their states and intended actions, improving overall coordination and reducing the likelihood of collisions.Safety and robustness are ensured by integrating safe reinforcement learning techniques and making the policy resilient to environmental uncertainties. Real-world challenges addressed include transferring learned policies from simulation to real-world applications, ensuring scalability across different traffic conditions, and compliance with traffic regulations. The effectiveness of the proposed solution is demonstrated through case studies and validation in various driving scenarios. This project aims to advance the development of safer and more efficient autonomous driving systems

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Artificial Intelligence and the Common Sense of Animals.

With the scale of E-commerce increasing year by year, the importance of recommender systems is getting increasing attention. Based on deep reinforcement learning, we can model the recommendation task as an interactional and sequential decision procedure between the system and users, instead of a static process. It can improve the recommendation quality to a large extent. By integrating a state representation module, the quality of modeling interaction between users and system can be improved. In this article, we propose a recommender system based on user-commodity state representation integrated deep reinforcement learning, named UCSRDRL, and conduct experiments on the datasets offered by FUXI AI Lab and the outcome performs better than the baseline. The score of UCSRDRL ranked third in the competition.

In passive ground target detection using Unmanned Aerial Vehicles (UAVs), some detrimental factors like occlusion significantly impact target detection performance. Active Object Detection offers an effective way to address it, which usually uses Deep Reinforcement Learning (DRL) to plan UAV’s viewpoint for favorable observations. However, existing DRL-based AOD methods often suffer from low sample efficiency and poor generalization due to inadequate state representation learned by the policy network. Inspired by human scene understanding where their spatial representation of the scene remains consistent despite different observation orders, we design a self-supervised state representation learning method based on Observations Temporal Permutation (OTP) to improve the state representation of the agent’s policy network. We require the policy network to output consistent action value estimates for observation sequences with the same content but different temporal orders. Besides, we use the state representation to predict the target orientation variations in the observation sequence, which further regularizes and facilitates the state representation learning process. Finally, we design multiple experiments based on the UEVAVD dataset to compare the proposed method with existing self-supervised state representation learning methods for the AOD task. The experimental results demonstrate that the OTP method can help the agent’s policy network learn a better state representation, thus achieving higher policy learning sample efficiency and stronger policy generalization.

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

Reinforcement learning (RL) is a machine learning paradigm concerned with how an agent learns to predict and control its own experience stream so as to maximize long-term cumulative reward. In the past decade, deep reinforcement learning (DeepRL), a subfield that aims to combine the sequential decision-making techniques in RL with the powerful non-linear function approximation tools offered by deep learning, has seen great success such as defeating human champions in the ancient board game Go and achieving expert-level performance in complex strategy games like Dota $2$ and Starcraft. It has also had an impact on real-world applications. Examples include robot control, stratospheric balloon navigation, and controlling nuclear fusion plasma. This thesis aims to further advance DeepRL techniques. Concretely, this thesis makes contributions in the following four directions: 1) In reward design, we develop a novel meta-learning algorithm for learning reward functions that facilitate policy optimization. Our algorithm improves the performance of policy-gradient methods and outperforms handcrafted heuristic reward functions. In a follow-up study, we show that the learned reward functions can capture knowledge about long-term exploration and exploitation and can generalize to different RL algorithms and changes in the environment dynamics. 2) In temporal credit assignment, we explore methods based on pairwise weights that are functions of the state in which the action was taken, the state in which the reward was received, and the time elapsed in between. We develop a metagradient algorithm for adapting these weights during policy learning. Our experiments show that our method achieves better performance than competing approaches. 3) In state representation learning, we investigate using random deep action-conditional prediction tasks as auxiliary tasks to help agents learn better state representations. Our experiments show that random deep action-conditional predictions can often yield better performance than handcrafted auxiliary tasks. 4) In model learning and planning, we develop a new method for learning value-equivalent models, a class of models that demonstrates strong empirical performance lately, that generalizes existing methods. Our experiments show that our method can improve both the model prediction accuracy and the control performance of the downstream planning procedure.