Cumulative Reward Research Articles

Improved SARSA and DQN algorithms for reinforcement learning

Reinforcement learning is a branch of machine learning in which an agent interacts with an environment to learn optimal actions that maximize cumulative rewards. This paper aims to enhance the SARSA and DQN algorithms in four key aspects: the ε-greedy policy, reward function, value iteration approach, and sampling probability. The experiments are conducted in three scenarios: path planning, CartPole, and MountainCar. The results show that, in these environments, the improved algorithms exhibit better convergence, higher rewards, and more stable training processes.

Reward deprivation is associated with elevated alcohol demand in emerging adults.

Policies vary substantially in terms of providing sources of psychosocial enrichment. Behavioral economic models of substance use and addiction emphasize that deficits in access to substance-free sources of reward increase substance reinforcing value and risk for addiction. The current study used an alcohol demand curve approach to test the hypothesis that various indices of reward deprivation would be associated with elevated alcohol reinforcing efficacy. We examined associations between alcohol demand indices and several facets of reward deprivation in a sample of young adults (N = 1,331; ages 19-25 years) recruited from the United States and Canada who reported recent binge drinking. Additionally, we created an index of cumulative reward deprivation that integrated the various reward facets and examined its association with alcohol demand intensity and maximum expenditure on alcohol. Our findings indicate that reward deprivation is associated with elevated alcohol demand and provide support for alcohol prevention and intervention approaches that emphasize environmental enrichment.

An algorithm that excavates suboptimal states and improves Q-learning

Abstract Reinforcement learning is inspired by the trial-and-error method in animal learning, where the reward values obtained from the interaction of the agent with the environment are used as feedback signals to train the agent. Reinforcement learning has attracted extensive attention in recent years. It is mainly used to solve sequential decision-making problems and has been applied to various aspects of life, such as autonomous driving, game gaming, and robotics. Exploration and exploitation are the main characteristics that distinguish reinforcement learning methods from other learning methods. Reinforcement learning methods need reward optimization algorithms to better balance exploration and exploitation. Aiming at the problems of unbalanced exploration and a large number of repeated explorations in the Q-learning algorithm in the MDP environment, an algorithm that excavates suboptimal states and improves Q-learning was proposed. It adopts the exploration idea of ‘exploring the potential of the second-best’, and explores the state with suboptimal state value, and calculates the exploration probability value according to the distance between the current state and the goal state. The larger the distance, the higher the exploration demand of the agent. In addition, only the immediate reward and the maximum action value of the next state are needed to calculate the Q value. Through the simulation experiments in two different MDP environments, The frozenLake8x8 environment and the CliffWalking environment, the results verify that the proposed algorithm obtains the highest average cumulative reward and the least total time consumption.

Applications and Advances of UCB Algorithms in Dynamic and Contaminated Environments

The Upper Confidence Bound (UCB) algorithm is a widely used approach in the Multi-Armed Bandit (MAB) problem, where the goal is to maximize cumulative rewards over time by selecting the best possible action among several options. The UCB algorithm uses confidence bounds to balance exploration and exploitation, guiding its decision-making. In recent years, researchers have identified significant challenges in applying the UCB algorithm to dynamic and contaminated environments. In such scenarios, the underlying conditions may change over time, making it difficult for standard UCB to adapt, or the data may be polluted by noise and outliers, leading to incorrect estimations of reward distributions. To address these challenges, several variants of the UCB algorithm have been developed. These new approaches are designed to better handle the complexities of changing environments and data contamination, ensuring more robust and reliable performance in these difficult settings. This paper aims to provide a comprehensive review of Robust-UCB (cr-UCB), Sliding Window UCB (SW-UCB) and bandit-over-bandit UCB (BOB-UCB). Focusing on their theoretical foundations, practical applications, and empirical performance. By examining how these algorithms have been adapted to handle the complexities of dynamic and contaminated environments, we found that the adaptability of these algorithms in dynamic environments is significantly improved, and they can effectively reduce decision-making errors caused by data pollution, thus providing a more reliable solution to the multi-armed bandit problem.

Traffic signal phase control at urban isolated intersections: an adaptive strategy utilizing the improved D3QN algorithm

Abstract The intelligent control of traffic signals at urban single intersections has emerged as an effective approach to mitigating urban traffic congestion. However, the existing fixed phase control strategy of traffic signal lights lacks capability to dynamically adjust signal phase switching based on real-time traffic conditions leading to traffic congestion. In this paper, an adaptive real-time control method employed by the traffic signal phase at a single intersection is considered based on the improved Double Dueling Deep Q Network (D3QN) algorithm. Firstly, the traffic signal phase control problem is modeled as a Markov decision process, with its state, action, and reward defined. Subsequently, to enhance the convergence speed and learning performance of the D3QN algorithm, attenuation action selection strategy and priority experience playback technology based on tree summation structure are introduced. Then, traffic flow data from various traffic scenarios are utilized to train the traffic signal control model based on the improved D3QN to obtain the optimal signal phase switch strategy. Finally, the effectiveness and optimal performance of the improved D3QN-based traffic signal control strategy are validated across diverse traffic scenarios. The simulation results show that, compared with the control strategy based on AC, DQN, DDQN, D3QN, and C-D3QN algorithms, the cumulative reward of the proposed I-D3QN strategy is increased by at least 6.57%, and the average queue length and average waiting time are reduced by at least 9.64% and 7.61%, which can effectively reduce the congestion at isolated intersections and significantly improve traffic efficiency.

Simulation of Coherent Excavator Operations in Earthmoving Tasks Based on Reinforcement Learning

Earthwork operations are critical to construction projects, with their safety and efficiency influenced by factors such as operator skill and working hours. Pre-construction simulation of these operations is essential for optimizing outcomes, providing key training for operators and improving safety awareness and operational efficiency. This study introduces a hierarchical cumulative reward mechanism that decomposes complex operational behaviors into simple, fundamental actions. The mechanism prioritizes reward function design elements, including order, size, and form, thus simplifying excavator operation simulation using reinforcement learning (RL) and enhancing policy network reusability. A 3D model of a hydraulic excavator was constructed with six degrees of freedom—comprising the boom, arm, bucket, base, and left/right tracks. The Proximal Policy Optimization (PPO) algorithm was applied to train four basic behaviors: scraping, digging, throwing, and turning back. Motion simulation was successfully achieved using diggable terrain resources. Results demonstrate that the simulated excavator, powered by RL neural networks, can perform coordinated actions and maintain smooth operational performance. This research offers practical implications by rapidly illustrating the full operational process before construction, delivering immersive movies, and enhancing worker safety and operational efficiency.

Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze.

In this study, we investigate the adaptability of artificial agents within a noisy T-maze that use Markov decision processes (MDPs) and successor feature (SF) and predecessor feature (PF) learning algorithms. Our focus is on quantifying how varying the hyperparameters, specifically the reward learning rate (αr) and the eligibility trace decay rate (λ), can enhance their adaptability. Adaptation is evaluated by analyzing the hyperparameters of cumulative reward, step length, adaptation rate, and adaptation step length and the relationships between them using Spearman's correlation tests and linear regression. Our findings reveal that an αr of 0.9 consistently yields superior adaptation across all metrics at a noise level of 0.05. However, the optimal setting for λ varies by metric and context. In discussing these results, we emphasize the critical role of hyperparameter optimization in refining the performance and transfer learning efficacy of learning algorithms. This research advances our understanding of the functionality of PF and SF algorithms, particularly in navigating the inherent uncertainty of transfer learning tasks. By offering insights into the optimal hyperparameter configurations, this study contributes to the development of more adaptive and robust learning algorithms, paving the way for future explorations in artificial intelligence and neuroscience.

Optimization of pressure management strategies for geological CO2 storage using surrogate model-based reinforcement learning

Injecting greenhouse gas (e.g. CO2) into deep underground reservoirs for permanent storage can inadvertently lead to fault reactivation, caprock fracturing and greenhouse gas leakage when the injection-induced stress exceeds the critical threshold. It is essential to monitor the evolution of pressure and the movement of the CO2 plume closely during the injection to allow for timely remediation actions or rapid adjustments to the storage design. Extraction of pre-existing fluids at various stages of the injection process, referred as pressure management, can mitigate associated risks and lessen environmental impact. However, identifying optimal pressure management strategies typically requires thousands of simulations, making the process computationally prohibitive. This paper introduces a novel surrogate model-based reinforcement learning method for devising optimal pressure management strategies for geological CO2 sequestration efficiently. Our approach comprises of two steps. The first step involves developing a surrogate model using the embed to control method, which employs an encoder-transition-decoder structure to learn dynamics in a latent or reduced space. The second step, leveraging this proxy model, utilizes reinforcement learning to find an optimal strategy that maximizes economic benefits while satisfying various control constraints. The reinforcement learning agent receives the latent state representation and immediate reward tailored for CO2 sequestration and choose real-time controls which are subject to predefined engineering constraints in order to maximize the long-term cumulative rewards. To demonstrate its effectiveness, this framework is applied to a compositional simulation model where CO2 is injected into saline aquifer. The results reveal that our surrogate model-based reinforcement learning approach significantly optimizes CO2 sequestration strategies, leading to notable economic gains compared to baseline scenarios.

Edge-Intelligence-Powered Joint Computation Offloading and Unmanned Aerial Vehicle Trajectory Optimization Strategy

UAV-based air-ground integrated networks offer a significant benefit in terms of providing ubiquitous communications and computing services for Internet of Things (IoT) devices. With the empowerment of edge intelligence (EI) technology, they can efficiently deploy various intelligent IoT applications. However, the trajectory of UAVs can significantly affect the quality of service (QoS) and resource optimization decisions. Joint computation offloading and UAV trajectory optimization bring many challenges, including coupled decision variables, information uncertainty, and long-term queue delay constraints. Therefore, this paper introduces an air-ground integrated architecture with EI and proposes a TD3-based joint computation offloading and UAV trajectory optimization (TCOTO) algorithm. Specifically, we use the principle of the TD3 algorithm to transform the original problem into a cumulative reward maximization problem in deep reinforcement learning (DRL) to obtain the UAV trajectory and offloading strategy. Additionally, the Lyapunov framework is used to convert the original long-term optimization problem into a deterministic short-term time-slot problem to ensure the long-term stability of the UAV queue. Based on the simulation results, it can be concluded that our novel TD3-based algorithm effectively solves the joint computation offloading and UAV trajectory optimization problems. The proposed algorithm improves the performance of the system energy efficiency by 3.77%, 22.90%, and 67.62%, respectively, compared to the other three benchmark schemes.

An Optimal Multibarrier Strategy for a Singular Stochastic Control Problem with a State-Dependent Reward

We consider a singular control problem that aims to maximize the expected cumulative rewards, where the instantaneous returns depend on the state of a controlled process. The contributions of this paper are twofold. Firstly, to establish sufficient conditions for determining the optimality of the one-barrier strategy when the uncontrolled process X follows a spectrally negative Lévy process with a Lévy measure defined by a completely monotone density. Secondly, to verify the optimality of the (2n+1)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$(2n+1)$$\\end{document}-barrier strategy when X is a Brownian motion with a drift. Additionally, we provide an algorithm to compute the barrier values in the latter case.

Enhancing network function parallelism in mobile edge computing using Deep Reinforcement Learning

This paper introduces a Deep Reinforcement Learning (DRL)-based framework to enhance Network Function Parallelism (NFP) in Mobile Edge Computing (MEC). Leveraging Network Function Virtualization (NFV), the proposed framework optimizes service delay by solving a fairness-aware throughput maximization problem for service function chain placement. It aims to maximize the long-term cumulative reward while satisfying Quality of Service (QoS) requirements. The framework also preserves resources for future requests by efficiently managing the initialized network functions distribution. Simulation results demonstrate the superior performance of the proposed framework across various metrics. Specifically, our framework improves the average delay and deployment rate by 1.2% and 2.4% compared to the existing best method.

Solving Flexible Job-Shop Scheduling Problem with Heterogeneous Graph Neural Network Based on Relation and Deep Reinforcement Learning

Driven by the rise of intelligent manufacturing and Industry 4.0, the manufacturing industry faces significant challenges in adapting to flexible and efficient production methods. This study presents an innovative approach to solving the Flexible Job-Shop Scheduling Problem (FJSP) by integrating Heterogeneous Graph Neural Networks based on Relation (HGNNR) with Deep Reinforcement Learning (DRL). The proposed framework models the complex relationships in FJSP using heterogeneous graphs, where operations and machines are represented as nodes, with directed and undirected arcs indicating dependencies and compatibilities. The HGNNR framework comprises four key components: relation-specific subgraph decomposition, data preprocessing, feature extraction through graph convolution, and cross-relation feature fusion using a multi-head attention mechanism. For decision-making, we employ the Proximal Policy Optimization (PPO) algorithm, which iteratively updates policies to maximize cumulative rewards through continuous interaction with the environment. Experimental results on four public benchmark datasets demonstrate that our proposed method outperforms four state-of-the-art DRL-based techniques and three common rule-based heuristic algorithms, achieving superior scheduling efficiency and generalization capabilities. This framework offers a robust and scalable solution for complex industrial scheduling problems, enhancing production efficiency and adaptability.

Automated architectural spatial composition via multi-agent deep reinforcement learning for building renovation

This paper focuses on utilizing deep reinforcement learning technology to explore how to achieve automated architectural space composition under established built environment conditions to address the extensive demands of old building renovation. A reinforcement learning platform has been develop0065d, centered around the multi-agent deep deterministic policy gradient (MADDPG) algorithm. The paper leverages functional blocks as agents within Grasshopper to simulate existing architectural structures and their interactions, establishing a task-based reward system to guide the design process. Furthermore, data exchange between algorithmic and modeling software is facilitated through UDP communication. Training results using typical frame structure spaces indicate a significant increase in cumulative rewards around the 650,000th training step, effectively achieving the given design tasks. The paper demonstrates the potential of automated architectural space composition methods, based on deep reinforcement learning, to enhance design efficiency, optimize spatial arrangements, and promote human-machine collaborative design in building renovations.

A soft actor-critic reinforcement learning framework for optimal energy management in electric vehicles with hybrid storage

The efficient energy management of electric vehicles (EVs) equipped with hybrid energy storage systems (HESS) poses a significant challenge due to its vast search space, numerous control variables, and intricate driving conditions. In response to this challenge, this paper introduces a novel deep reinforcement learning (DRL) algorithm, specifically the soft actor-critic (SAC), tailored for optimizing energy distribution within EVs equipped with battery-supercapacitor (SC) HESS. The proposed SAC-based energy management system (EMS) is designed to address inherent limitations present in most existing DRL algorithms, such as slower convergence rates, discretization errors, unstable training dynamics, and suboptimal optimization effects. The SAC-based EMS undergoes training in a continuous action space through self-play, utilizing a complex driving cycle and a novel reward function. The algorithm demonstrates its efficiency by rapidly maximizing cumulative rewards and refining its decision-making policies. Afterward, extensive experiments showcase the superiority of the proposed SAC-based EMS over rule-based (RB) techniques, deep deterministic policy gradient (DDPG), and battery-only configurations across various driving cycles. Notably, the approach effectively allocates high-power surges and braking energy to the SC, reducing the frequency of charge/discharge cycles and thereby extending the battery's lifespan. Additionally, the learned SAC-based EMS policy achieves substantial reductions in electricity consumption by 39.6 %, 47.87 %, and 45 % compared to battery-only configuration, and 42.56 %, 47.87 %, and 45.83 % compared to RB technique under the NYCC, LA92, and FTP cycles, respectively. In contrast, the DDPG algorithm tends to rely heavily on the SC to deliver the total power required, even under normal driving conditions.

Cognitive mechanisms of learning in sequential decision-making under uncertainty: an experimental and theoretical approach.

Learning to make adaptive decisions involves making choices, assessing their consequence, and leveraging this assessment to attain higher rewarding states. Despite vast literature on value-based decision-making, relatively little is known about the cognitive processes underlying decisions in highly uncertain contexts. Real world decisions are rarely accompanied by immediate feedback, explicit rewards, or complete knowledge of the environment. Being able to make informed decisions in such contexts requires significant knowledge about the environment, which can only be gained via exploration. Here we aim at understanding and formalizing the brain mechanisms underlying these processes. To this end, we first designed and performed an experimental task. Human participants had to learn to maximize reward while making sequences of decisions with only basic knowledge of the environment, and in the absence of explicit performance cues. Participants had to rely on their own internal assessment of performance to reveal a covert relationship between their choices and their subsequent consequences to find a strategy leading to the highest cumulative reward. Our results show that the participants' reaction times were longer whenever the decision involved a future consequence, suggesting greater introspection whenever a delayed value had to be considered. The learning time varied significantly across participants. Second, we formalized the neurocognitive processes underlying decision-making within this task, combining mean-field representations of competing neural populations with a reinforcement learning mechanism. This model provided a plausible characterization of the brain dynamics underlying these processes, and reproduced each aspect of the participants' behavior, from their reaction times and choices to their learning rates. In summary, both the experimental results and the model provide a principled explanation to how delayed value may be computed and incorporated into the neural dynamics of decision-making, and to how learning occurs in these uncertain scenarios.

Learning continuous multi-UAV controls with directed explorations for flood area coverage

Real time on-ground information is of critical value during any natural disaster such as floods. The disaster response teams require the latest ground information of the flooded areas to effectively plan and execute rescue operations. Unmanned Aerial Vehicles (UAVs) are increasingly becoming a tool to perform quick surveys of larger areas such as flood disasters. In this paper, we propose a method to perform critical area coverage of flood-struck regions using multiple autonomous UAVs. A Deep Reinforcement Learning algorithm is proposed to learn continuous multi-UAV controls, incorporating a directed exploration strategy for the DDPG’s target actor, which relies on the D-infinity (DINF) algorithm. The DINF water flow estimation technique utilizes surface elevation data to understand and predict the directed discharge of floodwater. Further, we introduce a Path scatter strategy for the multi-UAV system that inhibits the clustered formation of the UAVs over low-elevated regions. The performance of the proposed D3S (DDPG+DINF+Path Scatter) algorithm is evaluated using various performance metrics, such as average cumulative rewards, number of collisions, and UAVs’ spread observed over the environment. In comparison to the baseline algorithms and other prevalent approaches in the literature, the proposed method is found to be better placed as the results highlight a significantly improved performance by D3S across different metrics.

Reinforcement Learning in Social Sciences: A Survey

Reinforcement Learning (RL) has become one of the most prominent topics in artificial intelligence research. It is widely used in various fields, such as recommendation systems, psychology, economics, and natural language dialogue systems. Finding the best path of action to maximize cumulative reward is the long-term strategy of RL. Undertaking research may yield suboptimal immediate results but optimal long-term consequences. Economists can address difficult behavioral problems with knowledge, especially those generated by deep learning algorithms. We provide the most recent advancements in RL methods in this study, along with their applications in gaming, finance, and economics. The survey's last section discusses RL's present problems and potential future developments. Such open problems as sample efficiency, safety, and interpretability are currently being sought after by researchers. Moreover, several ambitious prospective applications of RL in a wide variety of domains are discussed. This study gives a comprehensive review of the many methods and uses of RL in social science. This study's results will give researchers a standard against which to evaluate the utility and efficacy of frequently used RL. Guide future investigations across several domains.

Enhancing SFC Placement with Parallelized Functions in MEC Using Deep Reinforcement Learning

This study presents an innovative architecture leveraging Deep Reinforcement Learning (DRL) to optimize online service delivery in Mobile Edge Computing (MEC) networks. Our proposed method, the DRL-based dynamic Service Function Chain (SFC) placement with parallelized Virtual Network Functions (VNFs), aims to maximize Long-Term Cumulative Reward (LTCR). By parallelizing VNFs, our approach achieves computational acceleration, enhancing the efficiency of online service provisioning. Furthermore, we enhance future request processing capabilities by extracting the distribution of initialized VNFs. Through comprehensive simulations, our proposed architecture demonstrates a notable improvement, with approximately a 9.5% increase in the average number of accepted requests.

Value-Based Subgoal Discovery and Path Planning for Reaching Long-Horizon Goals.

Learning to reach long-horizon goals in spatial traversal tasks is a significant challenge for autonomous agents. Recent subgoal graph-based planning methods address this challenge by decomposing a goal into a sequence of shorter-horizon subgoals. These methods, however, use arbitrary heuristics for sampling or discovering subgoals, which may not conform to the cumulative reward distribution. Moreover, they are prone to learning erroneous connections (edges) between subgoals, especially those lying across obstacles. To address these issues, this article proposes a novel subgoal graph-based planning method called learning subgoal graph using value-based subgoal discovery and automatic pruning (LSGVP). The proposed method uses a subgoal discovery heuristic that is based on a cumulative reward (value) measure and yields sparse subgoals, including those lying on the higher cumulative reward paths. Moreover, LSGVP guides the agent to automatically prune the learned subgoal graph to remove the erroneous edges. The combination of these novel features helps the LSGVP agent to achieve higher cumulative positive rewards than other subgoal sampling or discovery heuristics, as well as higher goal-reaching success rates than other state-of-the-art subgoal graph-based planning methods.

A survey of reinforcement learning theories in autonomous vehicles

This survey explores the core theories, techniques, and core applications of reinforcement learning (RL) in the domain of autonomous vehicles. RL has emerged as a significant contributor to decision-making techniques in this field, enabling agents to make informed decisions based on actions and rewards derived from the environment. The ultimate objective of RL in autonomous driving is to maximize cumulative rewards over time. However, RL faces challenges related to interpretability and sample efficiency, particularly in complex driving scenarios. This survey extensively investigates the utilization of RL in decision-making and control, encompassing various scenarios and addressing the challenges encountered within RL-based autonomous vehicles. By emphasizing the design of effective reward functions, enhancing sample efficiency, and improving model interpretability, future advancements in reinforcement learning for autonomous vehicles can foster the development of more robust, efficient, and trustworthy autonomous systems. Moreover, this survey provides valuable insights into the limitations of RL techniques in autonomous driving decision-making, highlighting areas that require further research and development.