Cumulative Reward Research Articles

Geological adaptive intelligent control of earth pressure balance shield machine based on deep reinforcement learning

Scientific and precise control of tunnelling parameters is of utmost importance during the construction of shield machines. Given the complexity of the working environment, manual operation is highly prone to causing safety accidents. Therefore, achieving intelligent control of the shield machine is crucial. Based on this, this paper proposes a geological adaptive intelligent control method of earth pressure balance shield machine using the Deep Deterministic Policy Gradient (DDPG) algorithm as the framework, with Actor-Critic as the basis. Firstly, DDPG agent is constructed to replace the screw conveyor control system as the main body of strategy implementation. Secondly, an environmental model is established by utilizing the mechanism model between the sealed cabin pressure and the screw conveyor speed. The real-time sealed cabin pressure, target pressure, and pressure error serve as the state space, while the screw conveyor speed is used as the action space. A combined reward function is set based on safety and accuracy. Finally, the Actor network interacts with the environment under the supervision of the reward function and Critic network. Successful training is achieved when the cumulative reward value is maximized, resulting in the output of optimal control strategy. In this paper, the method dynamically regulates the screw conveyor speed by interacting with the geological environment, to realize the precise control of the sealed cabin pressure and ensure the dynamic balance between sealed cabin pressure and excavation face pressure. The test results show that this method has a good control effect on the sealed cabin pressure under various geological conditions, and can complete 72 kinds of soil transition tasks. It has strong soil adaptability and can respond well to the dynamic changes of soil conditions. This approach enhances the intelligence of the shield machine, mitigating inaccuracies attributed to human operation, which provides a guarantee of safe shield machine operation, whilst exhibiting valuable engineering applications.

Online Surveillance of IoT Agents in Smart Cities Using Deep Reinforcement Learning

In the context of today's smart cities, the effective operation of online surveillance of IoT agents is crucial for maintaining public safety and security. To achieve this, collaboration and cooperation among these autonomous IoT agents are indispensable. While the existing research has focused on collaboration amongst the neighboring agents or implicit cooperation, real-world scenarios often necessitate broader forms of collaboration. In response to this need, we introduce a novel framework that leverages visual signals and observations to facilitate collaboration among online surveillance. Our proposed framework incorporates the Multi-Agent POsthumous Credit Assignment (MA-POCA) algorithm as a training mechanism. The empirical results demonstrate that our framework consistently outperforms the base model in various performance metrics. Specifically, it exhibits superior performance in group cumulative reward, cumulative reward, and episode length. Furthermore, our proposed model excels in policy loss performance measures when compared to base model.

DRL approach for online user-centric QoS-Aware SFC embedding with dynamic VNF placement

In this paper, we tackle online Service Function Chaining (SFC) embedding issue in Software Defined Network (SDN)/Network Function Virtualization (NFV) enabled networks using Deep Reinforcement Learning (DRL) approach. This approach investigates the problem of Quality of Experience (QoE)/Quality of Service (QoS) driven SFC embedding with dynamic Virtual Network Functions (VNF) placement. We particularly focus on video streaming traffic requiring a given Mean Opinion Score (MOS) quantified in terms of QoE. In this regard, we propose a DRL algorithm based on Deep-Q-Network (DQN) (named DQN_QoE/QoS_SFC algorithm) implementing an optimization problem maximizing QoE while meeting QoS requirements. Our algorithm named DQN_QoE/QoS_SFC improves previous work given in Chen et al. (2018) dealing with the same issue through bringing performance evaluation of the training phase by testing the learning quality. This is achieved through using a QoE Threshold Score (QoETh−Sc) to be attained on average by the DQN_QoE/QoS_SFC agent along the last 100 runs of the training phase while maximizing the expected cumulative reward. We conduct simulation experiments through two performance metrics related to the SFC request namely QoE (QoE) and Rejection Ratio (RR). We investigate also how these metrics evolve for the proposed algorithm when compared with two standard baseline algorithms (Violent search, Random search) during learning phase to achieve satisfactory training quality. We highlight how DQN_QoE/QoS_SFC algorithm behaves along the training process and how it attempts reaching a near-optimal solution as closer as possible to the optimal solution provided by Violent search algorithm.

Handling Efficient VNF Placement with Graph-Based Reinforcement Learning for SFC Fault Tolerance

Network functions virtualization (NFV) has become the platform for decomposing the sequence of virtual network functions (VNFs), which can be grouped as a forwarding graph of service function chaining (SFC) to serve multi-service slice requirements. NFV-enabled SFC consists of several challenges in reaching the reliability and efficiency of key performance indicators (KPIs) in management and orchestration (MANO) decision-making control. The problem of SFC fault tolerance is one of the most critical challenges for provisioning service requests, and it needs resource availability. In this article, we proposed graph neural network (GNN)-based deep reinforcement learning (DRL) to enhance SFC fault tolerance (GRL-SFT), which targets the chain graph representation, long-term approximation, and self-organizing service orchestration for future massive Internet of Everything applications. We formulate the problem as the Markov decision process (MDP). DRL seeks to maximize the cumulative rewards by maximizing the service request acceptance ratios and minimizing the average completion delays. The proposed model solves the VNF management problem in a short time and configures the node allocation reliably for real-time restoration. Our simulation result demonstrates the effectiveness of the proposed scheme and indicates better performance in terms of total rewards, delays, acceptances, failures, and restoration ratios in different network topologies compared to reference schemes.

Distributed resource optimisation using the Q-learning algorithm, in device-to-device communication: A reinforcement learning paradigm

In the context of wireless systems going forward, particularly in the Beyond 5G (B5G) era, where high data rates and low latency are critical, D2D communication is a pivotal technology that has many advantages. In D2D communication, the allocation of resources plays a critical role in achieving higher throughput while ensuring interference management, improved spectrum and energy efficiency, and system fairness. Conventional resource allocation methodologies encounter challenges in dynamically changing and diverse communication environments. To deal with the dynamic and unpredictable character of channel characteristics, we utilize a distributed iterative resource allocation technique based on the reinforcement learning (RL) approach that empowers the system to learn and adapt to the wireless environment autonomously. In this paper, we formulate a distributed Q learning-based RL method as its real-time learning capabilities, reduced communication overhead, and exploration efficiency make it well-suited for adapting to dynamic D2D environments compared to other RL methods. By applying the Q-learning method, the D2D devices act as learning agents striving to maximize the cumulative rewards. Through interactions with the environment and continuous learning from feedback, these agents adapt to real-time resource allocation decisions over time. Comparing our proposed Q-learning method to the state-of-the-art RL techniques, simulation results show improvements in energy and spectrum efficiency, latency, an increase in Jain's fairness index, and improvements in overall system throughput of about 6 %–8 %. The scalability is found to be 1.69 interpreting that Q-learning exhibits a good scalability as the throughput does not abruptly decrease for an increasing number of devices.

Selective policy transfer in multi-agent systems with sparse interactions

Previously trained single-agent strategies, which are considerably easier to acquire than multi-agent strategies, are instructive for multi-agent reinforcement learning, especially when the interactions between agents are sparse. Traditional methods of knowledge transfer, from single-agent source tasks to multi-agent target tasks, typically rely on a pre-designed Markov-decision-process similarity function or metric function for evaluating the task similarity. In this study, we propose a selective policy transfer (SPOT) algorithm that eliminates the need for the manual crafting of a metric function to assess the similarity between source and target tasks. The SPOT algorithm enables agents to autonomously determine when and which policy to transfer using well-trained single-agent policies as options in the training process. We introduced a multi-agent policy-learning option in the option library, thus allowing the SPOT algorithm to leverage the transferred knowledge while concurrently learning new policies. The SPOT algorithm efficiently transfers sequential strategies in a few steps, thereby capturing high-level semantics. Experimental results obtained in both multi-agent arcade and multi-agent particle environments have demonstrated that the proposed algorithm outperforms the state-of-the-art methods in terms of jump-start and convergence speeds. Our evaluation indicators included the jump start, steps to threshold, cumulative reward, and asymptotic performance. Furthermore, visualizations of the strategies and termination functions of the agents aid in elucidating the operational principles of the SPOT algorithm.

Expert-guided imitation learning for energy management: Evaluating GAIL’s performance in building control applications

The use of Deep Reinforcement Learning (DRL) in building energy management is often hampered by data efficiency and computational challenges. The long training time, unstable, and potentially harmful control performance limit DRL’s adaptability and practicality in building control applications. To address these issues, this study introduces a new method, called Generative Adversarial Imitation Learning (GAIL), which effectively utilizes expert knowledge and demonstrations. Expert demonstrations range from fine-tuned rule-based controls to strategies inspired by optimization algorithms. By combining the capabilities of the generative adversarial network and imitation learning, GAIL is known for effectively learning the optimal strategy from expert demonstrations through an adversarial training process. We conducted a comprehensive evaluation comparing GAIL’s performance with the DRL algorithm Proximal Policy Optimization (PPO) in the scenario of controlling a variable air volume system for load shifting in commercial buildings. Impressively, GAIL, guided by expert demonstrations based on model predictive control, achieved significantly improved computational efficiency and effectiveness. In terms of unified cumulative reward, GAIL with data augmentation achieved 95% expert performance, 22% higher than baseline rule-based control, in 100 training epochs; GAIL also outperformed PPO by 7%, resulting in 2% lower energy costs and notably improved thermal comfort. This improvement in thermal comfort is evidenced by a reduction of 18.65 unmet degree hours during the one-week operation. In comparison, PPO requires more training time and still lags behind GAIL in cumulative reward even after 500 epochs. These findings highlight the advantages of GAIL in enabling faster learning with fewer training samples, resulting in cost-effective solutions due to lower computational requirements. Overall, GAIL presents a promising approach to building energy management and provides a practical and flexible solution to the shortcomings of learning-based controllers that require extensive computational resources and training time.

Multi-Armed Bandit-Based User Network Node Selection.

In the scenario of an integrated space-air-ground emergency communication network, users encounter the challenge of rapidly identifying the optimal network node amidst the uncertainty and stochastic fluctuations of network states. This study introduces a Multi-Armed Bandit (MAB) model and proposes an optimization algorithm leveraging dynamic variance sampling (DVS). The algorithm posits that the prior distribution of each node's network state conforms to a normal distribution, and by constructing the distribution's expected value and variance, it maximizes the utilization of sample data, thereby maintaining an equilibrium between data exploitation and the exploration of the unknown. Theoretical substantiation is provided to illustrate that the Bayesian regret associated with the algorithm exhibits sublinear growth. Empirical simulations corroborate that the algorithm in question outperforms traditional ε-greedy, Upper Confidence Bound (UCB), and Thompson sampling algorithms in terms of higher cumulative rewards, diminished total regret, accelerated convergence rates, and enhanced system throughput.

Optimizing video click-through rates with bandit algorithms

In recent years, videos have increasingly influenced public perception, making video platforms a focal point of digital consumption. One critical challenge for platform operators is identifying videos that resonate most with users, as user ratings directly reflect viewer preferences and experiences. This study explores the use of bandit algorithms to predict and strategize the overall ratings of various anime videos on the Bilibili platform. Bandit algorithms, a subset of the multi-armed bandit model, dynamically adjust selection strategies based on prior feedback to maximize cumulative rewards. Our empirical research assessed multiple gambling algorithms, including the -greedy, Upper Confidence Bound (UCB), Explore-then-Commit (ETC), and Thompson Sampling (TS) algorithms. The findings indicate that the Thompson Sampling algorithm, in particular, achieved the lowest cumulative regret in selecting optimal videos on the Bilibili platform, showcasing its superior performance. This study highlights the potential of bandit algorithms to enhance video selection processes, ensuring that platforms can effectively cater to user preferences and enhance viewer satisfaction.

Model-Free Nonstationary Reinforcement Learning: Near-Optimal Regret and Applications in Multiagent Reinforcement Learning and Inventory Control

We consider model-free reinforcement learning (RL) in nonstationary Markov decision processes. Both the reward functions and the state transition functions are allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain variation budgets. We propose Restarted Q-Learning with Upper Confidence Bounds (RestartQ-UCB), the first model-free algorithm for nonstationary RL, and show that it outperforms existing solutions in terms of dynamic regret. Specifically, RestartQ-UCB with Freedman-type bonus terms achieves a dynamic regret bound of [Formula: see text], where S and A are the numbers of states and actions, respectively, [Formula: see text] is the variation budget, H is the number of time steps per episode, and T is the total number of time steps. We further present a parameter-free algorithm named Double-Restart Q-UCB that does not require prior knowledge of the variation budget. We show that our algorithms are nearly optimal by establishing an information-theoretical lower bound of [Formula: see text], the first lower bound in nonstationary RL. Numerical experiments validate the advantages of RestartQ-UCB in terms of both cumulative rewards and computational efficiency. We demonstrate the power of our results in examples of multiagent RL and inventory control across related products. This paper was accepted by Omar Besbes, revenue management and market analytics. Funding: The research of D. Simchi-Levi and R. Zhu was supported by the MIT Data Science Laboratory. The research of W. Mao, K. Zhang, and T. Başar was supported in part by the U.S. Army Research Laboratory (ARL) Cooperative Agreement W911NF-17-2-0196, in part by the Office of Naval Research (ONR) [MURI Grant N00014-16-1-2710], and in part by the Air Force Office of Scientific Research (AFOSR) [Grant FA9550-19-1-0353]. K. Zhang also acknowledges support from U.S. Army Research Laboratory (ARL) [Grant W911NF-24-1-0085]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.02533 .

AARF: Autonomous Attack Response Framework for Honeypots to Enhance Interaction Based on Multi-Agent Dynamic Game

Highly interactive honeypots can form reliable connections by responding to attackers to delay and capture intranet attacks. However, current research focuses on modeling the attacker as part of the environment and defining single-step attack actions by simulation to study the interaction of honeypots. It ignores the iterative nature of the attack and defense game, which is inconsistent with the correlative and sequential nature of actions in real attacks. These limitations lead to insufficient interaction of the honeypot response strategies generated by the study, making it difficult to support effective and continuous games with attack behaviors. In this paper, we propose an autonomous attack response framework (named AARF) to enhance interaction based on multi-agent dynamic games. AARF consists of three parts: a virtual honeynet environment, attack agents, and defense agents. Attack agents are modeled to generate multi-step attack chains based on a Hidden Markov Model (HMM) combined with the generic threat framework ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge). The defense agents iteratively interact with the attack behavior chain based on reinforcement learning (RL) to learn to generate honeypot optimal response strategies. Aiming at the sample utilization inefficiency problem of random uniform sampling widely used in RL, we propose the dynamic value label sampling (DVLS) method in the dynamic environment. DVLS can effectively improve the sample utilization during the experience replay phase and thus improve the learning efficiency of honeypot agents under the RL framework. We further couple it with a classic DQN to replace the traditional random uniform sampling method. Based on AARF, we instantiate different functional honeypot models for deception in intranet scenarios. In the simulation environment, honeypots collaboratively respond to multi-step intranet attack chains to defend against these attacks, which demonstrates the effectiveness of AARF. The average cumulative reward of the DQN with DVLS is beyond eight percent, and the convergence speed is improved by five percent compared to a classic DQN.

An algorithm for multi-armed bandit based on variance change sensitivity

The Multi-Arm Bandit problem is becoming increasingly popular as it enables real-world sequential decision making across application domains, including clinical trials, recommender systems, and online decision making. The Multi-Arm Bandit problem is a classical problem of exploration and exploitation dilemma in reinforcement learning and it needs to decide on the optimal strategy based on the reward situation of each rocker arm. However, the existing Multi-Arm Bandit algorithms have many shortcomings, such as blind exploration, weak generalization ability and endless exploration. Aiming at the shortcomings of the existing Multi-Armed Bandit algorithms, this paper proposes a Multi-Armed Bandit algorithm based on variance change sensitivity. The algorithm takes the variance change of reward as a clue, and adjusts the exploration probability by the average variance change of all actions, and selects the action with the largest variance change when exploring. At the same time, in order to reduce the waste of action selection times and maximize the cumulative reward, A parameter Ncon was introduced to record the number of consecutive selections of the same action. The exploration stops when Ncon reaches a certain value. Experiments show that the exploration algorithm can obtain higher reward value and lower regret value in the end.

REINFORCEMENT LEARNING: FRAMEWORK, APPLICATIONS AND CHALLENGES

Artificial Intelligence is the science of developing intelligent machines that can simulate human intelligence and perform complex tasks. Reinforcement learning (RL) is the trending interdisciplinary learning paradigm of machine learning and optimal control that mirrors the hit and trial learning mechanism of humans .The self-learning agents take actions in a dynamic environment, get rewarded or punished and use this feedback to learn optimal decision-making to maximize cumulative rewards over time. This objective of this paper is to present an insight into promising future technology. The introduction to the technology is followed by the key concepts that constitute the reinforcement framework, the different models of this paradigm, the applications that span a wide arena and the challenges facing this ever-evolving strategy.

REINFORCEMENT LEARNING: FRAMEWORK, APPLICATIONS AND CHALLENGES

Artificial Intelligence is the science of developing intelligent machines that can simulate human intelligence and perform complex tasks. Reinforcement learning (RL) is the trending interdisciplinary learning paradigm of machine learning and optimal control that mirrors the hit and trial learning mechanism of humans .The self-learning agents take actions in a dynamic environment, get rewarded or punished and use this feedback to learn optimal decision-making to maximize cumulative rewards over time. This objective of this paper is to present an insight into promising future technology. The introduction to the technology is followed by the key concepts that constitute the reinforcement framework, the different models of this paradigm, the applications that span a wide arena and the challenges facing this ever-evolving strategy.

Reinforcement learning in building controls: A comparative study of algorithms considering model availability and policy representation

Reinforcement Learning (RL) has presented considerable potential as an advanced control technique in building controls to enable buildings operating more energy-efficient. As various types of RL algorithms have been studied on their performance of building controls, benchmarking these algorithms across the entire spectrum of features is essential to provide an overview and deepen the understanding of RL applications. Therefore, this study aims to compare and analyze the effectiveness of various RL algorithms, encompassing the entire RL categories featured by value-based, policy gradient, actor-critic and model-based RL considering model availability and policy representation. To provide a comprehensive analysis, in addition to the control performance quantified by the cumulative rewards based on the cost function of RL, data demand and robustness of hyperparameter tuning were investigated. The open-source Gym-Eplus framework was selected as the virtual environment to train and test different RL agents. The results showed that both model-free and model-based RL agents outperformed the baseline rule-based control in terms of energy consumption and thermal comfort, and RL agents were capable of evaluating both short-term and long-term rewards to achieve adaptive control optimization continuously along with the online control process. Model-based RL agent improved the data sampling efficiency but presented a relatively sacrificed control performance during the tested summer days.

FuzzyBandit An Autonomous Personalized Model Based on Contextual Multi Arm Bandits Using Explainable AI

In the era of artificial cognizance, context-aware decision-making problems have attracted significant attention. Contextual bandit addresses these problems by solving the exploration versus exploitation dilemma faced to provide customized solutions as per the user’s liking. However, a high level of accountability is required, and there is a need to understand the underlying mechanism of the black box nature of the contextual bandit algorithms proposed in the literature. To overcome these shortcomings, an explainable AI (XAI) based FuzzyBandit model is proposed, which maximizes the cumulative reward by optimizing the decision at each trial based on the rewards received in previous observations and, at the same time, generates explanations for the decision made. The proposed model uses an adaptive neuro-fuzzy inference system (ANFIS) to address the vague nature of arm selection in contextual bandits and uses a feedback mechanism to adjust its parameters based on the relevance and diversity of the features to maximize reward generation. The FuzzyBandit model has also been empirically compared with the existing seven most popular art of literature models on four benchmark datasets over nine criteria, namely recall, specificity, precision, prevalence, F1 score, Matthews Correlation Coefficient (MCC), Fowlkes–Mallows index (FM), Critical Success Index (CSI) and accuracy.

The emergence of cooperation via Q-learning in spatial donation game

Decision-making often overlooks the feedback between agents and the environment. Reinforcement learning is widely employed through exploratory experimentation to address problems related to states, actions, rewards, decision-making in various contexts. This work considers a new perspective, where individuals continually update their policies based on interactions with the spatial environment, aiming to maximize cumulative rewards and learn the optimal strategy. Specifically, we utilize the Q-learning algorithm to study the emergence of cooperation in a spatial population playing the donation game. Each individual has a Q-table that guides their decision-making in the game. Interestingly, we find that cooperation emerges within this introspective learning framework, and a smaller learning rate and higher discount factor make cooperation more likely to occur. Through the analysis of Q-table evolution, we disclose the underlying mechanism for cooperation, which may provide some insights to the emergence of cooperation in the real-world systems.

Comparative Evaluation of Asymptotically Optimal and Standard Upper Confidence Bound Algorithms in Multi-Armed Bandit Scenarios

The Multi-Armed Bandit (MAB) problem stands as a cornerstone challenge in the realm of reinforcement learning, particularly in contexts where an agent must sequentially choose actions from multiple options to maximize cumulative rewards, often analogized to pulling levers on slot machines. A plethora of algorithms have been developed and rigorously examined to tackle this issue, each aiming to enhance the process of reward collection. Nonetheless, a significant research gap persists, notably in relation to the Upper Confidence Bound algorithm (UCB). This study delves into the intricacies of UCB and its asymptotically optimal variant, focusing on their performance within a movie recommendation system context. Employing this real-world application as an illustrative example, the analysis seeks to discern the nuanced distinctions between the standard UCB and its asymptotically optimal counterpart. Through detailed exploration and comparison, insights emerge regarding the influence of different parameter values on the UCB algorithm's efficacy. This in-depth examination provides a more profound comprehension of how variations in these parameters affect the algorithm's capacity to balance exploration and exploitation, a key element in effective decision-making under uncertainty. Contrary to prevailing assumptions, the findings suggest that the asymptotically optimal UCB algorithm does not universally surpass the standard UCB algorithm in practical applications. This revelation calls for a reconsideration of the effectiveness of these algorithms, highlighting the importance of a nuanced evaluation of their performance across various scenarios.

Security Assessment of Industrial Control System Applying Reinforcement Learning

Industrial control systems are often used to assist and manage an industrial operation. These systems’ weaknesses in the various hierarchical structures of the system components and communication backbones make them vulnerable to cyberattacks that jeopardize their security. In this paper, the security of these systems is studied by employing a reinforcement learning extended attack graph to efficiently reveal the subsystems’ flaws. Specifically, an attack graph that mimics the environment is constructed for the system using the state–action–reward–state–action technique, in which the agent is regarded as the attacker. Attackers may cause the greatest amount of system damage with the fewest possible actions if they have the highest cumulative reward. The worst-case assault scheme with a total reward of 42.9 was successfully shown in the results, and the most badly affected subsystems were recognized.

Exoatmospheric Evasion Guidance Law with Total Energy Limit via Constrained Reinforcement Learning

Due to the lack of aerodynamic forces, the available propulsion for exoatmospheric pursuit-evasion problem is strictly limited, which has not been thoroughly investigated. This paper focuses on the evasion guidance in an exoatmospheric environment with total energy limit. A Constrained Reinforcement Learning (CRL) method is proposed to solve the problem. Firstly, the acceleration commands of the evader are defined as cost and an Actor-Critic-Cost (AC2) network structure is established to predict the accumulated cost of a trajectory. The learning objective of the agent becomes to maximize cumulative rewards while satisfying the cost constraint. Secondly, a Maximum-Minimum Entropy Learning (M2EL) method is proposed to minimize the randomness of acceleration commands while preserving the agent’s exploration capability. Our approaches address two challenges in the application of reinforcement learning: constraint specification and precise control. The well-trained agent is capable of generating accurate commands while satisfying the specified constraints. The simulation results indicate that the CRL and M2EL methods can effectively control the agent’s energy consumption within the specified constraints. The robustness of the agent under information error is also validated.