Q-value Function Research Articles

Reference RL: Reinforcement learning with reference mechanism and its application in traffic signal control

This paper addresses the challenges of deploying reinforcement learning (RL) models for traffic signal control (TSC) in real-world environments. Real-world training can prevent mismatches between simulation environments and the actual traffic conditions, thereby achieving better performance of agent upon deployment. However, free explorations by agents during real-world training can disrupt traffic operations. To mitigate this, this paper proposes a reference mechanism to guide the decision-making process within the RL framework. A reference timing policy, typically a model-based signal strategy, is integrated into the learning process to provide agents with reference actions. Specifically, an additional Q-value function is introduced to evaluate both the agent’s actions and those of the reference policy, allowing for adjustments before the actions are executed in real traffic system. Numerical results indicate that the reference mechanism effectively enhances system performance early in the training process, thus accelerating learning. We also combine the reference RL method with a pretraining procedure and a jump-start algorithm, respectively. Experimental results demonstrate their effectiveness in further enhancing system performance and facilitating real-world training.

OPGW State Evaluation Method Based on MSIF and QPSO-DQN in Icing Scenarios

A new OPGW state evaluation method based on Multi-Source Information Fusion (MSIF) and Quantum Particle Swarm Optimization & Deep Q-learning (QPSO-DQN) is proposed. Firstly, using MSIF to integrate and unify historical data and real-time monitoring data of OPGW, more comprehensive and accurate OPGW status information was obtained. Then, utilizing the advantages of deep reinforcement learning (DRL) algorithm DQN in handling highly nonlinear problems, various influencing factors related to the operation of OPGW were addressed. Finally, DQN was improved by introducing the QPSO optimization algorithm, which transformed the Q-value function solving in DQN into a function fitting problem and used QPSO as an intelligent agent to fit the function, achieving accurate evaluation of the OPGW operating status. The simulation experiment results show that the proposed method has the highest accuracy in ice weight detection, temperature detection, frequency detection, and optical power detection on the same dataset, reaching 98.85%, 98.97%, 98.13%, and 98.97%, respectively.

Improved energy decay estimate for Dir-stationary Q-valued functions and its applications

In this paper, we establish an improved decay estimate for the Dirichlet energy of Dir-stationary Q-valued functions. As a direct application of this estimate, we derive a Liouville-type theorem for bounded Dir-stationary Q-valued functions defined on Rm. Additionally, in an attempt to establish the continuity of Dir-stationary Q-valued functions, we confirm that such functions exhibit the Lebesgue property at every point within their domain.

Optimizing data processing for edge-enabled IoT devices using deep learning based heterogeneous data clustering approach

- Edge-based Internet of Things devices have transformed smart farming, aiding in efficient data collection and processing for optimal resource utilization and crop yields. However, task scheduling and resource allocation pose significant challenges due to the dynamic nature of agricultural environments. Our research introduces a novel framework that integrates deep reinforcement learning algorithm into an edge-enabled wireless sensor network for multi-objective optimization of the functionality of the Deep Q-Networks (DQNs). This framework extends the traditional Q-learning method to manage large state-action spaces efficiently. It employs a deep neural network to approximate the Q-value function, rather than relying on a Q-table, making it more capable of handling complex problems with high-dimensional state spaces. It forms heterogenous data clusters supports an optimal task scheduling and resource allocation policies, sustains key objectives such as minimal energy consumption, latency, efficient resource utilization, and reduced task completion time. The framework's performance is evaluated in a simulated environment mimicking real-world smart farming applications. Results confirm its superiority in enhancing performance metrics and lowering energy consumption, as opposed to traditional networks.

A New Fourier Q Operator Network Based Reinforcement Learning Method for Continuous Action Space Decision-making in Manufacturing

The problems of continuous action space decision-making are widespread in industrial manufacturing. However, when dealing with these problems, existing reinforcement learning (RL) methods relies on a large number of training samples, which is always unacceptable given the limited availability or expensive nature of data, such as low-volume manufacturing. This paper proposes a new Fourier Q operator network (FQON) based RL method. The input of FQON is the expected state function and its output the Q-value function, and both functions take the action in RL as independent variables. The infinite-dimensional mapping between the function domains is established by a set of parameters that can be used with different discretization, which fixes the mapping complexity regardless of the action space resolution. By taking the advantages of the fast calculation using on Fourier kernel operator, the mapping complexity is highly reduced, and it enables that FQON can realize the decision-making in continuous action space using a small amount of training samples. Taking machining deformation control of an aero-engine casing as a case study, experimental results showed that FQON based RL method can control the deformation well with limited training samples.

Multiple Unmanned Aerial Vehicle Autonomous Path Planning Algorithm Based on Whale-Inspired Deep Q-Network

In emergency rescue missions, rescue teams can use UAVs and efficient path planning strategies to provide flexible rescue services for trapped people, which can improve rescue efficiency and reduce personnel risks. However, since the task environment of UAVs is usually complex, uncertain, and communication-limited, traditional path planning methods may not be able to meet practical needs. In this paper, we introduce a whale optimization algorithm into a deep Q-network and propose a path planning algorithm based on a whale-inspired deep Q-network, which enables UAVs to search for targets faster and safer in uncertain and complex environments. In particular, we first transform the UAV path planning problem into a Markov decision process. Then, we design a comprehensive reward function considering the three factors of path length, obstacle avoidance, and energy consumption. Next, we use the main framework of the deep Q-network to approximate the Q-value function by training a deep neural network. During the training phase, the whale optimization algorithm is introduced for path exploration to generate a richer action decision experience. Finally, experiments show that the proposed algorithm can enable the UAV to autonomously plan a collision-free feasible path in an uncertain environment. And compared with classic reinforcement learning algorithms, the proposed algorithm has a better performance in learning efficiency, path planning success rate, and path length.

Goal-Conditioned Q-learning as Knowledge Distillation

Many applications of reinforcement learning can be formalized as goal-conditioned environments, where, in each episode, there is a "goal" that affects the rewards obtained during that episode but does not affect the dynamics. Various techniques have been proposed to improve performance in goal-conditioned environments, such as automatic curriculum generation and goal relabeling. In this work, we explore a connection between off-policy reinforcement learning in goal-conditioned settings and knowledge distillation. In particular: the current Q-value function and the target Q-value estimate are both functions of the goal, and we would like to train the Q-value function to match its target for all goals. We therefore apply Gradient-Based Attention Transfer (Zagoruyko and Komodakis 2017), a knowledge distillation technique, to the Q-function update. We empirically show that this can improve the performance of goal-conditioned off-policy reinforcement learning when the space of goals is high-dimensional. We also show that this technique can be adapted to allow for efficient learning in the case of multiple simultaneous sparse goals, where the agent can attain a reward by achieving any one of a large set of objectives, all specified at test time. Finally, to provide theoretical support, we give examples of classes of environments where (under some assumptions) standard off-policy algorithms such as DDPG require at least O(d^2) replay buffer transitions to learn an optimal policy, while our proposed technique requires only O(d) transitions, where d is the dimensionality of the goal and state space. Code and appendix are available at https://github.com/alevine0/ReenGAGE.

Offline Quantum Reinforcement Learning in a Conservative Manner

Recently, to reap the quantum advantage, empowering reinforcement learning (RL) with quantum computing has attracted much attention, which is dubbed as quantum RL (QRL). However, current QRL algorithms employ an online learning scheme, i.e., the policy that is run on a quantum computer needs to interact with the environment to collect experiences, which could be expensive and dangerous for practical applications. In this paper, we aim to solve this problem in an offline learning manner. To be more specific, we develop the first offline quantum RL (offline QRL) algorithm named CQ2L (Conservative Quantum Q-learning), which learns from offline samples and does not require any interaction with the environment. CQ2L utilizes variational quantum circuits (VQCs), which are improved with data re-uploading and scaling parameters, to represent Q-value functions of agents. To suppress the overestimation of Q-values resulting from offline data, we first employ a double Q-learning framework to reduce the overestimation bias; then a penalty term that encourages generating conservative Q-values is designed. We conduct abundant experiments to demonstrate that the proposed method CQ2L can successfully solve offline QRL tasks that the online counterpart could not.

On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation

Sample-efficient offline reinforcement learning (RL) with linear function approximation has been studied extensively recently. Much of the prior work has yielded instance-independent rates that hold even for the worst-case realization of problem instances. This work seeks to understand instance-dependent bounds for offline RL with linear function approximation. We present an algorithm called Bootstrapped and Constrained Pessimistic Value Iteration (BCP-VI), which leverages data bootstrapping and constrained optimization on top of pessimism. We show that under a partial data coverage assumption, that of concentrability with respect to an optimal policy, the proposed algorithm yields a fast rate for offline RL when there is a positive gap in the optimal Q-value functions, even if the offline data were collected adaptively. Moreover, when the linear features of the optimal actions in the states reachable by an optimal policy span those reachable by the behavior policy and the optimal actions are unique, offline RL achieves absolute zero sub-optimality error when the number of episodes exceeds a (finite) instance-dependent threshold. To the best of our knowledge, these are the first results that give a fast rate bound on the sub-optimality and an absolute zero sub-optimality bound for offline RL with linear function approximation from adaptive data with partial coverage. We also provide instance-agnostic and instance-dependent information-theoretical lower bounds to complement our upper bounds.

Peer to Peer Distributed Solar Energy Trading

Solar module prices have dramatically dropped in recent years, which in turn has facilitated distributed solar energy resources (DSERs) in smart grids. To manage electricity real time supply and demand, the utilities has shown the strong interest in deploying virtual power plants(VPPs) which enable solar generated energy trading to reduce the intermittent DSERs impact in electric grid. Unfortunately, the current energy trading approaches in residential VPPs typically require a trusted third party to take on the role of the middleman, DSER users are not allowed to trade their surplus solar energy independently and simultaneously to maximize potential benefits. In addition, these approaches do not achieve "fair" for both VPPs and DSERs users. To address this issue, we build a multi-agent deep reinforcement learning based peer to peer (P2P) new solar energy trading system-SolarTrader+, which enables unsupervised, distributed, and long term fair solar energy trading in residential VPPs. We apply deep reinforcement learning with neural networks as Q-value function approximator.We implement SolarTrader+ and evaluate it using data from U.S. residential VPP communities that are comprised of - 119 residential DSERs. Our results show that SolarTrader+ can reduce the aggregated VPP energy consumption by 83.8% when compared against a non-trading approach.

Towards Real-Time Video Caching at Edge Servers: A Cost-Aware Deep Q-Learning Solution

Given the rapid growth of user-generated videos, internet traffic has been heavily dominated by online video streaming. Caching videos on edge servers in close proximity to users has been an effective approach to reduce the backbone traffic and the request response time, as well as to improve the video quality on the user side. Video popularity, however, can be highly dynamic over time. The cost of cache replacement at edge servers, particularly that related to service interruption during replacement, is not yet well understood. This paper presents a novel lightweight video caching algorithm for edge servers, seeking to optimize the hit rate with real-time decisions and minimized cost. Inspired by recent advances in deep Q-learning, our DQN-based online video caching (DQN-OVC) makes effective use of the rich and readily available information from users and networks. We decompose the Q-value function as a product of the video value function and the action function, which significantly reduces the state space. We instantiate the action function for cost-aware caching decisions with low complexity so that the cached videos can be updated continuously and instantly with dynamic video popularity. We used video traces from Tencent, one of the largest online video providers in China, to evaluate the performance of our DQN-OVC and to compare it with state-of-the-art solutions. The results demonstrate that DQN-OVC significantly outperforms the baseline algorithms in the edge caching context.

Modeling Collective Behavior for Fish School With Deep Q-Networks

Modeling collective behavior is a way to better understand the mechanisms that govern collective animal behaviors. Traditional rule-based modeling methods rely heavily on human prior knowledge and may not provide a proper explanation of the phenomenon of collective behaviors. This paper proposes a Deep Q-Networks (DQN)-based modeling method for fish school. Firstly, an individual’s state (continuous value) is expressed by the angle between its direction and the average direction of its perceived neighbors. An individual’s action is represented with discretized turning angle. Then, the reward function is constructed with the change in the number of neighbors. And finally, the neural network structure is constructed to represent the Q-value function and is trained by the DQN algorithm. The proposed approach is tested in two scenarios: single-learner and multi-learner. Results show that in both scenarios the proposed method can gradually converge and finally obtain a model that can produce collective behavior. On this basis, this paper also deeply analyzes the learned model from the perspectives of average order parameter and collective behavior patterns. It verifies that the behavior pattern generated by the learned model is a highly ordered collective behavior. In addition, we make a comparison between our proposed approach and the Q-Learning algorithm. The results show that our approach not only has a stronger ability to express policy and is better at handling continuous states but also has a more stable learning performance in training.

AQMDRL: Automatic Quality of Service Architecture Based on Multistep Deep Reinforcement Learning in Software-Defined Networking

Software-defined networking (SDN) has become one of the critical technologies for data center networks, as it can improve network performance from a global perspective using artificial intelligence algorithms. Due to the strong decision-making and generalization ability, deep reinforcement learning (DRL) has been used in SDN intelligent routing and scheduling mechanisms. However, traditional deep reinforcement learning algorithms present the problems of slow convergence rate and instability, resulting in poor network quality of service (QoS) for an extended period before convergence. Aiming at the above problems, we propose an automatic QoS architecture based on multistep DRL (AQMDRL) to optimize the QoS performance of SDN. AQMDRL uses a multistep approach to solve the overestimation and underestimation problems of the deep deterministic policy gradient (DDPG) algorithm. The multistep approach uses the maximum value of the n-step action currently estimated by the neural network instead of the one-step Q-value function, as it reduces the possibility of positive error generated by the Q-value function and can effectively improve convergence stability. In addition, we adapt a prioritized experience sampling based on SumTree binary trees to improve the convergence rate of the multistep DDPG algorithm. Our experiments show that the AQMDRL we proposed significantly improves the convergence performance and effectively reduces the network transmission delay of SDN over existing DRL algorithms.

Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Critic

Model-based reinforcement learning algorithms, which aim to learn a model of the environment to make decisions, are more sample efficient than their model-free counterparts. The sample efficiency of model-based approaches relies on whether the model can well approximate the environment. However, learning an accurate model is challenging, especially in complex and noisy environments. To tackle this problem, we propose the conservative model-based actor-critic (CMBAC), a novel approach that achieves high sample efficiency without the strong reliance on accurate learned models. Specifically, CMBAC learns multiple estimates of the Q-value function from a set of inaccurate models and uses the average of the bottom-k estimates---a conservative estimate---to optimize the policy. An appealing feature of CMBAC is that the conservative estimates effectively encourage the agent to avoid unreliable “promising actions”---whose values are high in only a small fraction of the models. Experiments demonstrate that CMBAC significantly outperforms state-of-the-art approaches in terms of sample efficiency on several challenging control tasks, and the proposed method is more robust than previous methods in noisy environments.

Q-Learning for Shift-Reduce Parsing in Indonesian Tree-LSTM-Based Text Generation

Tree-LSTM algorithm accommodates tree structure processing to extract information outside the linear sequence pattern. The use of Tree-LSTM in text generation problems requires the help of an external parser at each generation iteration. Developing a good parser demands the representation of complex features and relies heavily on the grammar of the corpus. The limited corpus results in an insufficient number of vocabs for a grammar-based parser, making it less natural to link the text generation process. This research aims to solve the problem of limited corpus by proposing the use of a Reinforcement Learning algorithm in the formation of constituency trees, which link the sentence generation process given a seed phrase as the input in the Tree-LSTM model. The tree production process is modeled as a Markov’s decision process, where a set of states consists of word embedding vectors, and a set of actions of {Shift, Reduce}. The Deep Q-Network model as an approximator of the Q-Learning algorithm is trained to obtain optimal weights in representing the Q-value function. The test results on perplexity-based evaluation show that the proposed Tree-LSTM and Q-Learning combination model achieves values 9.60 and 4.60 for two kinds of corpus with 205 and 1,000 sentences, respectively, better than the Shift-All model. Human evaluation of Friedman test and posthoc analysis showed that all five respondents tended to give the same assessment for the combination model of Tree-LSTM and Q-Learning, which on average outperforms two other nongrammar models, i.e., Shift-All and Reduce-All.

Multi-Agent Reinforcement Learning with Optimal Equivalent Action of Neighborhood

In a multi-agent system, the complex interaction among agents is one of the difficulties in making the optimal decision. This paper proposes a new action value function and a learning mechanism based on the optimal equivalent action of the neighborhood (OEAN) of a multi-agent system, in order to obtain the optimal decision from the agents. In the new Q-value function, the OEAN is used to depict the equivalent interaction between the current agent and the others. To deal with the non-stationary environment when agents act, the OEAN of the current agent is inferred simultaneously by the maximum a posteriori based on the hidden Markov random field model. The convergence property of the proposed methodology proved that the Q-value function can approach the global Nash equilibrium value using the iteration mechanism. The effectiveness of the method is verified by the case study of the top-coal caving. The experiment results show that the OEAN can reduce the complexity of the agents’ interaction description, meanwhile, the top-coal caving performance can be improved significantly.

Adaptive Double Fuzzy Systems Based Q-Learning for Pursuit-Evasion Game

The paper addresses the problem of obtaining the optimal solutions of pursuit and evasion game in continuous time system. A novel technique that combines double fuzzy systems with Q-learning is proposed. The system consists of two segments: fuzzy Q-learning (FQL) module and Q-value table fuzzy inference system (QTFIS). The linguistic rules are updated adaptively by Q-learning in FQL module. Local Q-value table which takes advantages of the sparsity of Q-learning is the kernel data structure in FQL. The Controllers of pursuer and evader are provided by LQT while the parameters of fuzzy inference system are updated adaptively. Q-value function is approximated by the fuzzy inference system according to QTFIS. The innovative method has an advantage of not relying on accurate controlled models. The difficult drawback of continuous state space and continuous actions for Q-learning is overcome by the novel technique. The limitation of low dimensional space for Q-learning is broken by the system. The proposed technique is applied to an imperfect information pursuit-evasion game between UAVs. Finally, computer simulations show the usefulness of the creative approach.

Double Deep Q-Learning for Optimal Execution

ABSTRACT Optimal trade execution is an important problem faced by essentially all traders. Much research into optimal execution uses stringent model assumptions and applies continuous time stochastic control to solve them. Here, we instead take a model free approach and develop a variation of Deep Q-Learning to estimate the optimal actions of a trader. The model is a fully connected Neural Network trained using Experience Replay and Double DQN with input features given by the current state of the limit order book, other trading signals, and available execution actions, while the output is the Q-value function estimating the future rewards under an arbitrary action. We apply our model to nine different stocks and find that it outperforms the standard benchmark approach on most stocks using the measures of (i) mean and median out-performance, (ii) probability of out-performance, and (iii) gain-loss ratios.

A learning search algorithm with propagational reinforcement learning

When reinforcement learning with a deep neural network is applied to heuristic search, the search becomes a learning search. In a learning search system, there are two key components: (1) a deep neural network with sufficient expression ability as a heuristic function approximator that estimates the distance from any state to a goal; (2) a strategy to guide the interaction of an agent with its environment to obtain more efficient simulated experience to update the Q-value or V-value function of reinforcement learning. To date, neither component has been sufficiently discussed. This study theoretically discusses the size of a deep neural network for approximating a product function of p piecewise multivariate linear functions. The existence of such a deep neural network with O(n + p) layers and O(dn + dnp + dp) neurons has been proven, where d is the number of variables of the multivariate function being approximated, 𝜖 is the approximation error, and n = O(p + log2(pd/𝜖)). For the second component, this study proposes a general propagational reinforcement-learning-based learning search method that improves the estimate h(.) according to the newly observed distance information about the goals, propagates the improvement bidirectionally in the search tree, and consequently obtains a sequence of more accurate V-values for a sequence of states. Experiments on the maze problems show that our method increases the convergence rate of reinforcement learning by a factor of 2.06 and reduces the number of learning episodes to 1/4 that of other nonpropagating methods.

Adaptive optimal control of stencil printing process using reinforcement learning

The stencil printing process (SPP) is a critical operation in surface mount technology (SMT) because it contributes to 60% of soldering defects. The complex relationships between solder paste volume transfer efficiency (TE) and the SPP variables make the control of the solder paste volume TE during production a challenging problem. This research aims to optimize the stencil printing parameters in real time to control the solder paste volume TE and increase the first-pass yields of printed circuit boards (PCBs). A Reinforcement learning (RL) approach, specifically Q-learning, is used to control and maintain the volume TE within the spec limits in an optimal adaptive control system. RL deals with the problem of building a control system or an autonomous agent that can learn how to take the proper actions to reach its objectives through interaction with its environment. The application of RL in SPP is not yet fully explored; therefore, this study investigates the impacts of applying Q-learning to control the volume TE in real time. The proposed control systems capture the induced variations in the SPP for two printing directions and consequently adjust the significant and easy-to-change printing parameters in real time. Two types of Q-learning are explored: Q-table that uses a tabular format to store the Q-values and Q-network that uses an artificial neural network (ANN) to approximate the Q-value function. Moreover, a new heterogeneous reward function-based clustering is proposed, which is integrated into the Q-network to enhance its performance. The results show that the developed control systems can learn the optimal policy and take the proper actions to transit from initial states to terminal states. The proposed control systems using Q-network with a function approximator and heterogeneous reward function converge fully much faster than Q-table using continuous state space. Moreover, Q-network control systems are capable to transit more states to terminal states with a lower number of actions when compared to Q-table control systems.