Continuous State Markov Decision Process Research Articles

A Q-learning algorithm for Markov decision processes with continuous state spaces

We propose an online algorithm for solving a class of continuous-state Markov decision processes. The algorithm combines classical Q-learning with an asynchronous averaging procedure, which allows Q-function estimates at sampled state–action pairs to be adaptively updated based on observations collected along a single sample trajectory. These estimates are then used to iteratively construct an interpolation-based function approximator of the Q-function. We prove the convergence of the algorithm and provide numerical results to illustrate its performance.

Kernel-based diffusion approximated Markov decision processes for autonomous navigation and control on unstructured terrains

We propose a diffusion approximation method to the continuous-state Markov decision processes that can be utilized to address autonomous navigation and control in unstructured off-road environments. In contrast to most decision-theoretic planning frameworks that assume fully known state transition models, we design a method that eliminates such a strong assumption that is often extremely difficult to engineer in reality. We first take the second-order Taylor expansion of the value function. The Bellman optimality equation is then approximated by a partial differential equation, which only relies on the first and second moments of the transition model. By combining the kernel representation of the value function, we design an efficient policy iteration algorithm whose policy evaluation step can be represented as a linear system of equations characterized by a finite set of supporting states. We first validate the proposed method through extensive simulations in 2D obstacle avoidance and 2.5D terrain navigation problems. The results show that the proposed approach leads to a much superior performance over several baselines. We then develop a system that integrates our decision-making framework with onboard perception and conduct real-world experiments in both cluttered indoor and unstructured outdoor environments. The results from the physical systems further demonstrate the applicability of our method in challenging real-world environments.

Max-Plus Linear Approximations for Deterministic Continuous-State Markov Decision Processes

We consider deterministic continuous-state Markov decision processes (MDPs). We apply a max-plus linear method to approximate the value function with a specific dictionary of functions that leads to an adequate state-discretization of the MDP. This is more efficient than a direct discretization of the state space, typically intractable in high dimension. We propose a simple strategy to adapt the discretization to a problem instance, thus mitigating the curse of dimensionality. We provide numerical examples showing that the method works well on simple MDPs.

A content search method for security topics in microblog based on deep reinforcement learning

Traditional methods treat the search problem as a process of selecting and ranking sequential documents. The methods have been proved effective and are widely used in the web search domain. However, due to the complexity and particularity of microblog text contents, the classical methods are rarely used microblog searches for specific topics. Focusing on the issue of searching for specific topics in microblog content, we present a microblog search method for security topics based on deep reinforcement learning by modeling the microblog search for specific topics as a continuous-state Markov decision process. We also design a novel deep Q network to evaluate the relevance of microblog content based on the target topic. We adopt reinforcement learning to solve the microblog search problem using an intelligent strategy and evaluate content relevance through deep learning. Experiments conducted on a real-world dataset show that our approach outperforms the selected baseline methods.

Sensing, Probing, and Transmitting Policy for Energy Harvesting Cognitive Radio With Two-Stage After-State Reinforcement Learning

This paper considers joint optimization of spectrum sensing, channel probing, and transmission power control for a single-channel secondary transmitter that operates with harvested energy from ambient sources. At each time slot, to maximize the expected secondary throughput, the transmitter needs to decide whether or not to perform the operations of spectrum sensing, channel probing, and transmission, according to energy status and channel fading status. First, we model this stochastic optimization problem as a two-stage continuous-state Markov decision process, with a sensing-and-probing stage and a transmit-power-control stage. We simplify this problem by a more useful after-state value function formulation. We then propose a reinforcement learning algorithm to learn the after-state value function from data samples when the statistical distributions of harvested energy and channel fading are unknown. Numerical results demonstrate learning characteristics and performance of the proposed algorithm.

Learning Datum-Wise Sampling Frequency for Energy-Efficient Human Activity Recognition

Continuous Human Activity Recognition (HAR) is an important application of smart mobile/wearable systems for providing dynamic assistance to users. However, HAR in real-time requires continuous sampling of data using built-in sensors (e.g., accelerometer), which significantly increases the energy cost and shortens the operating span. Reducing sampling rate can save energy but causes low recognition accuracy. Therefore, choosing adaptive sampling frequency that balances accuracy and energy efficiency becomes a critical problem in HAR. In this paper, we formalize the problem as minimizing both classification error and energy cost by choosing dynamically appropriate sampling rates. We propose Datum-Wise Frequency Selection (DWFS) to solve the problem via a continuous state Markov Decision Process (MDP). A policy function is learned from the MDP, which selects the best frequency for sampling an incoming data entity by exploiting a datum related state of the system. We propose a method for alternative learning the parameters of an activity classification model and the MDP that improves both the accuracy and the energy efficiency. We evaluate DWFS with three real-world HAR datasets, and the results show that DWFS statistically outperforms the state-of-the-arts regarding a combined measurement of accuracy and energy efficiency.

A learning method for energy optimization of the plug-in hybrid electric bus

The optimal energy management for a plug-in hybrid electric bus (PHEB) running along the fixed city bus route is an important technique to improve the vehicles' fuel economy and reduce the bus emission. Considering the inherently high regularities of the fixed bus routes, the continuous state Markov decision process (MDP) is adopted to describe a cost function as total gas and electric consumption fee. Then a learning algorithm is proposed to construct such a MDP model without knowing the all parameters of the MDP. Next, fitted value iteration algorithm is given to approximate the cost function, and linear regression is used in this fitted value iteration. Simulation results show that this approach is feasible in searching for the control strategy of PHEB. Simultaneously this method has its own advantage comparing with the CDCS mode. Furthermore, a test based on a real PHEB was carried out to verify the applicable of the proposed method.

Optimal day-ahead trading and storage of renewable energies—an approximate dynamic programming approach

A renewable power producer who trades on a day-ahead market sells electricity under supply and price uncertainty. Investments in energy storage mitigate the associated financial risks and allow for decoupling the timing of supply and delivery. This paper introduces a model of the optimal bidding strategy for a hybrid system of renewable power generation and energy storage. We formulate the problem as a continuous-state Markov decision process and present a solution based on approximate dynamic programming. We propose an algorithm that combines approximate policy iteration with Least Squares Policy Evaluation (LSPE) which is used to estimate the weights of a polynomial value function approximation. We find that the approximate policies produce significantly better results for the continuous state space than an optimal discrete policy obtained by linear programming. A numerical analysis of the response surface of rewards on model parameters reveals that supply uncertainty, imbalance costs and a negative correlation of market price and supplies are the main drivers for investments in energy storage. Supply and price autocorrelation, on the other hand, have a negative effect on the value of storage.