Partially Observable Markov Decision Process Research Articles

Integrated robust navigation and guidance for the kinetic impact of near-earth asteroids based on deep reinforcement learning

The defense against near-Earth asteroids (NEAs) using kinetic impact is faced with various challenges, including limited maneuverability of the impactor, inaccurate dynamic models, poor observability of relative navigation, and control execution errors. To address these challenges, an integrated robust navigation and guidance method for the kinetic impact of NEAs based on deep reinforcement learning (DRL) is proposed in this paper, which can directly map angle measurements by a monocular camera to guidance maneuvers. Firstly, a discrete partial observable Markov decision process (POMDP) is modeled for the integrated navigation and guidance problem of NEA interception. To address the situation where impactors are usually only equipped with aiming cameras and only the line of sight (LOS) angle is measured, past and current LOS measurements are concatenated into a one-dimensional state observation vector to directly introduce the memory of historical state information. A shaping reward function design based on potential energy has also been proposed to solve the problem of sparse reward as the main goal. Subsequently, proximal policy optimization (PPO) is used to solve the established POMDP model, obtaining an integrated navigation and guidance policy that directly maps from the original output of the navigation sensor to the guidance command. The potentially hazard asteroid (PHA) Bennu is used as the target and a typical kinetic impact defense scenario is designed to simulate and verify the model and method proposed in this paper, considering the influence of multiple factors. Numerical simulation results show that the proposed method achieves an interception accuracy of 203.58 m (mean value). The proposed method abandons the traditional separate design of navigation and guidance algorithms, and its robustness has been tested and verified in a wide range of uncertain environments.

Peer-to-peer trading in smart grid with demand response and grid outage using deep reinforcement learning

With the price of green energy now more reasonable, users can now produce enough electricity to meet their needs and make a profit by selling the surplus on the underground P2P energy market. For households, energy trading and demand management can reduce electricity costs. However, consumers generally obtain market offers based on their expectations and the forecasts of other households. However, the P2P exchange system is not able to quantify the gap between these offers and the best market. The objective of this paper is to apply deep reinforcement learning techniques to optimal energy trading and demand response (DR) methods within a peer-to-peer (P2P) market. The main objective is to maximize cost reductions. The best approach to achieve this objective was investigated as part of this project. The complexity of domestic energy trading and energy recovery is formally characterized as a partially observable Markov decision process (POMDP). Through decentralized training and performance-based learning, the strategy maximizes policy and value functions. In order to identify the most effective proactive solutions, a comparative analysis is carried out between the two parties. Based on the simulation results, it was observed that implementing the recommended reinforcement learning strategy to optimize peer-to-peer (P2P) energy exchange can lead to a significant improvement in the average household reward. Specifically, the average household reward can be increased by 7.6% and 12.08% by employing the aforementioned approach.

Simultaneous search and monitoring by multiple aerial robots

This paper studies simultaneous search and monitoring (SSM) between multiple unmanned aerial vehicles (UAVs) and multiple moving ground targets. Searching for unknown targets and monitoring known ones are two intrinsically related problems, but they have mostly been addressed in isolation. We combine the two tasks and exploit their interconnection as a synergy rather than a trade-off. We construct the single-robot SSM as a partially observable Markov decision process (POMDP) and the multi-robot SSM as a semi-decentralised POMDP (semi-Dec-POMDP). A novel heuristic reactive policy planning is proposed to solve the POMDP. It is then extended for semi-Dec-POMDP with game-theoretical methods. In simulations and experiments, the searchers will successfully locate unknown targets without losing known ones and cooperate by partitioning their tasks. With theoretical proofs, simulations, and experiments, we demonstrate that our method can perform better than conventional approaches and the state-of-the-art.

Stopping-Time Control of Multiple Smart Sensors Based on Tradeoffs Between Sensing Accuracy and Energy Consumption While Maintaining Energy Consumption Balance

This paper considers the stopping-time control of multiple smart sensors which independently sample the physical parameter of interests and request the desired data from the common server with probabilities. More samples may lead to higher sensing accuracy but cost more energy. More requests may facilitate the smart sensors to obtain the desired data with higher probability while yet cost more energy. Meanwhile, the heterogeneities of smart sensors issuing the requests may lead to early energy exhaustion for some smart sensors. Therefore, how to make an optimal tradeoff between sensing accuracy and energy consumption while keeping the energy consumption balance arises as an interesting problem. To address this issue, this paper first formulates the stopping-time control policy of an individual smart sensor as a partially observable Markov decision process (POMDP), then coordinates the stopping-time control policies of multiple smart sensors within a noncooperative game (NCG), and finally proposes a noncooperative PODMP game-based algorithm to make the above tradeoff. Theoretical analysis and field experiments jointly document the performance. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper addresses the interesting tradeoff between sensing accuracy and energy cost while balancing the energy consumption in a sensor network that operates in the environments where the available energy is limited and the early energy exhaustion for some of the smart sensors should be avoided. It helps to improve the operation efficiency of the sensor network with multiple smart sensors and a shared server setting in the area of Internet of Things (IoT) or Cyber-Physical Systems (CPS) to minimize the estimation error, while keeping the accumulative energy cost within a range and balancing the energy consumption. Field experimental investigations show that the solution proposed here outperforms existing solutions.

Improving Knowledge Learning Through Modelling Students’ Practice-Based Cognitive Processes

Practice is an essential means by which humans and animals engage in cognitive activities. Intelligent tutoring systems, with a crucial component of modelling learners’ cognitive processes during learning and optimizing their learning strategies, offer an excellent platform to investigate students’ practice-based cognitive processes. In related studies, modelling methods for cognitive processes have demonstrated commendable performance. Furthermore, researchers have extended their investigations using decision-theoretic approaches, such as a partially observable Markov decision process (POMDP), to induce learning strategies by modelling the students’ cognitive processes. However, the existing research has primarily centered around the modelling of macro-level instructional behaviors rather than the specific practice selection made by the students within the intricate realms of cognitive domains. In this paper, we adapt the POMDP model to represent relations between the student’s performance on cognitive tasks and his/her cognitive states. By doing so, we can predict his/her performance while inducing learning strategies. More specifically, we focus on question selection during the student’s real-time learning activities in an intelligent tutoring system. To address the challenges on modelling complex cognitive domains, we exploit the question types to automate parameter learning and subsequently employ information entropy techniques to refine learning strategies in the POMDP. We conduct the experiments in two real-world knowledge concept learning domains. The experimental results show that the performance of the learning strategies induced by our new model is superior to that of other learning strategies. Moreover, the new model has good reliability in predicting the student’s performance. Utilizing an intelligent tutoring system as the research platform, this article addresses the modelling and strategy induction challenges of practice-based cognitive processes with intricate structures, aiming to tutor students effectively. Our work provides a new approach of predicting the students’ performance as well as personalizing their learning strategies.

Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes

In applications of offline reinforcement learning to observational data, such as in healthcare or education, a general concern is that observed actions might be affected by unobserved factors, inducing confounding and biasing estimates derived assuming a perfect Markov decision process (MDP) model. In “Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes,” A. Bennett and N. Kallus tackle this by considering off-policy evaluation in a partially observed MDP (POMDP). Specifically, they consider estimating the value of a given target policy in an unknown POMDP, given observations of trajectories generated by a different and unknown policy, which may depend on the unobserved states. They consider both when the target policy value can be identified the observed data and, given identification, how best to estimate it. Both these problems are addressed by extending the framework of proximal causal inference to POMDP settings, using sequences of so-called bridge functions. This results in a novel framework for off-policy evaluation in POMDPs that they term proximal reinforcement learning, which they validate in various empirical settings.

Optimal Scheduling in IoT-Driven Smart Isolated Microgrids Based on Deep Reinforcement Learning

In this paper, we investigate the scheduling issue of diesel generators (DGs) in an Internet of Things (IoT)-Driven isolated microgrid (MG) by deep reinforcement learning (DRL). The renewable energy is fully exploited under the uncertainty of renewable generation and load demand. The DRL agent learns an optimal policy from history renewable and load data of previous days, where the policy can generate real-time decisions based on observations of past renewable and load data of previous hours collected by connected sensors. The goal is to reduce operating cost on the premise of ensuring supply-demand balance. In specific, a novel finite-horizon partial observable Markov decision process (POMDP) model is conceived considering the spinning reserve. In order to overcome the challenge of discrete-continuous hybrid action space due to the binary DG switching decision and continuous energy dispatch (ED) decision, a DRL algorithm, namely the hybrid action finite-horizon RDPG (HAFH-RDPG), is proposed. HAFH-RDPG seamlessly integrates two classical DRL algorithms, i.e., deep Q-network (DQN) and recurrent deterministic policy gradient (RDPG), based on a finite-horizon dynamic programming (DP) framework. Extensive experiments are performed with real-world data in an IoT-driven MG to evaluate the capability of the proposed algorithm in handling the uncertainty due to inter-hour and inter-day power fluctuation and to compare its performance with those of the benchmark algorithms.

Explainable production planning under partial observability in high-precision manufacturing

Conceptually, high-precision manufacturing is a sequence of production and measurement steps, where both kinds of steps require to use non-deterministic models to represent production and measurement tolerances. This paper demonstrates how to effectively represent these manufacturing processes as Partially Observable Markov Decision Processes (POMDP) and derive an offline strategy with state-of-the-art Monte Carlo Tree Search (MCTS) approaches. In doing so, we face two challenges: a continuous observation space and explainability requirements from the side of the process engineers. As a result, we find that a tradeoff between the quantitative performance of the solution and its explainability is required. In a nutshell, the paper elucidates the entire process of explainable production planning: We design and validate a white-box simulation from expert knowledge, examine state-of-the-art POMDP solvers, and discuss our results from both the perspective of machine learning research and as an illustration for high-precision manufacturing practitioners.

Optimality Guarantees for Particle Belief Approximation of POMDPs

Partially observable Markov decision processes (POMDPs) provide a flexible representation for real-world decision and control problems. However, POMDPs are notoriously difficult to solve, especially when the state and observation spaces are continuous or hybrid, which is often the case for physical systems. While recent online sampling-based POMDP algorithms that plan with observation likelihood weighting have shown practical effectiveness, a general theory characterizing the approximation error of the particle filtering techniques that these algorithms use has not previously been proposed. Our main contribution is bounding the error between any POMDP and its corresponding finite sample particle belief MDP (PB-MDP) approximation. This fundamental bridge between PB-MDPs and POMDPs allows us to adapt any sampling-based MDP algorithm to a POMDP by solving the corresponding particle belief MDP, thereby extending the convergence guarantees of the MDP algorithm to the POMDP. Practically, this is implemented by using the particle filter belief transition model as the generative model for the MDP solver. While this requires access to the observation density model from the POMDP, it only increases the transition sampling complexity of the MDP solver by a factor of O(C), where C is the number of particles. Thus, when combined with sparse sampling MDP algorithms, this approach can yield algorithms for POMDPs that have no direct theoretical dependence on the size of the state and observation spaces. In addition to our theoretical contribution, we perform five numerical experiments on benchmark POMDPs to demonstrate that a simple MDP algorithm adapted using PB-MDP approximation, Sparse-PFT, achieves performance competitive with other leading continuous observation POMDP solvers.

Simulation Evidence of Trust Calibration: Using POMDP with Signal Detection Theory to Adapt Agent Features for Optimised Task Outcome During Human-Agent Collaboration

Appropriately calibrated human trust is essential for successful Human-Agent collaboration. Probabilistic frameworks using a partially observable Markov decision process (POMDP) have been previously employed to model the trust dynamics of human behavior, optimising the outcomes of a task completed with a collaborative recommender system. A POMDP model utilising signal detection theory to account for latent user trust is presented, with the model working to calibrate user trust via the implementation of three distinct agent features: disclaimer message, request for additional information, and no additional feature. A simulation experiment is run to investigate the efficacy of the proposed POMDP model compared against a random feature model and a control model. Evidence demonstrates that the proposed POMDP model can appropriately adapt agent features in-task based on human trust belief estimates in order to achieve trust calibration. Specifically, task accuracy is highest with the POMDP model, followed by the control and then the random model. This emphasises the importance of trust calibration, as agents that lack considered design to implement features in an appropriate way can be more detrimental to task outcome compared to an agent with no additional features.

Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning

We present a mathematical framework and computational methods for optimally designing a finite sequence of experiments. This sequential optimal experimental design (sOED) problem is formulated as a finite-horizon partially observable Markov decision process (POMDP) under a Bayesian setting and with information-theoretic utilities. The formulation is general and may accommodate continuous random variables, non-Gaussian posteriors, and nonlinear forward models. The sOED design policy incorporates elements of feedback and lookahead simultaneously, and we show it to generalize the commonly-used batch and greedy design strategies. We solve for the sOED policy using the policy gradient (PG) method from reinforcement learning, and provide a derivation for the PG expression in the sOED context. Adopting an actor–critic approach, the policy and value functions are parameterized using deep neural networks and improved via PG estimates produced from simulated episodes of designs and observations. The new PG-sOED algorithm is first validated on a linear-Gaussian benchmark, and then compared against other design baselines on a sensor movement problem for contaminant source inversion in a convection–diffusion field. We provide explanation for the policy behaviors using knowledge of the underlying physical process.

Sensor Management with Dynamic Clustering for Bearings-Only Multi-Target Tracking via Swarm Intelligence Optimization

Sensor management is a crucial research subject for multi-sensor multi-target tracking in wireless sensor networks (WSNs) with limited resources. Bearings-only tracking produces further challenges related to high nonlinearity and poor observability. Moreover, energy efficiency and energy balancing should be considered for sensor management in WSNs, which involves networking and transmission. This paper formulates the sensor management problem in the partially observable Markov decision process (POMDP) framework and uses the cardinality-balanced multi-target multi-Bernoulli (CBMeMBer) filter for tracking. A threshold control method is presented to reduce the impact on tracking accuracy when using bearings-only measurements for sequential update. Moreover, a Cauchy–Schwarz divergence center is defined to construct a new objective function for efficiently finding the optimal sensor subset via swarm intelligence optimization. This is also conducive to dynamic clustering for the energy efficiency and energy balancing of the network. The simulation results illustrate that the proposed solution can achieve good tracking performance with less energy, and especially that it can effectively balance network energy consumption and prolong network lifetime.

Adaptive Discretization using Voronoi Trees for Continuous POMDPs

Solving continuous Partially Observable Markov Decision Processes (POMDPs) is challenging, particularly for high-dimensional continuous action spaces. To alleviate this difficulty, we propose a new sampling-based online POMDP solver, called A daptive D iscretization using V oronoi T rees (ADVT). It uses Monte Carlo Tree Search in combination with an adaptive discretization of the action space as well as optimistic optimization to efficiently sample high-dimensional continuous action spaces and compute the best action to perform. Specifically, we adaptively discretize the action space for each sampled belief using a hierarchical partition called Voronoi tree, which is a Binary Space Partitioning that implicitly maintains the partition of a cell as the Voronoi diagram of two points sampled from the cell. ADVT uses the estimated diameters of the cells to form an upper-confidence bound on the action value function within the cell, guiding the Monte Carlo Tree Search expansion and further discretization of the action space. This enables ADVT to better exploit local information with respect to the action value function, allowing faster identification of the most promising regions in the action space, compared to existing solvers. Voronoi trees keep the cost of partitioning and estimating the diameter of each cell low, even in high-dimensional spaces where many sampled points are required to cover the space well. ADVT additionally handles continuous observation spaces, by adopting an observation progressive widening strategy, along with a weighted particle representation of beliefs. Experimental results indicate that ADVT scales substantially better to high-dimensional continuous action spaces, compared to state-of-the-art methods.

Towards complementary operations of offshore wind farm and photovoltaic array: A centralized reinforcement learning enabled control approach

Integrated offshore wind and photovoltaic (PV) power generation has high potential in significantly improving renewable power utilization, but the complementary operation of the integrated power system is fundamentally challenging, especially under high uncertainties of meteorological conditions. In this paper, the joint regulation of the offshore wind farm and the PV array is considered where the generator torque of each wind turbine and the tilt angles of the PV array are regarded as the control actions to simultaneously maximize the overall power generation and improve power quality. The model of the offshore wind farm including wake models and the turbine drive-train dynamic model is established while model of the PV array using the tilt angle as the regulation variable is also constructed. By considering the control of the integrated power system of the wind farm and the PV array as a typical partially-observable Markov decision process (MDP), a twin-delayed deep deterministic policy gradient (TD3) algorithm is developed to jointly regulate the wind farm and PV array. Design experiments have been conducted to evaluate the potential and effectiveness of the proposed TD3 control method for the integrated offshore wind and PV power system. The test results demonstrate that the TD3 method has good convergence performances in the overall power output and power variation rate regulations. The control objectives of improving the overall power output and smoothing the power variations can be simultaneously achieved by using appropriate parameter settings in the TD3 algorithm.

Structural Estimation of Partially Observable Markov Decision Processes

Partially Observable Markov Decision Processes (POMDPs) is a well-developed framework for sequential decision making under uncertainty and partial information. This paper considers the (inverse) structural estimation of the primitives of a POMDP based upon data in the form of sequences of observables and implemented actions. We analyze the structural properties of an entropy regularized POMDP and specify conditions under which the model is identifiable without knowledge of the state dynamics. We consider a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">soft</i> policy gradient algorithm to compute a maximum likelihood estimator, and illustrate the approach with an equipment replacement problem.

Age-Optimal Low-Power Status Update Over Time-Correlated Fading Channel

In this paper, we consider transmission scheduling in a status update system, where updates are generated periodically and transmitted over a Gilbert-Elliott fading channel. The goal is to minimize the long-run average age of information (AoI) under a long-run average energy constraint. We consider two practical cases to obtain channel state information (CSI): (i) without channel sensing and (ii) with delayed channel sensing. For (i), CSI is revealed by the feedback (ACK/NACK) of a transmission, but when no transmission occurs, CSI is not revealed. Thus, we have to balance tradeoffs across energy, AoI, channel exploration, and channel exploitation. The problem is formulated as a constrained partially observable Markov decision process (POMDP). We show that the optimal policy is a randomized mixture of no more than two stationary deterministic policies each of which is of a threshold-type in the belief on the channel. For (ii), (delayed) CSI is available via channel sensing. Then, the tradeoff is only between the AoI and energy. The problem is formulated as a constrained MDP. The optimal policy is shown to have a similar structure as in (i) but with an AoI associated threshold. With these, we develop an optimal structure-aware algorithm for each case.

On Deep Recurrent Reinforcement Learning for Active Visual Tracking of Space Noncooperative Objects

Active tracking of space noncooperative object that merely relies on vision camera is greatly significant for autonomous rendezvous and debris removal. Considering its Partial Observable Markov Decision Process (POMDP) property, this paper proposes a novel deep recurrent neural network architecture, named as RAMAVT, incorporating Multi-Head Attention (MHA) module and Squeeze-and-Excitation (SE) layer that remarkably improve the representative ability of neural network with almost no extra computational cost. It has been successfully applied to value-based and policy gradient-based deep reinforcement learning algorithm, and learned to drive the chasing spacecraft to follow arbitrary space noncooperative object with high-frequency and near-optimal velocity control commands. Extensive experiments and robustness evaluations implemented on SNCOAT benchmark show the betterment and robustness of our method compared with other state-of-the-art active visual trackers. In addition, we make further ablation study and interpretability research on RAMAVT of which validity and rationality have been demonstrated. The source codes are available on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Dongzhou-1996/RAMAVT</uri> .

Monte Carlo Planning in Hybrid Belief POMDPs

Real-world problems often require reasoning about hybrid beliefs, over both discrete and continuous random vari- ables. Yet, such a setting has hardly been investigated in the context of planning. Moreover, existing online Partially Ob- servable Markov Decision Processes (POMDPs) solvers do not support hybrid beliefs directly. In particular, these solvers do not address the added computational burden due to an increasing number of hypotheses with the planning horizon, which can grow exponentially. As part of this work, we present a novel algorithm, Hybrid Belief Monte Carlo Planning (HB-MCP) that utilizes the Monte Carlo Tree Search (MCTS) algorithm to solve a POMDP while maintaining a hybrid belief. We illustrate how the upper confidence bound (UCB) exploration bonus can be leveraged to guide the growth of hypotheses trees alongside the belief trees. We then evaluate our approach in highly aliased simulated environments where unresolved data association leads to multi-modal belief hypotheses.

Approximability and efficient algorithms for constrained fixed-horizon POMDPs with durative actions

Partially Observable Markov Decision Process (POMDP) is a fundamental model for probabilistic planning in stochastic domains. More recently, constrained POMDP and chance-constrained POMDP extend the model allowing constraints to be specified on some aspects of the policy in addition to the objective function. Despite their expressive power, these models assume all actions take a fixed duration, which poses a limitation in modeling real-world planning problems. In this work, we propose a unified model for durative POMDP and its constrained extensions. First, we convert these extensions into an Integer Linear Programming (ILP) formulation, which can be solved using existing solvers in the ILP literature. Second, a heuristic search approach is provided that can efficiently prune the search space, guided by solving successive partial ILP programs. Third, we give a theoretical analysis of the problem: unlike short-horizon POMDPs, with policies of a constant depth, which can be solved in polynomial time, the constrained extensions are NP-Hard even with a planning horizon of two and non-negative rewards. To alleviate that, we propose a Fully Polynomial Time Approximation Scheme (FPTAS) that computes (near) optimal deterministic policies in polynomial time. The FPTAS is among the best achievable in theory in terms of approximation ratio. Finally, evaluation results show that our approach is empirically superior to the state-of-the-art fixed-horizon chance-constrained POMDP solver.

Model-based motion planning in POMDPs with temporal logic specifications

Partially observable Markov decision processes (POMDPs) have been used as mathematical models for sequential decision-making under uncertain and incomplete information. Since the state space is partially observable in a POMDP, the agent has to make a decision based on the integrated information over the past experiences of actions and observations. This study aims to solve probabilistic motion planning problems in which the agent is assigned a complex task under a partially observable environment. We employ linear temporal logic (LTL) to formulate the complex task and then convert it to a limit-deterministic generalized Büchi automaton (LDGBA). We reformulate the problem as finding an optimal policy on the product of POMDP and LDGBA based on model-checking techniques. This paper adopts and modifies two reinforcement learning (RL) approaches: value iteration and deep Q-learning. Both are model-based because the optimal policy is a function of belief states that need transition and observation probabilities to be updated. We illustrate the applicability of the proposed methods by addressing two simulations, including a grid-world problem with various sizes and a TurtleBot office path planning problem.