Belief State Space Research Articles

A New Post-Processing Method using Latent Structure Influence Models for Channel Fusion in Automatic Sleep Staging.

Human sleep stage dynamics can be adequately represented using Markov chain models, and the accuracy of sleep stage classification can be improved by considering these dynamics. The present study proposes a new post-processing method based on channel fusion using Latent Structure Influence Models (LSIMs). LSIMs can simultaneously model sequences of different channels to have a nonlinear dynamical fusion based on sleep stage dynamics. The proposed method develops and examines two channel-fusion algorithms: the standard LSIM fusion and the integrated LSIM fusion, in which the latter is more efficient and performs better. The proposed LSIM-based method simultaneously incorporates the nonlinear interactions between channels and the sleep stage dynamics. In the first step, existing sleep staging systems process every data channel independently and produce stage score sequences for each channel. These single-channel scores are then projected into belief space using the marginal one-slice parameter of all channels by LSIM fusion algorithms. The logarithms of marginal one-slice parameters are concatenated to obtain log-scale belief state space (LBSS) features in the standard LSIM fusion. In the integrated LSIM fusion, integrated LBSS (ILBSS) features are formed by combining the LBSS features of several LSIMs. Finally, a KNN classifier maps the LBSS and ILBSS features onto the sleep stages. By utilizing four recently developed sleep staging systems, the proposed method is applied to the publicly available SleepEDF-20 database that contains five AASM sleep stages (N1, N2, N3, REM, and W). Compared to single-channel (Fpz-Cz, Pz-Oz, and EOG) results, integrated LSIM fusion results have a statistically significant improvement of 1.5% in 2-channel fusion (Fpz-Cz and Pz-Oz) and 2.5% in 3-channel fusion (Fpz-Cz, Pz-Oz, and EOG). With an overall accuracy of 87.3% for 3-channel post-processing, the integrated LSIM fusion system offers one of the highest overall accuracy rates among existing studies.

Radar Anti-Jamming Countermeasures Intelligent Decision-Making: A Partially Observable Markov Decision Process Approach

Current electronic warfare jammers and radar countermeasures are characterized by dynamism and uncertainty. This paper focuses on a decision-making framework of radar anti-jamming countermeasures. The characteristics and implementation process of radar intelligent anti-jamming systems are analyzed, and a scheduling method for radar anti-jamming action based on the Partially Observable Markov Process (POMDP) is proposed. The sample-based belief distribution is used to reflect the radar’s cognition of the environment and describes the uncertainty of the recognition of jamming patterns in the belief state space. The belief state of jamming patterns is updated with Bayesian rules. The reward function is used as the evaluation criterion to select the best anti-jamming strategy, so that the radar is in a low threat state as often as possible. Numerical simulation combines the behavioral prior knowledge base of radars and jammers and obtains the behavioral confrontation benefit matrix from the past experience of experts. The radar controls the output according to the POMDP policy, and dynamically performs the best anti-jamming action according to the change of jamming state. The results show that the POMDP anti-jamming policy is better than the conventional policy. The POMDP approach improves the adaptive anti-jamming capability of the radar and can quickly realize the anti-jamming decision to jammers. This work provides some design ideas for the subsequent development of an intelligent radar.

Synthesis of Non-blocking Controllers for Linear Temporal Logic Tasks under Partial Observations⋆

In this paper, we investigate the formal synthesis of discrete controllers for linear temporal logic tasks under partial information. Existing works on this topic mainly focus on find sure winning or almost sure winning control strategies under some assumptions regarding the system's atomic propositions. In this work, we consider non-blockingness as the metric as the achievement of the task. Specifically, we require that at each instant, the controller maintains the possibility to achieve the LTL task under some environment's behavior. We first present an offline algorithm for the computation of the winning region over the belief state space. Then we present an online control algorithm that effective solves the control synthesis problem. The proposed control algorithm is also illustrated by a case study of robot task planning.

A reactive power optimization partially observable Markov decision process with data uncertainty using multi-agent actor-attention-critic algorithm

We present an innovative partially observable Markov decision process (POMDP) modelling method for the reactive power optimization process of the active distribution network (ADN) under the high permeability of the distributed generation. This model is tolerant of the uncertainty resulting from data uncertainty. We believe that the belief state space in the POMDP model corresponds to the state space in the Markov decision process (MDP) model, and we apply the multi-agent actor-attention-critic (MAAC) reinforcement learning (RL) algorithm to the proposed model. This technique extracts the most effective information with the highest quality from the huge historical measurement database, hence enhancing the learning effectiveness of agents and the stability of the optimization strategy. We simulate reactive power optimization in a modified IEEE-33 nodes ADN and a modified IEEE-123nodes ADN. The simulation demonstrates the stability and economic superiority of the proposed approach under varying degrees of data uncertainty relative to previous RL algorithms based on the MDP model. The simulation demonstrates that the proposed POMDP model is more appropriate for the real operation of the partially observable distribution network than the MDP models. And the optimal strategy obtained by the proposed MAAC algorithm is reliable with deteriorating data quality.

Deep Ensemble Reinforcement Learning with Multiple Deep Deterministic Policy Gradient Algorithm

Deep deterministic policy gradient algorithm operating over continuous space of actions has attracted great attention for reinforcement learning. However, the exploration strategy through dynamic programming within the Bayesian belief state space is rather inefficient even for simple systems. Another problem is the sequential and iterative training data with autonomous vehicles subject to the law of causality, which is against the i.i.d. (independent identically distributed) data assumption of the training samples. This usually results in failure of the standard bootstrap when learning an optimal policy. In this paper, we propose a framework of m-out-of-n bootstrapped and aggregated multiple deep deterministic policy gradient to accelerate the training process and increase the performance. Experiment results on the 2D robot arm game show that the reward gained by the aggregated policy is 10%–50% better than those gained by subpolicies. Experiment results on the open racing car simulator (TORCS) demonstrate that the new algorithm can learn successful control policies with less training time by 56.7%. Analysis on convergence is also given from the perspective of probability and statistics. These results verify that the proposed method outperforms the existing algorithms in both efficiency and performance.

MAC protocol for underwater acoustic sensor network based on belied state space

This study aims to solve time-space uncertainties due to the narrow network channel bandwidth and long transmission delay of an underwater acoustic sensor network when a node is using a channel. This study proposes a MAC protocol (BSPMDP-MAC) for an underwater acoustic sensor network based on the belief state space. This protocol can averagely divide the time axis of a sensor’s receiving nodes into n slots. The action state information of a sensor’s transmission node was divided by the grades of link quality and the residual energy of each node. The receiving nodes would obtain the decision strategy sequence of the usage rights of the competitive channels of the sensor’s transmission nodes according to the joint probability distributions of historical observations and action information of channel occupancy. The transmission nodes will transmit data packets to the receiving nodes in turns in allocated slots, according to the decision strategy sequence, and the receiving nodes will predict the channel occupancy and perceive the belief states and access actions in the next cycle, according to the present belief states and actions. These experimental simulation results show that this protocol can reduce the collision rate of data packets, improve the network throughput and transmission success rate of data packets, and reduce the energy overhead of the network.

Non-deterministic planning methods for automated web service composition

Web service composition (WSC) is the task of generating new composite web services that exhibit functionalities not supported by any single web service. In its simplest form this is achieved by linking existing web services in sequence. More complex forms link services in parallel or use alternative paths. WSC can be considered a planning task, with the web services being the planning operators and the initial state and the goals being provided by the user. Particularly, since web services operate in a stochastic environment, their output is not predictable, and the problem is formulated as a non-deterministic planning one. This article presents a critical, comprehensive and up-to-date review of the literature concerning alternative non-deterministic planning methods, including probabilistic planning, determinization methods, planning in the belief state space and translation based methods. Furthermore, the article reviews existing implementations of WSC systems, employing a variety of planning approaches, and discusses the degree in which the current achievements from the non-deterministic planning field have been adopted successfully. To the best of our knowledge, this is the first review of its kind, one that provides a thorough introduction to the vast area of automated web service composition.

Structured Possibilistic Planning Using Decision Diagrams

Qualitative Possibilistic Mixed-Observable MDPs (pi-MOMDPs), generalizing pi-MDPs and pi-POMDPs, are well-suited models to planning under uncertainty with mixed-observability when transition, observation and reward functions are not precisely known and can be qualitatively described. Functions defining the model as well as intermediate calculations are valued in a finite possibilistic scale L, which induces a finite belief state space under partial observability contrary to its probabilistic counterpart. In this paper, we propose the first study of factored pi-MOMDP models in order to solve large structured planning problems under qualitative uncertainty, or considered as qualitative approximations of probabilistic problems. Building upon the SPUDD algorithm for solving factored (probabilistic) MDPs, we conceived a symbolic algorithm named PPUDD for solving factored pi-MOMDPs. Whereas SPUDD's decision diagrams' leaves may be as large as the state space since their values are real numbers aggregated through additions and multiplications, PPUDD's ones always remain in the finite scale L via min and max operations only. Our experiments show that PPUDD's computation time is much lower than SPUDD, Symbolic-HSVI and APPL for possibilistic and probabilistic versions of the same benchmarks under either total or mixed observability, while still providing high-quality policies.

SHP-VI Method of Solving DEC-POMDP Problem

DEC-POMDP(Distributed Partially Observable Markov Decision Process) model is a multi-agent model of collaborative decision-making is important, but due to an alarming number of DEC-POMDP problem state space and great strategy solution space, so DEC-POMDP solution of the problem becomes very difficult. The agent from the initial state to the target state during the interaction with the environment, the system's maximum benefit is often only with some small amount of a higher reward states. This article by searching from the initial belief state to the target state to get a shortest Hamiltonian path, according to the corresponding sequence of actions on the path forward search to get faith belief state space trajectory, and then along the trajectory reverse convictions value function iteration, thus forming the state with the largest gains beliefs trajectory corresponding optimal strategy. In this paper, shortest Hamiltonian path-based value iteration to search the optimal path of faith so as to solve the state Hamiltonian larger DEC-POMDP problem.

Contingent Planning as AND/OR Forward Search with Disjunctive Representation

This paper introduces a highly competitive contingent planner, that uses the novel idea of encoding belief states as disjunctive normal form formulae (To et al. 2009), for the search for solutions in the belief state space. In (To et al. 2009), a complete transition function for computing successor belief states in the presence of incomplete information has been defined. This work extends the function to handle non-deterministic and sensing actions in the AND/OR forward search paradigm for contingent planning solutions. The function allows one, under reasonable assumptions, to compute successor belief states efficiently, i.e., in polynomial time. The paper also presents a novel variant of an AND/OR search algorithm, called PrAO (Pruning AND/OR search), which allows the planner to significantly prune the search space; furthermore, by the time a solution is found, the remaining search graph is also the solution tree for the contingent planing problem. The strength of these techniques is confirmed by the empirical results obtained from a large set of benchmarks available in the literature.

A Modified Memory-Based Reinforcement Learning Method for Solving POMDP Problems

Partially observable Markov decision processes (POMDP) provide a mathematical framework for agent planning under stochastic and partially observable environments. The classic Bayesian optimal solution can be obtained by transforming the problem into Markov decision process (MDP) using belief states. However, because the belief state space is continuous and multi-dimensional, the problem is highly intractable. Many practical heuristic based methods are proposed, but most of them require a complete POMDP model of the environment, which is not always practical. This article introduces a modified memory-based reinforcement learning algorithm called modified U-Tree that is capable of learning from raw sensor experiences with minimum prior knowledge. This article describes an enhancement of the original U-Tree's state generation process to make the generated model more compact, and also proposes a modification of the statistical test for reward estimation, which allows the algorithm to be benchmarked against some traditional model-based algorithms with a set of well known POMDP problems.

State agnostic planning graphs: deterministic, non-deterministic, and probabilistic planning

Planning graphs have been shown to be a rich source of heuristic information for many kinds of planners. In many cases, planners must compute a planning graph for each element of a set of states, and the naive technique enumerates the graphs individually. This is equivalent to solving a multiple-source shortest path problem by iterating a single-source algorithm over each source. We introduce a data-structure, the state agnostic planning graph, that directly solves the multiple-source problem for the relaxation introduced by planning graphs. The technique can also be characterized as exploiting the overlap present in sets of planning graphs. For the purpose of exposition, we first present the technique in deterministic (classical) planning to capture a set of planning graphs used in forward chaining search. A more prominent application of this technique is in conformant and conditional planning (i.e., search in belief state space), where each search node utilizes a set of planning graphs; an optimization to exploit state overlap between belief states collapses the set of sets of planning graphs to a single set. We describe another extension in conformant probabilistic planning that reuses planning graph samples of probabilistic action outcomes across search nodes to otherwise curb the inherent prediction cost associated with handling probabilistic actions. Finally, we show how to extract a state agnostic relaxed plan that implicitly solves the relaxed planning problem in each of the planning graphs represented by the state agnostic planning graph and reduces each heuristic evaluation to counting the relevant actions in the state agnostic relaxed plan. Our experimental evaluation (using many existing International Planning Competition problems from classical and non-deterministic conformant tracks) quantifies each of these performance boosts, and demonstrates that heuristic belief state space progression planning using our technique is competitive with the state of the art.

DYNAMIC PLANNING PROBLEM GENERATION IN A UAV DOMAIN

One of the most successful methods for planning in large partially observable stochastic domains is depth-limited forward search from the current belief state together with a utility estimation. However, when the environment is continuous and the number of possible actions is practically infinite, then abstractions have to be made before any forward search planning can be performed. The paper presents a method to dynamically generate such planning problem abstractions for a domain that is inspired by our research with unmanned aerial vehicles (UAVs). The planning problems are created by first stating the selection of points to fly to as an optimization problem. When the points have been selected, a set of possible paths between them are then created with a pathplanner and then forward search in the belief state space is applied. The method has been implemented and tested in simulation and the experiments show the importance of modelling both the dynamics of the environment and the limited computational resources of the architecture when searching for suitable parameters in the planning problem formulation procedure.