Belief Space Research Articles

POMHDP: Search-Based Belief Space Planning Using Multiple Heuristics

Robots operating in the real world encounter substantial uncertainty that cannot be modeled deterministically before the actual execution. This gives rise to the necessity of robust motion planning under uncertainty also known as belief space planning. Belief space planning can be formulated as Partially Observable Markov Decision Processes (POMDPs). However, computing optimal policies for non-trivial POMDPs is computationally intractable. Building upon recent progress from the search community, we propose a novel anytime POMDP solver, Partially Observable Multi-Heuristic Dynamic Programming (POMHDP), that leverages multiple heuristics to efficiently compute high-quality solutions while guaranteeing asymptotic convergence to an optimal policy. Through iterative forward search, POMHDP utilizes domain knowledge to solve POMDPs with specific goals and an infinite horizon. We demonstrate the efficacy of our proposed framework on a real-world, highly-complex, truck unloading application.

Successor Feature Sets: Generalizing Successor Representations Across Policies

Successor-style representations have many advantages for reinforcement learning: for example, they can help an agent generalize from past experience to new goals, and they have been proposed as explanations of behavioral and neural data from human and animal learners. They also form a natural bridge between model-based and model-free RL methods: like the former they make predictions about future experiences, and like the latter they allow efficient prediction of total discounted rewards. However, successor-style representations are not optimized to generalize across policies: typically, we maintain a limited-length list of policies, and share information among them by representation learning or GPI. Successor-style representations also typically make no provision for gathering information or reasoning about latent variables. To address these limitations, we bring together ideas from predictive state representations, belief space value iteration, successor features, and convex analysis: we develop a new, general successor-style representation, together with a Bellman equation that connects multiple sources of information within this representation, including different latent states, policies, and reward functions. The new representation is highly expressive: for example, it lets us efficiently read off an optimal policy for a new reward function, or a policy that imitates a new demonstration. For this paper, we focus on exact computation of the new representation in small, known environments, since even this restricted setting offers plenty of interesting questions. Our implementation does not scale to large, unknown environments --- nor would we expect it to, since it generalizes POMDP value iteration, which is difficult to scale. However, we believe that future work will allow us to extend our ideas to approximate reasoning in large, unknown environments. We conduct experiments to explore which of the potential barriers to scaling are most pressing.

PLGRIM: Hierarchical Value Learning for Large-scale Exploration in Unknown Environments

In order for an autonomous robot to efficiently explore an unknown environment, it must account for uncertainty in sensor measurements, hazard assessment, localization, and motion execution. Making decisions for maximal reward in a stochastic setting requires value learning and policy construction over a belief space, i.e., probability distribution over all possible robot-world states. However, belief space planning in a large spatial environment over long temporal horizons suffers from severe computational challenges. Moreover, constructed policies must safely adapt to unexpected changes in the belief at runtime. This work proposes a scalable value learning framework, PLGRIM (Probabilistic Local and Global Reasoning on Information roadMaps), that bridges the gap between (i) local, risk-aware resiliency and (ii) global, reward-seeking mission objectives. Leveraging hierarchical belief space planners with information-rich graph structures, PLGRIM addresses large-scale exploration problems while providing locally near-optimal coverage plans. We validate our proposed framework with high-fidelity dynamic simulations in diverse environments and on physical robots in Martian-analog lava tubes.

Stochastic Dynamic Games in Belief Space

Information gathering while interacting with other agents under sensing and motion uncertainty is critical in domains such as driving, service robots, racing, or surveillance. The interests of agents may be at odds with others, resulting in a stochastic noncooperative dynamic game. Agents must predict others’ future actions without communication, incorporate their actions into these predictions, account for uncertainty and noise in information gathering, and consider what information their actions reveal. Our solution uses local iterative dynamic programming in Gaussian belief space to solve a game-theoretic continuous POMDP. Solving a quadratic game in the backward pass of a game-theoretic belief-space variant of iterative linear-quadratic Gaussian control (iLQG) achieves a runtime polynomial in the number of agents and linear in the planning horizon. Our algorithm yields linear feedback policies for our robot, and predicted feedback policies for other agents. We present three applications: Active surveillance, guiding eyes for a blind agent, and autonomous racing. Agents with game-theoretic belief-space planning win 44% more races than without game theory and 34% more than without belief-space planning.

Toward competitive multi-agents in Polo game based on reinforcement learning

The learning of agents in a competitive space such as a game is a challenging task. The aim of the proposed research is to improve the reinforcement learning techniques in a competitive multi-agent for the Polo game. First, the video dataset is prepared. Then, the rules of the Polo game are extracted as a class diagram. An architecture is designed for multi-agent team in the Polo game. Therefore, an algorithm is proposed for the temporal difference in the game belief space for improving reward catching. The reward function is implemented in the agent team. Finally, the research improvement is evaluated by increasing 31 units in comparison with previous work. Therefore, competitive learning in the agent team has been improved.

MPTP: Motion-planning-aware task planning for navigation in belief space

We present an integrated Task-Motion Planning (TMP) framework for navigation in large-scale environments. Of late, TMP for manipulation has attracted significant interest resulting in a proliferation of different approaches. In contrast, TMP for navigation has received considerably less attention. Autonomous robots operating in real-world complex scenarios require planning in the discrete (task) space and the continuous (motion) space. In knowledge-intensive domains, on the one hand, a robot has to reason at the highest-level, for example, the objects to procure, the regions to navigate to in order to acquire them; on the other hand, the feasibility of the respective navigation tasks have to be checked at the execution level. This presents a need for motion-planning-aware task planners. In this paper, we discuss a probabilistically complete approach that leverages this task-motion interaction for navigating in large knowledge-intensive domains, returning a plan that is optimal at the task-level. The framework is intended for motion planning under motion and sensing uncertainty, which is formally known as belief space planning. The underlying methodology is validated in simulation, in an office environment and its scalability is tested in the larger Willow Garage world. A reasonable comparison with a work that is closest to our approach is also provided. We also demonstrate the adaptability of our approach by considering a building floor navigation domain. Finally, we also discuss the limitations of our approach and put forward suggestions for improvements and future work.

Uncertainty-Constrained Differential Dynamic Programming in Belief Space for Vision Based Robots

Most mobile robots follow a modular sense-plan-act system architecture that can lead to poor performance or even catastrophic failure for visual inertial navigation systems due to trajectories devoid of feature matches. Planning in belief space provides a unified approach to tightly couple the perception, planning and control modules, leading to trajectories that are robust to noisy measurements and disturbances. However, existing methods handle uncertainties as costs that require manual tuning for varying environments and hardware. We therefore propose a novel trajectory optimization formulation that incorporates inequality constraints on uncertainty and a novel Augmented Lagrangian based stochastic differential dynamic programming method in belief space. Furthermore, we develop a probabilistic visibility model that accounts for discontinuities due to feature visibility limits. Our simulation tests demonstrate that our method can handle inequality constraints in different environments, for holonomic and nonholonomic motion models with no manual tuning of uncertainty costs involved. We also show the improved optimization performance in belief space due to our visibility model.

Belief Space Planning for Mobile Robots With Range Sensors Using iLQG

In this work, we use iterative Linear Quadratic Gaussian (iLQG) to plan motions for a mobile robot with range sensors in belief space. We address two limitations that prevent applications of iLQG to the considered robotic system. First, iLQG assumes a differentiable measurement model, which is not true for range sensors. We show that iLQG only requires the differentiability of the belief dynamics. We propose to use a derivative-free filter to approximate the belief dynamics, which does not require explicit differentiability of the measurement model. Second, informative measurements from a range sensor are sparse. Uninformative measurements produce trivial gradient information, which prevent iLQG optimization from converging to a local minimum. We densify the informative measurements by introducing additional parameters in the measurement model. The parameters are iteratively updated in the optimization to ensure convergence to the true measurement model of a range sensor. We show the effectiveness of the proposed modifications through an ablation study. We also apply the proposed method in simulations of large scale real world environments, which show superior performance comparing to the state-of-the-art methods that either assume the separation principle or maximum likelihood measurements.

FT-BSP: Focused Topological Belief Space Planning

At its core, decision making under uncertainty can be regarded as sorting candidate actions according to a certain objective. While finding the optimal solution directly is computationally expensive, other approaches that produce the same ordering of candidate actions, will result in the same selection. With this motivation in mind, we present a computationally efficient approach for the focused belief space planning (BSP) problem, where reducing the uncertainty of only a predefined subset of variables is of interest. Our approach uses topological signatures, defined over the topology induced from factor graph representations of posterior beliefs, to rank candidate actions. In particular, we present two such signatures in the context of information theoretic focused decision making problems. We derive error bounds, with respect to the optimal solution, and prove that one of these signatures converges to the true optimal solution. We also derive a second set of bounds, which is more conservative, but is only a function of topological aspects and can be used online. We introduce the Von Neumann graph entropy for the focused case, which is based on weighted node degrees, and show that it supports incremental update. We then analyze our approach under two different settings, measurement selection and active focused 2D pose SLAM.

Towards Energy-Aware Feedback Planning for Long-Range Autonomous Underwater Vehicles

Ocean ecosystems have spatiotemporal variability and dynamic complexity that require a long-term deployment of an autonomous underwater vehicle for data collection. A new generation of long-range autonomous underwater vehicles (LRAUVs), such as the Slocum glider and Tethys-class AUV, has emerged with high endurance, long-range, and energy-aware capabilities. These new vehicles provide an effective solution to study different oceanic phenomena across multiple spatial and temporal scales. For these vehicles, the ocean environment has forces and moments from changing water currents which are generally on the order of magnitude of the operational vehicle velocity. Therefore, it is not practical to generate a simple trajectory from an initial location to a goal location in an uncertain ocean, as the vehicle can deviate significantly from the prescribed trajectory due to disturbances resulted from water currents. Since state estimation remains challenging in underwater conditions, feedback planning must incorporate state uncertainty that can be framed into a stochastic energy-aware path planning problem. This article presents an energy-aware feedback planning method for an LRAUV utilizing its kinematic model in an underwater environment under motion and sensor uncertainties. Our method uses ocean dynamics from a predictive ocean model to understand the water flow pattern and introduces a goal-constrained belief space to make the feedback plan synthesis computationally tractable. Energy-aware feedback plans for different water current layers are synthesized through sampling and ocean dynamics. The synthesized feedback plans provide strategies for the vehicle that drive it from an environment’s initial location toward the goal location. We validate our method through extensive simulations involving the Tethys vehicle’s kinematic model and incorporating actual ocean model prediction data.

An Integrated Localization, Motion Planning and Obstacle Avoidance Algorithm in Belief Space

As robots are being increasingly used in close proximity to humans and objects, it is imperative that robots operate safely and efficiently under real-world conditions. Yet, the environment is seldom known perfectly. Noisy sensors and actuation errors compound to the errors introduced while estimating features of the environment. We present a novel approach (1) to incorporate these uncertainties for robot state estimation and (2) to compute the probability of collision pertaining to the estimated robot configurations. The expression for collision probability is obtained as an infinite series and we prove its convergence. An upper bound for the truncation error is also derived and the number of terms required is demonstrated by analyzing the convergence for different robot and obstacle configurations. We evaluate our approach using two simulation domains which use a roadmap-based strategy to synthesize trajectories that satisfy collision probability bounds.

A comprehensive survey on cultural algorithms

Cultural Algorithms (CAs) are evolutionary algorithms (EAs) inspired by the conceptual models of the human cultural evolution process. In contrast to the conventional EAs, which work only based on the population space, CAs employ an additional space, called belief space, to collect the information about the behaviour of individuals in the search space. Since the emergence of CAs, they have been successfully extended to solve a wide variety of problems in different branches of science and technology. In this paper, a comprehensive survey on the recent advances in CAs is presented. Literature survey reveals some interesting challenges and future research directions.

Using data complexity measures and an evolutionary cultural algorithm for gene selection in microarray data

Cancer detection using gene expression data has been a major trend of research for the last decade. Microarray gene expression data is one of the most challenging types of data due to high dimensionality and rarity of available samples. Feature redundancy greatly contributes to the difficulty of prediction task. Therefore, it is essential to apply feature selection to datasets to reduce the number of features selected for the classification task. In this paper, a novel two-staged framework is proposed to confront curse of dimensionality in microarray data using data complexity measures and a customized cultural algorithm, incorporating a static belief space into the genetic algorithm in order to reduce the search space and prioritize important genes. Experimental results indicate highly improved accuracy and reduction in number of selected genes compared to the state-of-the-art methods on Gli85, Colon, DLBCL, SMK and CNS datasets.

Control Theory Meets POMDPs: A Hybrid Systems Approach

Partially observable Markov decision processes (POMDPs) provide a modeling framework for a variety of sequential decision making under uncertainty scenarios in artificial intelligence (AI). Since the states are not directly observable in a POMDP, decision making has to be performed based on the output of a Bayesian filter (continuous beliefs); hence, making POMDPs intractable to solve and analyze. To overcome the complexity challenge of POMDPs, we apply techniques from the control theory. Our contributions are fourfold. 1) We begin by casting the problem of analyzing a POMDP into analyzing the behavior of a discrete-time switched system. 2) Then, in order to estimate the reachable belief space of a POMDP, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.,</i> the set of all possible evolutions given an initial belief distribution over the states and a set of actions and observations, we find overapproximations in terms of sublevel sets of Lyapunov-like functions. 3) Furthermore, in order to verify safety and performance requirements of a given POMDP, we formulate a barrier certificate theorem, wherein we show that if there exists a barrier certificate satisfying a set of inequalities along the solutions to the belief update equation of the POMDP, the safety and performance properties are guaranteed to hold. In both cases 2) and 3), the calculations can be decomposed and solved in parallel. 4) Finally, we show that the conditions we formulate can be computationally implemented as a set of sum-of-squares programs. We illustrate the applicability of our method by addressing two problems in active ad scheduling and machine teaching.

A hybrid elephant herding optimization and cultural algorithm for energy‐balanced cluster head selection scheme to extend the lifetime in WSNs

SummaryClustering‐based optimal cluster head selection in wireless sensor networks (WSNs) is considered as the efficient technique essential for improving the network lifetime. But enforcing optimal cluster head selection based on energy stabilization, reduced delay, and minimized distance between sensor nodes always remain a crucial challenge for prolonging the network lifetime in WSNs. In this paper, a hybrid elephant herding optimization and cultural algorithm for optimal cluster head selection (HEHO‐CA‐OCHS) scheme is proposed to extend the lifetime. This proposed HEHO‐CA‐OCHS scheme utilizes the merits of belief space framed by the cultural algorithm for defining a separating operator that is potent in constructing new local optimal solutions in the search space. Further, the inclusion of belief space aids in maintaining the balance between an optimal exploitation and exploration process with enhanced search capabilities under optimal cluster head selection. This proposed HEHO‐CA‐OCHS scheme improves the characteristic properties of the algorithm by incorporating separating and clan updating operators for effective selection of cluster head with the view to increase the lifetime of the network. The simulation results of the proposed HEHO‐CA‐OCHS scheme were estimated to be superior in percentage of alive nodes by 11.21%, percentage of dead nodes by 13.84%, residual energy by 16.38%, throughput by 13.94%, and network lifetime by 19.42% compared to the benchmarked cluster head selection schemes.

Quantum-like influence diagrams for decision-making

This article proposes a novel and comprehensive framework on how to describe the probabilistic nature of decision-making process. We suggest extending the quantum-like Bayesian network formalism to incorporate the notion of maximum expected utility to model human paradoxical, sub-optimal and irrational decisions. What distinguishes this work is that we take advantage of the quantum interference effects produced in quantum-like Bayesian Networks during the inference process to influence the probabilities used to compute the maximum expected utility of some decision.The proposed quantum-like decision model is able to (1) predict the probability distributions found in different experiments reported in the literature by modelling uncertainty through quantum interference, (2) to identify decisions that the decision-makers perceive to be optimal within their belief space, but that are actually irrational with respect to expected utility theory, (3) gain an understanding of how the decision-maker’s beliefs evolve within a decision-making scenario. The proposed model has the potential to provide new insights in decision science, as well as having direct implications for decision support systems that deal with human data, such as in the fields of economics, finance, psychology, etc.

Simultaneous Control in Belief Space for Circular Insertion in Precision Assembly

Simultaneously inserting multiple objects is an essential topic in precision assembly to compose complicated shapes. This task involves the acquisition difficulty of unobservable interaction states, which makes it hard to plan the insertion. To solve it, this article investigates the circular assembly of multiple objects and proposes a strategy to control the simultaneous insertion in belief space. We first present the insertion state transition and observation models, in which the stochastic parts are modeled as Gaussian noise, and then estimate the belief state using an extended Kalman filter. An optimization approach is discussed, for the compensational movement planning, to decrease the estimated radial interaction forces and the toward-center movement is thus determined, considering the optimized compensational movement and the belief state. Experiments are carried out to demonstrate the validation of the proposed method.

Trustable Environmental Monitoring by Means of Sensors Networks on Swarming Autonomous Marine Vessels and Distributed Ledger Technology.

The article describes a highly trustable environmental monitoring system employing a small scalable swarm of small-sized marine vessels equipped with compact sensors and intended for the monitoring of water resources and infrastructures. The technological foundation of the process which guarantees that any third party can not alter the samples taken by the robot swarm is based on the Robonomics platform. This platform provides encrypted decentralized technologies based on distributed ledger tools, and market mechanisms for organizing the work of heterogeneous multi-vendor cyber-physical systems when automated economical transactions are needed. A small swarm of robots follows the autonomous ship, which is in charge of maintaining the secure transactions. The swarm implements a version of Reynolds' Boids model based on the Belief Space Planning approach. The main contributions of our work consist of: (1) the deployment of a secure sample certification and logging platform based on the blockchain with a small-sized swarm of autonomous vessels performing maneuvers to measure chemical parameters of water in automatic mode; (2) the coordination of a leader-follower framework for the small platoon of robots by means of a Reynolds' Boids model based on a Belief Space Planning approach. In addition, the article describes the process of measuring the chemical parameters of water by using sensors located on the vessels. Both technology testing on experimental vessel and environmental measurements are detailed. The results have been obtained through real world experiments of an autonomous vessel, which was integrated as the “leader” into a mixed reality simulation of a swarm of simulated smaller vessels.The design of the experimental vessel physically deployed in the Volga river to demonstrate the practical viability of the proposed methods is shortly described.

Reinforcement Learning for POMDP: Partitioned Rollout and Policy Iteration With Application to Autonomous Sequential Repair Problems

In this letter we consider infinite horizon discounted dynamic programming problems with finite state and control spaces, and partial state observations. We discuss an algorithm that uses multistep lookahead, truncated rollout with a known base policy, and a terminal cost function approximation. This algorithm is also used for policy improvement in an approximate policy iteration scheme, where successive policies are approximated by using a neural network classifier. A novel feature of our approach is that it is well suited for distributed computation through an extended belief space formulation and the use of a partitioned architecture, which is trained with multiple neural networks. We apply our methods in simulation to a class of sequential repair problems where a robot inspects and repairs a pipeline with potentially several rupture sites under partial information about the state of the pipeline.

Asymptotic Optimality of Finite Model Approximations for Partially Observed Markov Decision Processes With Discounted Cost

We consider finite model approximations of discrete-time partially observed Markov decision processes (POMDPs) under the discounted cost criterion. After converting the original partially observed stochastic control problem to a fully observed one on the belief space, the finite models are obtained through the uniform quantization of the state and action spaces of the belief space Markov decision process (MDP). Under mild assumptions on the components of the original model, it is established that the policies obtained from these finite models are nearly optimal for the belief space MDP, and so, for the original partially observed problem. The assumptions essentially require that the belief space MDP satisfies a mild weak continuity condition. We provide an example and introduce explicit approximation procedures for the quantization of the set of probability measures on the state space of POMDP (i.e., belief space).