Belief Space Research Articles

CCEO: cultural cognitive evolution optimization algorithm

Cognitive behavior is an indispensable factor in the history of human evolution; through cognitive behavior, human development has evolved from the Stone Age to the present high-tech information era. By simulating the process of cultural evolution, a cultural algorithm can reflect the evolutionary process of society accurately, which can speed up the process of solving problems. In this paper, cognitive behavior evolution and a cultural algorithm are combined to form the cultural cognitive evolution optimization (CCEO) algorithm. A cultural algorithm with the double-layer structure of the belief space and population space is transformed into a three-layer evolution mechanism in a CCEO so that the belief space improves on the next layer of careful guidance of the evolutionary process. Twenty-three benchmark test functions and two engineering problems were used to test the CCEO algorithm, and the results demonstrate that the culture cognitive evolution algorithm has a high convergence speed and calculation precision.

Inattention and belief polarization

Disagreement persists over issues that have objective truths. In the presence of increasing amounts of data, such disagreement should vanish, but it is nonetheless observable. This paper studies persistent disagreement in a model where rational Bayesian agents learn about an unobservable state of the world through noisy signals. We show that agents (i) choose signal structures that are more likely to reinforce their prior beliefs and (ii) choose less informative signals when their prior beliefs are more precise. For sufficiently precise beliefs, agents choose completely uninformative signals. We call the former the confirmation effect and the latter the complacency effect. Taken together, the two effects imply that the beliefs of ex ante identical agents over time can cluster in two distinct groups at opposite ends of the belief space. The complacency effect holds uniformly when information cost is proportional to channel capacity, but not when cost is proportional to reduction in entropy.

Speeding up Gaussian Belief Space Planning for Underwater Robots Through a Covariance Upper Bound

Existing belief space motion planning methods are not efficient for underwater robots that are subject to spatially varying motion and sensing uncertainties arising from the non-uniform current disturbances and landmark populations, respectively. Based on a closed-loop stochastic control framework, we propose a fast Gaussian belief space planning approach for coupled optimization of trajectory, localization and control, resulting in a non-linear programming problem (NLP). In particular, as opposed to advancing the covariance by a Kalman filter in the existing literature, we utilize an upper bound of the trace propagation of the covariance, thereby avoiding to solve Riccati equations and thus, reducing the computational complexity. The NLP is then efficiently solved by sequential quadratic programming based on the initial solutions obtained from RRT-connect. These initials lie in multiple homotopy classes guaranteed by H-signature discrimination, leading to global optimality with probability one as the number of samples in RRT-connect goes to infinity. Numerical simulations on holonomic and non-holonomic autonomous underwater vehicles (AUVs) and an Intervention-AUV with a manipulator in cluttered underwater environments demonstrate that optimal and collision-free trajectories with low localization uncertainty are obtained efficiently.

BSP-based Swarm Behaviour Implementation for Underwater Multiagent Environmental Low-frequency Sensing

The environmental chemical conditions - in terms of substances with positive and negative impacts on the sustainability of local ecological networks - of confined environments with weak currents change over time in probabilistically predictable ways. We are developing a reproducible and measurable approach to the deployment of complex adaptive networks of underwater gliders, ROVs and fixed or mobile sensors deployed on the sea floor. The approach exploits Voronoi maps, multisensory fusion of different kind of chemical and non chemical sensors framed by a multi agent Belief Space Planning methodology. In this paper we report the early results of the study of the swarm behaviour implementation of a set of gliders or ROVs that can work autonomously (in the case of ROVs semi-autonomously or autonomously for limited time frames). In particular we describe how we were able to implement a swarm behaviour by means of a novel multiagent implementation of a BSP control scheme that we have already studied in the context of the Project CADDY (7th Framework Program (2014)).

Distributed Hierarchical Control for State Estimation With Robotic Sensor Networks

This paper addresses active state estimation with a team of robotic sensors. The states to be estimated are represented by spatially distributed, uncorrelated, stationary vectors. Given a prior belief on the geographic locations of the states, we cluster the states in moderately sized groups and propose a new hierarchical dynamic programming framework to compute optimal sensing policies for each cluster that mitigates the computational cost of planning optimal policies in the combined belief space. Then, we develop a decentralized assignment algorithm that dynamically allocates clusters to robots based on the precomputed optimal policies at each cluster. The integrated distributed state estimation framework is optimal at the cluster level but also scales very well to large numbers of states and robot sensors. We demonstrate efficiency of the proposed method in both simulations and real-world experiments using stereoscopic vision sensors.

An efficient hybrid of elephant herding optimization and cultural algorithm for optimal design of trusses

In this article, a new hybrid algorithm is proposed which was based on the elephant herding optimization (EHO) and cultural algorithm (CA), known as elephant herding optimization cultural (EHOC) algorithm. In this process, the belief space defined by the cultural algorithm was used to improve the standard EHO. EHO is motivated by herding behavior of the elephant groups. These behaviors are modeled into two operators including clan updating operator and separating operator. In EHOC, based on belief space, the separating operator is defined, which is able to create new local optimums in search space, to improve the algorithm search ability and to create an algorithm with an optimal exploration–exploitation balance. The CA, EHO, and EHOC algorithms are applied to eight mathematical optimization problems and four truss weight minimization problems, and to assess the performance of the proposed algorithm, the results are compared. The results clearly indicate that EHOC is capable of accelerating the convergence rate effectively and can develop better solutions compared to the CA and EHO. In addition, it can produce competitive results in comparison with other metaheuristic algorithms in the literature.

SLAP: Simultaneous Localization and Planning Under Uncertainty via Dynamic Replanning in Belief Space

Simultaneous localization and planning (SLAP) is a crucial ability for an autonomous robot operating under uncertainty. In its most general form, SLAP induces a continuous partially observable Markov decision process (POMDP), which needs to be repeatedly solved online. This paper addresses this problem and proposes a dynamic replanning scheme in belief space. The underlying POMDP, which is continuous in state, action, and observation space, is approximated offline via sampling-based methods, but operates in a replanning loop online to admit local improvements to the coarse offline policy. This construct enables the proposed method to combat changing environments and large localization errors, even when the change alters the homotopy class of the optimal trajectory. It further outperforms the state-of-the-art Feedback-based Information RoadMap (FIRM) method by eliminating unnecessary stabilization steps. Applying belief space planning to physical systems brings with it a plethora of challenges. A key focus of this paper is to implement the proposed planner on a physical robot and show the SLAP solution performance under uncertainty, in changing environments and in the presence of large disturbances, such as a kidnapped robot situation.

Privacy Verification and Enforcement via Belief Abstraction

Privacy is a crucial concern in many practical systems. We consider a new notion of privacy based on beliefs of the system states, which is closely related to opacity in discrete event systems. To guarantee the privacy requirement, we propose to abstract the belief space whose dynamics is shown to be mixed monotone where efficient abstraction algorithm exists. Based on the abstraction, we propose two different approaches to preserve privacy with an illustrative example.

Combining Intentionality and Belief: Revisiting Believable Character Plans

In this paper we present two studies supporting a plan-based model of narrative generation that reasons about both intentionality and belief. First we compare the believability of agent plans taken from the spaces of valid classical plans, intentional plans, and belief plans. We show that the plans that make the most sense to humans are those in the overlapping regions of the intentionality and belief spaces. Second, we validate the model’s approach to representing anticipation, where characters form plans that involve actions they expect other characters to take. Using a short interactive scenario we demonstrate that players not only find it believable when NPCs anticipate their actions, but sometimes actively anticipate the actions of NPCs in a way that is consistent with the model.

Multi-goal motion planning using traveling salesman problem in belief space

In this paper, the multi-goal motion planning problem of an environment with some background information about its map is addressed in detail. The motion planning goal is to find a policy in belief space for the robot to traverse through a number of goal points. This problem is modeled as an asymmetric traveling salesman problem (TSP) in the belief space using Partially Observable Markov Decision Process (POMDP) framework. Then, feedback-based information roadmap (FIRM) algorithm is utilized to reduce the computational burden and complexity. By generating a TSP-FIRM graph, the search policy is obtained and an algorithm is proposed for online execution of the policy. Moreover, approaches to cope with challenges such as map updating, large deviations and high uncertainty in localization, which are more troublesome in a real implementation, are carefully addressed. Finally, in order to evaluate applicability and performance of the proposed algorithms, it is implemented in a simulation environment as well as on a physical robot in which some challenges such as kidnapping and discrepancies between real and computational models and map are examined.

Fast and Bounded Probabilistic Collision Detection for High-DOF Trajectory Planning in Dynamic Environments

We present a novel approach to perform probabilistic collision detection between a high-DOF robot and imperfect obstacle representations in dynamic and uncertain environments. Our formulation is designed for high-DOF robot trajectory planning in dynamic scenes, where the uncertainties are modeled using Gaussian distributions. We present an efficient algorithm to compute collision probabilities between the robot and the obstacles. Furthermore, we present a prediction algorithm for obstacle positions that takes into account spatial and temporal uncertainties and uses that for trajectory optimization. We highlight the performance of our trajectory planning algorithm in challenging simulated and real-world environments with robot arms operating next to dynamically moving human obstacles. Note to Practitioners —This paper suggests a novel trajectory planning approach for dynamic and uncertain environments. Existing planning approaches generally deal with uncertainties by performing collision checks using enlarged bounding shapes for the given confidence levels, which are conservative and tend to compute less optimal trajectories or fail to find feasible trajectories. In this paper, we suggest a new collision probability approximation of a robot and obstacles. Our approach guarantees that the computed probability is an upper bound on the actual probability. We then present a trajectory planning algorithm based on our probabilistic collision detection, and a practical belief space estimation algorithm. In our experimental results, we demonstrate that our approach can compute more efficient trajectories than the prior approaches, while our approach has a similar level of safety. Our experiments assume Gaussian distributions for the environment uncertainties, and we will extend our algorithm to non-Gaussian distributions in future research. Recently, we have extended our approach to general convex polytopes and improved the speed and accuracy of the collision probability computation using bounding volume hierarchies.

Optimal Reconfiguration of Distribution Network Using Modified Culture Algorithm

This paper proposes a meta heuristic optimization method called modified Culture Algorithm (CA) for reconfiguration of distribution network considering power loss. Modified CA is inspired by culture and culture is collection of different traits, social motivated behavior and individual’s experiences of several generations. These social motivated behavior and individual’s experiences are collected, merged and generalized in the belief space. This belief space is communicated with future generation, which forces the optimization problem towards the global optimal solutions. In the proposed methodology decimal based coding technique has been used to represent chromosomes. To get feasible chromosomes all the time, it is subjected to some graph theory based rules. Direct topological approach has been used for precise calculation of power losses. The proposed methodology has been tested on 16-bus and 33-bus distribution test cases. The obtained result has been compared with some previous paper results and it shows that the proposed method is better than the methods used in previous papers.

Planning Robust Manual Tasks in Hierarchical Belief Spaces

This paper introduces a technique for planning in hierarchical belief spaces and demonstrates the idea in an autonomous assembly task. The objective is to effectively propagate belief across multiple levels of abstraction to make control decisions that expose useful state information, manage uncertainty and risk, and actively satisfy task specifications. This approach is demonstrated by performing multiple instances of a simple class of assembly tasks using the uBot-6 mobile manipulator with a two-level hierarchy that manages uncertainty over objects and assembly geometry. The result is a method for composing a sequence of manual interactions that causes changes to the environment and exposes new information in order to support belief that the task is satisfied. This approach has the added virtue that it provides a natural way to accomplish tasks while simultaneously suppressing errors and mitigating risk. We compare performance in an example assembly task against a baseline hybrid approach that combines uncertainty management with a symbolic planner and show statistically significant improvement in task outcomes. In additional demonstrations that challenge the system we highlight useful artifacts of the approach---risk management and autonomous recovery from unexpected events.

Finite-horizon LQR controller for partially-observed Boolean dynamical systems

This paper proposes an approach for finite-horizon control of partially-observed Boolean dynamical systems (POBDS) with uncertain continuous control input and infinite observation space. To cope with the partial observability of states, the proposed method first maps the POBDS to an unnormalized belief space. The nonlinear dynamics in this continuous belief space are linearized over a nominal trajectory. Then, the optimal feedback controller is derived, based on the well-known linear quadratic regulator (LQR), to push the system to follow the nominal trajectory. This nominal trajectory is computed in a planning stage before starting execution, and updated efficiently during execution, whenever the system is found to deviate from the nominal trajectory. We prove that, under mild regularization conditions, the proposed controller approaches the cost of the nominal trajectory as the linearization error approaches zero. The performance of the proposed controller is demonstrated by numerical experiments with a Melanoma gene regulatory network observed through noisy gene expression measurements.

Optimal Operation of Cascade Hydropower Stations Based on Chaos Cultural Sine Cosine Algorithm

Due to the fact that Sine Cosine Algorithm(SCA) may easily premature convergence, by taking cultural algorithm as the framework of population space and using chaotic search to lead the evolvement of belief space, a new Chaos Cultural Sine Cosine Algorithm (CCSCA) was present in this paper. And the results of typical benchmark functions show that CCSCA has better global searching ability than SCA to avoid the premature phenomenon. Finally, CCSCA was employed for optimal operation of cascade hydropower stations (OOCHS), and case study showed that CCSCA could get satisfying results with high accuracy and fast efficient, thus providing a new efficient method to solve OOCHS.

POMDP and MOMDP solutions for structural life-cycle cost minimization under partial and mixed observability

Scheduling of inspection and maintenance policies during the life-cycle of operating infrastructure necessitates optimization of long-term objectives in stochastic environments. Modern answers to the problem should focus on quantitative decision-making techniques, taking advantage of informative but uncertain data that become available in time. As such, the problem is efficiently addressed within the framework of stochastic dynamic programming by means of Partially Observable Markov Decision Processes (POMDPs) and Mixed Observability Markov Decision Processes (MOMDPs). Although these methodologies can provide very sophisticated solutions with optimality guarantees, important computational challenges often emerge, mainly due to the continuity of the multidimensional belief space on the probability simplex. In response, recent value iteration algorithms based on point-based approaches have been suggested, focusing on reachable belief points that can support an accurate value function. In this work, several POMDP and MOMDP point-based algorithms, with various characteristics regarding the exploration of the belief space and the value function update procedures, are rigorously analyzed. The algorithms are compared and evaluated in terms of accuracy and performance in stationary and nonstationary problems of structural inspection and maintenance for life-cycle cost minimization. Results are thoroughly discussed and several insights along with practical suggestions for similar problems are provided.

POMDP-Based Coding of Child–Robot Interaction within a Robot-Assisted ASD Diagnostic Protocol

The existing procedures for autism spectrum disorder (ASD) diagnosis are often time consuming and tiresome both for highly-trained human evaluators and children, which may be alleviated by using humanoid robots in the diagnostic process. Hence, this paper proposes a framework for robot-assisted ASD evaluation based on partially observable Markov decision process (POMDP) modeling, specifically POMDPs with mixed observability (MOMDPs). POMDP is broadly used for modeling optimal sequential decision making tasks under uncertainty. Spurred by the widely accepted autism diagnostic observation schedule (ADOS), we emulate ADOS through four tasks, whose models incorporate observations of multiple social cues such as eye contact, gestures and utterances. Relying only on those observations, the robot provides an assessment of the child’s ASD-relevant functioning level (which is partially observable) within a particular task and provides human evaluators with readable information by partitioning its belief space. Finally, we evaluate the proposed MOMDP task models and demonstrate that chaining the tasks provides fine-grained outcome quantification, which could also increase the appeal of robot-assisted diagnostic protocols in the future.

Multi-agent reinforcement learning based on local communication

Aiming at the locality and uncertainty of observations in large-scale multi-agent application scenarios, the model of Decentralized Partially Observable Markov Decision Processes (DEC-POMDP) is considered, and a novel multi-agent reinforcement learning algorithm based on local communication is proposed. For a distributed learning environment, the elements of reinforcement learning are difficult to describe effectively in local observation situation, and the learning behaviour of each individual agent is influenced by its teammates. The local communication with consensus protocol is utilized to agree on the global observing environment, and thus that a part of strategies generated by repeating observations are eliminated, and the agent team gradually reach uniform opinion on the state of the event or object to be observed, they can thus approach a unique belief space regardless of whether each individual agent can perform a complete or partial observation. The simulation results show that the learning strategy space is reduced, and the learning speed is improved.

A balanced fuzzy Cultural Algorithm with a modified Levy flight search for real parameter optimization

Over the last few decades, a plethora of improved evolutionary algorithms was developed with exquisite performance on numerical and real-world problems. Among such algorithms, the Cultural Algorithm is a hyper-heuristic evolutionary algorithm, which explicitly utilizes the knowledge represented in the belief space as an essential component to guide the evolutionary search. In this paper, a new enhanced Cultural Algorithm incorporates a fuzzy system with a modified Levy flight search that is introduced as a new component. The new algorithm namely, b-fCA+mLF, utilizes a balanced search mode using a customized belief space with a quality function to harmonize how the knowledge sources work in parallel. The communication protocols between the population space and the belief space are established through the modified fuzzy acceptance and influence functions. Using these new functions, the best individuals are selected to create new knowledge in an effective manner. Similarly, the best knowledge is selected to evolve the individuals in the population space and guide the evolutionary search towards the promising regions. A modified Levy flight search is proposed and utilizes the information from the belief space as an input to support the evolution process to generate better solutions. The algorithm is tested on the benchmark suite taken from the IEEE-CEC’15 competition on learning-based real-parameter single objective optimization, and is compared with other algorithms including the best performer algorithms in this competition. The results suggest that the proposed algorithm is statistically better and is able to produce higher quality solutions than the other state-of-the-art algorithms. A case study on the well-known 120-bar dome truss design problem is also presented to test the validity of the proposed algorithm for the solution of complex design problems. The results of this problem show the ability of the proposed algorithm to generate good solutions with fewer function evaluations, compared to reported results in the literature and other well-known algorithms.

Efficient Reinforcement Learning via Probabilistic Trajectory Optimization.

We present a trajectory optimization approach to reinforcement learning in continuous state and action spaces, called probabilistic differential dynamic programming (PDDP). Our method represents systems dynamics using Gaussian processes (GPs), and performs local dynamic programming iteratively around a nominal trajectory in Gaussian belief spaces. Different from model-based policy search methods, PDDP does not require a policy parameterization and learns a time-varying control policy via successive forward-backward sweeps. A convergence analysis of the iterative scheme is given, showing that our algorithm converges to a stationary point globally under certain conditions. We show that prior model knowledge can be incorporated into the proposed framework to speed up learning, and a generalized optimization criterion based on the predicted cost distribution can be employed to enable risk-sensitive learning. We demonstrate the effectiveness and efficiency of the proposed algorithm using nontrivial tasks. Compared with a state-of-the-art GP-based policy search method, PDDP offers a superior combination of learning speed, data efficiency, and applicability.