Belief State Research Articles

Belief base change as priority change: A study based on dynamic epistemic logic

• Priority graphs have the same expressibility as injective preference models. • Relevant graph transformations are characterised by those transformations preserving P-graph equivalence. • A grounded P-graph is an extension of a P-graph with which we can reason about minimality. • Transformations on grounded priority graphs can express belief change postulates not representable with priority graphs. AGM's belief revision is one of the main paradigms in the study of belief change operations. In this context, belief bases (prioritised bases) have been primarily used to specify the agent's belief state. While the connection of iterated AGM-like operations and their encoding in dynamic epistemic logics have been studied before, few works considered how well-known postulates from iterated belief revision theory can be characterised by means of belief bases. Particularly, it has been shown that some postulates can be characterised through transformations in priority graphs, while others may not be represented that way. This work investigates the expressive power of prioritized bases, employing priority graphs as a representation for an agent's belief state and of transformations on such bases as representations for iterated relational belief change operators. As such, we propose a new representation for an agent's belief state, which we show can be used to characterize different iterated belief change postulates not previously representable using transformations on priority graphs.

UBAR: Towards Fully End-to-End Task-Oriented Dialog System with GPT-2

This paper presents our task-oriented dialog system UBAR which models task-oriented dialogs on a dialog session level. Specifically, UBAR is acquired by fine-tuning the large pre-trained unidirectional language model GPT-2 on the sequence of the entire dialog session which is composed of user utterance, belief state, database result, system act, and system response of every dialog turn. Additionally, UBAR is evaluated in a more realistic setting, where its dialog context has access to user utterances and all content it generated such as belief states, system acts, and system responses. Experimental results on the MultiWOZ datasets show that UBAR achieves state-of-the-art performances in multiple settings, improving the combined score of response generation, policy optimization, and end-to-end modeling by 4.7, 3.5, and 9.4 points respectively. Thorough analyses demonstrate that the session-level training sequence formulation and the generated dialog context are essential for UBAR to operate as a fully end-to-end task-oriented dialog system in real life. We also examine the transfer ability of UBAR to new domains with limited data and provide visualization and a case study to illustrate the advantages of UBAR in modeling on a dialog session level.

Sequential Generative Exploration Model for Partially Observable Reinforcement Learning

Many challenging partially observable reinforcement learning problems have sparse rewards and most existing model-free algorithms struggle with such reward sparsity. In this paper, we propose a novel reward shaping approach to infer the intrinsic rewards for the agent from a sequential generative model. Specifically, the sequential generative model processes a sequence of partial observations and actions from the agent's historical transitions to compile a belief state for performing forward dynamics prediction. Then we utilize the error of the dynamics prediction task to infer the intrinsic rewards for the agent. Our proposed method is able to derive intrinsic rewards that could better reflect the agent's surprise or curiosity over its ground-truth state by taking a sequential inference procedure. Furthermore, we formulate the inference procedure for dynamics prediction as a multi-step forward prediction task, where the time abstraction that has been incorporated could effectively help to increase the expressiveness of the intrinsic reward signals. To evaluate our method, we conduct extensive experiments on challenging 3D navigation tasks in ViZDoom and DeepMind Lab. Empirical evaluation results show that our proposed exploration method could lead to significantly faster convergence than various state-of-the-art exploration approaches in the testified navigation domains.

Knowledge norms of belief and belief formation: When the time is ripe to actualize one's epistemic potential

AbstractAccording to a currently popular view within the camp of knowledge lovers, knowledge is the norm of (full) belief. It says that no‐one should believe p without knowing that p, and thus governs the state of belief; it is a static norm. I will argue that we need an additional, structurally different knowledge norm in order to cover cases in which a subject is such that she ought to start forming a belief. So the standard view of how knowledge governs matters of belief must be supplemented by including a further, dynamic norm. This additional knowledge norm prescribes the initiation of knowledge acquistion on condition of the epistemic potential to know (plus some further qualifications). It thus expresses the plausible thought that we should actualize our epistemic potential (under certain conditions).

Targeting the Fragility Fracture Prevention Message

Abstract Background: The impact of fragility fractures on human suffering is underappreciated. Fragility fractures increase mortality and the risk of recurrent fracture. Notwithstanding these facts, only a minority of patients suffering osteoporotic fractures receive fracture preventing therapy. Education about the burden of disease and the benefits of prompt treatment are important means to increase the treatment rate. We propose that the focus of the educational message should be varied for distinct audiences. Method: We searched existing literature to determine perceptions about the importance of treating fractures and osteoporosis among the following groups: the general population, primary care physicians, specialist physicians, hospital administrators, and government health officials. Result: Extensive data focusing on patients and primary care physicians exist about barriers to treatment initiation. In these groups, lack of perceived treatment benefit is a widespread, common finding. To address this gap, we have produced a primary care education toolkit, patient booklet and education directory as Asia Pacific Fragility Fracture Alliance (APFFA)-sponsored initiatives. Data are more limited for specialist physicians, hospital administrators, and government officials. Health economic arguments have been found to be persuasive among policy makers. Conclusion: Information about the importance of treating fragility fractures and osteoporosis is far more abundant with reference to patients and primary care providers than to specialty physicians, hospital administrators, and government officials. Understanding the state of knowledge and belief is an essential first step in developing educational materials that address the concerns and misperceptions of each key constituency. Existing data suggest that patients will be more receptive to materials emphasising independence and quality of life, while policy makers will be more receptive to materials highlighting the burden of disease.

Observing a superposition

The bare theory is a no-collapse version of quantum mechanics which predicts certain puzzling results for the introspective beliefs of human observers of superpositions. The bare theory can be interpreted to claim that an observer can form false beliefs about the outcome of an experiment which produces a superpositional result. It is argued that, when careful consideration is given to the observer’s belief states and their evolution, the observer does not end up with the beliefs claimed. This result leads to questions about whether there can be any allure for no-collapse theories as austere as the bare theory.

Modest-vocabulary loop-closure detection with incremental bag of tracked words

A key feature in the context of simultaneous localization and mapping is loop-closure detection, a process determining whether the current robot’s environment perception coincides with previous observation. However, in long-term operations, both computational efficiency and memory requirements involved in an autonomous robot operation in uncontrolled environments, are of particular importance. The majority of approaches scale linearly with the environment’s size in terms of storage and query time. The article at hand presents an efficient appearance-based loop-closure detection pipeline, which encodes the traversed trajectory by a low amount of unique visual words generated on-line through feature tracking. The incrementally constructed visual vocabulary is referred to as the “Bag of Tracked Words.” A nearest-neighbor voting scheme is utilized to query the database and assign probabilistic scores to all visited locations. Exploiting the inherent temporal coherency in the loop-closure task, the produced scores are processed through a Bayesian filter to estimate the belief state about the robot’s location on the map. Also, a geometrical verification step ensures consistency between image matches. Management is also applied to the resulting vocabulary to reduce its growth rate and constraint the system’s computational complexity while improving its voting distinctiveness. The proposed approach’s performance is experimentally evaluated on several publicly available and challenging datasets, including hand-held, car-mounted, aerial, and ground trajectories. Results demonstrate the method’s adaptability, which retains high operational frequency in environments of up to 13 km and high recall rates for perfect precision, outperforming other state-of-the-art techniques. The system’s effectiveness is owed to the reduced vocabulary size, which is at least one order of magnitude smaller than other contemporary approaches. An open research-oriented source code has been made publicly available, which is dubbed as “BoTW-LCD.” • Fully probabilistic loop-closure detection pipeline. • Modest and incrementally generated visual vocabulary through feature tracking for on-line trajectory mapping. • Low computational complexity system, capable of detecting loop-closure events without the need of any indexing. • Exhaustive experimental parameter evaluation scheme based on the number of track points, descriptor accuracy and words’ generation method. • Open-source implementation following the article’s structure.

How beliefs are like colors

Double dissociations between perceivable colors and physical properties of colored objects have led many philosophers to endorse relationalist accounts of color. I argue that there are analogous double dissociations between attitudes of belief—the beliefs that people attribute to each other in everyday life—and intrinsic cognitive states of belief—the beliefs that some cognitive scientists posit as cogs in cognitive systems—pitched at every level of psychological explanation. These dissociations provide good reason to refrain from conflating attitudes of belief with intrinsic cognitive states of belief. I suggest that interpretivism provides an attractive account of the former (insofar as they are not conflated with the latter). Like colors, attitudes of belief evolved to be ecological signifiers, not cogs in cognitive systems.

Serial reproduction reveals the geometry of visuospatial representations

An essential function of the human visual system is to locate objects in space and navigate the environment. Due to limited resources, the visual system achieves this by combining imperfect sensory information with a belief state about locations in a scene, resulting in systematic distortions and biases. These biases can be captured by a Bayesian model in which internal beliefs are expressed in a prior probability distribution over locations in a scene. We introduce a paradigm that enables us to measure these priors by iterating a simple memory task where the response of one participant becomes the stimulus for the next. This approach reveals an unprecedented richness and level of detail in these priors, suggesting a different way to think about biases in spatial memory. A prior distribution on locations in a visual scene can reflect the selective allocation of coding resources to different visual regions during encoding ("efficient encoding"). This selective allocation predicts that locations in the scene will be encoded with variable precision, in contrast to previous work that has assumed fixed encoding precision regardless of location. We demonstrate that perceptual biases covary with variations in discrimination accuracy, a finding that is aligned with simulations of our efficient encoding model but not the traditional fixed encoding view. This work demonstrates the promise of using nonparametric data-driven approaches that combine crowdsourcing with the careful curation of information transmission within social networks to reveal the hidden structure of shared visual representations.

Adaptive Learning of Drug Quality and Optimization of Patient Recruitment for Clinical Trials with Dropouts

Problem definition: Clinical trials are crucial to new drug development. This study investigates optimal patient enrollment in clinical trials with interim analyses, which are analyses of treatment responses from patients at intermediate points. Our model considers uncertainties in patient enrollment and drug treatment effectiveness. We consider the benefits of completing a trial early and the cost of accelerating a trial by maximizing the net present value of drug cumulative profit. Academic/practical relevance: Clinical trials frequently account for the largest cost in drug development, and patient enrollment is an important problem in trial management. Our study develops a dynamic program, accurately capturing the dynamics of the problem, to optimize patient enrollment while learning the treatment effectiveness of an investigated drug. Methodology: The model explicitly captures both the physical state (enrolled patients) and belief states about the effectiveness of the investigated drug and a standard treatment drug. Using Bayesian updates and dynamic programming, we establish monotonicity of the value function in state variables and characterize an optimal enrollment policy. We also introduce, for the first time, the use of backward approximate dynamic programming (ADP) for this problem class. We illustrate the findings using a clinical trial program from a leading firm. Our study performs sensitivity analyses of the input parameters on the optimal enrollment policy. Results: The value function is monotonic in cumulative patient enrollment and the average responses of treatment for the investigated drug and standard treatment drug. The optimal enrollment policy is nondecreasing in the average response from patients using the investigated drug and is nonincreasing in cumulative patient enrollment in periods between two successive interim analyses. The forward ADP algorithm (or backward ADP algorithm) exploiting the monotonicity of the value function reduced the run time from 1.5 months using the exact method to a day (or 20 minutes) within 4% of the exact method. Through an application to a leading firm’s clinical trial program, the study demonstrates that the firm can have a sizable gain of drug profit following the optimal policy that our model provides. Managerial implications: We developed a new model for improving the management of clinical trials. Our study provides insights of an optimal policy and insights into the sensitivity of value function to the dropout rate and prior probability distribution. A firm can have a sizable gain in the drug’s profit by managing its trials using the optimal policies and the properties of value function. We illustrated that firms can use the ADP algorithms to develop their patient enrollment strategies.

Ignorance, Introspection, and Epistemic Modals

Embedded epistemic modals are infelicitous under desire predicates when they are anchored to the belief state of the attitude holder (see, esp., Anand & Hacquard 2013). We present two ways of deriving this observation from an inde- pendently motivated property of desire predicates (Heim 1992; von Fintel 1999).

Sophisticated Inference.

Active inference offers a first principle account of sentient behavior, from which special and important cases-for example, reinforcement learning, active learning, Bayes optimal inference, Bayes optimal design-can be derived. Active inference finesses the exploitation-exploration dilemma in relation to prior preferences by placing information gain on the same footing as reward or value. In brief, active inference replaces value functions with functionals of (Bayesian) beliefs, in the form of an expected (variational) free energy. In this letter, we consider a sophisticated kind of active inference using a recursive form of expected free energy. Sophistication describes the degree to which an agent has beliefs about beliefs. We consider agents with beliefs about the counterfactual consequences of action for states of affairs and beliefs about those latent states. In other words, we move from simply considering beliefs about "what would happen if I did that" to "what I would believe about what would happen if I did that." The recursive form of the free energy functional effectively implements a deep tree search over actions and outcomes in the future. Crucially, this search is over sequences of belief states as opposed to states per se. We illustrate the competence of this scheme using numerical simulations of deep decision problems.

Neural and social correlates of attitudinal brokerage: using the complete social networks of two entire villages.

To avoid polarization and maintain small-worldness in society, people who act as attitudinal brokers are critical. These people maintain social ties with people who have dissimilar and even incompatible attitudes. Based on resting-state functional magnetic resonance imaging (n = 139) and the complete social networks from two Korean villages (n = 1508), we investigated the individual-level neural capacity and social-level structural opportunity for attitudinal brokerage regarding gender role attitudes. First, using a connectome-based predictive model, we successfully identified the brain functional connectivity that predicts attitudinal diversity of respondents' social network members. Brain regions that contributed most to the prediction included mentalizing regions known to be recruited in reading and understanding others’ belief states. This result was corroborated by leave-one-out cross-validation, fivefold cross-validation and external validation where the brain connectivity identified in one village was used to predict the attitudinal diversity in another independent village. Second, the association between functional connectivity and attitudinal diversity of social network members was contingent on a specific position in a social network, namely, the structural brokerage position where people have ties with two people who are not otherwise connected.

SUMBT+LaRL: Effective Multi-Domain End-to-End Neural Task-Oriented Dialog System

The recent advent of neural approaches for developing each dialog component in task-oriented dialog systems has remarkably improved, yet optimizing the overall system performance remains a challenge. Besides, previous research on modeling complicated multi-domain goal-oriented dialogs in end-to-end fashion has been limited. In this paper, we present an effective multi-domain end-to-end trainable neural dialog system <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SUMBT+LaRL</i> that incorporates two previous strong models and facilitates them to be fully differentiable. Specifically, the SUMBT+ estimates user-acts as well as dialog belief states, and the LaRL models latent system action spaces and generates responses given the estimated contexts. We emphasize that the training framework of three steps significantly and stably increase dialog success rates: separately pretraining the SUMBT+ and LaRL, fine-tuning the entire system, and then reinforcement learning of dialog policy. We also introduce new reward criteria of reinforcement learning for dialog policy training. Then, we discuss experimental results depending on the reward criteria and different dialog evaluation methods. Consequently, our model achieved the new state-of-the-art success rate of 85.4% on corpus-based evaluation, and a comparable success rate of 81.40% on simulator-based evaluation provided by the DSTC8 challenge. To our best knowledge, our work is the first comprehensive study of a modularized E2E multi-domain dialog system that learning from each component to the entire dialog policy for task success.

HDRS: Hindi Dialogue Restaurant Search Corpus for Dialogue State Tracking in Task-Oriented Environment

Due to the rapid increase in the development of Task-oriented dialogue systems, the need for labelled dialogue corpus has become inevitable. For the Hindi language, there is no such dialogue corpus yet available. As a first attempt, we release a Hindi Dialogue Restaurant Search (HDRS) corpus and compare various state-of-the-art dialogue state tracking (DST) models on it. The corpus consists of 1.4 k human-to-human typed dialogues collected using Wizard-of-Oz paradigm. The paper starts with a brief description of the corpus by providing the details of features, corpus collection process and statistical analysis, then the performance of baseline NLU and DST models are investigated. Further, we experimented two categories of state-of-the-art belief state trackers: (1) Non-contextual pre-trained word embedding based DST models; (2) Contextual pre-trained BERT based DST models. All belief trackers follow a three-layered generic architecture. The category-1 models use the static domain ontology, while category-2 models have the capability to handle the dynamic ontology. The DST models are compared on joint-goal and turn-request accuracy. Global encoder and Slot-ATtentive decoders (GSAT) outperforms all the models with 83.25% joint-goal accuracy, followed by SUMBT.

Modeling combination of question order effect, response replicability effect, and QQ-equality with quantum instruments

We continue to analyze basic constraints on the human decision making from the viewpoint of quantum measurement theory (QMT). As it has been found, the conventional QMT based on the projection postulate cannot account for the combination of the question order effect (QOE) and the response replicability effect (RRE). This was an alarming finding for quantum-like modeling of decision making. Recently, it was shown that this difficulty can be resolved by using of the general QMT based on quantum instruments. In the present paper we analyze the problem of the combination of QOE, RRE, and the well-known QQ-equality (QQE). This equality was derived by Busemeyer and Wang, and it was shown (in a joint paper with Solloway and Shiffrin) that statistical data from many social opinion polls satisfy it. Here we construct quantum instruments satisfying QOE, RRE and QQE. The general features of our approach are formalized with postulates that generalize (the Wang–Busemeyer) postulates for quantum-like modeling of decision making. Moreover, we show that our model closely reproduces the statistics of the well-known Clinton–Gore Poll data with a prior belief state independent of the question order. This model successfully corrects for the order effect in the data to determine the “genuine” distribution of the opinions in the Poll. The paper also provides an accessible introduction to the theory of quantum instruments — the most general mathematical framework for quantum measurements.

Behavioral, Physiological, and Neural Signatures of Surprise during Naturalistic Sports Viewing

Surprise signals a discrepancy between past and current beliefs. It is theorized to be linked to affective experiences, the creation of particularly resilient memories, and segmentation of the flow of experience into discrete perceived events. However, the ability to precisely measure naturalistic surprise has remained elusive. We used advanced basketball analytics to derive a quantitative measure of surprise and characterized its behavioral, physiological, and neural correlates in human subjects observing basketball games. We found that surprise was associated with segmentation of ongoing experiences, as reflected by subjectively perceived event boundaries and shifts in neocortical patterns underlying belief states. Interestingly, these effects differed by whether surprising moments contradicted or bolstered current predominant beliefs. Surprise also positively correlated with pupil dilation, activation in subcortical regions associated with dopamine, game enjoyment, and long-term memory. These investigations support key predictions from event segmentation theory and extend theoretical conceptualizations of surprise to real-world contexts.

Dopamine, Updated: Reward Prediction Error and Beyond.

Dopamine neurons have been intensely studied for their roles in reinforcement learning. A dominant theory of how these neurons contribute to learning is through the encoding of a reward prediction error (RPE) signal. Recent advances in dopamine research have added nuance to RPE theory by incorporating the ideas of sensory prediction error, distributional encoding, and belief states. Further nuance is likely to be added shortly by convergent lines of research on dopamine neuron diversity. Finally, a major challenge is to reconcile RPE theory with other current theories of dopamine function to account for dopamine's role in movement, motivation, and goal-directed planning.

Programming and reasoning with partial observability

Computer programs are increasingly being deployed in partially-observable environments. A partially observable environment is an environment whose state is not completely visible to the program, but from which the program receives partial observations. Developers typically deal with partial observability by writing a state estimator that, given observations, attempts to deduce the hidden state of the environment. In safety-critical domains, to formally verify safety properties developers may write an environment model. The model captures the relationship between observations and hidden states and is used to prove the software correct. In this paper, we present a new methodology for writing and verifying programs in partially observable environments. We present belief programming , a programming methodology where developers write an environment model that the program runtime automatically uses to perform state estimation. A belief program dynamically updates and queries a belief state that captures the possible states the environment could be in. To enable verification, we present Epistemic Hoare Logic that reasons about the possible belief states of a belief program the same way that classical Hoare logic reasons about the possible states of a program. We develop these concepts by defining a semantics and a program logic for a simple core language called BLIMP. In a case study, we show how belief programming could be used to write and verify a controller for the Mars Polar Lander in BLIMP. We present an implementation of BLIMP called CBLIMP and evaluate it to determine the feasibility of belief programming.

Using POMDPs for learning cost sensitive decision trees

In classification, an algorithm learns to classify a given instance based on a set of observed attribute values. In many real world cases testing the value of an attribute incurs a cost. Furthermore, there can also be a cost associated with the misclassification of an instance. Cost sensitive classification attempts to minimize the expected cost of classification, by deciding after each observed attribute value, which attribute to measure next. In this paper we suggest Partially Observable Markov Decision Processes (POMDPs) as a modeling tool for cost sensitive classification. POMDPs are typically solved through a policy over belief states. We show how a relatively small set of potentially important belief states can be identified, and define an MDP over these belief states. To identify these potentially important belief states, we construct standard decision trees over all attribute subsets, and the leaves of these trees become the state space of our tree-based MDP. At each phase we decide on the next attribute to measure, balancing the cost of the measurement and the classification accuracy. We compare our approach to a set of previous approaches, showing our approach to work better for a range of misclassification costs.