Stochastic Action Research Articles

Mice exhibit stochastic and efficient action switching during probabilistic decision making

In probabilistic and nonstationary environments, individuals must use internal and external cues to flexibly make decisions that lead to desirable outcomes. To gain insight into the process by which animals choose between actions, we trained mice in a task with time-varying reward probabilities. In our implementation of such a two-armed bandit task, thirsty mice use information about recent action and action–outcome histories to choose between two ports that deliver water probabilistically. Here we comprehensively modeled choice behavior in this task, including the trial-to-trial changes in port selection, i.e., action switching behavior. We find that mouse behavior is, at times, deterministic and, at others, apparently stochastic. The behavior deviates from that of a theoretically optimal agent performing Bayesian inference in a hidden Markov model (HMM). We formulate a set of models based on logistic regression, reinforcement learning, and sticky Bayesian inference that we demonstrate are mathematically equivalent and that accurately describe mouse behavior. The switching behavior of mice in the task is captured in each model by a stochastic action policy, a history-dependent representation of action value, and a tendency to repeat actions despite incoming evidence. The models parsimoniously capture behavior across different environmental conditionals by varying the stickiness parameter, and like the mice, they achieve nearly maximal reward rates. These results indicate that mouse behavior reaches near-maximal performance with reduced action switching and can be described by a set of equivalent models with a small number of relatively fixed parameters.

Stochastic Path-Integral Analysis of the Continuously Monitored Quantum Harmonic Oscillator

We consider the evolution of a quantum simple harmonic oscillator in a general Gaussian state under simultaneous time-continuous weak position and momentum measurements. We deduce the stochastic evolution equations for position and momentum expectation values and the covariance matrix elements from the system's characteristic function. By generalizing the Chantasri-Dressel-Jordan (CDJ) formalism (Chantasri et al.~2013 and 2015) to this continuous variable system, we construct its stochastic Hamiltonian and action. Action extremization gives us the equations for the most-likely readout paths and quantum trajectories. For steady states of the covariance matrix elements, the analytical solutions for these most-likely paths are obtained. Using the CDJ formalism we calculate final state probability densities exactly starting from any initial state. We also demonstrate the agreement between the optimal path solutions and the averages of simulated clustered stochastic trajectories. Our results provide insights into the time dependence of the mechanical energy of the system during the measurement process, motivating their importance for quantum measurement engine/refrigerator experiments.

Quantum-Thermal Fluctuations of Effective Macroparameters and Their Correlations

The application of the conventional theory of macroparameter fluctuations has been shown to go beyond the framework of the thermodynamic description in a number of cases. The principles of the theory of quantum-thermal fluctuations of effectivemacroparameters and their correlations have been formulated. The theory satisfies the applicability conditions of equilibriumthermodynamics and is based on effective macroparameters, which take the integral stochastic action of the environment into account at any temperatures. The correlator of conjugate macroparameters, namely, the effective entropy and the effective temperature, has been calculated. The correlator was found to be proportional to the effective action which characterizes the stochastic environment. The pair correlators for the conjugate effective parameters "entropy–temperature" and "coordinate–momentum" have been demonstrated to depend linearly on the effective action, with their minimum values being determined by Planck's constant.

A Survey of Opponent Modeling in Adversarial Domains

Opponent modeling is the ability to use prior knowledge and observations in order to predict the behavior of an opponent. This survey presents a comprehensive overview of existing opponent modeling techniques for adversarial domains, many of which must address stochastic, continuous, or concurrent actions, and sparse, partially observable payoff structures. We discuss all the components of opponent modeling systems, including feature extraction, learning algorithms, and strategy abstractions. These discussions lead us to propose a new form of analysis for describing and predicting the evolution of game states over time. We then introduce a new framework that facilitates method comparison, analyze a representative selection of techniques using the proposed framework, and highlight common trends among recently proposed methods. Finally, we list several open problems and discuss future research directions inspired by AI research on opponent modeling and related research in other disciplines.

PROBABILISTIC SAFETY OF ESTUARY VESSELS BASED ON NONLINEAR ROLLING IN WIND AND WAVES

The author was previously involved in the development of the risk-based stability analysis which is now further extended, and used for the safety assessment of estuary container vessels subjected to stochastic action of beam wind and irregular waves. The study was motivated by the new set of safety regulations for estuary vessels issued by Belgian authorities in cooperation with Lloyd’s Register. These regulations introduce very innovative probabilistic ideas to ship stability regulations, and therefore present a significant step forward compared to the classical approach. Still, they do not account properly some important influences, such as wind gusts and motion nonlinearities, so considerably simplify the problem. The present investigation models the vessel motion much more realistically, analyzes the influence of beam wind and beam waves on the probability of a stability failure, and argues whether simplifications proposed by the regulations were justified. It is believed that presented method is not limited to the safety of estuary vessels only, but also gives important guidelines for a more general investigation of ship safety in wind and waves.

A method for along-wind vibration control of chimneys by tuning liners

Reinforced concrete chimneys with steel liners are widely used in waste gas discharge of industrial facilities, and guaranteeing their safety performance in harsh environments is important for industries and society. This paper proposes a novel method for the reduction of along-wind vibration in chimneys with liners by tuning the movement of the suspended liners to the response of the outer cylinder, and the conventional rigid supporting platform is replaced by a combination of radial horizontal tuning systems and vertical suspension systems. This ‘tuned liners’ method is applied to a simplified beam-like model that is able to capture the liners/chimney interaction and is validated against a more detailed finite-element shell model. A design method is proposed to obtain parameters of the tuning system that lead to significant reductions of along-wind vibrations whilst satisfying the relative response requirement. A comprehensive study of the structural vibration under stochastic wind actions is performed to demonstrate the effectiveness of the proposed system. The characteristics of the relative vibration of the outer cylinder and the liners are studied. Comparison with conventional TMD solution is conducted to further explore the advantage of the tuned-liners system under stochastic wind actions. The results indicate that the top displacement and acceleration of the outer cylinder are effectively reduced by 62% and 70% with the tuned liners, respectively. A similar performance using a conventional TMD would require an auxiliary vibrating mass that is approximately 300 tons, which is avoided with the proposed tuned liners. Results show that the proposed technique could be effective in the along-wind control of chimneys in multiple directions, even with some unintentional eccentricities.

To alkalinize or acidify, that is the question

Neurons are engineering marvels that respond to environmental stimuli with both stochastic and predictable actions. One stimulus that has been intensely studied over the years is extracellular pH. Extracellular pH, or more specifically synaptic pH, is predicted to decrease after neural activity because of the release of neurotransmitters that are packaged into acidic vesicles (1). Although this hypothesis is biologically sound, the experimental data to support it have been mixed.

Efficient Approximate Value Iteration for Continuous Gaussian POMDPs

We introduce a highly efficient method for solving continuous partially-observable Markov decision processes (POMDPs) in which beliefs can be modeled using Gaussian distributions over the state space. Our method enables fast solutions to sequential decision making under uncertainty for a variety of problems involving noisy or incomplete observations and stochastic actions. We present an efficient approach to compute locally-valid approximations to the value function over continuous spaces in time polynomial (O[n^4]) in the dimension n of the state space. To directly tackle the intractability of solving general POMDPs, we leverage the assumption that beliefs are Gaussian distributions over the state space, approximate the belief update using an extended Kalman filter (EKF), and represent the value function by a function that is quadratic in the mean and linear in the variance of the belief. Our approach iterates towards a linear control policy over the state space that is locally-optimal with respect to a user defined cost function, and is approximately valid in the vicinity of a nominal trajectory through belief space. We demonstrate the scalability and potential of our approach on problems inspired by robot navigation under uncertainty for state spaces of up to 128 dimensions.

SACBP: Belief space planning for continuous-time dynamical systems via stochastic sequential action control

We propose a novel belief space planning technique for continuous dynamics by viewing the belief system as a hybrid dynamical system with time-driven switching. Our approach is based on the perturbation theory of differential equations and extends sequential action control to stochastic dynamics. The resulting algorithm, which we name SACBP, does not require discretization of spaces or time and synthesizes control signals in near real-time. SACBP is an anytime algorithm that can handle general parametric Bayesian filters under certain assumptions. We demonstrate the effectiveness of our approach in an active sensing scenario and a model-based Bayesian reinforcement learning problem. In these challenging problems, we show that the algorithm significantly outperforms other existing solution techniques including approximate dynamic programming and local trajectory optimization.

Experimental Measurement of Relative Path Probabilities and Stochastic Actions

For diffusive stochastic dynamics, the probability to observe any individual trajectory is vanishingly small, making it unclear how to experimentally validate theoretical results for ratios of path probabilities. We provide the missing link between theory and experiment, by establishing a protocol to extract ratios of path probabilities from measured time series. For experiments on a single colloidal particle in a microchannel, we extract both ratios of path probabilities, and the most probable path for a barrier crossing, and find excellent agreement with independently calculated predictions based on the Onsager-Machlup stochastic action. Our experimental results at room temperature are found to be inconsistent with the low-noise Freidlin-Wentzell stochastic action, and we discuss under which circumstances the latter is expected to describe the most probable path. Furthermore, while the experimentally accessible ratio of path probabilities is uniquely determined, the formal path-integral action is known to depend on the time-discretization scheme used for deriving it; we reconcile these two seemingly contradictory facts by careful analysis of the time-slicing derivation of the path integral. Our experimental protocol enables us to probe probability distributions on path space, and allows us to relate theoretical single-trajectory results to measurement.

Ontological Knowledge for Goal-Driven Autonomy Agents in Starcraft

Starcraft, a commercial Real-Time Strategy (RTS) game that has enjoyed world-wide popularity (including televised professional matches), is a challenging domain for automated computer agents. Evidence of this difficulty comes not only from characteristics of the game (massive state space, stochastic actions, partial visibility, etc.) but also from three years of competitive entries in tournaments (i.e. AIIDE Annual Starcraft Competition) in which the best automated entry performs poorly against a human expert. We are interested in taking a new research direction: using semantic knowledge, such as description logic, to represent the game state with abstract concepts in order to perform high level actions.

The Organization of the Sinoatrial Node Microvasculature Varies Regionally to Match Local Myocyte Excitability.

The cardiac cycle starts when an action potential is produced by pacemaking cells in the sinoatrial node. This cycle is repeated approximately 100 000 times in humans and 1 million times in mice per day, imposing a monumental metabolic demand on the heart, requiring efficient blood supply via the coronary vasculature to maintain cardiac function. Although the ventricular coronary circulation has been extensively studied, the relationship between vascularization and cellular pacemaking modalities in the sinoatrial node is poorly understood. Here, we tested the hypothesis that the organization of the sinoatrial node microvasculature varies regionally, reflecting local myocyte firing properties. We show that vessel densities are higher in the superior versus inferior sinoatrial node. Accordingly, sinoatrial node myocytes are closer to vessels in the superior versus inferior regions. Superior and inferior sinoatrial node myocytes produce stochastic subthreshold voltage fluctuations and action potentials. However, the intrinsic action potential firing rate of sinoatrial node myocytes is higher in the superior versus inferior node. Our data support a model in which the microvascular densities vary regionally within the sinoatrial node to match the electrical and Ca2+ dynamics of nearby myocytes, effectively determining the dominant pacemaking site within the node. In this model, the high vascular density in the superior sinoatrial node places myocytes with metabolically demanding, high-frequency action potentials near vessels. The lower vascularization and electrical activity of inferior sinoatrial node myocytes could limit these cells to function to support sinoatrial node periodicity with sporadic voltage fluctuations via a stochastic resonance mechanism.

MADRaS : Multi Agent Driving Simulator

Autonomous driving has emerged as one of the most active areas of research as it has the promise of making transportation safer and more efficient than ever before. Most real-world autonomous driving pipelines perform perception, motion planning and action in a loop. In this work we present MADRaS, an open-source multi-agent driving simulator for use in the design and evaluation of motion planning algorithms for autonomous driving. Given a start and a goal state, the task of motion planning is to solve for a sequence of position, orientation and speed values in order to navigate between the states while adhering to safety constraints. These constraints often involve the behaviors of other agents in the environment. MADRaS provides a platform for constructing a wide variety of highway and track driving scenarios where multiple driving agents can be trained for motion planning tasks using reinforcement learning and other machine learning algorithms. MADRaS is built on TORCS, an open-source car-racing simulator. TORCS offers a variety of cars with different dynamic properties and driving tracks with different geometries and surface. MADRaS inherits these functionalities from TORCS and introduces support for multi-agent training, inter-vehicular communication, noisy observations, stochastic actions, and custom traffic cars whose behaviors can be programmed to simulate challenging traffic conditions encountered in the real world. MADRaS can be used to create driving tasks whose complexities can be tuned along eight axes in well-defined steps. This makes it particularly suited for curriculum and continual learning. MADRaS is lightweight and it provides a convenient OpenAI Gym interface for independent control of each car. Apart from the primitive steering-acceleration-brake control mode of TORCS, MADRaS offers a hierarchical track-position – speed control mode that can potentially be used to achieve better generalization. MADRaS uses a UDP based client server model where the simulation engine is the server and each client is a driving agent. MADRaS uses multiprocessing to run each agent as a parallel process for efficiency and integrates well with popular reinforcement learning libraries like RLLib. We show experiments on single and multi-agent reinforcement learning with and without curriculum

Time evolution with symmetric stochastic action

Quantum dynamical time-evolution of bosonic fields is shown to be equivalent to a stochastic trajectory in space-time, corresponding to samples of a statistical mechanical steady-state in a higher dimensional quasi-time. This is proved using the Q-function of quantum theory with time-symmetric diffusion, that is equivalent to a forward-backward stochastic process in both the directions of time. The resulting probability distribution has a positive, time-symmetric action principle and path integral, whose solution corresponds to a classical field equilibrating in an additional dimension. Comparisons are made to stochastic quantization and other higher dimensional physics proposals. Five-dimensional space-time was originally introduced by Kaluza and Klein, and is now widely proposed in cosmology and particle physics. Time-symmetric action principles for quantum fields are also related to electrodynamical absorber theory, which is known to be capable of violating a Bell inequality. We give numerical methods and examples of solutions to the resulting stochastic partial differential equations in a higher time-dimension, giving agreement with exact solutions for soluble boson field quantum dynamics. This approach may lead to useful computational techniques for quantum field theory, as the action principle is real, as well as to ontological models of physical reality.

The effect of high-frequency stochastic actions on the low-frequency behaviour of dynamic systems.

The dynamic system described by a finite number of first-order differential equations is considered. The right side of each equation is a sum of a slow, deterministic and, in the general case, nonlinear function of dynamic variables and a stochastic excitation. The stochastic action is a superposition of a finite number of independent random processes with coefficients depending on dynamic variables and slow time. The problem statement is oriented to applications in the field of driven systems. The analysis is based on the concept of vibration mechanics proposed by I. I. Blekhman. The modified method of direct separation of slow and fast motions uses the explicit introduction of a small parameter and some ideas of the two-scale technique. The general formulas for vibrational forces (or fluxes) are obtained. These additional terms appear in the resulting system for averaged motion instead of the stochastic terms to make the averaged system equivalent to the initial stochastic system with respect to slow motions and, in particular, to low-frequency resonances. As an example, the model of a vibration machine for bulk material processing is considered. The stochastic effect is caused by random oscillations of the bulk material mass. It is transformed into a modification of the machine's frequency characteristics leading to a specific stochastic resonance. This article is part of the theme issue 'Vibrational and stochastic resonance in driven nonlinear systems (part 1)'.

Characterization of an Equivalent Coupled Flexural-Torsional Beam Model for the Analysis of Tall Buildings under Stochastic Actions

AbstractAn equivalent beam model to estimate the three-dimensional response of tall buildings to random loads is proposed in this paper. The model fully describes coupling between bending and torsi...

Nonequilibrium steady state and heat transport in nonlinear open quantum systems: Stochastic influence action and functional perturbative analysis

In this paper, we show that a nonequilibrium steady state (NESS) exists at late times in open quantum systems with weak nonlinearity by following its nonequilibrium dynamics with a perturbative analysis. We consider an oscillator chain containing three-types of anharmonicity: cubic α- and quartic β-type Fermi–Pasta–Ulam–Tsingou (FPUT) nearest-oscillator interactions and the on-site (pinned) Klein–Gordon (KG) quartic self-interaction. Assuming weak nonlinearity, we introduce a stochastic influence action approach to the problem and obtain the energy flows in different junctures across the chain. The formal results obtained here can be used for quantum transport problems in weakly nonlinear quantum systems. For α-type anharmonicity, we observe that the first-order corrections do not play any role in the thermal transport in the NESS of the configuration we considered. For KG and β-types anharmonicity, we work out explicitly the case of two weakly nonlinearly coupled oscillators, with results scalable to any number of oscillators. We examine the late-time energy flows from one thermal bath to the other via the coupled oscillators, and show that both the zeroth- and the first-order contributions of the energy flows become constant in time at late times, signaling the existence of a late-time NESS to first order in nonlinearity. Our perturbative calculations provide a measure of the strength of nonlinearity for nonlinear open quantum systems, which may help control the mesoscopic heat transport distinct from or close to linear transport. Furthermore, our results also give a benchmark for the numerical challenge of simulating heat transport. Our setup and predictions can be implemented and verified by investigating heat flow in an array of Josephson junctions in the limit of large Josephson energy with the platform of circuit QED.

Variational integrators for stochastic dissipative Hamiltonian systems

Abstract Variational integrators are derived for structure-preserving simulation of stochastic forced Hamiltonian systems. The derivation is based on a stochastic discrete Hamiltonian, which approximates a type-II stochastic generating function for the stochastic flow of the Hamiltonian system. The generating function is obtained by introducing an appropriate stochastic action functional and considering a stochastic generalization of the deterministic Lagrange–d’Alembert principle. Our approach presents a general methodology to derive new structure-preserving numerical schemes. The resulting integrators satisfy a discrete version of the stochastic Lagrange–d’Alembert principle, and in the presence of symmetries, they also satisfy a discrete counterpart of Noether’s theorem. Furthermore, mean-square and weak Lagrange–d’Alembert Runge–Kutta methods are proposed and tested numerically to demonstrate their superior long-time numerical stability and energy behaviour compared to nongeometric methods. The Vlasov–Fokker–Planck equation is considered as one of the numerical test cases, and a new geometric approach to collisional kinetic plasmas is presented.

Stochastic action for tubes: Connecting path probabilities to measurement

This paper calculates the probability that a stochastic trajectory remains within a small but finite distance of a given smooth reference path. The authors explain how this measurable probability is closely related to the Onsager-Machlup stochastic action Lagrangian, thus transforming the latter from a purely mathematical concept into a physical observable.

Privacy Preserving Planning in Stochastic Environments

Collaborative privacy preserving planning (cppp) has gained much attention in the past decade. To date, cppp has focused on domains with deterministic action effects. In this paper, we extend cppp to domains with stochastic action effects. We show how such environments can be modeled as an mdp. We then focus on the popular Real-Time Dynamic Programming (RTDP) algorithm for computing value functions for mdps, extending it to the stochastic cppp setting. We provide two versions of RTDP: a complete version identical to executing centralized RTDP, and an approximate version that sends significantly fewer messages and computes competitive policies in practice. We experiment on domains adapted from the deterministic cppp literature.