Discrete State Space Research Articles

Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning

Quantum machine learning (QML) has been identified as one of the key fields that could reap advantages from near-term quantum devices, next to optimization and quantum chemistry. Research in this area has focused primarily on variational quantum algorithms (VQAs), and several proposals to enhance supervised, unsupervised and reinforcement learning (RL) algorithms with VQAs have been put forward. Out of the three, RL is the least studied and it is still an open question whether VQAs can be competitive with state-of-the-art classical algorithms based on neural networks (NNs) even on simple benchmark tasks. In this work, we introduce a training method for parametrized quantum circuits (PQCs) that can be used to solve RL tasks for discrete and continuous state spaces based on the deep Q-learning algorithm. We investigate which architectural choices for quantum Q-learning agents are most important for successfully solving certain types of environments by performing ablation studies for a number of different data encoding and readout strategies. We provide insight into why the performance of a VQA-based Q-learning algorithm crucially depends on the observables of the quantum model and show how to choose suitable observables based on the learning task at hand. To compare our model against the classical DQN algorithm, we perform an extensive hyperparameter search of PQCs and NNs with varying numbers of parameters. We confirm that similar to results in classical literature, the architectural choices and hyperparameters contribute more to the agents' success in a RL setting than the number of parameters used in the model. Finally, we show when recent separation results between classical and quantum agents for policy gradient RL can be extended to inferring optimal Q-values in restricted families of environments.

Polarisation of Russia’s spatial development: quo vadis?

Inequality is an inherent property of spatial development, but too high spatial inequality can have a number of serious consequences, from increased fiscal spending aimed at smoothing fiscal space to slowing national economic development and increasing separatist sentiment. The problem of heterogeneity in spatial development is therefore relevant, especially for countries with a large spatial extent, where such inequalities are naturally higher. In Russia the spatial inequality started to grow since the end of the 20th century and is still considered to be high. The aim of the work was to identify the features of the process of polarization of Russia’s spatial development by analyzing the dynamics of economic activity in Russian regions in the 21st century and obtaining extrapolation forecast of interregional distribution of economic activity.The study was carried out with the help of distribution dynamics analysis methods, using the theory of Markov process with discrete time and continuous state space. The above methods have allowed us to: assess the actual distribution of economic activity indicators of the regions and to conclude about its changes in the first two decades of the 21st century; to obtain extrapolation forecast of the distribution of regional economic activity indicators in the long-run, i.e. to forecast long-run spatial equilibrium; to identify and characterize groups of regions forming “convergence groups” in the long-run. The obtained results led to a conclusion about the increasing polarization of Russia’s regions by the level of economic activity both in the study period and in the long run, while maintaining the current trends, to substantiate the need to expand the practice of applying tools of place-based regional policy.

Data-driven state estimation toward blade individual load-reduction of wind turbines1

As wind turbines tend to be build larger every year, the importance of load-reducing controls to damp the rotor blade oscillations increases. For a control algorithm to perform blade individual load-reducing strategies, online information about each rotor blade’s deflection is needed. An approach for wind turbine state estimation with special attention to the rotor blade deflections is given using a Hammerstein model driven by simulation data in a Kalman Filter (KF). The Hammerstein’s linear submodel is mainly derived as time discrete state space system via least squares, while nonlinear effects are defined in terms of their proportionality. The model noise definition required by the KF is calculated by the system model’s one-step prediction error. The simulation results show the estimate to have a standard deviation of less than 2 % in edge and 9% in flap-wise direction when using an appropriate sensor configuration. Using state-of-the-art sensors, standard deviations of less than 4% and 25% respectively are achieved. The approach is particularly attractive due to the low level of tuning required. The usage of the state estimation in a closed control loop as well as the Hammerstein model in a model-based control system offer room for promising further investigations.

Path Planning Method of Mobile Robot Using Improved Deep Reinforcement Learning

A mobile robot path planning method based on improved deep reinforcement learning is proposed. First, in order to conform to the actual kinematics model of the robot, the continuous environmental state space and discrete action state space are designed. In addition, an improved deep Q-network (DQN) method is proposed, which takes the directly collected information as the training samples and combines the environmental state characteristics of the robot and the target point to be reached as the input of the network. DQN method takes the Q value at the current position as the output of the network model and uses ε -greedy strategy for action selection. Finally, the reward function combined with the artificial potential field method is designed to optimize the state-action space. The reward function solves the problem of sparse reward in the environmental state space and makes the action selection of the robot more accurate. Experiments show that compared with the classical DQN method, the average loss function value is reduced by 36.87% and the average reward value is increased by 12.96%, which can effectively improve the working efficiency of mobile robot.

Desynchronous learning in a physics-driven learning network.

In a neuron network, synapses update individually using local information, allowing for entirely decentralized learning. In contrast, elements in an artificial neural network are typically updated simultaneously using a central processor. Here, we investigate the feasibility and effect of desynchronous learning in a recently introduced decentralized, physics-driven learning network. We show that desynchronizing the learning process does not degrade the performance for a variety of tasks in an idealized simulation. In experiment, desynchronization actually improves the performance by allowing the system to better explore the discretized state space of solutions. We draw an analogy between desynchronization and mini-batching in stochastic gradient descent and show that they have similar effects on the learning process. Desynchronizing the learning process establishes physics-driven learning networks as truly fully distributed learning machines, promoting better performance and scalability in deployment.

Continuum and thermodynamic limits for a simple random-exchange model

We discuss various limits of a simple random exchange model that can be used for the distribution of wealth. We start from a discrete state space — discrete time version of this model and, under suitable scaling, we show its functional convergence to a continuous space — discrete time model. Then, we show a thermodynamic limit of the empirical distribution to the solution of a kinetic equation of Boltzmann type. We solve this equation and we show that the solutions coincide with the appropriate limits of the invariant measure for the Markov chain. In this way we complete Boltzmann’s program of deriving kinetic equations from random dynamics for this simple model. Three families of invariant measures for the mean field limit are discovered and we show that only two of those families can be obtained as limits of the discrete system while the third is extraneous.

The Influence of Total Catastrophe on Simple Birth-Death-Immigration Processes

Markov Models are very much popular in studying population dynamics subject to catastrophes. So, emphasis is given on a discrete state space models specically when the population change occurs as a result of births, deaths and immigration in presence of catastrophes. A transient solution of simple birth-death-immigration process under the influence of total catastrophes has been found where the catastrophes occur at a constant rate and the population size reduces to zero if the catastrophes occur. Since, such processes are completely explained in terms of probability generating function, the mean and variance, we have analyzed the process focusing on these aspects. A simulation study was also carried out to gather evidences towards the results.

Optimal chance-constrained pension fund management through dynamic stochastic control

We apply a dynamic stochastic control (DSC) approach based on an open-loop linear feedback policy to a classical asset-liability management problem as the one faced by a defined-benefit pension fund (PF) manager. We assume a PF manager seeking an optimal investment policy under random market returns and liability costs as well as stochastic PF members’ survival rates. The objective function is formulated as a risk-reward trade-off function resulting in a quadratic programming problem. The proposed methodology combines a stochastic control approach, due to Primbs and Sung (IEEE Trans Autom Control 54(2):221–230, 2009), with a chance constraint on the PF funding ratio (FR) and it is applied for the first time to this class of long-term financial planning problems characterized by stochastic asset and liabilities. Thanks to the DSC formulation, we preserve the underlying risk factors continuous distributions and avoid any state space discretization as is typically the case in multistage stochastic programs (MSP). By distinguishing between a long-term PF liability projection horizon and a shorter investment horizon for the FR control, we avoid the curse-of-dimensionality, in-sample instability and approximation errors that typically characterize MSP formulations. Through an extended computational study, we present in- and out-of-sample results which allows us to validate the proposed methodology. The collected evidences confirm the potential of this approach when applied to a stylized but sufficiently realistic long-term PF problem.

Estimating Wildlife Density as a Function of Environmental Heterogeneity Using Unmarked Data

Recent developments to spatial-capture recapture models have allowed their use on species whose members are not uniquely identifiable from photographs by including individual identity as a latent, unobserved variable in the model. These ‘unmarked’ spatial capture recapture (uSCR) models have also been extended to presence-absence data and modified to allow categorical environmental covariates on density, but a uSCR model, which allows fitting continuous environmental covariates to density, has yet to be formulated. In this paper, we fill this gap and present an extension to the uSCR modeling framework by modeling animal density on a discrete state space as a function of continuous environmental covariates and investigate a form of Bayesian variable selection to improve inference. We used an elk population in their winter range within Karuk Indigenous Territory in Northern California as a case study and found a positive credible effect of increasing forb/grass cover on elk density and a negative credible effect of increasing tree cover on elk density. We posit that our extensions to uSCR modeling increase its utility in a wide range of ecological and management applications in which spatial counts of wildlife can be derived and environmental heterogeneity acts as a control on animal density.

Obtaining the long-term behavior of master equations with finite state space from the structure of the associated state transition network

The master equation describes the time evolution of the probabilities of a system with a discrete state space. This time evolution approaches for long times a stationary state that will in general depend on the initial probability distribution. Conditions under which the stationary state is unique are usually given as remarks appended to more comprehensive theories in the mathematical literature. We provide a direct and complete derivation of a necessary and sufficient criterion for when this steady state is unique. We translate this problem into the language of graph theory and show that there is a one-to-one correspondence between minimal absorbing sets within the state-transition network and linearly independent stationary states of the master equation.

Risk-sensitive semi-Markov decision problems with discounted cost and general utilities

In this article we consider risk-sensitive control of semi-Markov processes with a discrete state space. We consider general utility functions and discounted cost in the optimization criteria. We consider random finite horizon and infinite horizon problems. Using a state augmentation technique we characterize the value functions and also prescribe optimal controls.

Deep reinforcement learning approach for solving joint pricing and inventory problem with reference price effects

Reference prices, developed by consumers who frequently buy their desired products or services and form psychological price expectations as a benchmark, have significant impacts on customers’ purchase behaviors and firms’ operational strategies. Therefore, to determine an appropriate pricing and ordering strategy to maximize the total discounted revenues of a retailer, we consider joint pricing and inventory management system under reference price effects in an infinite horizon. Such a system involves uncertain market turbulence and customers’ sensitivities to gains and losses (loss-averse, gain-seeking and loss-neutral). We aggregate those factors into a general value function model with only a few realistic constraints on the variables and structure parameters. A deep reinforcement learning approach based on Double Deep Q-Networks with a target network (TN-DDQN) algorithm is proposed and forms the core of the expert decision system. Two ground truth algorithms (value iteration and real-demand-response policy) and two classical RL algorithms (Double Q-learning and Q-learning) are compared with the TN-DDQN algorithm in discrete and continuous state spaces respectively. Through a sequence of experiments, we find that the retailer should not ignore the impact of current prices on future demand, and a myopic policy will cause the retailer’s profits to decrease through reference price effects. Moreover, we also find that if customers have high abilities to remember previous prices, the retailer must bring sales prices down and raise the order-up-to level accordingly. Our system with the TN-DDQN algorithm provides a new way to handle complicated behavioral science and operation problems, which can be applied in a broader field.

Economic dispatch of a single micro gas turbine under CHP operation with uncertain demands

This work considers the economic dispatch problem for a single micro gas turbine, governed by a discrete state–space model, under combined heat and power (CHP) operation and coupled with a utility. If the exact power and heat demands are given, existing algorithms can be used to give a quick optimal solution to the economic dispatch problem. However, in practice, the power and heat demands cannot be known deterministically, but are rather predicted, resulting in an estimate and a bound on the estimation error. We consider the case in which the power and heat demands are unknown, and present a robust optimization-based approach for scheduling the turbine’s heat and power generation, in which the demand is assumed to be inside an uncertainty set. We consider two different choices of the uncertainty set relying on the ℓ∞- and the ℓ1-norms, each with different advantages, and consider the associated robust economic dispatch problems. We recast these as robust shortest-path problems on appropriately defined graphs. For the first choice, we provide an exact linear-time algorithm for the solution of the robust shortest-path problem, and for the second, we provide an exact quadratic-time algorithm and an approximate linear-time algorithm. The efficiency and usefulness of the algorithms are demonstrated using a detailed case study that employs real data on energy demand profiles and electricity tariffs.

Inference of Aggregate Hidden Markov Models With Continuous Observations

We consider a class of inference problems for large populations where each individual is modeled by the same hidden Markov model (HMM). We focus on aggregate inference problems in HMMs with discrete state space and continuous observation space. The continuous observations are aggregated in a way such that the individuals are indistinguishable from measurements. We propose an aggregate inference algorithm called continuous observation collective forward-backward algorithm. It extends the recently proposed collective forward-backward algorithm for aggregate inference in HMMs with discrete observations to the case of continuous observations.

Stochastic mathematical models for the spread of COVID-19: a novel epidemiological approach.

In this paper, three stochastic mathematical models are developed for the spread of the coronavirus disease (COVID-19). These models take into account the known special characteristics of this disease such as the existence of infectious undetected cases and the different social and infectiousness conditions of infected people. In particular, they include a novel approach that considers the social structure, the fraction of detected cases over the real total infected cases, the influx of undetected infected people from outside the borders, as well as contact-tracing and quarantine period for travellers. Two of these models are discrete time-discrete state space models (one is simplified and the other is complete) while the third one is a continuous time-continuous state space stochastic integro-differential model obtained by a formal passing to the limit from the proposed simplified discrete model. From a numerical point of view, the particular case of Lebanon has been studied and its reported data have been used to estimate the complete discrete model parameters, which can be of interest in estimating the spread of COVID-19 in other countries. The obtained simulation results have shown a good agreement with the reported data. Moreover, a parameters' analysis is presented in order to better understand the role of some of the parameters. This may help policy makers in deciding on different social distancing measures.

Intelligent fault detection of reciprocating compressor using a novel discrete state space

Aiming to the intelligent fault detection of reciprocating compressors (RCs), a novel method based on discrete state space construction and state space transformation is proposed in this paper. Conventional fault detection methods were conducted with continuous data, which tend to cause overfitting on establishing a model. This paper proposes a novel framework of fault detection on the basis of discrete data. High-dimensional features are extracted from raw signals and discretized into labels representing potential states of operating conditions. These discrete features of normal data are constructed into raw state space, and then transformed into a new state space by latent Dirichlet allocation subsequently. The transformed state space and its distribution can reflect operating conditions with more representative information. Then the distributions of real-time data under the transformed normal state space are calculated. The divergences of real-time state space distribution and normal state space are determined to identify the operating condition of reciprocating compressors, including normal condition and abnormal condition. This proposed method was validated on four cases of RC faults. The result demonstrates the effectiveness of the proposed method on detecting abnormal conditions of RCs.

Discrete phase space and continuous time relativistic quantum mechanics II: Peano circles, hyper-tori phase cells, and fiber bundles

The discrete phase space and continuous time representation of relativistic quantum mechanics are further investigated here as a continuation of paper I.1 The main mathematical construct used here will be that of an area filling Peano curve. We show that the limit of a sequence of a class of Peano curves is a Peano circle denoted as [Formula: see text], a circle of radius [Formula: see text] where [Formula: see text]. We interpret this two-dimensional (2D) Peano circle in our framework as a phase cell inside our 2D discrete phase plane. We postulate that a first quantized Planck oscillator, being very light, and small beyond current experimental detection, occupies this phase cell [Formula: see text]. The time evolution of this Peano circle sweeps out a 2D vertical cylinder analogous to the worldsheet of string theory. Extending this to 3D space, we introduce a [Formula: see text]-dimensional phase space hyper-tori [Formula: see text] as the appropriate phase cell in the physical dimensional discrete phase space. A geometric interpretation of this structure in state space is given in terms of product fiber bundles. We also study free scalar Bosons in the background [Formula: see text]-dimensional discrete phase space and continuous time state space using the relativistic partial difference-differential Klein–Gordon equation. The second quantized field quanta of this system can cohabit with the tiny Planck oscillators inside the [Formula: see text] phase cells for eternity. Finally, a generalized free second quantized Klein–Gordon equation in a higher [Formula: see text]-dimensional discrete state space is explored. The resulting discrete phase space dimension is compared to the significant spatial dimensions of some of the popular models of string theory.

Stationary GE-Process and its Application in Analyzing Gold Price Data

In this paper we introduce a new discrete time and continuous state space stationary process {Xn;n = 1,2,…}, such that Xn follows a two-parameter generalized exponential (GE) distribution. Joint distribution functions, characterization and some dependency properties of this new process have been investigated. The GE-process has three unknown parameters, two shape parameters and one scale parameter, and due to this reason it is more flexible than the existing exponential process. In presence of the scale parameter, if the two shape parameters are equal, then the maximum likelihood estimators of the unknown parameters can be obtained by solving one non-linear equation and if the two shape parameters are arbitrary, then the maximum likelihood estimators can be obtained by solving a two dimensional optimization problem. Two synthetic data sets, and one real gold-price data set have been analyzed to see the performance of the proposed model in practice. Finally some generalizations have been indicated.

Gradient-bounded dynamic programming for submodular and concave extensible value functions with probabilistic performance guarantees

We consider stochastic dynamic programming problems with high-dimensional, discrete state-spaces and finite, discrete-time horizons that prohibit direct computation of the value function from a given Bellman equation for all states and time steps due to the “curse of dimensionality”. For the case where the value function of the dynamic program is concave extensible and submodular in its state-space, we present a new algorithm that computes deterministic upper and stochastic lower bounds of the value function in the realm of dual dynamic programming. We show that the proposed algorithm terminates after a finite number of iterations. Furthermore, we derive probabilistic guarantees on the value accumulated under the associated policy for a single realisation of the dynamic program and for the expectation of this value. Finally, we demonstrate the efficacy of our approach on a high-dimensional numerical example from delivery slot pricing in attended home delivery.

Connection among Stochastic Hamilton–Jacobi–Bellman Equation, Path-Integral, and Koopman Operator on Nonlinear Stochastic Optimal Control

The path-integral control, which stems from the stochastic Hamilton-Jacobi-Bellman equation, is one of the methods to control stochastic nonlinear systems. This paper gives a new insight into nonlinear stochastic optimal control problems from the perspective of Koopman operators. When a finite-dimensional dynamical system is nonlinear, the corresponding Koopman operator is linear. Although the Koopman operator is infinite-dimensional, adequate approximation makes it tractable and useful in some discussions and applications. Employing the Koopman operator perspective, it is clarified that only a specific type of observable is enough to be focused on in the control problem. This fact becomes easier to understand via path-integral control. Furthermore, the focus on the specific observable leads to a natural power-series expansion; coupled ordinary differential equations for discrete-state space systems are derived. A demonstration for nonlinear stochastic optimal control shows that the derived equations work well.