Control Of Stochastic Processes Research Articles

Synchronization in a Kuramoto mean field game

The classical Kuramoto model is studied in the setting of an infinite horizon mean field game. The system is shown to exhibit both synchronization and phase transition. Incoherence below a critical value of the interaction parameter is demonstrated by the stability of the uniform distribution. Above this value, the game bifurcates and develops self-organizing time homogeneous Nash equilibria. As interactions get stronger, these stationary solutions become fully synchronized. Results are proved by an amalgam of techniques from nonlinear partial differential equations, viscosity solutions, stochastic optimal control and stochastic processes.

Combining neural networks and control: potentialities, patterns and perspectives

Machine learning tools are widely used for knowledge extraction, modeling, and decision tasks; a range of problems that Control Theory also tackles. Their relations have been largely explored by looking at stochastic control and Markov Decision Processes, due to the proximity of their formulations. However, novel links between machine learning and deterministic control are emerging; combining both approaches, e.g. by performing identification with learning, or controlling the training process. The recent flourishing literature is vast: there is a need to identify challenges, trends and opportunities on this interface. This survey contributes i) to the compared analysis of both fields. ii) Based on literature review, a categorization of combinations of learning and control is drawn. In the control framework, learning has been used for modeling, controllers tuning or adaptation, generating a controller or as a controller itself, for translating complex objectives, or checking controlled systems. Conversely, in the learning framework, control is used for tuning hyperparameters, selecting or generating training data, as the training or decision-making algorithm itself or to guarantee learning properties. iii) Finally, discussions on the literature open novel promising combinations to be explored, such as control of neural networks’ training process.

An Expanded Framework for Situation Control.

There is an extensive body of literature on the topic of estimating situational states, in applications ranging from cyber-defense to military operations to traffic situations and autonomous cars. In the military/defense/intelligence literature, situation assessment seems to be the sine qua non for any research on surveillance and reconnaissance, command and control, and intelligence analysis. Virtually all of this work focuses on assessing the situation-at-the-moment; many if not most of the estimation techniques are based on Data and Information Fusion (DIF) approaches, with some recent schemes employing Artificial Intelligence (AI) and Machine Learning (ML) methods. But estimating and recognizing situational conditions is most often couched in a decision-making, action-taking context, implying that actions may be needed so that certain goal situations will be reached as a result of such actions, or at least that progress toward such goal states will be made. This context thus frames the estimation of situational states in the larger context of a control-loop, with a need to understand the temporal evolution of situational states, not just a snapshot at a given time. Estimating situational dynamics requires the important functions of situation recognition, situation prediction, and situation understanding that are also central to such an integrated estimation + action-taking architecture. The varied processes for all of these combined capabilities lie in a closed-loop “situation control” framework, where the core operations of a stochastic control process involve situation recognition—learning—prediction—situation “error” assessment—and action taking to move the situation to a goal state. We propose several additional functionalities for this closed-loop control process in relation to some prior work on this topic, to include remarks on the integration of control-theoretic principles. Expanded remarks are also made on the state of the art of the schemas and computational technologies for situation recognition, prediction and understanding, as well as the roles for human intelligence in this larger framework.

Reinforcement Learning Approaches to Optimal Market Making

Market making is the process whereby a market participant, called a market maker, simultaneously and repeatedly posts limit orders on both sides of the limit order book of a security in order to both provide liquidity and generate profit. Optimal market making entails dynamic adjustment of bid and ask prices in response to the market maker’s current inventory level and market conditions with the goal of maximizing a risk-adjusted return measure. This problem is naturally framed as a Markov decision process, a discrete-time stochastic (inventory) control process. Reinforcement learning, a class of techniques based on learning from observations and used for solving Markov decision processes, lends itself particularly well to it. Recent years have seen a very strong uptick in the popularity of such techniques in the field, fueled in part by a series of successes of deep reinforcement learning in other domains. The primary goal of this paper is to provide a comprehensive and up-to-date overview of the current state-of-the-art applications of (deep) reinforcement learning focused on optimal market making. The analysis indicated that reinforcement learning techniques provide superior performance in terms of the risk-adjusted return over more standard market making strategies, typically derived from analytical models.

Balancing trade-offs in one-stage production with processing time uncertainty

Production scheduling faces three challenges, two of which are trade-offs and the third is processing time uncertainty. The two sources of trade-offs are between inconsistent key performance indicators (KPIs), and between the expected return and the risk of KPI portfolios. Given the KPIs of total completion time (TCT) and variance of completion times (VCT) are inconsistent for one-stage production, we propose our trade-off balancing (ToB) heuristics. Based on comprehensive case studies, we show that our ToB heuristics efficiently and effectively balance the trade-offs from these two sources. Daniels and Kouvelis (DK) proposed a scheduling scheme to optimize the worst-case scenarios against processing time uncertainty, and they designed the endpoint product (EP) and endpoint sum (ES) heuristics for robust scheduling accordingly. Using 5 levels of coefficients of variation (CVs) to represent processing time uncertainty, we show that our ToB heuristics are robust as well, and even better than the EP and ES heuristics at high levels of processing time uncertainty. In addition, our ToB heuristics generate undominated solution spaces of KPIs, which provides a solid base in deciding control and specification limits for stochastic process control (SPC). Moreover, based on the normalized deviations from optima, our trade-off balancing scheme can be generalized to balance any inconsistent KPIs.

Optimal Market Making under Partial Information with General Intensities

ABSTRACT Starting from the Avellaneda–Stoikov framework, we consider a market maker who wants to optimally set bid/ask quotes over a finite time horizon, to maximize her expected utility. The intensities of the orders she receives depend not only on the spreads she quotes but also on unobservable factors modelled by a hidden Markov chain. We tackle this stochastic control problem under partial information with a model that unifies and generalizes many existing ones under full information, combining several risk metrics and constraints, and using general decreasing intensity functionals. We use stochastic filtering, control and piecewise-deterministic Markov processes theory, to reduce the dimensionality of the problem and characterize the reduced value function as the unique continuous viscosity solution of its dynamic programming equation. We then solve the analogous full information problem and compare the results numerically through a concrete example. We show that the optimal full information spreads are biased when the exact market regime is unknown, and the market maker needs to adjust for additional regime uncertainty in terms of P&L sensitivity and observed order flow volatility. This effect becomes higher, the longer the waiting time in between orders.

A Stochastic Control Approach to Fight COVID-19

In this paper, we present several optimal control approaches which are designed to control and regulate the evolution of infection numbers in the COVID-19 disease. In our setting, the number of infected people at a certain location is modeled by a continuous-time stochastic process which can be affected by a related stochastic control process. We use mathematical tools from stochastic analysis and optimal control theory. In particular, we prove an innovative stochastic maximum principle for continuous-state branching processes with immigration (so-called CBI processes) and apply the result to a stochastic control problem stemming from epidemiology.

Pervasive and diverse collateral sensitivity profiles inform optimal strategies to limit antibiotic resistance.

Evolved resistance to one antibiotic may be associated with "collateral" sensitivity to other drugs. Here, we provide an extensive quantitative characterization of collateral effects in Enterococcus faecalis, a gram-positive opportunistic pathogen. By combining parallel experimental evolution with high-throughput dose-response measurements, we measure phenotypic profiles of collateral sensitivity and resistance for a total of 900 mutant–drug combinations. We find that collateral effects are pervasive but difficult to predict because independent populations selected by the same drug can exhibit qualitatively different profiles of collateral sensitivity as well as markedly different fitness costs. Using whole-genome sequencing of evolved populations, we identified mutations in a number of known resistance determinants, including mutations in several genes previously linked with collateral sensitivity in other species. Although phenotypic drug sensitivity profiles show significant diversity, they cluster into statistically similar groups characterized by selecting drugs with similar mechanisms. To exploit the statistical structure in these resistance profiles, we develop a simple mathematical model based on a stochastic control process and use it to design optimal drug policies that assign a unique drug to every possible resistance profile. Stochastic simulations reveal that these optimal drug policies outperform intuitive cycling protocols by maintaining long-term sensitivity at the expense of short-term periods of high resistance. The approach reveals a new conceptual strategy for mitigating resistance by balancing short-term inhibition of pathogen growth with infrequent use of drugs intended to steer pathogen populations to a more vulnerable future state. Experiments in laboratory populations confirm that model-inspired sequences of four drugs reduce growth and slow adaptation relative to naive protocols involving the drugs alone, in pairwise cycles, or in a four-drug uniform cycle.

On Stochastic Optimal Control in Ferromagnetism

A model is proposed to, e.g., control the domain wall motion in ferromagnets in the presence of thermal fluctuations, and the existence of an optimal stochastic control process is proved. The convergence of a finite element approximation of the problem is shown in space-dimension one, which then allows one to apply Pontryagin’s maximum principle for this finite dimensional setting. The resulting coupled system of forward-backward stochastic differential equations is numerically solved by means of the stochastic gradient method to enable practical simulations.

Adaptive Energy Aware Quality of Service for Reliable Data Transfer in Under Water Acoustic Sensor Networks

Currently, reliable data transfer, and energy management have been considered as a significant research challenge in the underwater acoustic sensor networks (UWASN) owing to high packet loss, limited ratio of bandwidth with significant incur of energy, network life time with high propagation delay, less precision with high data hold time and so on. Energy saving and maintaining quality of service (QoS) is more important for UWASN owing to QoS application necessity and limited sensor nodes. To address this issue, several existing algorithms such as adaptive data forwarding algorithms, QoS-based congestion control algorithms and several methodologies were proposed with high throughput and less network lifetime as well as the less utilization of energy in UWASN by choosing sensor nodes data based on data transfer and link reliability. However, all the conventional algorithms have fixed data hold time, which incurs more end-to-end delay with less reliability of data and consumption of high energy due to high data transfer reachability. This high end research proposes adaptive energy aware quality of service (AEA-QoS) algorithm for reliable data delivery by formulating discrete times stochastic control process and deep learning techniques for UWSAN to overcome these issues. The proposed algorithm has been validated with conventional state-of-the-art methods and results show that the proposed approach exhibits its effectiveness in terms of less network overhead and propagation delay with high throughput and less energy consumption for every reliable packet transmission.

Optimal Market Making Under Partial Information With General Intensities

Starting from the Avellaneda–Stoikov framework, we consider a market maker who wants to optimally set bid/ask quotes over a finite time horizon, to maximize her expected utility. The intensities of the orders she receives depend not only on the spreads she quotes, but also on unobservable factors modelled by a hidden Markov chain. We tackle this stochastic control problem under partial information with a model that unifies and generalizes many existing ones under full information, combining several risk metrics and constraints, and using general decreasing intensity functionals. We use stochastic filtering, control and piecewise-deterministic Markov processes theory, to reduce the dimensionality of the problem and characterize the reduced value function as the unique continuous viscosity solution of its dynamic programming equation. We then solve the analogous full information problem and compare the results numerically through a concrete example. We show that the optimal full information spreads are biased when the exact market regime is unknown, and the market maker needs to adjust for additional regime uncertainty in terms of P&L sensitivity and observed order flow volatility. This effect becomes higher, the longer the waiting time in between orders.

The Theory of Optimal Stochastic Control as Applied to Insurance Underwriting Cycles

We use the theories of optimal stochastic control and engineering process control to analyze the well-known phenomenon of insurance underwriting cycles in continuous time. We show in a continuous time framework that underwriting cycles can be explained with a model where premiums are set rationally, but where there are various reporting and regulatory lags. We find that the observed cycle length depends on the length of these underlying lags. Our result can be seen as consistent with previous empirical work showing underwriting cycles varying across countries and lines of insurance. In the event that no lags exist, our result is also consistent with more recent literature suggesting that insurance cycles may not exist.

Variational and optimal control representations of conditioned and driven processes

.We have shown recently that a Markov process conditioned on rare events involving time-integrated random variables can be described in the long-time limit by an effective Markov process, called the driven process, which is given mathematically by a generalization of Doob’s h-transform. We show here that this driven process can be represented in two other ways: first, as a process satisfying various variational principles involving large deviation functions and relative entropies and, second, as an optimal stochastic control process minimizing a cost function also related to large deviation functions. These interpretations of the driven process generalize and unify many previous results on maximum entropy approaches to nonequilibrium systems, spectral characterizations of positive operators, and control approaches to large deviation theory. They also lead, as briefly discussed, to new methods for analytically or numerically approximating large deviation functions.

Cross-Layer Handoff Design in MIMO-Enabled WLANs for Communication-Based Train Control (CBTC) Systems

Communication-Based Train Control (CBTC) system is an automated train control system using bidirectional train-ground communications to ensure the safe operation of rail vehicles. Handoff design has significant impacts on the train control performance in CBTC systems based on multi-input and multi-output (MIMO)-enabled WLANs. Most of previous works use traditional design criteria, such as network capacity and communication latency, in handoff designs. However, these designs do not necessarily benefit the train control performance. In this paper, we take an integrated design approach to jointly optimize handoff decisions and physical layer parameters to improve the train control performance in CBTC systems. We use linear quadratic cost for the train controller as the performance measure. The handoff decision and physical layer parameters adaptation problem is formulated as a stochastic control process. Simulation result shows that the proposed approach can significantly improve the control performance in CBTC systems.

About Our Authors

About Our Authors

Generalized solutions to nonlinear stochastic differential equations with vector--valued impulsive controls

We develop a notion of generalized solution to a stochastic differential equation depending in a nonlinear way on a vector--valued stochastic control process $\{U_t\},$ merely of bounded variation, and on its derivative. Our results rely on the concept of Lipschitz continuous graph completion of $\{U_t\}$ and the generalized solution turns out to coincide a.e. with the limit of classical solutions to (1). In the linear case our notion of solution is equivalent to the usual one in distributional sense. We prove that the generalized solution does not depend on the particular graph-completion of the control process $\{U_t\}$ both for vector-valued controls under a suitable commutativity condition and for scalar controls.

An importance sampling procedure for estimating failure probabilities of non-linear dynamic systems subjected to random noise

The first passage problem for linear and non-linear oscillators excited by white and coloured noise are considered. An iterative variance reduction scheme is used in a framework of a measure change in the space of sample functions according to the Girsanov transformation, which is based on introducing a Markov control process. It is proved that a good approximation to the optimal stochastic control process can be obtained from an equivalent white noise excited linear oscillator. It is shown that this leads to very accurate estimates of the failure probability of the original system. The advantage of this procedure is that expressions for the parameters of the equivalent linear system and the design point oscillations, which are needed to find the control process, are available analytically. The number of samples, the variance of the failure probability estimates and the computational time are reduced significantly compared with direct Monte Carlo simulations.

Dependence of Left-Skewed Payoff Distributions on Risky-Asset Price Uncertainty

A left-skewed payoff distribution ascribes more probability to higher payoffs than a right-skewed distribution. The former is a desirable feature of payoff spreading if losses are strongly avoided. Therefore, investment policies leading to left-skewed payoff distributions are of interest to practical fund managers where the probability of loss needs be minimal (e.g., in pension funds). In this paper, features of stochastic optimal control processes, under which left-skewed payoff distributions can be obtained, are examined. In particular, the diffusion parameter that represents risky-asset price uncertainty is varied, and sensitivity of the payoff distribution due to this variation is studied. Interestingly, left-skewed payoff distributions can be obtained for relatively large values of the diffusion parameter.

Control chart and stochastic control processes

In this paper, we shall discuss the relation between the expected cost model for a process and the probabilistic control model, whose mean is controlled by an X chart. T. Odanaka has exploited the problem of minimizing the membership function or the probability that the performance level of the machine will run out of range in a fuzzy environment. This paper considers the same basic problem of the expected cost model, modified to include the set up costs for controlling the action in any period. The expected cost comprises the fixed and variable cost of taking a control action, the cost of sampling, the cost of investigating the process when the control chart indicates that the process parameters exceed the specified bound, and the cost of producing defective units. Dynamic programming can be used to determine optimal control policies for models where control produces economically measurable benefits and/or costs. When the controlling action incurs a set up charge, the optimal policies are found to be characterized by a pair ( s n , S n , S′ n , s′ n ), where the action is made in a period n to state S n or S′ n ( S n < S′ n ), if the state is found to be outside s n or s′ n ( s n < s′ n ). This statement is subject to a set of basic assumptions such that proportional costs of changing the state variable are zero, the two fixed costs are equal, the loss function is symmetric quasi-convex and the problem's probability densities are quasi-concave.

Stochastic control processes and management science

In the stochastic control processes, the criterion which minimizes the probability that the state variables of the overall stage exceed the fixed level is probably of greater importance. Although it does not resolve the arbitrariness inherent in the choice of a criterion of performance, this is a conforming result. The paper is divided into three parts. First, the mathematical formulation, the theorem of existence and uniqueness of solution, and the optimal policy in the equations created from our stochastic control processes are considered. Second, problems involving some stochastic control processes in a fuzzy environment are considered. Third, applications of the inventory control problem, the statistical quality control processes, and a control process connected with the Van der Pol equation are presented. Finally, the future problems are discussed.