Ergodic Problem Research Articles

Time-optimal ergodic search: Multiscale coverage in minimum time

Search and exploration capabilities are essential for robots to inspect hazardous areas, support scientific expeditions in extreme environments, and potentially save human lives in natural disasters. The variability of scale in these problems requires robots to reason about time alongside their dynamics and sensor capabilities to effectively assess and explore for information. Recent advances in ergodic search methods have shown promise in supporting trajectory planning for exploration in continuous, multiscale environments with dynamics consideration. However, these methods are still limited by their inability to effectively reason about and adapt the time to explore in response to their environment. This ability is crucial for adapting exploration to variable-resolution information-gathering tasks. To address this limitation, this paper poses the time-optimal ergodic search problem and investigates solutions for fast, multiscale, and adaptive robotic exploration trajectories. The problem is formulated as a minimum-time problem with an ergodic inequality constraint whose upper bound specifies the amount of coverage needed. We show the existence of optimal solutions using Pontryagin’s conditions of optimality, and we demonstrate effective, minimum-time coverage numerically through a direct transcription optimization approach. The efficacy of the approach in generating time-optimal search trajectories is demonstrated in simulation under several nonlinear dynamic constraints, and in a physical experiment using a drone in a cluttered environment. We find that constraints such as obstacle avoidance are readily integrated into our formulation, and we show through an ablation study the flexibility of search capabilities at various scales. Last, we contribute a receding-horizon formulation of time-optimal ergodic search for sensor-driven information-gathering and demonstrate improved adaptive sampling capabilities in localization tasks.

Drift Control of High-Dimensional Reflected Brownian Motion: A Computational Method Based on Neural Networks

Motivated by applications in queueing theory, we consider a stochastic control problem whose state space is the d-dimensional positive orthant. The controlled process Z evolves as a reflected Brownian motion whose covariance matrix is exogenously specified, as are its directions of reflection from the orthant’s boundary surfaces. A system manager chooses a drift vector [Formula: see text] at each time t based on the history of Z, and the cost rate at time t depends on both [Formula: see text] and [Formula: see text]. In our initial problem formulation, the objective is to minimize expected discounted cost over an infinite planning horizon, after which we treat the corresponding ergodic control problem. Extending the earlier work by Han et al. [Han J, Jentzen A, Weinan E (2018) Solving high-dimensional partial differential equations using deep learning. Proc. Natl. Acad. Sci. USA 115(34):8505–8510], we develop and illustrate a simulation-based computational method that relies heavily on deep neural network technology. For the test problems studied thus far, our method is accurate to within a fraction of 1% and is computationally feasible in dimensions up to at least [Formula: see text].

Some Remarks on Linear-Quadratic Closed-Loop Games with Many Players

AbstractWe identify structural assumptions which provide solvability of the Nash system arising from a linear-quadratic closed-loop game, with stable properties with respect to the number of players. In a setting of interactions governed by a sparse graph, both short-time and long-time existence of a classical solution for the Nash system set in infinitely many dimensions are addressed, as well as convergence to the solution to the respective ergodic problem as the time horizon goes to infinity; in addition, equilibria for the infinite-dimensional game are shown to provide $$\epsilon $$ ϵ -Nash closed-loop equilibria for the N-player game. In a setting of generalized mean-field type (where the number of interactions is large but not necessarily symmetric), directly from the N-player Nash system estimates on the value functions are deduced on an arbitrary large time horizon, which should pave the way for a convergence result as N goes to infinity.

A numerical method for ergodic optimal control of switching diffusions with reflection

The ergodic or long run average cost control problem for diffusions is one of the classical problems of stochastic control that still eludes a completely satisfactory treatment. This is certainly true for the setting in which the systems to be controlled is modeled by the solution of a switching stochastic differential equation with reflection (SSDER). In this paper, we advance in this rather difficult problem by setting forth preliminary results for the unidimensional case. Besides carving out a numerical method, our treatment of the ergodic control in this scenario straddles issues of existence and uniqueness of solution of the SSDER and a verification theorem for the associated HJB equation. We conclude by illustrating the effectiveness of the method considering the control of energy consumption in a large parallel processing computer system composed of one queue and several processing stations.

Optimal Power Allocation in Optical GEO Satellite Downlinks Using Model-Free Deep Learning Algorithms

Geostationary (GEO) satellites are employed in optical frequencies for a variety of satellite services providing wide coverage and connectivity. Multi-beam GEO high-throughput satellites offer Gbps broadband rates and, jointly with low-Earth-orbit mega-constellations, are anticipated to enable a large-scale free-space optical (FSO) network. In this paper, a power allocation methodology based on deep reinforcement learning (DRL) is proposed for optical satellite systems disregarding any channel statistics knowledge requirements. An all-FSO, multi-aperture GEO-to-ground system is considered and an ergodic capacity optimization problem for the downlink is formulated with transmitted power constraints. A power allocation algorithm was developed, aided by a deep neural network (DNN) which is fed channel state information (CSI) observations and trained in a parameterized on-policy manner through a stochastic policy gradient approach. The proposed method does not require the channels’ transition models or fading distributions. To validate and test the proposed allocation scheme, experimental measurements from the European Space Agency’s ARTEMIS optical satellite campaign were utilized. It is demonstrated that the predicted average capacity greatly exceeds other baseline heuristic algorithms while strongly converging to the supervised, unparameterized approach. The predicted average channel powers differ only by 0.1 W from the reference ones, while the baselines differ significantly more, about 0.1–0.5 W.

A Viscous Ergodic Problem with Unbounded and Measurable Ingredients, Part 1: HJB Equation

A Viscous Ergodic Problem with Unbounded and Measurable Ingredients, Part 1: HJB Equation

Ergodic problems for second-order mean field games with state constraints

Ergodic problems for second-order mean field games with state constraints

Nonlocal ergodic control problem in $${\mathbb {R}}^d$$

Nonlocal ergodic control problem in $${\mathbb {R}}^d$$

Dual stochastic descriptions of streamflow dynamics under model ambiguity through a Markovian embedding

Hamilton–Jacobi–Bellman equation (HJBE) and backward stochastic differential equation (BSDE) are the two faces of stochastic control. We explore their equivalence focusing on a system of self-exciting and affine stochastic differential equations (SDEs) arising in streamflow dynamics. Our SDE is a finite-dimensional Markovian embedding of an infinite-dimensional jump-driven process called the superposition of continuous-state branching processes (a supCBI process). We formulate new ergodic control problems to evaluate the worst-case streamflow discharge in the long run and derive their HJBEs and ergodic BSDEs. The constant ambiguity aversion classically used in assessing model ambiguity must be modified in our case so that the optimality equations become well-posed. With a suitable modification of the ambiguity-aversion coefficient depending on the distributed reversion speed, we demonstrate that the solutions to the optimality equations are equivalent to each other in the sense that they lead to the same result. Finally, we apply the proposed framework to the computation of realistic cases with an existing record of discharge through a numerical Markovian embedding.

A New Monotonicity Condition for Ergodic Backward SDEs and Ergodic Control with Superquadratic Hamiltonians

We establish the existence (and in an appropriate sense uniqueness) of Markovian solutions for ergodic backward SDEs (BSDEs) under a novel monotonicity condition. Our monotonicity condition allows us to prove existence even when the driver has arbitrary (in particular superquadratic) growth in , which reveals an interesting trade-off between monotonicity and growth for ergodic BSDEs. The technique of proof is to establish a probabilistic representation of the derivative of the Markovian solution and then use this representation to obtain a priori estimates. Our study is motivated by applications to ergodic control, and we use our existence result to prove the existence of optimal controls for a class of ergodic control problems with potentially superquadratic Hamiltonians. We also treat a class of drivers coming from the construction of forward performance processes and interpret our monotonicity condition in this setting.

Ergodic control of McKean–Vlasov SDEs and associated Bellman equation

We consider the ergodic control problem for McKean-Vlasov stochastic differential equations and prove the existence and uniqueness of the viscosity solution to the associated fully nonlinear HJB equation in a lifted sense. Furthermore, as the time horizon goes to infinity, we show that the solutions of finite-horizon time-averaging optimal control problems converge to that of the ergodic control problem. Our results require dissipativity conditions and dissipativity-like conditions on distribution variables of both drift and diffusion coefficients.

Boundary value problems of potential theory for the exterior ball and the approximation and ergodic behaviour of the solutions

The paper is concerned with the interconnection of the boundary behaviour of the solutions of the exterior Dirichlet, Neumann and Robin problems of harmonic analysis for the unit ball in R3 with the corresponding behaviour of the associated ergodic inverse problems for the entire space. Rates of approximation play a basic role.The solutions themselves are evaluated by means of Fourier expansions with respect to spherical harmonics. In case of the first two problems, the basis for the investigation of the approximation and ergodic behaviour is the theory of semigroups of linear operators mapping a Banach space X into itself. The connection between the semigroup property and the major premise of Huygens’ principle is emphasized.Another tool is a Drazin-like inverse operator Aad for the infinitesimal generator A of a semigroup that arises naturally in ergodic theory. This operator is a closed, not necessarily bounded, operator. It was introduced in a paper with U. Westphal (Butzer and Westphal, 1970/71) and extended to a generalized setting with J.J. Koliha (Butzer and Koliha, 2009).Unlike the latter two problems, the solution of Robin’s problem does not have the semigroup property and therefore the semigroup methods applied to Dirichlet’s and Neumann’s problem do not work. The authors give several hints how to overcome these difficulties.

Ergodic switching control for diffusion-type processes

We consider the control of the discrete component $ n_t $ of a switching Markov process $ x_t = ( z_t, n_t) $ when there is a running cost and an immediate cost $ c(i, j) $ for switching $ n_t $ from $ i $ to $ j $. We study the minimization of the ergodic (or long-term average) total cost. Essentially, this paper treats the case where, for $ n_t=n $ fixed, $ z_t $ is a reflected diffusion or a reflected diffusion with jumps, $ n_t $ being, for fixed $ z $, a continuous-time Markov chain. Using the vanishing discount approach, we extend existing results dealing with the situation where $ n_t $ evolves only by the switching control action and the diffusion is non-degenerate. Moreover, we solve the ergodic problem for a class of diffusions which can be degenerate and for an example with absorbing state.

Feedback stabilization of quasi nonintegrable Hamiltonian systems under combined Gaussian and Poisson white noise excitations

A procedure for designing a feedback control to asymptotic Lyapunov stability with probability one of quasi nonintegrable Hamiltonian systems under combined Gaussian and Poisson white noise excitations is proposed. First, a one dimensional partially averaged Itô stochastic differential equation for controlled Hamiltonian is derived from the motion equations of the system by using the stochastic averaging method. Second, the dynamical programming equation for the ergodic control problem of the averaged system with undetermined cost function is set up based on the dynamical programming principle and the jump–diffusion chain stochastic differential rules. The optimal control law is obtained by solving the dynamical programming equation. Third, the analytical expression for the largest Lyapunov exponent of the averaged system is derived. Finally, the asymptotic Lyapunov stability with probability one of the originally controlled system is analyzed approximately by using the largest Lyapunov exponent. The cost function and optimal control forces are determined by the requirements of stabilizing the system. An example is worked out in detail to illustrate the effectiveness of the proposed method for stabilization control, and the control effect of the proposed feedback stabilization varies with the change of parameters is also studied in this paper, such as, the greater the excitation intensity of Gaussian and Poisson white noise, the better the stabilization control effect.

Ergodic risk-sensitive control for regime-switching diffusions

In this article, we study the ergodic risk-sensitive control problem for controlled regime-switching diffusions. Under a blanket stability hypothesis, we solve the associated nonlinear eigenvalue problem for weakly coupled systems and characterize the optimal stationary Markov controls via a suitable verification theorem. We also consider the near-monotone case and obtain the existence of principal eigenfunction and optimal stationary Markov controls.

Stochastic stabilization of multi‐degree‐of‐freedom nonlinear random‐time‐delay controlled systems

AbstractThe random‐time‐delay (RTD) occurs inevitably in feedback control, caused by the utilization of a multi‐user network with random demands. This paper is devoted to the stochastic stabilization of multi‐degree‐of‐freedom (MDOF) nonlinear RTD controlled systems. Firstly, a two‐step approximated method is proposed to deal with RTD: by modeling the RTD as a Markov jump process, the RTD control is formulated as the one with discrete jump varying delay; utilizing the randomly periodic characteristic, the Markov jump delay is taken out of the control as a parameter. The stochastic averaging principle is applied to generate a less‐dimensional averaged governing equation of system's energies. Secondly, the dynamical programming principle is applied to the ergodic control problem of the average systems with an undetermined cost function, which is determined by the demand of stabilizing the averaged systems, then the optimal control law is deduced. A largest Lyapunov exponent is introduced to the averaged systems to evaluate the asymptotic stability with probability one. Finally, an example is worked out to demonstrate the application and effectiveness of the proposed method.

Stochastic optimization of a mixed moving average process for controlling non-Markovian streamflow environments

We investigated a cost-constrained static ergodic control problem of the variance of measure-valued affine processes and its application in streamflow management. The controlled system is a jump-driven mixed moving average process that generates realistic subexponential autocorrelation functions, and the “static” nature of the control originates from a realistic observability assumption in the system. The Markovian lift was effectively used to discretize the system into a finite-dimensional process, which is easier to analyze. The resolution of the problem is based on backward Kolmogorov equations and a quadratic solution ansatz. The control problem has a closed-form solution, and the variance has both strict upper and lower bounds, indicating that the variance cannot take an arbitrary value even when it is subject to a high control cost. The correspondence between the discretized system based on the Markovian lift and the original infinite-dimensional one is discussed. Then, a convergent Markovian lift is presented to approximate the infinite-dimensional system. Finally, the control problem was applied to real cases using available data for a river reach. An extended problem subject to an additional constraint on maintaining the flow variability was also analyzed without significantly degrading the tractability of the proposed framework.

Intelligent Surface Aided D2D-V2X System for Low-Latency and High-Reliability Communications

With low-cost energy consumption, the reconfigurable intelligent surface (RIS) technique is a potential solution to the real-time data processing for intelligent transportation systems (ITSs). In this paper, an intelligent transmissive surface is introduced into the vehicular communications, enabling vehicle-to-infrastructure (V2I) signals to penetrate the intelligent RIS to access the base station (BS) on the opposite side of the vehicle. Considering that the vehicle-to-vehicle (V2V) communication reuses the spectrum spanned for V2I link, we investigate the ergodic capacity optimization problem for the vehicle performing V2I communications with the assistance of RIS, while meeting the low-latency and high-reliability requirements of the V2V link. The RIS transmission coefficients and power allocation of vehicles are jointly optimized, for the management of the desired and undesired vehicular communication links. Moreover, the expression of optimal phase shifts is derived in a closed-form, which reveals that the performance gain brought by RIS is proportional to the number of intelligent elements, while inversely proportional to the distance from vehicle-to-BS, in a quadratic form. Moreover, in the case of discrete phase shifts, an intelligent algorithm is proposed for the beamforming design at RIS. Afterwards, with the objective to maximize the ergodic capacity of the V2I link, the optimal power allocation is also proposed. Simulation results confirm the accuracy of the proposed resource allocation strategy, and that the system performance in terms of the ergodic V2I capacity can be significantly improved by the RIS.

Multi-Pair D2D Communications Aided by an Active RIS Over Spatially Correlated Channels With Phase Noise

This paper investigates a multi-pair device-to-device (D2D) communication system aided by an active reconfigurable intelligent surface (RIS) with phase noise and direct link. The approximate closed-form expression of the ergodic sum rate is derived over spatially correlated Rician fading channels with statistical channel state information (CSI). When the Rician factors go to infinity, the asymptotic expressions of the ergodic sum rates are presented to give insights in poor scattering environment. The power scaling law for the special case of a single D2D pair is presented without phase noise under uncorrelated Rician fading condition. Then, to solve the ergodic sum rate maximization problem, a method based on genetic algorithm (GA) is proposed for joint power control and discrete phase shifts optimization. Simulation results verify the accuracy of our derivations, and also show that the active RIS outperforms the passive RIS.

Enhanced Gravitational Search Optimization with Hybrid Deep Learning Model for COVID-19 Diagnosis on Epidemiology Data.

Effective screening provides efficient and quick diagnoses of COVID-19 and could alleviate related problems in the health care system. A prediction model that combines multiple features to assess contamination risks was established in the hope of supporting healthcare workers worldwide in triaging patients, particularly in situations with limited health care resources. Furthermore, a lack of diagnosis kits and asymptomatic cases can lead to missed or delayed diagnoses, exposing visitors, medical staff, and patients to 2019-nCoV contamination. Non-clinical techniques including data mining, expert systems, machine learning, and other artificial intelligence technologies have a crucial role to play in containment and diagnosis in the COVID-19 outbreak. This study developed Enhanced Gravitational Search Optimization with a Hybrid Deep Learning Model (EGSO-HDLM) for COVID-19 diagnoses using epidemiology data. The major aim of designing the EGSO-HDLM model was the identification and classification of COVID-19 using epidemiology data. In order to examine the epidemiology data, the EGSO-HDLM model employed a hybrid convolutional neural network with a gated recurrent unit based fusion (HCNN-GRUF) model. In addition, the hyperparameter optimization of the HCNN-GRUF model was improved by the use of the EGSO algorithm, which was derived by including the concepts of cat map and the traditional GSO algorithm. The design of the EGSO algorithm helps in reducing the ergodic problem, avoiding premature convergence, and enhancing algorithm efficiency. To demonstrate the better performance of the EGSO-HDLM model, experimental validation on a benchmark dataset was performed. The simulation results ensured the enhanced performance of the EGSO-HDLM model over recent approaches.