Discrete-time Process Research Articles

A direct discretization recurrent neurodynamics method for time-variant nonlinear optimization with redundant robot manipulators

Discrete time-variant nonlinear optimization (DTVNO) problems are commonly encountered in various scientific researches and engineering application fields. Nowadays, many discrete-time recurrent neurodynamics (DTRN) methods have been proposed for solving the DTVNO problems. However, these traditional DTRN methods currently employ an indirect technical route in which the discrete-time derivation process requires to interconvert with continuous-time derivation process. In order to break through this traditional research method, we develop a novel DTRN method based on the inspiring direct discrete technique for solving the DTVNO problem more concisely and efficiently. To be specific, firstly, considering that the DTVNO problem emerging in the discrete-time tracing control of robot manipulator, we further abstract and summarize the mathematical definition of DTVNO problem, and then we define the corresponding error function. Secondly, based on the second-order Taylor expansion, we can directly obtain the DTRN method for solving the DTVNO problem, which no longer requires the derivation process in the continuous-time environment. Whereafter, such a DTRN method is theoretically analyzed and its convergence is demonstrated. Furthermore, numerical experiments confirm the effectiveness and superiority of the DTRN method. In addition, the application experiments of the robot manipulators are presented to further demonstrate the superior performance of the DTRN method.

Limit theorems for discrete multitype branching processes counted with a characteristic

For a discrete time multitype supercritical Galton–Watson process (Zn)n∈N and corresponding genealogical tree T, we associate a new discrete time process (ZnΦ)n∈N such that, for each n∈N, the contribution of each individual u∈T to ZnΦ is determined by a (random) characteristic Φ evaluated at the age of u at time n. In other words, ZnΦ is obtained by summing over all u∈T the corresponding contributions Φu, where (Φu)u∈T are i.i.d. copies of Φ. Such processes are known in the literature under the name of Crump–Mode–Jagers (CMJ) processes counted with characteristicΦ. We derive a LLN and a CLT for the process (ZnΦ)n∈N in the discrete time setting, and in particular, we show a dichotomy in its limit behavior. By applying our main result, we also obtain a generalization of the results in Kesten and Stigum (1966).

Risk of Stochastic Systems for Temporal Logic Specifications

The wide availability of data coupled with the computational advances in artificial intelligence and machine learning promise to enable many future technologies such as autonomous driving. While there has been a variety of successful demonstrations of these technologies, critical system failures have repeatedly been reported. Even if rare, such system failures pose a serious barrier to adoption without a rigorous risk assessment. This article presents a framework for the systematic and rigorous risk verification of systems. We consider a wide range of system specifications formulated in signal temporal logic (STL) and model the system as a stochastic process, permitting discrete-time and continuous-time stochastic processes. We then define the STL robustness risk as the risk of lacking robustness against failure . This definition is motivated as system failures are often caused by missing robustness to modeling errors, system disturbances, and distribution shifts in the underlying data generating process. Within the definition, we permit general classes of risk measures and focus on tail risk measures such as the value-at-risk and the conditional value-at-risk. While the STL robustness risk is in general hard to compute, we propose the approximate STL robustness risk as a more tractable notion that upper bounds the STL robustness risk. We show how the approximate STL robustness risk can accurately be estimated from system trajectory data. For discrete-time stochastic processes, we show under which conditions the approximate STL robustness risk can even be computed exactly. We illustrate our verification algorithm in the autonomous driving simulator CARLA and show how a least risky controller can be selected among four neural network lane-keeping controllers for five meaningful system specifications.

PROPERTIES OF DISCRETE-TIME CONDITIONAL LINEAR CYCLOSTATIONARY RANDOM PROCESSES IN THE PROBLEMS OF ENERGY INFORMATICS

Modern challenges in the energy industry require comprehensive research in the field of energy informatics, which combines computer science, control systems, and energy management systems within a single methodology. An important area of energy informatics is the study of problems of systems and processes modeling in energy, including energy loads and consumption. Linear and conditional linear random processes (CLRP) are mathematical models of signals represented as the sum of a large number of random impulses occurring at random times. The energy consumption, vibration signals of energy objects, etc. can be modeled using this approach. A variant of the CLRP model with discrete time, taking into account the cyclic properties of energy consumption, has been investigated in the paper. The goal is to justify the conditions for the discrete-time CLRP to be a periodically correlated random process, as well as a cyclostationary process. It has been shown that the corresponding conditions depend on the periodicity of the probability distributions of the kernel and the generating white noise of the CLRP representation. To achieve the goal, the properties of mathematical expectation and covariance function of CLRP, as well as the method of characteristic functions, have been used. The paper proves that the discrete-time CLRP is a periodically correlated random sequence if the generating white noise has periodic mathematical expectation and variance, and the kernel is a periodically correlated random field. Based on the analysis of the multivariate characteristic function, it has been proven that the discrete-time CLRP is cyclostationary if the generating white noise is a cyclostationary process and the kernel is a cyclostationary random field. The properties of discrete-time conditional linear cyclostationary random processes are important for mathematical modeling, simulation, statistical analysis, and forecasting of energy consumption. Keywords: mathematical model, energy informatics, conditional linear random process, cyclostationary process, white noise, characteristic function.

Significant wave height modelling and simulation of the monsoon-influenced South China Sea coast

The modelling of ocean wave dynamics and future possible scenario simulations is crucial as coastal communities with offshore and shipping industries can be impacted. Therefore, this study modelled and simulated the significant wave height (SWH) dynamics for both long-term (decadal) and northeast monsoons using a discrete-time stochastic process. These processes were successfully performed by applying the data from the European Centre for Medium-Range Weather Forecasts (ECMWF) database at the offshore point of Terengganu, Malaysia, in the South China Sea. The SWH time series were analysed and modelled by decomposing the series into non-stationary and stationary components. Consequently, the five-term Fourier series was adequate to model the non-stationary component for long-term and northeast monsoon time series. Meanwhile, the stationary component followed the autoregressive AR(2) process with stationary white noise residual fitted to t location-scale distribution. The model simulated the significant wave, which well-represented the time series behaviour. Thus, the simulation outcome could contribute to coastal response studies. These contributions included long-term prediction of coastal response, investigation of coastal response during the northeast monsoon season, and further development of other seasonal component models influencing the South China Sea.

Risk-Sensitive Average Optimality for Discrete-Time Markov Decision Processes

In this paper we study the risk-sensitive average optimality for discrete-time Markov decision processes with denumerable states and unbounded costs. We derive the multiplicative Poisson equation under the suitable ergodicity conditions via an approximation method. Moreover, we prove the existence of a unique solution to the risk-sensitive average cost optimality equation and give an equivalent characterization of the set of all optimal stationary policies. Finally, we present the policy iteration algorithm and show its convergence.

Multiplexing or Diversity: AoI-Oriented Short-Packet Transmission Over Fading Channels

Multiplexing or transmission diversity via dual links enables the reduction in age of information (AoI). We address the open issue of selection between the two transmission modes for AoI-oriented short-packet system over fading channels, with a comprehensive analysis. Closed-form expressions for the average AoI and peak AoI (PAoI) are derived based on the discrete-time Markov-chain process. Then, to obtain the explicit region of preference (RoP) and quantitative PAoI gains of multiplexing/diversity over the single-queue case, we derive the signal-to-noise ratio (SNR) threshold for transmission mode selection, which is shown to be a decreasing function of the arrival rate and saturates at high arrival rate. Also, the monotonicity of the PAoI gains by multiplexing and diversity, and their achievable gains, are analyzed in a comprehensive manner. It is shown that diversity is able to achieve a PAoI gain of more than 3 dB over the single-queue case at low SNR, while multiplexing has a larger RoP than diversity and is selected at high SNR and high arrival rate. Furthermore, both throughput and PAoI violation probability are considered alongside the average PAoI for a wide range of tradeoff in system design.

Semi-Markovian Discrete-Time Telegraph Process with Generalized Sibuya Waiting Times

In a recent work we introduced a semi-Markovian discrete-time generalization of the telegraph process. We referred to this random walk as the ‘squirrel random walk’ (SRW). The SRW is a discrete-time random walk on the one-dimensional infinite lattice where the step direction is reversed at arrival times of a discrete-time renewal process and remains unchanged at uneventful time instants. We first recall general notions of the SRW. The main subject of the paper is the study of the SRW where the step direction switches at the arrival times of a generalization of the Sibuya discrete-time renewal process (GSP) which only recently appeared in the literature. The waiting time density of the GSP, the ‘generalized Sibuya distribution’ (GSD), is such that the moments are finite up to a certain order r≤m−1 (m≥1) and diverging for orders r≥m capturing all behaviors from broad to narrow and containing the standard Sibuya distribution as a special case (m=1). We also derive some new representations for the generating functions related to the GSD. We show that the generalized Sibuya SRW exhibits several regimes of anomalous diffusion depending on the lowest order m of diverging GSD moment. The generalized Sibuya SRW opens various new directions in anomalous physics.

Determination of the characteristic function of discrete-time conditional linear random process and its application

The discrete-time conditional linear random process is defined, and its properties in the context of application for mathematical modelling of information signals in energy and medicine are analyzed. The relation to the continuous-time counterpart is considered on the basis of time sampling and aggregation. One-dimensional and multidimensional characteristic functions of discrete-time conditional linear random process are obtained using conditional characteristic function approach. The conditions for the investigated model to be strict sense stationary are justified.

The optimal probability of the risk for finite horizon partially observable Markov decision processes

<abstract><p>This paper investigates the optimality of the risk probability for finite horizon partially observable discrete-time Markov decision processes (POMDPs). The probability of the risk is optimized based on the criterion of total rewards not exceeding the preset goal value, which is different from the optimal problem of expected rewards. Based on the Bayes operator and the filter equations, the optimization problem of risk probability can be equivalently reformulated as filtered Markov decision processes. As an advantage of developing the value iteration technique, the optimality equation satisfied by the value function is established and the existence of the risk probability optimal policy is proven. Finally, an example is given to illustrate the effectiveness of using the value iteration algorithm to compute the value function and optimal policy.</p></abstract>

Energy-Efficient Optimal Sensor Scheduling for State Estimation Over Multihop Sensor Networks.

In this article, we consider the power scheduling problem of the multihop transmission with limited power resources. For a discrete-time linear time-invariant process, we consider a more practical scenario where the forward-error-correcting (FEC) coding scheme is utilized. An approximate communication model is introduced to formulate the nonanalytical relationship between the consumption of power and the successful-decoding-probability. For the single-hop transmission, we propose an analytical method to figure out the optimal offline scheduling for the finite-time case and the optimal periodic schedule for the infinite-time case. We consider the process and terminal errors simultaneously, and explicitly discuss how different values of parameters affect the optimality. Moreover, we extend our conclusions to the multihop case. In order to deal with the difficulty and complexity brought by the multihop scenario, a novel method based on the equivalent-scheduling matrix (ESM) is proposed to describe the accumulated effects through the multihop transmission. Meanwhile, explicit solutions of the multihop case are provided for finite- and infinite-time cases, respectively. Numerical examples are provided to demonstrate the effectiveness of the proposed methods.

General criteria for the study of quasi-stationarity

For Markov processes with absorption, we provide general criteria ensuring the existence and the exponential non-uniform convergence in weighted total variation norm to a quasi-stationary distribution. We also characterize a subset of its domain of attraction by an integrability condition, prove the existence of a right eigenvector for the semigroup of the process and the existence and exponential ergodicity of the Q-process. These results are applied to one-dimensional and multi-dimensional diffusion processes, to pure jump continuous time processes, to reducible processes with several communication classes, to perturbed dynamical systems and discrete time processes evolving in discrete state spaces.

The Moving Firefighter Problem

The original formulation of the firefighter problem defines a discrete-time process where a fire starts at a designated subset of the vertices of a graph G. At each subsequent discrete time unit, the fire propagates from each burnt vertex to all of its neighbors unless they are defended by a firefighter that can move between any pair of vertices in a single time unit. Once a vertex is burnt or defended, it remains in that state, and the process terminates when the fire can no longer spread. In this work, we present the moving firefighter problem, which is a generalization of the firefighter problem where the time it takes a firefighter to move from a vertex u to defend vertex v is determined by a function τ. This new formulation models situations such as a wildfire or a flood, where firefighters have to physically move from their current position to the location of an entity they intend to defend. It also incorporates the notion that entities modeled by the vertices are not necessarily instantaneously defended upon the arrival of a firefighter. We present a mixed-integer quadratically constrained program (MIQCP) for the optimization version of the moving firefighter problem that minimizes the number of burnt vertices for the case of general finite graphs, an arbitrary set F⊂V of vertices where the fire breaks out, a single firefighter, and metric time functions τ.

On Subgrouping Continuous Processes in Discrete Time

"On Subgrouping Continuous Processes in Discrete Time." Multivariate Behavioral Research, ahead-of-print(ahead-of-print), pp. 1–2

Estimation and Analysis of the Electric Arc Furnace Model Coefficients

This paper is devoted to electric arc furnace (EAF) modeling using a random differential equation based on the power balance equation. The proposed approach broadens and improves the model through the introduction of stochastic processes in place of existing coefficients. The paper presents a method which enables the estimation of EAF model coefficients with the help of measurement data - voltage and current waveforms recorded during the melting stage of an EAF work cycle. The estimation process is conducted with a Monte Carlo method and genetic algorithm, which is applied iteratively to each of the defined frames of the input signal. The estimated coefficients have been analyzed with respect to their time variability as well as the probability distributions of their values and increments. The results have been extensively visualized. Next, the identification of the stochastic processes representing the model coefficients has been carried out. Based on the previous results and autocorrelation functions, the density functions and parameters of discrete-time stochastic processes were identified. The paper presents solutions validated with statistical tests.

Distance evolutions in growing preferential attachment graphs

We study the evolution of the graph distance and weighted distance between two fixed vertices in dynamically growing random graph models. More precisely, we consider preferential attachment models with power-law exponent τ∈(2,3), sample two vertices ut, vt uniformly at random when the graph has t vertices and study the evolution of the graph distance between these two fixed vertices as the surrounding graph grows. This yields a discrete-time stochastic process in t′≥t, called the distance evolution. We show that there is a tight strip around the function 4loglog(t)−log(log(t′/t)∨1)|log(τ−2)|∨2 that the distance evolution never leaves with high probability as t tends to infinity. We extend our results to weighted distances, where every edge is equipped with an i.i.d. copy of a nonnegative random variable L.

Increasing Biomanufacturing Yield with Bleed–Feed: Optimal Policies and Insights

Problem definition: Bleed–feed is a novel technology that allows biomanufacturers to skip intermediary bioreactor setups. However, the specific time at which bleed–feed is performed is critical for success. The process is stringently regulated, and its implementation involves unique tradeoffs in operational decision making. Our analysis formalizes the operational challenges related to bleed–feed decisions, and our results inform biomanufacturers and policymakers on the potential impact of this technology on current practice. Academic/practical relevance: Operations management (OM) methodologies have not yet been widely adopted in the biomanufacturing industry. This research presents one of the first attempts to demonstrate how OM can complement biomanufacturing to improve operational decisions. We present a practically relevant problem and a rigorous solution approach that is relevant to both OM and biomanufacturing. Methodology: We develop a finite-horizon, discrete-time Markov decision processes model and analyze the structural characteristics of optimal bleed–feed policies. Moreover, we characterize the behavior of the value function as a function of regulatory restrictions. As a salient feature, the MDP model captures both the biological dynamics of fermentation and the operational tradeoffs in biomanufacturing. Results: We show that optimal bleed–feed policies have a three-way control-limit structure under mild conditions that were validated with industry data. Moreover, our analysis reveals that the marginal benefits of bleed–feed diminish as additional bleed–feeds are performed. As a practically relevant benchmark, we consider a risk-averse heuristic and identify sufficient conditions for its optimality. Managerial implications: Our analysis (supported with an industry case study) shows that bleed–feed implementation can provide benefits. Real-world implementation at MSD resulted in an 82.5% improvement in the batch yield per setup (using one bleed–feed). We find that low-risk fermentation systems benefit the most from bleed–feed implementation. We also find that the performance gap between optimal policies and the risk-averse heuristic is higher when the failure risks or the critical biomass levels are lower. Funding: This work was supported by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek, NWO-VENI scheme. Supplemental Material: The online appendices are available at https://doi.org/10.1287/msom.2022.1163 .

A stochastic dynamic programming approach for the machine replacement problem

This paper addresses both the modeling and the resolution of the replacement problem for a population of machines. The main objective is the computation of a minimum cost replacement policy, which, based on the status of each machine, determines whether one or more machines have to be replaced over a given finite time horizon.The replacement problem of a set of machines can be regarded as a sequential decision-making problem under uncertainty. Thanks to this, we propose a novel formulation for such problems consisting of a composition of discrete-time multi-state Markov Decision Processes (MDPs), one for each specific machine. The underlying optimization problem is formulated as a stochastic Dynamic Programming (DP), and then solved by using the principles of the backward DP algorithm. Moreover, to deal with the curse of dimensionality due to the high-cardinality state–space of real-world/industrial applications, a new generalized multi-trajectory Least-Squares Temporal Difference (LSTD) based method is introduced. The resulting algorithm computes an approximate optimal cost function by: (i) running Monte Carlo simulations over different trajectories of a given length; (ii) embedding the policy improvement step within the recursive LSTD iterations; (iii) enforcing an off-policy mechanism to improve the LSTD exploration capabilities. A study on the convergence properties of the proposed approach is also provided. Several numerical examples are given to illustrate its effectiveness in terms of parametric sensitivity, computational burden, and performance of the computed policies compared with some heuristics defined in the literature.

Squirrels can remember little: A random walk with jump reversals induced by a discrete-time renewal process

We consider a class of discrete-time random walks with directed unit steps on the integer line. The direction of the steps is reversed at the time instants of events in a discrete-time renewal process and is maintained at uneventful time instants. This model represents a discrete-time semi-Markovian generalization of the telegraph process. We derive exact formulae for the propagator using generating functions. We prove that for geometrically distributed waiting times in the diffusive limit, this walk converges to the classical telegraph process. We consider the large-time asymptotics of the expected position: For waiting time densities with finite mean the walker remains in the average localized close to the departure site whereas escapes for fat-tailed waiting-time densities (i.e. densities with infinite mean) by a sublinear power-law. We explore anomalous diffusion features by accounting for the ‘aging effect’ as a hallmark of non-Markovianity where the discrete-time version of the ‘aging renewal process’ comes into play. By deriving pertinent distributions of this process we obtain explicit formulae for the variance when the waiting-times are Sibuya-distributed. In this case and generally for fat-tailed waiting time PDFs a t2-ballistic superdiffusive scaling emerges in the large time limit. In contrast if the waiting time PDF between the step reversals is light-tailed (‘narrow’ with finite mean and variance) the walk exhibits normal diffusion and for ‘broad’ waiting time PDFs (with finite mean and infinite variance) superdiffusive large time scaling. We also consider time-changed versions where the walk is subordinated to a continuous-time point process such as the time-fractional Poisson process. This defines a new class of biased continuous-time random walks exhibiting several regimes of anomalous diffusion.

Entropy rate of random walks on complex networks under stochastic resetting.

Stochastic processes under resetting at random times have attracted a lot of attention in recent years and served as illustrations of nontrivial and interesting static and dynamic features of stochastic dynamics. In this paper, we aim to address how the entropy rate is affected by stochastic resetting in discrete-time Markovian processes, and we explore nontrivial effects of the resetting in the mixing properties of a stochastic process. In particular, we consider resetting random walks (RRWs) with a single resetting node on three different types of networks: degree-regular random networks, a finite-size Cayley tree, and a Barabási-Albert (BA) scale-free network, and we compute the entropy rate as a function of the resetting probability γ. Interestingly, for the first two types of networks, the entropy rate shows a nonmonotonic dependence on γ. There exists an optimal value of γ at which the entropy rate reaches a maximum. Such a maximum is larger than that of maximal-entropy random walks (MREWs) and standard random walks (SRWs) on the same topology, while for the BA network the entropy rate of RRWs either shows a unique maximum or decreases monotonically with γ, depending upon the choice of the resetting node. When the maximum entropy rate of RRWs exists, it can be higher or lower than that of MREWs or SRWs. Our study reveals a nontrivial effect of stochastic resetting on nonequilibrium statistical physics.