Discrete-time Process Research Articles

Burning the Plane

Graph burning is a discrete-time process on graphs, where vertices are sequentially burned, and burned vertices cause their neighbours to burn over time. We consider extremal properties of this process in the new setting where the underlying graph is also changing at each time-step. The main focus is on the possible densities of burning vertices when the sequence of underlying graphs are growing grids in the Cartesian plane, centred at the origin. If the grids are of height and width $2cn+1$ at time $n$, then all values in $\left [ \frac{1}{2c^2} , 1 \right ]$ are possible densities for the burned set. For faster growing grids, we show that there is a threshold behaviour: if the size of the grids at time $n$ is $\omega(n^{3/2})$, then the density of burned vertices is always $0$, while if the grid sizes are $\Theta(n^{3/2})$, then positive densities are possible. Some extensions to lattices of arbitrary but fixed dimension are also considered.

Reachability and Safety Objectives in Markov Decision Processes on Long but Finite Horizons

We consider discrete-time Markov decision processes in which the decision maker is interested in long but finite horizons. First we consider reachability objective: the decision maker’s goal is to reach a specific target state with the highest possible probability. A strategy is said to overtake another strategy, if it gives a strictly higher probability of reaching the target state on all sufficiently large but finite horizons. We prove that there exists a pure stationary strategy that is not overtaken by any pure strategy nor by any stationary strategy, under some condition on the transition structure and respectively under genericity. A strategy that is not overtaken by any other strategy, called an overtaking optimal strategy, does not always exist. We provide sufficient conditions for its existence. Next we consider safety objective: the decision maker’s goal is to avoid a specific state with the highest possible probability. We argue that the results proven for reachability objective extend to this model.

On discrete-time semi-Markov processes

In the last years, several authors studied a class of continuous-time semi-Markov processes obtained by time-changing Markov processes by hitting times of independent subordinators. Such processes are governed by integro-differential convolution equations of generalized fractional type. The aim of this paper is to develop a discrete-time counterpart of such a theory and to show relationships and differences with respect to the continuous time case. We present a class of discrete-time semi-Markov chains which can be constructed as time-changed Markov chains and we obtain the related governing convolution type equations. Such processes converge weakly to those in continuous time under suitable scaling limits.

Efficient spatio-temporal Gaussian regression via Kalman filtering

We study the non-parametric reconstruction of spatio-temporal dynamical processes via Gaussian Processes (GPs) regression from sparse and noisy data. GPs have been mainly applied to spatial regression where they represent one of the most powerful estimation approaches also thanks to their universal representing properties. Their extension to dynamical processes has been instead elusive so far since classical implementations lead to unscalable algorithms or require some sort of approximation. We propose a novel procedure to address this problem by coupling GPs regression and Kalman filtering. In particular, assuming space/time separability of the covariance (kernel) of the process and rational time spectrum, we build a finite-dimensional discrete-time state-space process representation amenable to Kalman filtering. With sampling over a finite set of fixed spatial locations, our major finding is that the current Kalman filter state represents a sufficient statistic to compute the minimum variance estimate of the process at any future time over the entire spatial domain. In machine learning, a representer theorem states that an important class of infinite-dimensional variational problems admits a computable and finite-dimensional exact solution. In view of this, our result can be interpreted as a novel Dynamic Representer Theorem for GPs. We then extend the study to situations where the spatial input locations set varies over time. The proposed algorithms are tested on both synthetic and real field data, providing comparisons with standard GP and truncated GP regression techniques.

Personality-Guided Cloud Pricing via Reinforcement Learning

As an efficient commercial computing paradigm, cloud computing provides various computing and storage resources to users in a pay-as-you-go manner. However, existing cloud pricing models and mechanisms are deterministic to some degree, thus, may not work well in a real-world environment where user perceived values with respect to cloud services are dynamically changing and highly personalized. In this article, we develop a reinforcement learning (RL)-based dynamic cloud pricing scheme to optimize both cloud provider’s profit and costs of heterogeneous users with distinct personalities. Specifically, we first propose a novel personality-guided user perceived value prediction scheme to proactively capture the dynamics of the users’ perceived values with respect to cloud services. The prediction scheme models the relationship among user personality, service price, quality of service (QoS), user satisfaction and perceived value in the cloud service market. Second, on the basis of the prediction model, a RL-based cloud pricing mechanism is developed to learn sequential service pricing decision-making for profit and costs optimization. Particularly, the profit and costs optimization problem is modeled as a discrete-time Markov decision process (MDP) that is solved by using Q-learning. Finally, extensive simulation experiments have been conducted to verify our user perceived value prediction scheme and RL-based cloud service pricing mechanism. Simulation results show that our perceived value prediction scheme can achieve up to 87.50 percent prediction accuracy, and our RL-based pricing mechanism can obtain up to 19.39 percent more profit than the state-of-the-art scheme.

On Dynamic Service Function Chain Reconfiguration in IoT Networks

Network function virtualization (NFV) technology continues to gain more attention as a paradigm shift, and telecommunication services can be flexibly deployed and managed. Any service can be represented by a service function chain (SFC) that is a set of virtual network functions (VNFs) to be executed based on the strict order. The NFV-enabled SFCs applied in the future Internet-of-Things (IoT) networks emerge a challenging problem, particularly more and more IoT devices are trying to access their telecommunication services whenever and wherever, SFCs are needed to be dynamically and adaptively reconfigured, thus adapting to the service requests’ dynamics for lower resource consumption and higher revenue for Internet service providers (ISPs). In this article, we study the SFC dynamic reconfiguration problem (SFC-DRP) in the IoT networks, a discrete-time Markov decision process (DTMDP)-based IoT SFC-DRP is formulated by guaranteeing the QoS and resource constraints. We subsequently propose a novel deep Dyna-Q (DDQ) approach to solve this model. Our proposal has been evaluated with the obtained results demonstrating an average CPU root-mean-square error (RMSE) of 0.17, compared to 0.75 obtained while using the original approach. Moreover, our proposed SFC reconfiguration technique can approximate the performance of the integer linear programming (ILP) model within a polynomial time, and outperform the existing benchmarks in terms of the reconfiguration overhead and the resource utilization ratio from service provisioning, respectively.

SET-VALUED DYNAMIC RISK MEASURES FOR BOUNDED DISCRETE-TIME PROCESSES

In this paper, we study how to evaluate the risk of a financial portfolio, whose components may be dependent and come from different markets or involve more than one kind of currencies, while we also take into consideration the uncertainty about the time value of money. Namely, we introduce a new class of risk measures, named set-valued dynamic risk measures for bounded discrete-time processes that are adapted to a given filtration. The time horizon can be finite or infinite. We investigate the representation results for them by making full use of Legendre–Fenchel conjugation theory for set-valued functions. Finally, some examples such as the set-valued dynamic average value at risk and the entropic risk measure for bounded discrete-time processes are also given.

Fundamental properties of process distances

To quantify the difference of distinct stochastic processes it is not sufficient to consider the distance of their states and corresponding probabilities. Instead, the information, which evolves and accumulates over time and which is mathematically encoded by filtrations, has to be accounted for as well. The nested distance, also known as bicausal Wasserstein distance, recognizes this component and involves the filtration properly. This distance is of emerging importance due to its applications in stochastic analysis, stochastic programming, mathematical economics and other disciplines.This paper investigates the basic metric and topological properties of the nested distance on the space of discrete-time processes. In particular we prove that the nested distance generates a Polish topology, although the genuine space is not complete. Moreover we identify its completion to be the space of nested distributions, a space of generalized stochastic processes.

Generalized pseudo Bayesian algorithms for tracking of multiple model underwater maneuvering target

The strength of Generalized Pseudo Bayesian (GPB) algorithms is exploited in the presented study to enhance the target tracking precision, effective model approximation and rapid convergence of multimodel maneuvering object tracking. The GPB methods are considered to be suitable for approximating systems whose dynamics follow discrete-time and fixed state Markov process. Underwater maneuvering target tracking problems are usually solved with nonlinear Bayesian algorithms, in which kinetics of object are associated with passive bearings using state-space modeling. Here accuracy and convergence of GPB methods based on Interacting Multiple Model Extended Kalman Filter (IMMEKF), Interacting Multiple Model Extended Kalman Smoother (IMMEKS), Interacting Multiple Model Unscented Kalman Filter (IMMUKF) and Interacting Multiple Model Unscented Kalman Smoother (IMMUKS) are efficiently analyzed for tracking of multimodel maneuvering target in complex ocean environment. Application of these algorithms is systematically presented for estimating the real-time state of a maneuvering object that follows a coordinated turn trajectory. Performance analysis of IMM Kalman filters and smoothers is done with variations in the standard deviation of white Gaussian measurement noise by following Bearings Only Tracking (BOT) phenomena. Least Mean Square Error (MSE) between approximated and the real position of maneuvering target in rectangular coordinates is calculated for analyzing the performance of filtering and smoothing techniques. Simulation results of the Monte Carlo runs validate the effectiveness of IMMEKS and IMMUKS over IMMEKF and IMMUKF for scenario of given framework.

Sequential hypothesis tests under random horizon

We consider a problem of sequential testing a simple hypothesis against a simple alternative, based on observations of a discrete-time stochastic process in the presence of a random horizon H. At any time n of the experiment, the statistician is only informed whether H > n or not. In this latter case, the experiment should be terminated and the final decision on the acceptance or rejection of the hypothesis should be taken on the basis of the available observations (). H is assumed to be independent of the observations, and its distribution is known to the statistician. Under the random horizon, we consider a variant of the modified Kiefer-Weiss problem: given restrictions on the probabilities of errors, minimize the average sample size calculated under the assumption that the observations follow a fixed distribution, not necessarily one of those hypothesized. Under suitable conditions on the process and/or the horizon, we characterize the structure of all optimal sequential tests in this problem. Then, we apply these results to characterize optimal tests in the case of independent observations. On the basis of the general theory, more specific results are obtained for independent and identically distributed (i.i.d.) observations with a geometrically distributed horizon. In a simple sampling model, we solve the Kiefer-Weiss problem under the random horizon model. We also discuss the questions of Wald-Wolfowitz optimality in the presence of the random horizon. In particular, we show that the stopping rules of the optimal tests, minimizing the average sample size under one of the hypotheses, are randomized versions of those of Wald’s sequential probability ratio tests.

Age of Information Aware Radio Resource Management in Vehicular Networks: A Proactive Deep Reinforcement Learning Perspective

In this paper, we investigate the problem of age of information (AoI)-aware radio resource management for expected long-term performance optimization in a Manhattan grid vehicle-to-vehicle network. With the observation of global network state at each scheduling slot, the roadside unit (RSU) allocates the frequency bands and schedules packet transmissions for all vehicle user equipment-pairs (VUE-pairs). We model the stochastic decision-making procedure as a discrete-time single-agent Markov decision process (MDP). The technical challenges in solving the optimal control policy originate from high spatial mobility and temporally varying traffic information arrivals of the VUE-pairs. To make the problem solving tractable, we first decompose the original MDP into a series of per-VUE-pair MDPs. Then we propose a proactive algorithm based on long short-term memory and deep reinforcement learning techniques to address the partial observability and the curse of high dimensionality in local network state space faced by each VUE-pair. With the proposed algorithm, the RSU makes the optimal frequency band allocation and packet scheduling decision at each scheduling slot in a decentralized way in accordance with the partial observations of the global network state at the VUE-pairs. Numerical experiments validate the theoretical analysis and demonstrate the significant performance improvements from the proposed algorithm.

A Large Deviation Perspective on Ratio Observables in Reset Processes: Robustness of Rate Functions

We study large deviations of a ratio observable in discrete-time reset processes. The ratio takes the form of a current divided by the number of reset steps and as such it is not extensive in time. A large deviation rate function can be derived for this observable via contraction from the joint probability density function of current and number of reset steps. The ratio rate function is differentiable and we argue that its qualitative shape is ‘robust’, i.e. it is generic for reset processes regardless of whether they have short- or long-range correlations. We discuss similarities and differences with the rate function of the efficiency in stochastic thermodynamics.

On the Rate-Distortion Function of Sampled Cyclostationary Gaussian Processes.

Man-made communications signals are typically modelled as continuous-time (CT) wide-sense cyclostationary (WSCS) processes. As modern processing is digital, it is applied to discrete-time (DT) processes obtained by sampling the CT processes. When sampling is applied to a CT WSCS process, the statistics of the resulting DT process depends on the relationship between the sampling interval and the period of the statistics of the CT process: When these two parameters have a common integer factor, then the DT process is WSCS. This situation is referred to as synchronous sampling. When this is not the case, which is referred to as asynchronous sampling, the resulting DT process is wide-sense almost cyclostationary (WSACS). The sampled CT processes are commonly encoded using a source code to facilitate storage or transmission over wireless networks, e.g., using compress-and-forward relaying. In this work, we study the fundamental tradeoff between rate and distortion for source codes applied to sampled CT WSCS processes, characterized via the rate-distortion function (RDF). We note that while RDF characterization for the case of synchronous sampling directly follows from classic information-theoretic tools utilizing ergodicity and the law of large numbers, when sampling is asynchronous, the resulting process is not information stable. In such cases, the commonly used information-theoretic tools are inapplicable to RDF analysis, which poses a major challenge. Using the information-spectrum framework, we show that the RDF for asynchronous sampling in the low distortion regime can be expressed as the limit superior of a sequence of RDFs in which each element corresponds to the RDF of a synchronously sampled WSCS process (yet their limit is not guaranteed to exist). The resulting characterization allows us to introduce novel insights on the relationship between sampling synchronization and the RDF. For example, we demonstrate that, differently from stationary processes, small differences in the sampling rate and the sampling time offset can notably affect the RDF of sampled CT WSCS processes.

Bispectral unfolding of the skewness of correlated additive and multiplicative noise processes.

Correlated additive and multiplicative (CAM) noise processes are well-established as general "null hypothesis" models of non-Gaussian variability in atmospheric and oceanic quantities. In this study, analytic expressions for the bispectral density (which partitions the third statistical moment into triad frequency interactions in a manner analogous to the partitioning of variance by the spectral density) are developed for discrete and continuous-time CAM processes. It is then demonstrated that under lowpass filtering, while the absolute skewness of a discrete-time CAM process may increase or decrease with decreasing cutoff frequency, the absolute skewness of continuous-time CAM processes decreases monotonically. This second result provides a test to assess the degree to which an observed time series is consistent with continuous-time CAM dynamics.

A Renewal Generated Geometric Catastrophe Model with Discrete-Time Markovian Arrival Process

Any event that results in sudden change of the normal functioning of a system may be thought of as a catastrophe. Stochastic processes involving catastrophes have very rich application in modeling of a dynamic population in areas of ecology, marketing, queueing theory, etc. When the size of the population reduces abruptly as a whole, due to a catastrophe, it is termed as the total catastrophe. However, in many real-life circumstances the catastrophes have a mild influence on the population and have a sequential effect on the individuals. This paper presents a discrete-time catastrophic model in which the catastrophes occur according to renewal process, and it eliminates each individual of the population in sequential order with probability p until the one individual survives or the entire population wipes out. The individuals arrive according to the discrete-time Markovian arrival process. Using the supplementary variable technique, we obtain the steady-state vector generating function (VGF) of the population size at various epochs. Further using the inversion method of VGF, the population size distribution is expressed in terms of the roots of the associated characteristic equation. We further give a detailed computational procedure by considering inter-catastrophe time distributions as discrete phase-type as well as arbitrary. Finally, a few numerical results in form of tables and graphs are presented. Moreover, the impact of the correlation of arrival process on the mean population size is also investigated.

Stabilization of Linear Systems Across a Time-Varying AWGN Fading Channel

This technical note investigates the minimum average transmit power required for mean-square stabilization of a discrete-time linear process across a time-varying additive white Gaussian noise (AWGN) fading channel that is presented between the sensor and the controller. We assume channel state information at both the transmitter and the receiver, and allow the transmit power to vary with the channel state to obtain the minimum required average transmit power via optimal power adaptation. We consider both the case of independent and identically distributed fading and fading subject to a Markov chain. Based on the proposed necessary and sufficient conditions for mean-square stabilization, we show that the minimum average transmit power to ensure stabilizability can be obtained by solving a geometric program.

Chain-referral sampling on stochastic block models

The discovery of the “hidden population”, whose size and membership are unknown, is made possible by assuming that its members are connected in a social network by their relationships. We explore these groups by a chain-referral sampling (CRS) method, where participants recommend the people they know. This leads to the study of a Markov chain on a random graph where vertices represent individuals and edges connecting any two nodes describe the relationships between corresponding people. We are interested in the study of CRS process on the stochastic block model (SBM), which extends the well-known Erdös-Rényi graphs to populations partitioned into communities. The SBM considered here is characterized by a number of verticesN, a number of communities (blocks)m, proportion of each communityπ= (π1, …,πm) and a pattern for connection between blocksP= (λkl∕N)(k,l)∈{1,…,m}2. In this paper, we give a precise description of the dynamic of CRS process in discrete time on an SBM. The difficulty lies in handling the heterogeneity of the graph. We prove that when the population’s size is large, the normalized stochastic process of the referral chain behaves like a deterministic curve which is the unique solution of a system of ODEs.

A Convex Programming Approach for Discrete-Time Markov Decision Processes under the Expected Total Reward Criterion

In this work, we study discrete-time Markov decision processes (MDPs) under constraints with Borel state and action spaces and where all the performance functions have the same form of the expected...

Increasing Biomanufacturing Yield with Bleed-Feed: Optimal Policies and Insights

Problem definition: Bleed-feed is a novel technology that allows biomanufacturers to skip intermediary bioreactor setups. However, the specific time at which bleed-feed is performed is critical for success. The process is stringently regulated, and its implementation involves unique trade-offs in operational decision-making. Our analysis formalizes the operational challenges related to bleed-feed decisions, and our results inform biomanufacturers and policy-makers on the potential impact of this technology on current practice. Academic/Practical Relevance: Operations Management (OM) methodologies have not yet been widely adopted in the biomanufacturing industry. This research presents one of the first attempts to demonstrate how OM can complement biomanufacturing to improve operational decisions. We present a practically-relevant problem and a rigorous solution approach that is relevant to both OM and biomanufacturing communities. Methodology: We develop a finite-horizon, discrete-time Markov decision process, and analyze the structural characteristics of optimal bleed-feed policies. Results: We show that optimal bleed-feed policies have a two-way control-limit structure under mild conditions that were validated with industry data. Unlike common industry opinions, our analysis reveals that regulatory requirements do not necessarily limit the potential benefits of bleed-feed. Managerial Implications: Our analysis (supported with an industry case study) shows that bleed-feed implementation can provide benefits. Slow growing and/or high-risk cell cultures benefit the most from the use of the bleed-feed technology. Real-world test runs at MSD (using one bleed--feed) resulted in 85% improvement in the batch yield.

Deep Deterministic Policy Gradient (DDPG)-Based Resource Allocation Scheme for NOMA Vehicular Communications

This paper investigates the resource allocation problem in vehicular communications based on multi-agent Deep Deterministic Policy Gradient (DDPG), in which each Vehicle-to-Vehicle (V2V) communication acts as agent and adopts Non-Orthogonal Multiple Access (NOMA) technology to share the frequency spectrum that pre-allocated to Vehicle-to-Infrastructure (V2I) communications. Different with conventional D2D communications, the fast varying channel condition due to the high mobility in vehicular environment causes the difficulty of collecting instantaneous Channel State Information (CSI) at base station. Meanwhile, one tremendous challenge faced by vehicular communications is how to maximize the sum-rate of V2I communications simultaneously guaranteeing the latency and reliability requirements for the transmission of safety-critical information in V2V communications. In response, we formulate the resource allocation problem as a decentralized Discrete-time and Finite-state Markov Decision Process (DFMDP), in which allocation decisions are made by multiple agents that do not have complete and global network information. Due to the complexity of the problem, we propose a DDPG algorithm which is capable of handling continuous high dimensional action spaces to find the optimal allocation strategy. Numerical results verify that each agent can effectively learn from the environment by means of the proposed DDPG algorithm to maximize the sum-rate of V2I communications while satisfying the stringent latency and reliability constraints of V2V communications.