non-Markovian Processes Research Articles

Temporal Logic Guided Meta Q-Learning of Multiple Tasks

Reinforcement learning (RL) based approaches have enabled robots to perform various tasks. However, most existing RL algorithms focus on learning a particular task, without considering generalization to new tasks. To address this issue, by combining meta learning and reinforcement learning, we develop a meta Q-learning of multi-task (MQMT) framework where the robot effectively learns a meta model from a diverse set of training tasks and then generalizes the learned model to a new set of tasks that have never been encountered during training using only a small amount of additional data. Particularly, the multiple tasks are specified by co-safe linear temporal logic specification. As a semantics-preserving rewriting operation, LTL progression is exploited to decompose training tasks into learnable sub-goals, which not only enables simultaneous learning of multiple tasks, but also facilitates reward design by converting non-Markovian reward process to Markovian ones. Reward shaping is further incorporated into the reward design to relax the sparse reward issue. The simulation and experiment results demonstrate the effectiveness of the MQMT framework.

Fluctuation theorems for multitime processes

In this paper, we extend the fluctuation theorems used for quantum channels to multitime processes. The fluctuation theorems for quantum channels are less restrictive. We show that the given entropy production can be equal to the result of a closed system environment. The assumption that the system evolves under a completely positive and trace preserving map is quite general, but it is more specific for cases in which the system is initially correlated with the environment. System-environment correlations arise naturally in multitime processes, with which we can give clear and physical interpretations regarding the effects of correlations. Multitime processes can provide many-body channels. The Choi state of such a many-body channel is called a process tensor. One can derive channels by executing the process tensor on a set of operations. We establish a general quantum fluctuation theorem framework for a many-body channel and its derived channels. In this framework, the effects of correlations are reflected in a Markovian property. For Markovian processes, we can extend the two-point measurement to a three-point measurement and obtain that the fluctuation theorems contain complete information about the intermediate state. For non-Markovian processes, the complete measurement of the intermediate state leads to conflicts. Therefore, we use a general measurement, which only provides partial information, for the intermediate state. The corresponding fluctuation theorems show that memory effects can reduce these fluctuations. This is consistent with the fact that system states can be recovered under non-Markovian processes.

A sequential direct hybrid algorithm to compute stationary distribution of continuous-time Markov chain

In modeling a wide variety of problems of the real world using Markov processes, we usually get large sparse Markov chains. In the case of non-Markovian process, the behavior of a stochastic phenomenon or system can be approximated by continuous-time Markov chains (CTMC) using Phase-type distributions. However, this substantially increases the number of states, which requires a large amount of computation time and operational memory resources. Even the most recent computer hardware is not enough to provide a solution in almost real time, unless a specific algorithm is designed.In this paper, a new hybrid algorithm for the computation of a stationary distribution of a large ergodic homogeneous Markov chains with a finite state space and continuous time is suggested. This algorithm would solve the aforementioned challenges. Depending on model size and sparsity, it significantly reduces a computational time when compared with other recent methods, such as the SparseLU solver from Eigen library (version 3.9.9) and the MUMPS solver (MUltifrontal Massively Parallel sparse direct Solver, version 5.4.0). It is important to note that the algorithm works well independently on the structure of the generator matrix of a continuous-time Markov chain. For the study of the developed hybrid algorithm, the generator matrix is constructed randomly using a special algorithm, based on combining two methods: the slightly modified Grassmann, Taskar and Heyman (GTH) algorithm and the Gauss elimination method. The new idea is to remove the majority of states in a near-optimal order by the GTH method and complete computations for the remaining states by the Gauss method. The robust approach to identify a switching position from one algorithm to another one has been proposed and tested.

Origin of the fractional derivative and fractional non-Markovian continuous-time processes

A complex fractional derivative can be derived by formally extending the integer $k$ in the $k\mathrm{th}$ derivative of a function, computed via Cauchy's integral, to complex $\ensuremath{\alpha}$. This straightforward approach reveals fundamental problems due to inherent nonanalyticity. A consequence is that the complex fractional derivative is not uniquely defined. We explain in detail the anomalies (not closed paths, branch cut jumps) and try to interpret their meaning physically in terms of entropy, friction and deviations from ideal vector fields. Next, we present a class of non-Markovian continuous-time processes by replacing the standard derivative by a Caputo fractional derivative in the classical Chapman-Kolmogorov governing equation of a continuous-time Markov process. The fractional derivative leads to a replacement of the set of exponential base functions by a set of Mittag-Leffler functions, but also creates a complicated dependence structure between states. This fractional non-Markovian process may be applied to generalize the Markovian SIS epidemic process on a contact graph to a more realistic setting.

A unified formalism of the GE-GDEE for generic continuous responses and first-passage reliability analysis of multi-dimensional nonlinear systems subjected to non-white-noise excitations

The stochastic response analysis and first-passage reliability evaluation of multi-dimensional nonlinear systems subjected to non-white-noise engineering dynamic excitations have long been challenging problems. The recently developed globally-evolving-based generalized density evolution equation (GE-GDEE) provides a promising tool and is extended in the present paper for these purposes. Firstly, a more general derivation of the GE-GDEE is given for generic Markov or non-Markov path-continuous stochastic processes. In this unified formalism, the GE-GDEE can be established with respect to one- or two-dimensional path-continuous response(s) of interest in a multi-dimensional system subjected to non-white-noise excitations. Further, for the first-passage reliability evaluation, a new process absorbed at the boundary (ABP) corresponding to failure criterion is constructed, and the GE-GDEE of ABP can be established as a one- or two-dimensional partial differential equation (PDE). This equation can then be solved to obtain the probability density in the safety domain, and the first-passage reliability can thereby be obtained via an integral. Meanwhile, the eligibility of imposing an absorbing boundary condition for the first-passage reliability is rigorously proved. Numerical algorithms are elaborated. Several examples of first-passage reliability analysis of multi-dimensional linear/nonlinear systems subjected to white/non-white noise are illustrated, demonstrating the efficiency and accuracy of the proposed method.

Non-Markovian entanglement dynamics of three independent qubits within atomic-cavity system

All realistic quantum systems will interact with their environments. This fact presents challenges to the implementation of quantum systems for practical systems. One of the most important quantum resources is the implementation of entanglement. In this paper, we investigate the interplay between the effects of noisy environments and the dynamics of entanglement. The open quantum system model that will be used is cavity quantum electrodynamics (CQED) with three-qubit interacting independently with a variation of environmental model. We use master equation to describe the dynamics of the open quantum system and a lower bound concurrence (CLBC ) to measure the entanglement of tripartite qubit. We derive the exact dynamics of each model and use CLBC method proposed by Li et.al to visualize entanglement dynamics. By adjusting the environmental parameters, we also found revivals and dark-periods of entanglement which are unique to non-Markovian processes.

Parameter-dependent filtering of Gaussian processes in Hilbert spaces

The filtering problem for non-Markovian Gaussian processes on rigged Hilbert spaces is considered. Continuous dependence of the filter and observation error on parameters which may be present both in the signal and observation processes is proved. The general results are applied to signals governed by stochastic heat equations driven by distributed or pointwise fractional noise. The observation process may be a noisy observation of the signal at given points in the domain, the position of which may depend on the parameter.

Non-Markovian Inverse Hawkes Processes

Hawkes processes are a class of self-exciting point processes with a clustering effect whose jump rate is determined by its past history. They are generally regarded as continuous-time processes and have been widely applied in a number of fields, such as insurance, finance, queueing, and statistics. The Hawkes model is generally non-Markovian because its future development depends on the timing of past events. However, it can be Markovian under certain circumstances. If the exciting function is an exponential function or a sum of exponential functions, the model can be Markovian with a generator of the model. In contrast to the general Hawkes processes, the inverse Hawkes process has some specific features and self-excitation indicates severity. Inverse Markovian Hawkes processes were introduced by Seol, who studied some asymptotic behaviors. An extended version of inverse Markovian Hawkes processes was also studied by Seol. With this paper, we propose a non-Markovian inverse Hawkes process, which is a more general inverse Hawkes process that features several existing models of self-exciting processes. In particular, we established both the law of large numbers (LLN) and Central limit theorems (CLT) for a newly considered non-Markovian inverse Hawkes process.

Simulating a catalyst induced quantum dynamical phase transition of a Heyrovsky reaction with different models for the environment

Through an appropriate election of the molecular orbital basis, we show analytically that the molecular dissociation occurring in a Heyrovsky reaction can be interpreted as a quantum dynamical phase transition, i.e., an analytical discontinuity in the molecular energy spectrum induced by the catalyst. The metallic substrate plays the role of an environment that produces an energy uncertainty on the adatom. This broadening induces a critical behavior not possible in a quantum closed system. We use suitable approximations on symmetry, together with both Lanczos and canonical transformations, to give analytical estimates for the critical parameters of molecular dissociation. This occurs when the bonding to the surface is times the molecular bonding. This value is slightly weakened for less symmetric situations. However simple, this conclusion involves a high order perturbative solution of the molecule-catalyst system. This model is further simplified to discuss how an environment-induced critical phenomenon can be evaluated through an idealized perturbative tunneling microscopy set-up. In this case, the energy uncertainties in one or both atoms are either Lorentzian or Gaussian. The former results from the Fermi golden rule, i.e., a Markovian approximation. The Gaussian uncertainty, associated with non-Markovian decoherent processes, requires the introduction of a particular model of a spin bath. The partially coherent tunneling current is obtained from the generalized Landauer–Büttiker equations. The resonances observed in these transport parameters reflect, in many cases, the critical properties of the resonances in the molecular spectrum.

Systematic inference identifies a major source of heterogeneity in cell signaling dynamics: The rate-limiting step number.

Identifying the sources of cell-to-cell variability in signaling dynamics is essential to understand drug response variability and develop effective therapeutics. However, it is challenging because not all signaling intermediate reactions can be experimentally measured simultaneously. This can be overcome by replacing them with a single random time delay, but the resulting process is non-Markovian, making it difficult to infer cell-to-cell heterogeneity in reaction rates and time delays. To address this, we developed an efficient and scalable moment-based Bayesian inference method (MBI) with a user-friendly computational package that infers cell-to-cell heterogeneity in the non-Markovian signaling process. We applied MBI to single-cell expression profiles from promoters responding to antibiotics and discovered a major source of cell-to-cell variability in antibiotic stress response: the number of rate-limiting steps in signaling cascades. This knowledge can help identify effective therapies that destroy all pathogenic or cancer cells, and the approach can be applied to precision medicine.

Joint statistics of space and time exploration of one-dimensional random walks.

The statistics of first-passage times of random walks to target sites has proved to play a key role in determining the kinetics of space exploration in various contexts. In parallel, the number of distinct sites visited by a random walker and related observables has been introduced to characterize the geometry of space exploration. Here, we address the question of the joint distribution of the first-passage time to a target and the number of distinct sites visited when the target is reached, which fully quantifies the coupling between the kinetics and geometry of search trajectories. Focusing on one-dimensional systems, we present a general method and derive explicit expressions of this joint distribution for several representative examples of Markovian search processes. In addition, we obtain a general scaling form, which holds also for non-Markovian processes and captures the general dependence of the joint distribution on its space and time variables. We argue that the joint distribution has important applications to various problems, such as a conditional form of the Rosenstock trapping model, and the persistence properties of self-interacting random walks.

Effects of void nodes on epidemic spreads in networks

We present the pair approximation models for susceptible–infected–recovered (SIR) epidemic dynamics in a sparse network based on a regular network. Two processes are considered, namely, a Markovian process with a constant recovery rate and a non-Markovian process with a fixed recovery time. We derive the implicit analytical expression for the final epidemic size and explicitly show the epidemic threshold in both Markovian and non-Markovian processes. As the connection rate decreases from the original network connection, the epidemic threshold in which epidemic phase transits from disease-free to endemic increases, and the final epidemic size decreases. Additionally, for comparison with sparse and heterogeneous networks, the pair approximation models were applied to a heterogeneous network with a degree distribution. The obtained phase diagram reveals that, upon increasing the degree of the original random regular networks and decreasing the effective connections by introducing void nodes accordingly, the final epidemic size of the sparse network is close to that of the random network with average degree of 4. Thus, introducing the void nodes in the network leads to more heterogeneous network and reduces the final epidemic size.

Global correlation and local information flows in controllable non-Markovian open quantum dynamics

In a fully-controllable experiment platform for studying non-Markovian open quantum dynamics, we show that the non-Markovianity could be investigated from the global and local aspects. By mixing random unitary dynamics, we demonstrate non-Markovian and Markovian open quantum dynamics. From the global point of view, by tuning the base frequency we demonstrate the transition from the Markovianity to the non-Markovianity as measured by the quantum mutual information (QMI). In a Markovian open quantum process, the QMI decays monotonically, while it may rise temporarily in a non-Markovian process. However, under some circumstances, it is not sufficient to globally investigate the non-Markovianity of the open quantum dynamics. As an essential supplement, we further utilize the quantum Fisher information (QFI) flow to locally characterize the non-Markovianity in different channels. We demonstrate that the QMI in combination with the QFI flow are capable of measuring the non-Markovianity for a multi-channel open quantum dynamics.

Fokker–Planck equation for Feynman–Kac transform of anomalous processes

In this paper, we develop a novel and rigorous approach to the Fokker–Planck equation, or Kolmogorov forward equation, for the Feynman–Kac transform of non-Markov anomalous processes. The equation describes the evolution of the density of the anomalous process Yt=XEt under the influence of potentials, where X is a strong Markov process on a Lusin space X that is in weak duality with another strong Markov process X̂ on X and {Et,t≥0} is the inverse of a driftless subordinator S that is independent of X and has infinite Lévy measure. We derive a probabilistic representation of the density of the anomalous process under the Feynman–Kac transform by the dual Feynman–Kac transform in terms of the weak dual process X̂t and the inverse subordinator {Et;t≥0}. We then establish the regularity of the density function, and show that it is the unique mild solution as well as the unique weak solution of a non-local Fokker–Planck equation that involves the dual generator of X and the potential measure of the subordinator S. During the course of the study, we are naturally led to extend the notation of Riemann–Liouville integral to measures that are locally finite on [0,∞).

The symmetric coalescent and Wright–Fisher models with bottlenecks

We define a new class of Ξ-coalescents characterized by a possibly infinite measure over the nonnegative integers. We call them symmetric coalescents since they are the unique family of exchangeable coalescents satisfying a symmetry property on their coagulation rates: they are invariant under any transformation that consists of moving one element from one block to another without changing the total number of blocks. We illustrate the diversity of behaviors of this family of processes by introducing and studying a one parameter subclass, the (β,S)-coalescents. We also embed this family in a larger class of Ξ-coalescents arising as the limit genealogies of Wright–Fisher models with bottlenecks. Some convergence results rely on a new Skorokhod type metric, that induces the Meyer–Zheng topology, which allows us to study the scaling limit of non-Markovian processes using standard techniques.

Nonparametric tests for multistate processes with clustered data.

In this work, we propose nonparametric two-sample tests for population-averaged transition and state occupation probabilities for continuous-time and finite state space processes with clustered, right-censored, and/or left-truncated data. We consider settings where the two groups under comparison are independent or dependent, with or without complete cluster structure. The proposed tests do not impose assumptions regarding the structure of the within-cluster dependence and are applicable to settings with informative cluster size and/or non-Markov processes. The asymptotic properties of the tests are rigorously established using empirical process theory. Simulation studies show that the proposed tests work well even with a small number of clusters, and that they can be substantially more powerful compared to the only, to the best of our knowledge, previously proposed test for this problem. The tests are illustrated using data from a multicenter randomized controlled trial on metastatic squamous-cell carcinoma of the head and neck.

Non-Markovian measure independent of initial states of open systems

In recent years, quantifying non-Markovian effect in open quantum system has become an important subject in the quantum decoherence control field. In this paper, a non-Markovian measure independent of the initial state of open system is proposed, thereby extending non-Markovian measure based on quantum Fisher information from the case where the initial state of the system is a pure state to the case where the initial state of the system is an arbitrary mixed state. As its application, the non-Markovian process is quantified by quantum Fisher information about a two-level system undergoing the three well-known dissipative channels, i.e. amplitude dissipative channel, phase damping channel, and random unitary channel. The results show that the conditions of non-Markovian processes in the three dissipative channels are independent of the selection of the initial state of the system by means of the quantum Fisher information of a phase parameter. Further, for amplitude dissipation channel and phase damping channel, the conditions for the non-Markovian processes to occur are equivalent to those given by trace distance, divisibility, quantum mutual information, quantum Fisher-information matrix, et al. As expected, for the case of amplitude dissipation channel, the corresponding results can reduce to the one in other paper (Lu X M, Wang X G, Sun C P <ext-link ext-link-type="uri" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://doi.org/10.1103/PhysRevA.82.042103">2010 <i>Phys. Rev. A</i><italic/> <b>82</b> 042103</ext-link>) by selecting the initial state of the system as an optimal pure state. However, for random unitary channel, the conditions of non-Markovian process are not equivalent to those for other measures. In addition, we also obtain an interesting relationship between quantum Fisher information and quantum coherence of the open system in the three dissipative channels, namely the square of quantum <inline-formula><tex-math id="M2">\begin{document}$l_1$\end{document}</tex-math><alternatives><graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="21-20221053_M2.jpg"/><graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="21-20221053_M2.png"/></alternatives></inline-formula> coherence for the evolved state of system is exactly equal to the quantum Fisher information of the phase parameter. In a word, the obtained results not only improve the application scope of using the quantum Fisher information to detect non-Markovian effects in open systems, but also further highlight its important role in quantum information processing.

Conservative random walk

Recently, in [11], the “coin-turning walk” was introduced on Z. It is a non-Markovian process where the steps form a (possibly) time-inhomogeneous Markov chain. In this article, we follow up the investigation by introducing analogous processes in Zd,d≥2: at time n the direction of the process is “updated” with probability pn; otherwise the next step repeats the previous one. We study some of the fundamental properties of these walks, such as transience/recurrence and scaling limits. Our results complement previous ones in the literature about “correlated” (or “Newtonian”) and “persistent” random walks.

A Convolutional Neural Network Particle Filter for UUV Target State Estimation

The challenges facing unmanned underwater vehicle (UUV) target state estimation are observation uncertainty, the unpredictability of the target motion model, and the complex relative motion of the moving target to the observer. Aiming at the above problems, a convolutional neural network particle filter (CNNPF) is proposed and applied to UUV target state estimation using forward-looking sonar. First, we design a target state prediction network based on convolutional neural network (CNN) to describe the nonlinear measurement and non-Markov target motion process models. Then, a target state prediction dataset is established for the prediction network to build the state space of UUV target state estimation and extract target motion features from the nonsequential measurements. Moreover, particles are sampled from the observations with disturbed time series to approximate the non-Gaussian distribution of measurements. The sampling particles in the target measured state space are mapped to target predictive state space through the prediction network. Convolutional and pooling layers impart invariance to the prediction network and improve its adaptability to uncertain measurement. Finally, the predicted states of particles are used to estimate the target states. The statistical experiment, simulation cases, and hardware-in-the-loop experiment proved the applicability and performance of the CNNPF.

Likelihood-Free Dynamical Survival Analysis applied to the COVID-19 epidemic in Ohio.

The Dynamical Survival Analysis (DSA) is a framework for modeling epidemics based on mean field dynamics applied to individual (agent) level history of infection and recovery. Recently, the Dynamical Survival Analysis (DSA) method has been shown to be an effective tool in analyzing complex non-Markovian epidemic processes that are otherwise difficult to handle using standard methods. One of the advantages of Dynamical Survival Analysis (DSA) is its representation of typical epidemic data in a simple although not explicit form that involves solutions of certain differential equations. In this work we describe how a complex non-Markovian Dynamical Survival Analysis (DSA) model may be applied to a specific data set with the help of appropriate numerical and statistical schemes. The ideas are illustrated with a data example of the COVID-19 epidemic in Ohio.