Discrete-time Markov Research Articles

A measure of the importance of moment for ball-strike counts in a baseball plate appearance

ABSTRACT This study constructs a discrete-time Markov Chain (DTMC) model for a baseball plate appearance (PA) employing Major League Baseball’s pitch-by-pitch dataset. Based on the DTMC model, we propose a novel measure for a baseball PA, termed the Importance of Moment (IOM). The IOM quantifies the criticality of each ball-strike count situation, by assessing the probabilistic difference between the pitcher’s and hitter’s favourable outcomes (out vs reaching base). If the favours significantly vary right after a particular ball-strike count, then the count is deemed critical and is assigned a high IOM value. We empirically verify that IOM explains pitchers’ behaviour of fastball speed. We then further investigate whether the behaviour of ace pitchers differs significantly from the majority. Several interesting properties are found from the analysis. Firstly, the path independence assumption generally holds, with the exception of the ball-strike count of 2B1S. Second, pitchers tend to throw the faster fastball at counts with higher IOM values. Lastly, ace pitchers are capable of pitching even faster fastball in two-strike situations in which IOM is high. The DTMC effectively models the probabilistic structure of a baseball PA, and the proposed IOM measure serves as a useful tool for explaining player behaviour.

Forecasting Freight on Board for Gonggong Export in Batam Using Markov Chain

As an archipelagic country, Indonesia has great potential in the fisheries sector. As a free trade zone, Batam is important in exporting fishery products. One of the fishery export products in Batam City is gonggong snails. It is a favorite seafood item in Riau Islands Province and has high economic value. However, previous studies focused more on the content of gonggong snails and their industrial feasibility; there has been no specific research on the analysis of gonggong snail exports in Batam City, even though gonggong snails are one of Batam City's export products. In this research, we will forecast the freight on board (FoB) value for gonggong exports in Batam City using a discrete-time Markov chain with two states: above and below the moving average. There are several types of moving averages, including simple moving averages and weighted moving averages. An initial analysis will determine the moving averages' type and duration following the Gonggong export FOB data in Batam. The data used is the Gonggong export FoB data for Batam City from January 2020 to November 2023. Based on this data, the transition probability matrix will be calculated based on the number of export transitions below and above the Weighted Moving Average 6 (WMA 6) value. Limiting probability from the Markov chain will be used to predict the long-term FOB value of fishery product exports up to steady-state conditions. It was found that steady-state conditions would be reached after 17 months, with a probability of FOB exports below WMA6 of 55.06% and FOB exports above WMA6 of 44.94%.

Analyzing the impact of cyber-attacks on the performance of digital twin-based industrial organizations

Digital twin is a virtual copy of their “real-world counterparts”, which refer to physical objects, systems, or entities that exist in the physical world. They share valuable information about how the system works by connecting the replica with the real product via the internet and sensors. In digital twin organizations, industrial control systems and the internet are the main sources of data collection and synchronization, resulting in an increase in cyber-attacks. As a result, they could cause data leaks and be used by hackers to launch attacks without being detected. Researching the appropriate defenses against such attacks is of paramount interest. To that end, in this study, we build an integrated model utilizing the combination of a Dynamic Bayesian Network (DBN) and a Discrete-Time Markov Chain (DTMC) to analyze the considerable consequences of cyberattacks on a digital twin-based organization over different time period. DTMC models recovery and vulnerability for DT-based organizations, and then it is paired with the DBN model to simulate the cyberattack propagation behavior in the organization's IT performance. Next, we calculate the organization's work level and total expected utility under two different cyber-attack scenarios to determine its IT performance and financial behavior. Finally, we evaluate the organization's IT system's reliability and resilience after implementing two prevention mechanisms. Result indicates that Distributed Denial-of-Service (DDoS) cyberattack can cause more destruction to the digital twin organization's IT system than a malware attack in terms of cyberattack propagation, work and utility level, reliability, and resilience capability. The findings of the analysis hold practical applicability in real-world settings, enabling the identification of high-risk cyberattacks within DT-based organizations, the analysis and prediction of an organization's IT performance during actual cyberattacks, and the formulation of effective prevention strategies to address cyber security concerns within DT-based organizations.

Equilibrium multi-period investment strategy for a DC pension plan with incomplete information: Hidden Markov model

In the financial market with both observable and unobservable market states, this article explores the equilibrium investment strategy for a DC pension plan with the mean-variance criterion in a discrete-time setting. The dynamics of the partially observed market state are described by a discrete-time finite-state hidden Markov chain. There is a riskless asset and a risky asset in the financial market, where the return rate of the risky asset depends both on the observable and unobservable market states. Meanwhile, the stochastic salary process is also modulated by the observable and unobservable market states. Under the framework of non cooperative game, we first define the equilibrium investment strategy for the multi-period mean-variance DC pension plan. By adopting the sufficient statistics method, the investment problem for the mean-variance DC pension plan with incomplete information is transformed into the one with complete information. The closed-form equilibrium investment strategy is derived by solving the extended Bellman equation. Finally, numerical results show that the incomplete information has significant impacts on the equilibrium investment strategy and the equilibrium efficient frontier. Neglecting the reality of incomplete information in the financial market will reduce the investment benefit of the DC pension plan. The longer the investment horizon, the greater the investment benefit loss for the DC pension plan.

Qualitative analysis of solutions for a degenerate partial differential equations model of epidemic spread dynamics

Compartmental models are widely used in mathematical epidemiology to describe the dynamics of infectious diseases or in mathematical models of population genetics. In this study, we study a time-dependent Susceptible-Infectious-Susceptible (SIS) Partial Differential Equation (PDE) model that is based on a diffusion-drift approximation of a probability density from a well-known discrete-time Markov chain model. This SIS-PDE model is conservative due to the degeneracy of the diffusion term at the origin. The main results of this article are the qualitative behavior of weak solutions, the dependence of the local asymptotic property of these solutions on initial data, and the existence of Dirac delta function type solutions. Moreover, we study the long-term behavior of solutions and confirm our analysis with numerical computations.

Stabilization and Optimal Control for Discrete-Time Markov Jump Linear System With Multiplicative Noises and Input Delays: A Complete Solution

Stabilization and Optimal Control for Discrete-Time Markov Jump Linear System With Multiplicative Noises and Input Delays: A Complete Solution

Existence and uniqueness of quasi-stationary and quasi-ergodic measures for absorbing Markov chains: A Banach lattice approach

We establish the existence and uniqueness of quasi-stationary and quasi-ergodic measures for almost surely absorbed discrete-time Markov chains under weak conditions. We obtain our results by exploiting Banach lattice properties of transition functions under natural regularity assumptions.

Event-triggered sliding mode control for singular discrete-time fuzzy Markov jump networked systems

In this present paper, the event-triggered sliding mode control (SMC) issue for a type of singular networked Markov jump systems with partly unknown transition probabilities (TPs) is mainly investigated, and it is modeled by discrete-time Takagi–Sugeno fuzzy systems. To save network resources, a mode-dependent event-triggered scheme (ETS) is applied to improve transmission efficiency. Subsequently, a common sliding surface is designed, which can effectively reduce the impact of the jumping process. Therein, the stochastic admissibility criterion for the system with [Formula: see text] performance [Formula: see text] is derived in the sense of partly unknown TPs. Moreover, a novel SMC law that guarantees the reachability of the quasi-sliding mode is given in view of the reaching condition. Finally, the validity of the presented theorems is demonstrated by a numerical example.

Gerber-Shiu theory for discrete risk processes in a regime switching environment

In this paper we develop the Gerber-Shiu theory for the classic and dual discrete risk processes in a Markovian (regime switching) environment. In particular, by expressing the Gerber-Shiu function in terms of potential measures of an upward (downward) skip-free discrete-time and discrete-space Markov Additive Process (MAP), we derive closed form expressions for the Gerber-Shiu function in terms of the so-called (discrete) Wv and Zv scale matrices, which were introduced in [27]. We show that the discrete scale matrices allow for a unified approach for identifying the Gerber-Shiu function as well as the value function of the associated constant dividend barrier problems.

Non-fragile [formula omitted] filtering for delayed discrete-time Markov jump systems: An adaptive event-triggered strategy

This paper deals with the problem of non-fragile filter for delayed discrete-time Markov jump systems. The purpose is to design a mode-dependent filter such that the result filtering error system is stochastically stable and satisfies a prescribed H∞ performance lever γ. In order to improve the data transmission efficient, an adaptive event-triggered strategy is introduced. Then, with the help of the Lyapunov stability theory and matrix inequality, the sufficient conditions for the solution of the filter design problem proposed are established. Finally, two numerical examples are presented to verification the effectiveness of the proposed method.

Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces

This paper attempts to study the risk-sensitive discounted discrete-time Markov decision processes in Borel spaces, in which the reward functions are allowed to be unbounded from above and from below. We find mild conditions imposed on the primitive data of the decision processes, which not only ensure the existence of a solution to the optimality equation (OE in short), but also are the generalization of the bounded reward case. Furthermore, using the OE and a novel technique, we prove the existence of an optimal policy out of the class of randomized history-dependent policies. Finally, we illustrate our results with an inventory system.

Stochastic LQ optimal control for Markov jumping systems with multiplicative noise using reinforcement learning

The stochastic linear quadratic (LQ) problem of discrete-time linear Markov jumping systems with multiplicative noise is investigated in this paper. Two reinforcement learning algorithms, one model-based and one model-free, are designed. The algorithms can be readily adapted to handle nonlinear cases by employing an appropriate function approximator. In the linear case, the network structure is designed to enable the problem to be solved without succumbing to modeling errors. For the model-free algorithm, an approach based on moving averages is proposed to reduce estimation variance. A proof of convergence is also provided. Finally, the effectiveness of the proposed algorithms is demonstrated through a numerical example.

Discrete-time Markov decision process for performance analysis of virtual machine allocation schemes in C-RAN

Cloud Radio Access Network (C-RAN) has been proposed as a cloud architecture to provide a common connected resource pool management. It separates the functionalities of the traditional Base Station (BS) into two parts: the Base Band Unit (BBU) and the Remote Radio Head (RRH). BBUs functions are implemented on the Virtual Machines (VMs) in the cloud over commodity hardware, serving User Equipments (UEs) by generating VMs. In this regard, a more accurate adjustment of the number of VMs in a BBU is a challenge. This paper proposes a VM allocation scheme in C-RAN using the hysteresis and migration mechanisms jointly. The goal is to ensure the system’s load balancing and preserve energy. The hysteresis mechanism with different thresholds adjusts the number of active VMs required to serve all UEs by putting idle VMs in sleep mode and activating them as needed. Whereas, the load balancing is achieved by the proposed allocation scheme and the migration of calls from most loaded to least loaded BBU. We use Discrete-Time Markov Decision Processes (MDPs) to enhance both the performance and energy efficiency in the VM allocation schemes. These objectives are achieved by the different decisions of a controller, who should choose the best decision at each time slot. We analyze the performance of the proposed scheme by Probabilistic Model Checking (PMC). Performance requirements are specified using Probabilistic Computation Tree Logic (PCTL) suitable for MDP models. We also check multi-objective properties, by generating Pareto curves, to evaluate the tradeoff based on which the controller can decide. We use the PRISM model checker to build and analyze the MDP models.

Quantization-Based Event-Triggered H∞ Consensus for Discrete-Time Markov Jump Fractional-Order Multiagent Systems with DoS Attacks

This paper investigates the H∞ consensus problem of discrete-time Markov jump fractional-order multiagent systems (DTMJFOMASs) under denial-of-service (DoS) attacks. By applying the short-memory principle, we can obtain discrete-time Markov jump multiagent systems with partially unknown probabilities. A novel quantized event-triggering mechanism (QETM), based on a mode-dependent logarithmic quantizer, is proposed to enhance transmission efficiency among multiagents. A distributed controller with quantized output is developed. Sufficient conditions are provided to ensure the system achieves H∞ consensus through Lyapunov stability theory. Finally, two examples are given to verify the effectiveness of the proposed model.

Synchronization and Random Attractors in Reaction Jump Processes

AbstractThis work explores a synchronization-like phenomenon induced by common noise for continuous-time Markov jump processes given by chemical reaction networks. Based on Gillespie’s stochastic simulation algorithm, a corresponding random dynamical system is formulated in a two-step procedure, at first for the states of the embedded discrete-time Markov chain and then for the augmented Markov chain including random jump times. We uncover a time-shifted synchronization in the sense that—after some initial waiting time—one trajectory exactly replicates another one with a certain time delay. Whether or not such a synchronization behavior occurs depends on the combination of the initial states. We prove this partial time-shifted synchronization for the special setting of a birth-death process by analyzing the corresponding two-point motion of the embedded Markov chain and determine the structure of the associated random attractor. In this context, we also provide general results on existence and form of random attractors for discrete-time, discrete-space random dynamical systems.

Parameter synthesis for Markov models: covering the parameter space

Markov chain analysis is a key technique in formal verification. A practical obstacle is that all probabilities in Markov models need to be known. However, system quantities such as failure rates or packet loss ratios, etc. are often not—or only partially—known. This motivates considering parametric models with transitions labeled with functions over parameters. Whereas traditional Markov chain analysis relies on a single, fixed set of probabilities, analysing parametric Markov models focuses on synthesising parameter values that establish a given safety or performance specification φ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varphi $$\\end{document}. Examples are: what component failure rates ensure the probability of a system breakdown to be below 0.00000001?, or which failure rates maximise the performance, for instance the throughput, of the system? This paper presents various analysis algorithms for parametric discrete-time Markov chains and Markov decision processes. We focus on three problems: (a) do all parameter values within a given region satisfy φ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varphi $$\\end{document}?, (b) which regions satisfy φ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varphi $$\\end{document} and which ones do not?, and (c) an approximate version of (b) focusing on covering a large fraction of all possible parameter values. We give a detailed account of the various algorithms, present a software tool realising these techniques, and report on an extensive experimental evaluation on benchmarks that span a wide range of applications.

Simulation-based evaluation of personalized dosing approaches for anti-FGFR/KLB bispecific antibody fazpilodemab.

Personalized dosing approaches play important roles in clinical practices to improve benefit: risk profiles. Whereas this is also important for drug development, especially in the context of drugs with narrow therapeutic windows, such approaches have not been fully evaluated during clinical development. Fazpilodemab (BFKB8488A) is an agonistic bispecific antibody which was being developed for the treatment of nonalcoholic steatohepatitis. The objective of this study was to characterize the exposure-response relationships of fazpilodemab with the purpose of guiding dose selection for a phase II study, as well as to evaluate various personalized dosing strategies to optimize the treatment benefit. Fazpilodemab exhibited clear exposure-response relationships for a pharmacodynamic (PD) biomarker and gastrointestinal adverse events (GIAEs), such as nausea and vomiting. Static exposure-response analysis, as well as longitudinal adverse event (AE) analysis using discrete-time Markov model, were performed to characterize the observations. Clinical trial simulations were performed based on the developed exposure-response models to evaluate probability of achieving target PD response and the frequency of GIAEs to inform phase II dose selection. Dynamic simulation of personalized dosing strategies demonstrated that the AE-based personalized dosing is the most effective approach for optimizing the benefit-risk profiles. The approach presented here can be a useful framework for quantifying the benefit of personalized dosing for drugs with narrow therapeutic windows.

Probabilistic Model Checking GitHub Repositories for Software Project Analysis

GitHub serves as a platform for collaborative software development, where contributors engage, evolve projects, and shape the community. This study presents a novel approach to analyzing GitHub activity that departs from traditional methods. Using Discrete-Time Markov Chains and probabilistic Computation Tree Logic for model checking, we aim to uncover temporal dynamics, probabilities, and key factors influencing project behavior. By explicitly modeling state transitions, our approach provides transparency and explainability for sequential properties. The application of our method to five repositories demonstrates its feasibility and scalability and provides insights into the long-term probabilities of various activities. In particular, the analysis provides valuable perspectives for project managers to optimize team dynamics and resource allocation. The query specifications developed for model checking allow users to generate and execute queries for specific aspects, demonstrating scalability beyond the queries we present. In conclusion, our analysis provides an understanding of GitHub repository properties, branch management, and subscriber behavior. We anticipate its applicability to various open-source projects, revealing trends among contributors based on the unique characteristics of repositories.

Co-offending and the Persistence of Violence: A Dynamic Analysis

ObjectivesPrevious research underscores the influence of prior violent co-offending on subsequent violent behavior, linking it to a social contagion mechanism akin to the internalization of violence. However, these studies are limited by disregarding the entirety of a criminal career and overlooking diverse co-offending dynamics beyond co-offenders’ characteristics. This study examines the longitudinal impact of prior violent (solo and co-) offending on future individual-level violent behavior among Italian organized crime offenders.MethodsLeveraging criminal career data from 9819 Italian organized crime offenders, we model offending choices through a discrete-time Markov process. Subsequently, employing dynamic random-effects probit models, we quantify the influence of prior violent (solo and co-) offending on future violence, considering various confounders and unobserved individual-level effects.ResultsViolence is a persistent and long-lasting behavior among organized crime offenders. Prior violent co-offending has a greater impact than prior violent solo offending on the probability of future violence. Prior violent co-offending increases the probability of future violent co-offending but does not impact the probability of future violent solo offending.ConclusionsThe results show that co-offending promotes the transmission of violence but fail to support the internalization of violent behavior postulated by prior studies. We propose possible alternative mechanisms of violence transmission that operate through self-sustaining dynamics of violent co-offending within criminal groups. Although limited data on individual characteristics constrains interpretation, our results imply that violence transmission dynamics are independent from the individual characteristics of the co-offenders and more directly connected to group effects.

Formula omitted]th moment stability of discrete-time Markov jump systems by extended system method

This paper studies the pth moment stability of discrete-time Markov jump systems by utilizing the extended system method, which is composed of the operator spectrum technique and the ℋ-representation technique. Firstly, some extended systems are constructed by the operator spectrum technique and the ℋ-representation technique to transform the discrete-time Markov jump systems into the extended systems. Furthermore, the relationship in pth moment stability between the extended systems and the stochastic systems is discussed. Next, with the help of the extended systems, the necessary and sufficient conditions for the even-order moment stability, and several sufficient conditions/necessary conditions for the odd-order moment stability are obtained, respectively. Additionally, as applications of the extended system method, observability and detectability in pth moment sense are discussed, and the necessary and sufficient conditions for observability and detectability in even-order moment are obtained, respectively. Finally, several examples are given to illustrate the validity of the results.