State Transition Probabilities Research Articles

DpGuard: A Lightweight Attack Detection Method for an Industrial Bus Network

In industrial control systems (ICSs), the PROFIBUS-DP (decentralized peripherals) protocol is widely used for communication between devices. Because PROFIBUS-DP is an unencrypted and insecure bus protocol, attackers can connect to the PROFIBUS-DP system and arbitrarily manipulate I/O process values, which may interrupt the normal operation of industrial equipment, or have more serious consequences. However, due to the complex structures of bus networks and the large number of attack areas, the existing scheme does not monitor all the messages in the industrial head office network and cannot effectively detect semantic attacks. To solve this problem, we propose a novel attack detection system DpGuard. DpGuard automatically builds a finite-state machine model of normal ICS behavior through a large number of historical ICS traffic data. The model includes state events, state transitions, state transition probability, and other normal behavior information. In addition, DpGuard records the execution status of the context data package, uses the real-time captured data package as the input of the model, and judges whether the state event and state transition probability conform to the constraints of the finite-state machine model, so as to identify the legitimate normal behavior of the ICS. Our proposal was evaluated using two Siemens PLCs (programmable logic controllers) deployed on the PROFIBUS-DP system. The experimental results demonstrated that the scheme could accurately detect fault injection and semantic attacks. Compared with other detection models, our scheme presented an improved detection performance, with a detection accuracy of 99.80%.

An approach towards the implementation of a reliable resilience model based on machine learning

Machine Learning tools to enhance systems’ resilience received an increased impetus driven by energy transition, climate change and digitalization, but critical challenges on system requirement definition and reliability of learning processes need to be addressed. This study proposes a systematic framework based on system engineering and focused on the reliability of the learning process of the Hidden Markov Model (HMM) coupled with the Baum-Welsh algorithm. The HMM hidden states may represent the precursors of accidental events, being the states between a regular performance and a failure of a sub-system. The Baum-Welch algorithm, estimating the parameters of the HMM, iteratively updates the estimates of the state transition and observation probabilities. The framework was applied to a real case of LNG bunkering, showing that the system can learn from incomplete data, improve the learning quality given a new set of observations, make predictions about the latent states and enhance system resilience. The novelty of this work lies in ensuring the learning process and contributing to the attainment of an explainable, robust, and interpretable data-driven approach.

Energy Dispatching Based on an Improved PSO-ACO Algorithm

In order to improve the comprehensive performance of energy dispatching between different sites, the optimization research of particle swarm optimization (PSO) algorithm and ant colony optimization (ACO) algorithm is carried out. We proposed a new improved PSO-ACO algorithm based on the idea of hybrid algorithm to solve the problem of poor energy dispatching efficiency between sites. First, the multiobjective performance indicators were introduced to transform the sites’ energy dispatching problem into a multiobjective optimization problem. Second, the vitality factor was introduced into the PSO strategy to solve the local optimal problem, and in the PSO-ACO fusion strategy, the PSO routes were transformed into the ant colony enhancement pheromone to accelerate the accumulation speed of the ACO initial pheromone. Then, the angle guidance function was introduced into the state transition probability of the ACO strategy to improve the global search capability, and a high-quality pheromone update rule was proposed to improve the convergence speed of the algorithm. Finally, simulation experiments were carried out on the improved PSO-ACO algorithm, Min–Max Ant System (MMAS) algorithm, ACO algorithm, PSO algorithm, and PSO update algorithm in a variety of complex site scenarios. The simulation results show that the improved PSO-ACO algorithm can plan a site energy dispatching route with shorter route, less time-consuming, and higher security and realize the comprehensive and global optimization of energy dispatching.

Geo/D/1/1 모형에서의 실시간 원격 추정값의 오차 분석

In this paper, we study real-time error in the context of monitoring a binary information source through a delay system. To derive the average real-time error, we model the delay system as a discrete time Geo/D/1/1 queueing model. Using a discrete time three-dimensional Markov chain with finite state space, we analyze the queueing model. We also perform some numerical analysis on various system parameters: state transition probabilities of binary information source; transmission times; and transmission frequencies. When the state changes of the information source are positively correlated and negatively correlated, we investigate the relationship between transmission time and transmission frequency.

Infrastructure deterioration modeling with an inhomogeneous continuous time Markov chain: A latent state approach with analytic transition probabilities

AbstractMarkov chains have been widely used to characterize performance deterioration of infrastructure assets, to model maintenance effectiveness, and to find the optimal intervention strategies. For long‐lived assets such as bridges, the time‐homogeneity assumptions of Markov chains should be carefully checked. For this purpose, this research proposes a regime‐switching continuous‐time Markov chain of which the state transition probabilities depend on another, latent, Markov chain that characterizes the overall aging regime of an asset. With the aid of a state‐augmentation technique, closed‐form solutions for the transition probabilities are analytically derived, making the statistical analysis simple. A case study is presented using the open Ontario Bridge Condition data for provincial highway bridges. The case study demonstrates that the proposed method allows to (1) estimate a statistically superior model to the homogeneous Markov chain and (2) obtain results with comparable accuracy in approximately 48% of the computation time of the state‐of‐the‐art inhomogeneous Markov chain.

Proselfs depend more on model-based than model-free learning in a non-social probabilistic state-transition task

Humans form complex societies in which we routinely engage in social decision-making regarding the allocation of resources among ourselves and others. One dimension that characterizes social decision-making in particular is whether to prioritize self-interest or respect for others—proself or prosocial. What causes this individual difference in social value orientation? Recent developments in the social dual-process theory argue that social decision-making is characterized by its underlying domain-general learning systems: the model-free and model-based systems. In line with this “learning” approach, we propose and experimentally test the hypothesis that differences in social preferences stem from which learning system is dominant in an individual. Here, we used a non-social state transition task that allowed us to assess the balance between model-free/model-based learning and investigate its relation to the social value orientations. The results showed that proselfs depended more on model-based learning, whereas prosocials depended more on model-free learning. Reward amount and reaction time analyses showed that proselfs learned the task structure earlier in the session than prosocials, reflecting their difference in model-based/model-free learning dependence. These findings support the learning hypothesis on what makes differences in social preferences and have implications for understanding the mechanisms of prosocial behavior.

A Loosely Coupled Model for Simulating and Predicting Land Use Changes

The analysis and modeling of spatial and temporal changes in land use can reveal changing urban spatial patterns and trends. In this paper, we introduce a linear transformation optimization Markov (LTOM) model that can be exploited to estimate the state transition probability matrix of land use, building a loosely coupled ANN-CA-LTOM model for simulating and predicting land use changes. The advantages of this model are that it is flexible and high expansibility; it can maintain semantic coupling between the Artificial Neural Networks (ANN), Cellular Automata (CA), and LTOM model and enhance their functions; and it can break the limitation of requiring two periods of land use data when calculating the transition probability matrix. We also construct a suitability atlas of land use as the transition rules into the CA-LTOM model, taking into account the regional natural and socioeconomic driver factors, by exploiting the ANN model. The ANN-CA-LTOM model is employed to simulate the distribution of the three major types of land use, i.e., construction land, agricultural land, and unused land, in the Nansha District, China, in 2018 and 2020. The results show that the model performs well and the overall accuracy of the land use simulation was 97.72%, with a kappa coefficient of 0.962761. Furthermore, the simulated and predicted results of land use changes from 2021 to 2023 in Nansha District show changing trends in construction, agricultural, and unused land use. This study provides an approach for estimating a Markov transition probability matrix and a coupled mode of the models for simulating and predicting land use changes.

Deep reinforcement learning-based adaptive modulation for OFDM underwater acoustic communication system

Due to the time-varying and space-varying characteristics of the underwater acoustic channel, the communication process may be seriously disturbed. Thus, the underwater acoustic communication system is facing the challenges of alleviating interference and improving communication quality and communication efficiency through adaptive modulation. In order to select the optimal modulation mode adaptively and maximize the system throughput ensuring that the bit error rate (BER) meets the transmission requirements, this paper introduces deep reinforcement learning (DRL) into orthogonal frequency division multiplexing acoustic communication system. The adaptive modulation is mapped into a Markov decision process with unknown state transition probability. Thereby, the underwater communication channel environment is regarded as the state of DRL, and the modulation mode is regarded as action. The system returns channel state information (CSI) and signal–noise ratio in every time slot through the feedback link. Because the Deep Q-Network optimizes in the changing state space of each time slot, it is suitable for a variety of different CSI. Finally, simulations in different underwater environments (SWellEx-96) show that the proposed adaptive modulation scheme can obtain lower BER and improve the system throughput effectively.

Research on the co-evolution of competitive public opinion and intervention strategy based on Markov process

Under Omni-media environment, Online Social Networks (OSN) have gradually become the most momentous platform for information propagation. Considering the interaction and coexistence of both positive and negative public opinion information (referred to as public opinion), it is of great significance for social development and economic stability to understand the co-evolution process of competitive public opinion and compress the spreading space of negative public opinion. Allowing for this point, this paper constructed a two-stage spreading model of competitive public opinion combing with the actual case of public opinion propagation, analysed the main factors influencing the co-evolution process, such as netizens’ intimacy, network literacy, and so on, and redefined netizens’ state transition probability matrix with the help of Markov process. Then, the effectiveness of the spreading model was verified and the propagation rule of public opinion was discussed in open and closed OSN through simulation experiments. Finally, the intervention strategies were proposed and optimised with the limitation of cost. The results show that the propagation of public opinion mainly depends on netizens’ behaviour with low literacy and presents difference characteristics in two types of OSN. During the intervention process of public opinion propagation, there exists an effective intervention interval and the best intervention strategy varies with the change of network topology. Our research provided a cornerstone for further understanding of the co-evolution process of competitive public opinion and the research conclusions also provided a certain decision-making reference for enterprises, governments and other regulators to reasonably respond to the propagation of public opinion.

Policy Synthesis for Switched Linear Systems With Markov Decision Process Switching

In this article, we study the synthesis of mode switching protocols for a class of discrete-time switched linear systems in which the mode jumps are governed by Markov decision processes (MDPs). We call such systems MDP-JLS for brevity. Each state of the MDP corresponds to a mode in the switched system. The probabilistic state transitions in the MDP represent the mode transitions. We focus on finding a policy that selects the switching actions at each mode such that the switched system is guaranteed to be stable. Given a policy in the MDP, the considered MDP-JLS reduces to a Markov jump linear system (MJLS). We consider both mean-square stability and stability with probability one. For mean-square stability, we leverage existing stability conditions for MJLSs and propose efficient semidefinite programming formulations to find a stabilizing policy in the MDP. For stability with probability one, we derive new sufficient conditions and compute a stabilizing policy using linear programming. We also extend the policy synthesis results to MDP-JLS with uncertain mode transition probabilities.

Microstates of Dynamic Directed Connectivity Networks Revealing Visual Color Influences on the Brain Information Processing During Learning

Color has the exceptional ability to capture visual attention and is also capable of enhancing positive emotions, leading to a significant impact on human learning and memory. However, the influence of color on the spatiotemporal dynamics of brain connectivity networks during learning has remained unexplored. This study aimed to propose an analytical approach based on time-frequency decomposition and microstate analysis to capture temporal variations in dynamic directed connectivity networks using electroencephalography (EEG) signals for investigating the influence of visual color on network dynamics of the brain during a learning task. Wavelet transform and phase slope index were employed to estimate the dynamic directed connectivity networks of EEG signals. The estimated dynamic directed connectivity networks were then characterized using graph theoretical analysis. The recurring patterns of dynamic directed connectivity networks were classified using cluster analysis before the temporal dynamics of directed connectivity networks were quantified using microstate analysis. Forty-five healthy participants participated in the experiment, which included memorizing learning materials presented in three different colors (achromatic, cool, and warm). The results revealed that the dynamic directed connectivity networks could be grouped into several quasi-stable states and the presence of common and unique brain states repetitive across frequency bands under individual conditions. A joint analysis of all conditions revealed that the temporal dynamics (coverage, mean duration, and state transition probability) differed significantly between the achromatic and colored conditions. Few dynamic brain states were shared between conditions and tended to remain in particular brain states for a longer duration in specific frequency bands. Our observations provided the first evidence of temporal dynamics of frequency-specific directed connectivity networks in the brain during multimedia learning tasks, that is, increased coverage of top-down interactions in the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\theta $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\alpha $ </tex-math></inline-formula> bands and switching between top-down and bottom-up interactions (information flow from anterior to posterior regions and vice versa) in the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\alpha $ </tex-math></inline-formula> band, in colored conditions compared to that of achromatic conditions. Therefore, these results suggest that several frequency-specific directed connectivity networks cooperate during knowledge acquisition and may change over time (from one state to another). The proposed framework captures the temporal dynamics of directed connectivity networks, and provides implications for monitoring and assessing emotional and cognitive processes in various contexts.

Research and experiment on global path planning for indoor AGV via improved ACO and fuzzy DWA.

In order to obtain an optimal trajectory for indoor AGV, this paper combined an improved ACO and fuzzy DWA (IACO-DWA) algorithm, which can provide an optimal path with collision-free under higher optimization efficiency. The highlights of this paper are detailed as follows: Firstly, an improved adaptive pseudo-random transition strategy is adopted in the state transition probability with an angle factor. A reward and punishment mechanism is introduced in the pheromone updating strategy, then a path optimization strategy called IACO is proposed for the more optimized path. Secondly, IDWA adopted three fuzzy controllers of direction, security and adjustment coefficients through evaluating directional and safety principles, then improving the angular velocity by processing the linear velocity with linear normalization. By adapting to the changes of the environment, the IDWA parameters can be dynamically adjusted to ensure the optimal running speed and reasonable path of AGV. Thirdly, aiming to deal with the path-planning problem in complex environments, we combined IACO with IDWA, the hybrid algorithm involves dividing the globally optimal path obtained from IACO planning into multiple virtual sub-target points. IDWA completes the path planning by switching between these local target points, thereby improving the efficiency of the path planning. Finally, simulations is verified by Matlab and experiment results on the QBot2e platform are given to verify IACO-DWA algorithm's effectiveness and high performance.

Effectiveness of Antiretroviral Treatment on the Transition Probability of Immunological State of HIV/AIDS Patients: A Markov Chain Model on the Iranian “National Registry of HIV/AIDS Care” Database

Antiretroviral therapy (ART) has improved survival and clinical course amongst HIV/AIDS patients. CD4 cell count is one of the most critical indicators of the disease progression. With respect to the dynamic nature of CD4 cell count during the clinical history of HIV/AIDS, modeling the CD4 cell count changes, which represents the likelihood of disease progression, is of interest to establish or revise treatment strategies and, specifically, to determine the stage at which giving ART is more clinically effective. In this historical cohort study on 917 HIV/AIDS patients in the Iranian “National Registry of HIV/AIDS Care” database, we used the Markov chain model to predict the effectiveness of the ART based on the transition probability of CD4 cell count, measured before and after initiating ART. We found that when the ART was initiated in the earlier stages of HIV infection, good prognosis might be more accessible; that is, after initiating ART at state CD4 ≥ 500, the probability of staying at this state was statistically increased than before the treatment (P &lt; 0.001). Also, it was found that initiating ART significantly decreased the probability of CD4 cell count transition to the lower counts (PCD4≥500‐to‐350≤CD4&lt;500 &lt; 0.001, P350≤CD4&lt;500‐to‐200≤CD4&lt;350 &lt; 0.001, and P200≤CD4&lt;350‐to‐CD4&lt;200 &lt; 0.001). In addition, initiating ART showed a statistically significant increase in the probability of CD4 cell count transition from a lower state (350 ≤ CD4 &lt; 500) to a higher state (CD4 ≥ 500) (P = 0.009). Furthermore, When CD4 count reaches under 200, even after the initiation of therapy, the probability of CD4 cell count transition to higher levels was not significant (P &gt; 0.05). In summary, these results have a message for primary healthcare organizations to extensively identify HIV patients and initiating ART as early as possible.

РОЗРАХУНОК ХАРАКТЕРИСТИК СИСТЕМ МАСОВОГО ОБСЛУГОВУВАННЯ ІЗ ЗАСТОСУВАННЯМ МЕТОДУ ЕРЛАНГА І ЗАКОНІВ ЗБЕРЕЖЕННЯ

Any human activity (economic, entrepreneurial, commercial) is associated with the performance of many operations at the stages of movement of raw materials, goods, products from the supply sector to the sphere of production and consumption. Such operations are transportation, storage, processing, sales, etc. These types of activities are characterized by the massive receipt of products, goods, money, and clients at random times; their sequential servicing is carried out by corresponding operations, the execution time of which, as a rule, is also random. All this leads to inequalities in work, creates idle time, underload and overload in operations. Therefore, the tasks arise of analyzing existing options for performing a certain set of operations, identifying bottlenecks and reserves for developing and making management decisions to improve the operating efficiency of any organization. Such problems are successfully solved using queuing theory. The basis for making management decisions are parameters that characterize different aspects of the operation of both the system as a whole and its individual elements. To describe individual parts of the system, the distribution of random variables and their numerical properties are traditionally used. The main characteristics of the action and state of the QS are the average number of requests in the queue or system, the average waiting time for service and others, as well as the value of some probabilities (the probability of service denial, the probability that there are at least a certain number of requirements in the system, the probability that the system is free from maintenance, etc.).The article describes some methods and techniques for studying Markov systems (the processes of which have no history), which turn out to be applicable to more general systems. A Markov queuing system with a restriction on the largest number of requirements in the system is considered. The probabilities of state transitions for closed and open-loop systems are described and calculated using Erlang methods and using the laws of conservation of system probabilities and the Laplace transform. Calculations of queue safety under stationary operating modes of queuing systems are demonstrated.

Byte-Level Function-Associated Method for Malware Detection

The byte stream is widely used in malware detection due to its independence of reverse engineering. However, existing methods based on the byte stream implement an indiscriminate feature extraction strategy, which ignores the byte function difference in different segments and fails to achieve targeted feature extraction for various byte semantic representation modes, resulting in byte semantic confusion. To address this issue, an enhanced adversarial byte function associated method for malware backdoor attack is proposed in this paper by categorizing various function bytes into three functions involving structure, code, and data. The Minhash algorithm, grayscale mapping, and state transition probability statistics are then used to capture byte semantics from the perspectives of text signature, spatial structure, and statistical aspects, respectively, to increase the accuracy of byte semantic representation. Finally, the three-channel malware feature image is constructed based on different function byte semantics, and a convolutional neural network is applied for detection. Experiments on multiple data sets from 2018 to 2021 show that the method can effectively combine byte functions to achieve targeted feature extraction, avoid byte semantic confusion, and improve the accuracy of malware detection.

Ecological spatial intensive use optimization modeling with framework of cellular automata for coordinating ecological protection and economic development.

With the exposure of excessive intensive use of urban and agricultural space, the optimization of intensive use of ecological space provides a new way to coordinate the global problem of spatial conflict between ecological protection and economic development. However, the coupling accuracy of the existing structure-spatial coupling optimization model is low, which cannot provide method support for the optimization of intensive use of ecological space. To solve this problem, we propose a new model of ecological spatial intensive use optimization (ESIUO) based on the non-stationarity of the Markov state transition probability of the dominant ecosystem service functions (DESFs) and their suitability, and with the help of the framework of cellular automata (CA). We took Qionglai City as an empirical study area, and compared the results of this model with those of CA-Markov and CLUE-S models with the same parameters. The results show that: (i) The quantitative structure corresponding to the spatial layout of each dominant ecosystem service function (DESF) optimized by the ESIUO model has the smallest relative error (δk≤0.04%) with the optimal quantitative structure. (ii) The layout of DESFs optimized by the ESIUO model maximizes the supply capacity of ecosystem services. The minimum matching degree between the distribution of each DESF and the high-value area of its suitability is 92.06 %, and the spatial distribution is more compact, and the comprehensive effect of spatial layout is the best. Further analysis confirmed that the model can establish the spatial layout of DESFs that can realize the high precision coupling with the optimal quantitative structure of DESFs in terms of quantitative structure, and can support the construction of the layout of intensive use of ecological space to alleviate the pressure of non-ecological space expansion in these areas, and then provide a new way to coordinate ecological protection and economic development.

State Aggregation for Distributed Value Iteration in Dynamic Programming

We propose a distributed algorithm to solve a dynamic programming problem with multiple agents, where each agent has only partial knowledge of the state transition probabilities and costs. We provide consensus proofs for the presented algorithm and derive error bounds of the obtained value function with respect to what is considered as the "true solution" obtained from conventional value iteration. To minimize communication overhead between agents, state costs are aggregated and shared between agents only when the updated costs are expected to influence the solution of other agents significantly. We demonstrate the efficacy of the proposed distributed aggregation method to a large-scale urban traffic routing problem. Individual agents compute the fastest route to a common access point and share local congestion information with other agents allowing for fully distributed routing with minimal communication between agents.

Impact of Channel Memory on the Data Freshness

In this letter, we investigate the impact of channel memory on the average age of information (AoI) for networks with various packet arrival models under the first-come-first-served (FCFS) and the preemptive last-generated-first-served (pLGFS) policies over the Gilbert-Elliott (GE) erasure channel. For networks with Bernoulli and generate-at-will arrival models, the AoI performances under the FCFS and pLGFS policies are derived explicitly as functions of the channel state transition probabilities. For networks with periodic arrival model, we derive the closed-form expression for the average AoI under pLGFS and propose a numerical algorithm for efficient evaluation of the AoI under FCFS. It is revealed that for pLGFS policy, the average AoI increases monotonically with channel memory <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\eta $ </tex-math></inline-formula> at <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\frac {\eta }{1-\eta }$ </tex-math></inline-formula> over the symmetric GE channel. For FCFS, the average AoI increases even faster due to the queuing delay, with an additional term related to the packet arrival rate.

The impact of customer clubs on lifetime value of banking customers

Today, with the abundance of private banks, state-owned banks and financial and credit institutions, banks are trying to satisfy customers more than ever in order to be preferred by them. In this regard, they use customer clubs as a communication network to improve their up-selling, cross-selling and word of mouth marketing. Therefore, this study aims to investigate the short-term impact of customer clubs on the lifetime value of customers as one of the most important indicators that reflects customer satisfaction, loyalty and thereby financial gain. To this end, the RFM model was used for clustering customers based on their behaviours. Then customers' lifetime value for each cluster was calculated in four periods. The state transition probability matrix of a Markov chain was also used to analyse the customers' transitions and change in their values through the seasons. The results indicated that customers' lifetime value and the number of profitable customers increased in the last period in which the customer club was used.

Research on the application of big data visualization technology in urban road congestion

ABSTRACT In order to improve the effect of urban road congestion analysis, this paper aims to study the mechanism of traffic congestion diffusion under the condition that users have real-time traffic information, and analyze the urban road congestion situation combined with big data visualization technology. According to the characteristics of traffic flow propagation, this paper conducts multi-granularity abstraction and multi-scale modeling of node-intersection-link-network for the complex and dynamic traffic congestion process, and establishes an improved SIS virus propagation model for traffic congestion propagation. In addition, this paper uses the method of state transition probability to construct an interactive dynamic model of traffic congestion propagation and early warning information propagation in a multi-layer network. The experimental research results show that the big data visualization technology introduced in this paper can play an important role in urban road congestion.