Optimal Transition Probabilities Research Articles

Finite-State Markov-Chain Approximations: A Hidden Markov Approach

This paper proposes a novel finite-state Markov chain approximation method for Markov processes with continuous support, providing both an optimal grid and transition probability matrix. The method can be used for multivariate processes, as well as non-stationary processes such as those with a life-cycle component. The method is based on minimizing the information loss between a Hidden Markov Model and the true data-generating process. We provide sufficient conditions under which this information loss can be made arbitrarily small if enough grid points are used. We compare our method to existing methods through the lens of an asset-pricing model, and a life-cycle consumption-savings model. We find our method leads to more parsimonious discretizations and more accurate solutions, and the discretization matters for the welfare costs of risk, the marginal propensities to consume, and the amount of wealth inequality a life-cycle model can generate.

Optimization of Mobile Robotic Relay Operation for Minimal Average Wait Time

This paper considers trajectory planning for a mobile robot which persistently relays data between pairs of far-away communication nodes. Data accumulates stochastically at each source, and the robot must move to appropriate positions to enable data offload to the corresponding destination. The robot needs to minimize the average time that data waits at a source before being serviced. We are interested in finding optimal robotic routing policies consisting of 1) locations where the robot stops to relay (relay positions) and 2) conditional transition probabilities that determine the sequence in which the pairs are serviced. We first pose this problem as a non-convex problem that optimizes over both relay positions and transition probabilities. To find approximate solutions, we propose a novel algorithm which alternately optimizes relay positions and transition probabilities. For the former, we find efficient convex partitions of the non-convex relay regions, then formulate a mixed-integer second-order cone problem. For the latter, we find optimal transition probabilities via sequential least squares programming. We extensively analyze the proposed approach and mathematically characterize important system properties related to the robot’s long-term energy consumption and service rate. Finally, through extensive simulation with real channel parameters, we verify the efficacy of our approach.

Probability bounds for reflecting diffusion processes

A solution to the optimal problem for determining vector fields which maximize (resp. minimize) the transition probabilities for a class of reflecting diffusion processes is obtained in this paper. The approach is based on a representation for the transition probability density functions. The optimal transition probabilities under the constraint that the drift vector field is bounded are studied in terms of the HJB equation. We demonstrate by simulations that, even in one dimension, by considering the nodal set of the solutions to the HJB equation, the optimal diffusion processes exhibit an interesting feature of phase transitions.

Adaptive Transition Probability Matrix Learning for Multiview Spectral Clustering.

Multiview clustering as an important unsupervised method has been gathering a great deal of attention. However, most multiview clustering methods exploit the self-representation property to capture the relationship among data, resulting in high computation cost in calculating the self-representation coefficients. In addition, they usually employ different regularizers to learn the representation tensor or matrix from which a transition probability matrix is constructed in a separate step, such as the one proposed by Wu et al.. Thus, an optimal transition probability matrix cannot be guaranteed. To solve these issues, we propose a unified model for multiview spectral clustering by directly learning an adaptive transition probability matrix (MCA2M), rather than an individual representation matrix of each view. Different from the one proposed by Wu et al., MCA2M utilizes the one-step strategy to directly learn the transition probability matrix under the robust principal component analysis framework. Unlike existing methods using the absolute symmetrization operation to guarantee the nonnegativity and symmetry of the affinity matrix, the transition probability matrix learned from MCA2M is nonnegative and symmetric without any postprocessing. An alternating optimization algorithm is designed based on the efficient alternating direction method of multipliers. Extensive experiments on several real-world databases demonstrate that the proposed method outperforms the state-of-the-art methods.

A Systematic Literature Review on prioritizing software test cases using Markov chains

Software Testing is a costly activity since the size of the test case set tends to increase as the construction of the software evolves. Test Case Prioritization (TCP) can reduce the effort and cost of software testing. TCP is an activity where a subset of the existing test cases is selected in order to maximize the possibility of finding defects. On the other hand, Markov Chains representing a reactive system, when solved, can present the occupation time of each of their states. The idea is to use such information and associate priority to those test cases that consist of states with the highest probabilities. The objective of this paper is to conduct a survey to identify and understand key initiatives for using Markov Chains in TCP. Aspects such as approaches, developed techniques, programming languages , analytical and simulation results, and validation tests are investigated. A Systematic Literature Review (SLR) was conducted considering studies published up to July 2021 from five different databases to answer the three research questions. From SLR, we identified 480 studies addressing Markov Chains in TCP that have been reviewed in order to extract relevant information on a set of research questions. The final 12 studies analyzed use Markov Chains at some stage of test case prioritization in a distinct way, that is, we found that there is no strong relationship between any of the studies, not only on how the technique was used but also in the context of the application. Concerning the fields of application of this subject, 6 forms of approach were found: Controlled Markov Chain, Usage Model, Model-Based Test, Regression Test , Statistical Test, and Random Test. This demonstrates the versatility and robustness of the tool. A large part of the studies developed some prioritization tool, being its validation done in some cases analytically and in others numerically, such as: Measure of the software specification, Optimal Test Transition Probabilities, Adaptive Software Testing, Automatic Prioritization, Ant Colony Optimization , Model Driven approach, and Monte Carlo Random Testing. • Software Testing is a costly activity since the size of the test case set tends to increase as the construction of the software evolves. • Test Case Prioritization can reduce the effort and cost of software testing. • Markov Chains can be efficiently used in Test Case Prioritization.

An Adaptive Offloading Mechanism for Mobile Cloud Computing: A Niching Genetic Algorithm Perspective

The fast evolution of mobile applications demonstrates the growing need for more resources and processing power on mobile devices. Mobile Cloud Computing (MCC) combines cloud computing with mobile devices, enabling sophisticated and resource-intensive applications to operate on mobile devices regardless of their performance limits (e.g., battery life, memory utilization, and computation). Overcoming these constraints is accomplished using a well-known technique called computation offloading, which entails offloading heavy processing to resourceful servers and getting the results from these servers. Many studies have been done on mobile code offloading, with the goal of avoiding the stated limits and reducing execution time or battery consumption through single- or multiple-site offloading, so that people can use their phones and tablets more. However, the majority of existing techniques make offloading choices based on profile data, which implies a stable network environment and forces an object to be offloaded to the same site. As a consequence of these challenges, this research proposes a novel strategy for enhancing the multisite offloading mechanism by combining a Niching Genetic Algorithm (NGA) with a Markov Decision Process (MDP). MDP is used to determine the most optimal location for each application’s modules to be executed. GA was used to determine the optimal transition probability for components operating on several sites. To aid in initial population selection, the proposed model employs a niche model in the form of a context-based clearing (CBC) technique to improve the variability of the genes inside the chromosome in order to minimize their association. The simulation results reveal that the proposed technique consumes little power and executes quickly while determining the optimal offloading decision with lower generation numbers.

Design of biased random walks on a graph with application to collaborative recommendation

This work investigates a paths-based statistical physics formalism, inspired from the bag-of-paths framework, for the design of random walks on a graph in which the transition probabilities (the policy) are biased in favor of some node features. More precisely, given a weighted directed graph G and a non-negative cost assigned to each edge, the biased random walk is defined as the policy minimizing the expected cost rate along the walks while maintaining a constant relative entropy rate. As for the standard bag-of-paths and the randomized shortest paths frameworks, the model assigns a Gibbs–Boltzmann distribution to the set of infinite walks and allows to recover known results from the literature, derived here from a different perspective. Examples of quantities of interest are the partition function of the system, the cost rate, the optimal transition probabilities, etc. In addition, the same formalism allows the introduction of capacity constraints on the expected node visit rates and an algorithm for computing the optimal policy subject to such capacity constraints is developed. Simulation results indicate that the proposed procedure can be effectively used in order to define a Markov chain driving the walk towards nodes having some specific properties, like seniority, education level or low node degree (hub-avoiding walk). An application relying on this last property is proposed as a tool for improving serendipity in collaborative recommendation, and is tested on the MovieLens data.

Optimal novel approach for estimating the pavement transition probabilities used in Markovian prediction models

ABSTRACT An optimal novel approach is proposed to estimate the transition probabilities associated with both homogeneous and non-homogeneous Markov chains. The approach applies an exhaustive optimisation technique to search for the optimal transition probabilities associated with minimal sum of squared errors (SSE), wherein the error defined as the difference between predicted and observed pavement condition ratings. Three state transitions are allowed in constructing the relevant transition probability matrix (TPM). In the homogenous chain, the approach yields one optimal TPM applicable to an analysis period of (n) years. Whereas, one distinct optimal TPM can be derived for each year if non-homogenous chain is deployed. A sequential iterative optimisation approach is proposed wherein the optimal TPM for a given year becomes the input for the subsequent year. Sample results are presented for two projects (A & B) with superior and inferior performances, respectively. The sample results indicate that the non-homogenous chain provided significant reduction in the SSE compared to the homogeneous one. However, the use of 10 condition states instead of 5 resulted in moderate reduction in the SSE considering both homogeneous and non-homogeneous Markov chains. Also, the results indicate that the use of three state transitions made significant impact when deploying 10 condition states instead of 5 especially in the case of inferior performance.

A Random Walk-Based Energy-Aware Compressive Data Collection for Wireless Sensor Networks

The energy efficiency for data collection is one of the most important research topics in wireless sensor networks (WSNs). As a popular data collection scheme, the compressive sensing- (CS-) based data collection schemes own many advantages from the perspectives of energy efficiency and load balance. Compared to the dense sensing matrices, applications of the sparse random matrices are able to further improve the performance of CS-based data collection schemes. In this paper, we proposed a compressive data collection scheme based on random walks, which exploits the compressibility of data vectors in the network. Each measurement was collected along a random walk that is modeled as a Markov chain. The Minimum Expected Cost Data Collection (MECDC) scheme was proposed to iteratively find the optimal transition probability of the Markov chain such that the expected cost of a random walk could be minimized. In the MECDC scheme, a nonuniform sparse random matrix, which is equivalent to the optimal transition probability matrix, was adopted to accurately recover the original data vector by using the nonuniform sparse random projection (NSRP) estimator. Simulation results showed that the proposed scheme was able to reduce the energy consumption and balance the network load.

Rate-distortion model for grayscale-invariance reversible data hiding

Recently, a novel scheme called grayscale-invariance reversible data hiding (RDH) has been developed, in which the generated color marked image will have the same grayscale as that of the host color image. Due to the property of grayscale invariance, for many cases, the further applications and image processing of the marked image will not be interfered with. However, the performance of current grayscale-invariance RDH is unsatisfactory, and the corresponding theoretical model is also lacking. For grayscale-invariance RDH, the modifications of the three color channels are interrelated. By decomposing the joint modification and associating the modification distortions of the green scales and blue scales with those of the red scales, we first approximate the modification-independent rate-distortion model for grayscale-invariance RDH. Subsequently, the optimal transition probability matrix can be estimated, according to which we implement the theoretically proven optimal modification, i.e., recursive code construction (RCC), to finish message embedding. The experimental results show that the presented method extends the applications of RCC and can significantly improve the performance of grayscale-invariance RDH.

Fastest Mixing Reversible Markov Chain: Clique Lifted Graphs and Subgraphs

Markov chains are one of the well-known tools for modeling and analyzing stochastic systems. At the same time, they are used for constructing random walks that can achieve a given stationary distribution. This paper is concerned with determining the transition probabilities that optimize the mixing time of the reversible Markov chains towards a given equilibrium distribution. This problem is referred to as the Fastest Mixing Reversible Markov Chain (FMRMC) problem. It is shown that for a given base graph and its clique lifted graph, the FMRMC problem over the clique lifted graph is reducible to the FMRMC problem over the base graph, while the optimal mixing times on both graphs are identical. Based on this result and the solution of the semidefinite programming formulation of the FMRMC problem, the problem has been addressed over a wide variety of topologies with the same base graph. Second, the general form of the FMRMC problem is addressed on stand-alone topologies as well as subgraphs of an arbitrary graph. For subgraphs, it is shown that the optimal transition probabilities over edges of the subgraph can be determined independent of rest of the topology.

On the Capacity of a Two-Hop Half-Duplex Relay Channel With a Markovian Constrained Relay

Consider a two-hop half-duplex relay channel (source–relay–destination) with a Markovian-constrained relay. The capacity of such channel is shown to be equal to the well-known cut-set upper bound. For the case where the relay-to-destination link is noise-free, the optimal state transition probabilities that give rise to the capacity are determined. This result links the relay channel to Shannon’s entropy maximization by introducing a relay adjacency matrix . For the case where both source-to-relay and relay-to-destination links are noisy, lower bounds on the achievable information rates for various constrained sequences are computed. We conjecture that our numerical bounds are tight. The numerically computed capacities and optimized information rates are significantly higher than the rate achieved by the traditional predetermined time-sharing scheme.

Reversible Data Hiding under Inconsistent Distortion Metrics.

Recursive code construction (RCC), based on the optimal transition probability matrix (OTPM), approaching the rate-distortion bound of reversible data hiding (RDH) has been proposed. Using the existing methods, OTPM can be effectively estimated only for a consistent distortion metric, i.e., if the host elements at different positions share the same distortion metric. However, in many applications, the distortion metrics are position dependent and should thus be inconsistent. Inconsistent distortion metrics can usually be quantified as a multi-distortion metric. In this paper, we first formulate the rate-distortion problem of RDH under a multi-distortion metric and subsequently propose a general framework to estimate the corresponding OTPM, with which RCC is extended to approach the rate-distortion bound of RDH under the multi-distortion metric. We apply the proposed framework to two examples of inconsistent distortion metrics: RDH in color image and reversible steganography. The experimental results show that the proposed method can efficiently improve upon the existing techniques.

Minimum Structural Similarity Distortion for Reversible Data Hiding

Until now, most Reversible data hiding (RDH) techniques have been evaluated by Peak signal-tonoise ratio (PSNR), which based on Mean squared error (MSE). Unfortunately, MSE turns out to be an extremely poor measure when the purpose is to predict perceived signal fidelity or quality. The Structural similarity (SSIM) index has gained widespread popularity as an alternative motivating principle for the design of image quality measures. How to utilize the characterize of SSIM to design RDH algorithm is very critical. We propose an optimal RDH algorithm under structural similarity constraint. We deduce the metric of the structural similarity constraint, and further we prove it does not hold Non-crossing-edges (NCE) property. We construct the rate-distortion function of optimal structural similarity constraint, which is equivalent to minimize the average distortion for a given embedding rate, and then we can obtain the optimal transition probability matrix under the structural similarity constraint. Experiments show that our proposed method can be used to improve the performance of previous RDH schemes evaluated by SSIM.

Model-Free Deep Inverse Reinforcement Learning by Logistic Regression

This paper proposes model-free deep inverse reinforcement learning to find nonlinear reward function structures. We formulate inverse reinforcement learning as a problem of density ratio estimation, and show that the log of the ratio between an optimal state transition and a baseline one is given by a part of reward and the difference of the value functions under the framework of linearly solvable Markov decision processes. The logarithm of density ratio is efficiently calculated by binomial logistic regression, of which the classifier is constructed by the reward and state value function. The classifier tries to discriminate between samples drawn from the optimal state transition probability and those from the baseline one. Then, the estimated state value function is used to initialize the part of the deep neural networks for forward reinforcement learning. The proposed deep forward and inverse reinforcement learning is applied into two benchmark games: Atari 2600 and Reversi. Simulation results show that our method reaches the best performance substantially faster than the standard combination of forward and inverse reinforcement learning as well as behavior cloning.

Optimal structural similarity constraint for reversible data hiding

Until now, most reversible data hiding techniques have been evaluated by peak signal-to-noise ratio(PSNR), which based on mean squared error(MSE). Unfortunately, MSE turns out to be an extremely poor measure when the purpose is to predict perceived signal fidelity or quality. The structural similarity (SSIM) index has gained widespread popularity as an alternative motivating principle for the design of image quality measures. How to utilize the characterize of SSIM to design RDH algorithm is very critical. In this paper, we propose an optimal RDH algorithm under structural similarity constraint. Firstly, we deduce the metric of the structural similarity constraint, and further we prove it does't hold non-crossing-edges property. Secondly, we construct the rate-distortion function of optimal structural similarity constraint, which is equivalent to minimize the average distortion for a given embedding rate, and then we can obtain the optimal transition probability matrix under the structural similarity constraint. Comparing with previous RDH, our method have gained the improvement of SSIM about 1.89 % on average. Experiments show that our proposed method outperforms the state-of-arts performance in SSIM.

A Dynamic Gesture Trajectory Recognition Based on Key Frame Extraction and HMM

Aiming at changing high computational complexity, underdeveloped real time, low recognition rate of dynamic gesture recognition algorithms, this paper present a real-time dynamic gesture trajectory recognition method based on key frame extraction and HMM. Key frames are selected without keeping track of all the details of one dynamic gesture, which is based on difference degree between frames. The trajectory data stream, sorted by the time-warping algorithm, is used to construct the Hidden Markov Method model of dynamic gesture. Finally, optimal transition probabilities are employed to implement dynamic gesture recognition. The result of this experiment implies that this method has high robustness and real time. The average recognition rate of dynamic gesture (0~9) is up to 87.67%, and average time efficiency is 0.46s.

Optimal transition probability of reversible data hiding for general distortion metrics and its applications.

Recently, a recursive code construction (RCC) approaching the rate-distortion bound of reversible data hiding (RDH) was proposed. However, to estimate the rate-distortion bound or execute RCC, one should first estimate the optimal transition probability matrix (OTPM). By previous methods, OTPM can be effectively estimated only for some specific distortion metrics, such as square error distortion or L1-Norm. In this paper, we proposed a unified framework of estimating the OTPM for general distortion metrics, with which we can calculate the rate-distortion bound of RDH for general cases and extend RCC to improve state-of-the-art RDH schemes based on any distortion metrics.

Emotional Agents at the Square Lattice

agents. Group emotions in the considered models are periodically fluctuating between two opposite valency levels and as result the mean value of such group emotions is zero. The oscillations amplitude depends strongly on probability ps of the individual spontaneous arousal. For small values of relaxation times ? we observed a stochastic resonance, i.e. the signal to noise ratio SNR is maximal for a non-zero ps parameter. The amplitude increases with the probability p of local aective interactions while the mean oscillations period increases with the relaxation time ? and is only weakly dependent on other system parameters. Presence of emotional antenna can enhance positive or negative emotions and for the optimal transition probability the antenna can change agents emotions at longer distances. The stochastic resonance was also observed for the influence of emotions on task execution eciency.

Importance Sampling Method of Software Reliability Estimation

This paper proposes an effective method for computing optimal state transition probabilities for software reliability estimation based on a Markov usage model.This method uses Cross-Entropy to measure the differences between the operational profile and the sampling distribution with zero variance.By adjusting the probabilities of state transitions during test,an iterative method based on the Cross-Entropy is proposed for this choice,and an unbiased reliability estimator with zero variance is obtained.Simulation results show that the testing profile with Cross-Entropy method performs significantly better than the simulated annealing algorithm.Moreover,this strategy can more effectively accelerate software statistical testing.