Markov Model Parameters Research Articles

Parameter synthesis for Markov models: covering the parameter space

Markov chain analysis is a key technique in formal verification. A practical obstacle is that all probabilities in Markov models need to be known. However, system quantities such as failure rates or packet loss ratios, etc. are often not—or only partially—known. This motivates considering parametric models with transitions labeled with functions over parameters. Whereas traditional Markov chain analysis relies on a single, fixed set of probabilities, analysing parametric Markov models focuses on synthesising parameter values that establish a given safety or performance specification φ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varphi $$\\end{document}. Examples are: what component failure rates ensure the probability of a system breakdown to be below 0.00000001?, or which failure rates maximise the performance, for instance the throughput, of the system? This paper presents various analysis algorithms for parametric discrete-time Markov chains and Markov decision processes. We focus on three problems: (a) do all parameter values within a given region satisfy φ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varphi $$\\end{document}?, (b) which regions satisfy φ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varphi $$\\end{document} and which ones do not?, and (c) an approximate version of (b) focusing on covering a large fraction of all possible parameter values. We give a detailed account of the various algorithms, present a software tool realising these techniques, and report on an extensive experimental evaluation on benchmarks that span a wide range of applications.

Parallel EM algorithm on Hadoop for large-scale hidden Markov model parameter estimation

This paper aims to propose a parallel EM algorithm based on Hadoop for parameter estimation of large-scale hidden Markov models (HMM). HMM is a commonly used statistical model. However, since the parameter estimation of HMM involves the storage and processing of large-scale data sets, traditional serial algorithms have certain limitations in efficiency. This paper introduces the Hadoop parallel computing framework, divides the task into multiple subtasks through the MapReduce programming model and assigns them to different machines for parallel computing, which improves the efficiency and scalability of parameter estimation. The results show that using parallel EM algorithm for large-scale hidden Markov model parameter estimation on Hadoop is feasible and effective.

Time-Adaptive Expectation Maximization Learning Framework for HMM Based Data-Driven Gas Sensor Calibration

In this paper, data-driven self-calibration algorithms for the low-cost gas sensors are designed. The sensor measurement errors happen due to the imperfect compensation for the variation of sensor component behavior that is caused by changing of environmental factors. To calibrate the sensors, the hidden Markov model is utilized to characterize the statistical dependency between the environmental factors and the variation of sensor component behavior. Considering the time-varying property of this dependency, a time-adaptive learning framework is further designed to update the hidden Markov model so that the time-varying drift process can be better tracked over a long term. More specifically, a time-adaptive expectation maximization learning approach is proposed to efficiently update the hidden Markov model parameters. A closed form of the convergence rate of this time-adaptive learning approach is derived, which provides a theoretical guarantee on the time efficiency as well as the computational efficiency. The performance of the scheme is illustrated in numerical experiments utilizing real data, which shows that long-term stable calibration performance can be achieved.

Computational issues in parameter estimation for hidden Markov models with template model builder

A popular way to estimate the parameters of a hidden Markov model (HMM) is direct numerical maximization (DNM) of the (log-)likelihood function. The advantages of employing the TMB [Kristensen K, Nielsen A, Berg C, et al. TMB: automatic differentiation and Laplace approximation. J Stat Softw Articles. 2016;70(5):1–21] framework in R for this purpose were illustrated recently [Bacri T, Berentsen GD, Bulla J, et al. A gentle tutorial on accelerated parameter and confidence interval estimation for hidden Markov models using template model builder. Biom J. 2022 Oct;64(7):1260–1288]. In this paper, we present extensions of these results in two directions. First, we present a practical way to obtain uncertainty estimates in form of confidence intervals (CIs) for the so-called smoothing probabilities at moderate computational and programming effort via TMB. Our approach thus permits to avoid computer-intensive bootstrap methods. By means of several examples, we illustrate patterns present for the derived CIs. Secondly, we investigate the performance of popular optimizers available in R when estimating HMMs via DNM. Hereby, our focus lies on the potential benefits of employing TMB. Investigated criteria via a number of simulation studies are convergence speed, accuracy, and the impact of (poor) initial values. Our findings suggest that all optimizers considered benefit in terms of speed from using the gradient supplied by TMB. When supplying both gradient and Hessian from TMB, the number of iterations reduces, suggesting a more efficient convergence to the maximum of the log-likelihood. Last, we briefly point out potential advantages of a hybrid approach.

Selection of features and hidden Markov model parameters for English word recognition from Leap Motion air‐writing trajectories

AbstractAir‐writing recognition is relevant in areas such as natural human–computer interaction, augmented reality, and virtual reality. A trajectory is the most natural way to represent air writing. We analyze the recognition accuracy of words written in air considering five features, namely, writing direction, curvature, trajectory, orthocenter, and ellipsoid, as well as different parameters of a hidden Markov model classifier. Experiments were performed on two representative datasets, whose sample trajectories were collected using a Leap Motion Controller from a fingertip performing air writing. Dataset contains 840 English words from 21 classes, and dataset contains 1600 English words from 40 classes. A genetic algorithm was combined with a hidden Markov model classifier to obtain the best subset of features. Combination trajectory, orthocenter, writing direction, curvature provided the best feature set, achieving recognition accuracies on datasets and of 98.81% and 83.58%, respectively.

Reliability Assessment of Distribution Power Transformers Considering Load Transfer Capability

A number of reliability models based on stationary Markov processes were developed at Hydro One to evaluate the reliability of a group of similar distribution power transformers. Those models accounted for a number of factors that affected the reliability of the group. The factors included, in addition to the basic Markov model parameters, the redundancy and mobile unit transformer capability at each individual station. There was one factor namely the load transfer capability between the stations that was not considered in the reliability assessment. The objective of load transfer is to further reduce the impact of transformer outages on customers and therefore improve the overall reliability of the system. The policy of load transfer between distribution stations can be simple or complicated and that can have a significant impact on the overall system reliability. This paper describes a probabilistic method based on stationary Markov models for evaluating the impacts of load transfers on the reliability of distribution stations. An example of a real distribution system is used to illustrate the proposed method of assessment.

The machine learning pipeline to unravel the functional effects of interactions between sodium channels.

Recent reports obtained using patch clamp approach in Na+ channel (NaV) pairs have suggested cooperative gating of two NaVs, which has been linked to their homodimerization. Yet, the existing methods of the cooperative gating analysis are not effective in unraveling alterations in kinetics of individual NaVs result from their interactions in dimers. To address this issue, we developed a machine learning pipeline to remove capacitance drift and noise within the signal, and detect channel activity in order to infer Markov models parameters from single and multi-channel recordings.

МЕТОДИ ПІДВИЩЕННЯ ЯКОСТІ ПЕРЕТВОРЕННЯ МОВИ НА ТЕКСТ В СИСТЕМАХ БІОМЕТРИЧНОЇ АУТЕНТИФІКАЦІЇ

The article discusses the methods and algorithms of speech-to-text conversion, modern open and commercial systems for creating systems, as well as the use of these technologies in the field of cyber security. It is proposed to create a high-quality speech-to-text conversion system. An analysis of the mathematical algorithms used to reduce the error rate, which makes it possible to create unique voice prints and increase protection against forgery, has been carried out. The structure of modern speech-to-text conversion systems is described. By changing datasets, parameters of hidden Markov models, a high-quality dictionary of phonemes, and the use of language models, there is an opportunity to reduce the percentage of errors in language recognition, as well as the use of a system for multilingualism such as "surzhyk". The mathematical methods of assessing the quality of the system of speech to text (WER), as well as various methods of calculation, which is important for their further improvement and optimization, are considered. The structure of modern systems is considered, namely, signal pre-processing, feature extraction, acoustic modeling, speech modeling, decoding, post-processing. For each of the stages, study vectors have been proposed that can reduce the error rate of the system as a whole. Reducing speech recognition errors and the ability to fake a voice is achieved using various methods: deep neural networks, hidden Markov models, Baum-Welch algorithm, N-gram models, models with attention, creation of a high-quality phonemes dictionary, dataset, and fillers. Speech-to-text conversion technology can be used in biometric authentication systems to detect and analyze the unique features of the user's voice. However, modern speech-to-text conversion systems for Ukrainian, Russian, and "surzhyk" need improvement in acoustic and language units. Scientific works, which are devoted to research and optimization of these systems for biometric authentication, do not fully cover these issues. This became the reason for further research in this direction, so this work aims to create a speech recognition system with a minimum error rate.

Prediction of schizophrenia from activity data using hidden Markov model parameters

In this paper, we address the problem of predicting schizophrenia based on a persons measured motor activity over time. A key challenge to achieve this is how to extract features from the activity data that can efficiently separate schizophrenia patients from healthy subjects. To achieve this, we suggest to fit time dependent hidden Markov models with and without integrated covariates and letting the estimated model parameters represent our features. To further evaluate the efficiency of these features, we suggest to use them as features in a classification method (logistic regression) to separate schizophrenia patients from healthy subjects. The results show that the estimated hidden Markov model parameters are well-performing in predicting schizophrenia, and outperform features derived from other methods in the literature in terms of goodness-of-fit and classification performance.

Identification of Continuous-Discrete Hidden Markov Models with Multiplicative Observation Noise

The paper aims to identify hidden Markov model parameters. The unobservable state represents a finite-state Markov jump process. The observations contain Wiener noise with state-dependent intensity. The identified parameters include the transition intensity matrix of the system state, conditional drift and diffusion coefficients in the observations. We propose an iterative identification algorithm based on the fixed-interval smoothing of the Markov state. Using the calculated state estimates, we restore all required system parameters. The paper contains a detailed description of the numerical schemes of state estimation and parameter identification. The comprehensive numerical study confirms the high precision of the proposed identification estimates.

Characterising Eye Movement Events with an Unsupervised Hidden Markov Model.

Eye-tracking allows researchers to infer cognitive processes from eye movements that are classified into distinct events. Parsing the events is typically done by algorithms. Here we aim at developing an unsupervised, generative model that can be fitted to eye-movement data using maximum likelihood estimation. This approach allows hypothesis testing about fitted models, next to being a method for classification. We developed gazeHMM, an algorithm that uses a hidden Markov model as a generative model, has few critical parameters to be set by users, and does not require human coded data as input. The algorithm classifies gaze data into fixations, saccades, and optionally postsaccadic oscillations and smooth pursuits. We evaluated gazeHMM's performance in a simulation study, showing that it successfully recovered hidden Markov model parameters and hidden states. Parameters were less well recovered when we included a smooth pursuit state and/or added even small noise to simulated data. We applied generative models with different numbers of events to benchmark data. Comparing them indicated that hidden Markov models with more events than expected had most likely generated the data. We also applied the full algorithm to benchmark data and assessed its similarity to human coding and other algorithms. For static stimuli, gazeHMM showed high similarity and outperformed other algorithms in this regard. For dynamic stimuli, gazeHMM tended to rapidly switch between fixations and smooth pursuits but still displayed higher similarity than most other algorithms. Concluding that gazeHMM can be used in practice, we recommend parsing smooth pursuits only for exploratory purposes. Future hidden Markov model algorithms could use covariates to better capture eye movement processes and explicitly model event durations to classify smooth pursuits more accurately.

Towards optimum smoking cessation interventions during pregnancy: a household model to explore cost-effectiveness.

Background and AimsPrevious economic evaluations of smoking cessation interventions for pregnant women are limited to single components, which do not in isolation offer sufficient potential impact to address smoking cessation targets. To inform the development of more appropriate complex interventions, we (1) describe the development of the Economics of Smoking in Pregnancy: Household (ESIP.H) model for estimating the life‐time cost‐effectiveness of smoking cessation interventions aimed at pregnant women and (2) use a hypothetical case study to demonstrate how ESIP.H can be used to identify the characteristics of optimum smoking cessation interventions.MethodsThe hypothetical intervention was based on current evidence relating to component elements, including financial incentives, partner smoking, intensive behaviour change support, cigarettes consumption and duration of support to 12 months post‐partum. ESIP.H was developed to assess the life‐time health and cost impacts of multi‐component interventions compared with standard National Health Service (NHS) care in England. ESIP.H considers cigarette consumption, partner smoking and some health conditions (e.g. obesity) that were not included in previous models. The Markov model's parameters were estimated based on published literature, expert judgement and evidence‐based assumptions. The hypothetical intervention was evaluated from an NHS perspective.ResultsThe hypothetical intervention was associated with an incremental gain in quitters (mother and partner) at 12 months postpartum of 249 [95% confidence interval (CI) = 195–304] per 1000 pregnant smokers. Over the long‐term, it had an incremental negative cost of £193 (CI = –£779 to 344) and it improved health, with a 0.50 (CI = 0.36–0.69) increase in quality‐adjusted life years (QALYs) for mothers, partners and offspring, with a 100% probability of being cost‐effective.ConclusionsThe Economics of Smoking in Pregnancy: Household model for estimating cost‐effectiveness of smoking cessation interventions aimed at pregnant women found that a hypothetical smoking cessation intervention would greatly extend reach, reduce smoking and be cost‐effective.

A gentle tutorial on accelerated parameter and confidence interval estimation for hidden Markov models using Template Model Builder.

A very common way to estimate the parameters of a hidden Markov model (HMM) is the relatively straightforward computation of maximum likelihood (ML) estimates. For this task, most users rely on user-friendly implementation of the estimation routines via an interpreted programming language such as the statistical software environment R. Such an approach can easily require time-consuming computations, in particular for longer sequences of observations. In addition, selecting a suitable approach for deriving confidence intervals for the estimated parameters is not entirely obvious, and often the computationally intensive bootstrap methods have to be applied. In this tutorial, we illustrate how to speed up the computation of ML estimates significantly via the R package TMB. Moreover, this approach permits simple retrieval of standard errors at the same time. We illustrate the performance of our routines using different data sets: first, two smaller samples from a mobile application for tinnitus patients and a well-known data set of fetal lamb movements with 87 and 240 data points, respectively. Second, we rely on larger data sets of simulated data of sizes 2000 and 5000 for further analysis. This tutorial is accompanied by a collection of scripts, which are all available in the Supporting Information. These scripts allow any user with moderate programming experience to benefit quickly from the computational advantages of TMB.

Relationship between risk, cumulative burden of exacerbations and mortality in patients with COPD: modelling analysis using data from the ETHOS study

BackgroundThe major drivers of cost-effectiveness for chronic obstructive pulmonary disease (COPD) therapies are the occurrence of exacerbations and deaths. Exacerbations, including acute and long-term events, can cause worsening of COPD and lead to an increased risk of further exacerbations, and ultimately may elevate the risk of death. In contrast to this, health economic models are based on COPD severity progression. In this post hoc analysis of the ETHOS study, we focus on the progression of COPD due to exacerbations and deaths.MethodsWe fitted semi-parametric and fully parametric multi-state Markov models with the following five progressive states: State 1, no exacerbation; State 2, 1 moderate exacerbation; State 3, ≥ 2 moderate exacerbations; State 4, ≥ 1 severe exacerbations; State 5, death. The models only allowed a patient to transition to a worsened health state, and transitions did not necessarily have to be to the next adjacent state. We used the multi-state models to analyse data from ETHOS, a phase III, 52-week study assessing the efficacy and safety of triple therapy with budesonide/glycopyrronium/formoterol fumarate dihydrate in moderate-to-very severe COPD.ResultsThe Weibull multi-state Markov model showed good fit of the data. In line with clinical evidence, we found a higher mortality risk after a severe exacerbation (11.4-fold relative ratio increase [95% CI, 7.7–17.0], 6.4-fold increase [95% CI, 3.8–10.8] and 5.4-fold increase [95% CI, 2.9–10.3] relative to no exacerbations, 1 moderate exacerbation or ≥ 2 moderate exacerbations, respectively). One moderate exacerbation increased mortality risk 1.8-fold (95% CI, 1.1–2.9) vs no exacerbations. We also found a higher risk of severe exacerbation and mortality following ≥ 2 moderate exacerbations.ConclusionMulti-state modelling of patients with COPD in ETHOS found an acute and chronic effect of severe exacerbations on mortality risk. Risk was also increased after a moderate exacerbation. Clinical management with effective pharmacotherapies should be optimised to avoid even moderate exacerbations. Modelling with exacerbations could be an alternative to current COPD models focused on disease progression.Trial registrationNCT02465567

Object-Based Scene Classification Modeled by Hidden Markov Models Architecture

Multiclass classification problems such as document classification, medical diagnosis or scene classification are very challenging to address due to similarities between mutual classes. The use of reliable tools is necessary to get good classification results. This paper addresses the scene classification problem using objects as attributes. The process of classification is modeled by a famous mathematical tool: The Hidden Markov Models. We introduce suitable relations that scale the parameters of the Hidden Markov Model into variables of scene classification. The construction of Hidden Markov Chains is supported with weight measures and sorting functions. Lastly, inference algorithms extract most suitable scene categories from the Discrete Markov Chain. A parallelism approach constructs several Discrete Markov Chains in order to improve the accuracy of the classification process. We provide numerous tests on different datasets and compare classification accuracies with some state of the art methods. The proposed approach distinguishes itself by outperforming the other.

Data-driven Steering Torque Behaviour Modelling with Hidden Markov Models

Modern Advanced Driver Assistance Systems (ADAS) are limited in their ability to consider the driver's intention, resulting in unnatural guidance and low customer acceptance. In this research, we focus on a novel data-driven approach to predict driver steering torque. In particular, driver behavior is modeled by learning the parameters of a Hidden Markov Model (HMM) and estimation is performed with Gaussian Mixture Regression (GMR). An extensive parameter selection framework enables us to objectively select the model hyper-parameters and prevents overfitting. The final model behavior is optimized with a cost function balancing between accuracy and smoothness. Naturalistic driving data covering seven participants is obtained using a static driving simulator at Toyota Motor Europe for the training, evaluation, and testing of the proposed model. The results demonstrate that our approach achieved a 92% steering torque accuracy with a 37% increase in signal smoothness and 90% fewer data compared to a baseline. In addition, our model captures the complex and nonlinear human behavior and inter-driver variability from novice to expert drivers, showing an interesting potential to become a steering performance predictor in future user-oriented ADAS.

Эвристическая методика настройки скрытых марковских моделей для распознавания образов стохастических процессов

Предложена эвристическая методика настройки параметров скрытых марковских моделей для реализации в алгоритмах распознавания образов стохастических процессов, регистрируемых в форме последовательностей наблюдений. Методика позволяет получить рабочие модели по небольшому числу обучающих реализаций и включает выполнение двух этапов: предварительная настройка начальных параметров модели с использованием априорных данных и обучение с применением алгоритма Баума - Уэлча. На обоих этапах используется дополнительная процедура корректировки параметров модели, позволяющая устранять проблемы численного характера при их реализации в алгоритмах распознавания. Представлены результаты экспериментальной апробации методики на примере построения моделей для алгоритма распознавания образов трех стохастических процессов.

Learning hidden Markov models from aggregate observations

In this paper, we propose an algorithm for estimating the parameters of a time-homogeneous hidden Markov model (HMM) from aggregate observations. This problem arises when only the population level counts of the number of individuals at each time step are available, and one seeks to learn the individual HMM from these observations. Our algorithm is built upon the classical expectation–maximization algorithm and the recently proposed aggregate inference algorithm (Sinkhorn belief propagation). We present the parameter learning algorithm for two different settings of HMMs: one with discrete observations and one with continuous observations, and the algorithm exhibits convergence guarantees in both cases. Moreover, our learning framework naturally reduces to the standard Baum–Welch learning algorithm for HMMs when the population size is 1. The efficacy of our algorithm is demonstrated through several numerical experiments.

Object-Based Scene Classification Modeled by Hidden Markov Models Architecture

Multiclass classification problems such as document classification, medical diagnosis or scene classification are very challenging to address due to similarities between mutual classes. The use of reliable tools is necessary to get good classification results. This paper addresses the scene classification problem using objects as attributes. The process of classification is modeled by a famous mathematical tool: The Hidden Markov Models. We introduce suitable relations that scale the parameters of the Hidden Markov Model into variables of scene classification. The construction of Hidden Markov Chains is supported with weight measures and sorting functions. Lastly, inference algorithms extract most suitable scene categories from the Discrete Markov Chain. A parallelism approach constructs several Discrete Markov Chains in order to improve the accuracy of the classification process. We provide numerous tests on different datasets and compare classification accuracies with some state of the art methods. The proposed approach distinguishes itself by outperforming the other.

Clinical outcomes and cost associated with HRD biomarker guided first line maintenance therapy in advanced ovarian cancer

<h3>Objectives:</h3> Olaparib is approved as a first line (1L) maintenance (mtx) option, as monotherapy for <i>BRCA</i> mutated (BRCAm) ovarian cancer (OC) and in combination with bevacizumab (Bev) for homologous recombination deficient (HRD) OC. Niraparib monotherapy is approved as 1L mtx irrespective of BRCA/HRD status. Despite its broad approval, niraparib demonstrated relatively shorter progression free survival (PFS) gains outside of an HRD test positive population in the PRIMA trial. Given the recent PARPi data in frontline mtx and the cost of HRD testing beyond standard of care g<i>BRCA</i> testing, there is debate over using an HRD biomarker guided (BMG) or non-biomarker guided (NBMG) approach to prescribing 1L mtx. The objective of this model is to compare the PFS and costs per month of PFS between: (1) BMG 1L mtx approach which restricts the prescribing of 1L mtx PARPi and PARPi combination based on Myriad myChoice results vs (2) NBMG PARPi mtx for all approach. <h3>Methods:</h3> This model is a parametric Markov model for a cohort of 100 newly diagnosed stage 3/4 OC patients (pts) prescribed 1L mtx according to HRD BMG and NBMG approaches. The model focuses on maintenance after first line treatment and evaluates costs and PFS after first line chemotherapy. To focus on the impact of testing, drugs are treated as classes. Treatment costs include 1L mtx PARP inhibitor (PARPi), Bev, and PARPi + Bev, subsequent therapy, management of adverse events, and administration costs over 60 months. PFS is estimated by applying an exponential curve to adjusted trial data. The BMG pt group undergoes HRD tumor testing and has a test positive (HRD+) rate of 47% (20% BRCAm, 27% non-BRCAm). Germline <i>BRCA</i>m testing cost is not included as it is assumed to be the standard of care and equally conducted in both groups. An estimated 65% are eligible for a PARPi (received and responded to 1L platinum chemotherapy). Calculations for the BMG PARPi eligible group assume all HRD+ pts receive PARPi + Bev and 30% of HRD test negative pts receive Bev (70% receive routine surveillance). The model assumes all pts in the NBMG group receive a PARPi. <h3>Results:</h3> In the BMG group 31 pts receive PARPi + Bev mtx and in the NBMG arm 65 pts receive PARPi mtx. The 31 BMG PARPi + Bev pts averaged 36.6 months of 1L PFS compared to 26.3 months in the 65 NBMG PARPi pts. The overall 1L PFS for the total cohorts were similar but slightly favored the BMG approach (1715.4 vs 1711.7 months). Over a 60 month horizon, the BMG approach was $1,002 less per month of 1L PFS gained ($15,475 vs. $16,477). HRD biomarker testing represented 1.5% of the total costs for the cohort. <h3>Conclusions:</h3> Despite initial testing costs, this model projects that use of an HRD BMG approach with PARPi + bev may minimize long-term costs and afford at least similar, if not greater, 1L PFS compared to an approach which omits HRD tumor testing and prescribes PARPi to all-comers.