Input-output Hidden Markov Model Research Articles

Hierarchical framework for interpretable and specialized deep reinforcement learning-based predictive maintenance

Deep reinforcement learning holds significant potential for application in industrial decision-making, offering a promising alternative to traditional physical models. However, its black-box learning approach presents challenges for real-world and safety-critical systems, as it lacks interpretability and explanations for the derived actions. Moreover, a key research question in deep reinforcement learning is how to focus policy learning on critical decisions within sparse domains. This paper introduces a novel approach that combines probabilistic modeling and reinforcement learning, providing interpretability and addressing these challenges in the context of safety-critical predictive maintenance. The methodology is activated in specific situations identified through the input–output hidden Markov model, such as critical conditions or near-failure scenarios. To mitigate the challenges associated with deep reinforcement learning in safety-critical predictive maintenance, the approach is initialized with a baseline policy using behavioral cloning, requiring minimal interactions with the environment. The effectiveness of this framework is demonstrated through a case study on predictive maintenance for turbofan engines, outperforming previous approaches and baselines, while also providing the added benefit of interpretability. Importantly, while the framework is applied to a specific use case, this paper aims to present a general methodology that can be applied to diverse predictive maintenance applications.

RUL management by production reference loopback

Online remaining useful life (RUL) assessment is a significant asset in prognostic and health management (PHM) in many industrial domains where safety, reliability, and cost reduction are of high importance. It is not easy to predict the breakdown state of a system when it operates under multiple operating conditions, because system degradation varies with the dynamics of the operations. This paper presents an Input-Output Hidden Markov Model (IOHMM) that estimates the RUL in real time based on available measurements. The model learns the impact of the operating condition on the RUL and allows to manage the system RUL by changing the corresponding operating conditions. A reference managing algorithm is presented to match the estimated RUL to a given target RUL. In addition, well-known algorithms are adapted from HMM to IOHMM and are used for model training and health state diagnostics. A numerical application is proposed to show the importance of obtaining good predictions from a limited amount of data sequences. Specifically, since degradation is a slow process, it is difficult to have a large amount of data sequences in order to predict the RUL more accurately until the failure. Therefore, the bootstrap method with data resampling and replacement is used to train the IOHMM model to improve estimation accuracy.

Integrated energy management of hybrid power supply based on short-term speed prediction

The energy management strategy and output limitation of energy storage system affect the actual regenerative braking recovery. In order to optimize the performance and energy efficiency of vehicle energy storage system in the process of braking energy recovery, an integrated energy management strategy based on short-term speed prediction is proposed in this paper. Firstly, support vector machine classification method is used for off-line training and on-line recognition of driving patterns, and then input-output hidden Markov model and Gaussian mixture model are fused to establish speed prediction models under urban, suburban, and high-speed conditions for real-time prediction of speed sequences. Then, the power is distributed by the dynamic programming algorithm according to the short-term speed prediction results, and the distribution coefficient is corrected through the integrated strategy of braking force distribution and power distribution. Finally, the simulation results under continuously changing driving conditions show that the pattern recognition with changeable span is conducive to improving its accuracy during the transition period, and the proposed integrated energy management strategy is helpful to optimizing the performance of the hybrid energy storage system. The renewable energy utilization efficiency index indicates the efficiency of the integrated energy management strategy.

Individual Mobility Prediction in Mass Transit Systems Using Smart Card Data: An Interpretable Activity-Based Hidden Markov Approach

Individual mobility is driven by demand for activities with diverse spatiotemporal patterns, but existing methods for mobility prediction often overlook the underlying activity patterns. Knowledge of activity patterns can improve the performance and interpretability of existing individual mobility models, leading to more informed policy design and better user experience in intelligent transportation systems. This study develops an activity-based modeling framework for individual mobility prediction in mass transit systems. Specifically, an input-output hidden Markov model (IOHMM) approach is proposed to simultaneously predict the (continuous) time and (discrete) location of an individual’s next trip using transit smart card data. The prediction task can be transformed into predicting the hidden activity duration and end location. Based on a case study of Hong Kong’s metro system, we show that the proposed model can achieve similar prediction performance as the state-of-the-art long short-term memory (LSTM) model. Unlike LSTM, the proposed IOHMM approach can also be used to analyze hidden activity patterns, which provides meaningful behavioral interpretation for why an individual makes a certain trip. Therefore, the activity-based prediction framework offers a way to preserve the predictive power of advanced machine learning methods while enhancing our ability to generate insightful behavioral explanations, which is useful for user-centric policy design and intelligent transportation applications such as personalized traveler information.

Learning Dynamics and Control of a Stochastic System under Limited Sensing Capabilities.

The operation of a variety of natural or man-made systems subject to uncertainty is maintained within a range of safe behavior through run-time sensing of the system state and control actions selected according to some strategy. When the system is observed from an external perspective, the control strategy may not be known and it should rather be reconstructed by joint observation of the applied control actions and the corresponding evolution of the system state. This is largely hurdled by limitations in the sensing of the system state and different levels of noise. We address the problem of optimal selection of control actions for a stochastic system with unknown dynamics operating under a controller with unknown strategy, for which we can observe trajectories made of the sequence of control actions and noisy observations of the system state which are labeled by the exact value of some reward functions. To this end, we present an approach to train an Input–Output Hidden Markov Model (IO-HMM) as the generative stochastic model that describes the state dynamics of a POMDP by the application of a novel optimization objective adopted from the literate. The learning task is hurdled by two restrictions: the only available sensed data are the limited number of trajectories of applied actions, noisy observations of the system state, and system state; and, the high failure costs prevent interaction with the online environment, preventing exploratory testing. Traditionally, stochastic generative models have been used to learn the underlying system dynamics and select appropriate actions in the defined task. However, current state of the art techniques, in which the state dynamics of the POMDP is first learned and then strategies are optimized over it, frequently fail because the model that best fits the data may not be well suited for controlling. By using the aforementioned optimization objective, we try to to tackle the problems related to model mis-specification. The proposed methodology is illustrated in a scenario of failure avoidance for a multi component system. The quality of the decision making is evaluated by using the collected reward on the test data and compared against the previous literature usual approach.

Generalized Input-Output Hidden-Markov-Models for Supervising Industrial Processes

Often in industrial processes, batches of material are processed sequentially and repeatedly through a deterministic sequence of process steps. The possibly large number of sensor measurements collected throughout such industrial processes require supervisory modeling techniques that allow for characterizing the operating conditions (or states) of these process steps. This would allow for inference and predictions over the operating conditions of the current and upcoming process steps (e.g., abnormal behavior, failures, etc.) which may significantly assist the operators of the process. Traditionally, such inference questions in repeated processes can be handled in a rather straightforward manner via Input-Output Hidden Markov Models (IOHMM). However, standard IOHMM are limited to repeated and identical processes, while industrial processes may comprise multiple non-identical process steps and possibly with non-standard interdependencies between process steps. For this reason, in this paper we propose a generalization of standard IOHMM that is more appropriate for modeling industrial processes. Furthermore, we derive the update recursions for training such models and we show analytically that such update recursions guarantee an increase in the likelihood of the observation sequences.

Probabilistic domain-knowledge modeling of disorder pathogenesis for dynamics forecasting of acute onset

Disease pathogenesis, a type of domain knowledge about biological mechanisms leading to diseases, has not been adequately encoded in machine-learning-based medical diagnostic models because of the inter-patient variabilities and complex dependencies of the underlying pathogenetic mechanisms. We propose 1) a novel pathogenesis probabilistic graphical model (PPGM) to quantify the dynamics underpinning patient-specific data and pathogenetic domain knowledge, 2) a Bayesian-based inference paradigm to answer the medical queries and forecast acute onsets. The PPGM model consists of two components: a Bayesian network of patient attributes and a temporal model of pathogenetic mechanisms. The model structure was reconstructed from expert knowledge elicitation, and its parameters were estimated using Variational Expectation-Maximization algorithms. We benchmarked our model with two well-established hidden Markov models (HMMs) – Input-output HMM (IO-HMM) and Switching Auto-Regressive HMM (SAR-HMM) – to evaluate the computational costs, forecasting performance, and execution time. Two case studies on Obstructive Sleep Apnea (OSA) and Paroxysmal Atrial Fibrillation (PAF) were used to validate the model. While the performance of the parameter learning step was equivalent to those of IO-HMM and SAR-HMM models, our model forecasting ability was outperforming those two models. The merits of the PPGM model are its representation capability to capture the dynamics of pathogenesis and perform medical inferences and its interpretability for physicians. The model has been used to perform medical queries and forecast the acute onset of OSA and PAF. Additional applications of the model include prognostic healthcare and preventive personalized treatments.

Modeling the dynamics of hurricane evacuation decisions from twitter data: An input output hidden markov modeling approach

Evacuations play a critical role in saving human lives during hurricanes. But individual evacuation decision-making is a complex dynamic process, often studied using post-hurricane survey data. Alternatively, ubiquitous use of social media generates a massive amount of data that can be used to predict evacuation behavior in real time. In this paper, we present a method to infer individual evacuation behaviors (e.g., evacuation decision, timing, destination) from social media data. We develop an input output hidden Markov model (IO-HMM) to infer evacuation decisions from user tweets. To extract the underlying evacuation context from tweets, we first estimate a word2vec model from a corpus of more than 100 million tweets collected over four major hurricanes. Using input variables such as evacuation context, time to landfall, type of evacuation order, and the distance from home, the proposed model infers what activities are made by individuals, when they decide to evacuate, and where they evacuate to. To validate our results, we have created a labeled dataset from 38,256 tweets posted between September 2, 2017 and September 19, 2017 by 2,571 users from Florida during hurricane Irma. Our findings show that the proposed IO-HMM method can be useful for inferring evacuation behavior in real time from social media data. Since traditional surveys are infrequent, costly, and often performed at a post-hurricane period, the proposed approach can be very useful for predicting evacuation demand as a hurricane unfolds in real time.

An IOHMM-Based Framework to Investigate Drift in Effectiveness of IoT-Based Systems.

IoT-based systems, when interacting with the physical environment through actuators, are complex systems difficult to model. Formal verification techniques carried out at design-time being often ineffective in this context, these systems have to be quantitatively evaluated for effectiveness at run-time, i.e., for the extent to which they behave as expected. This evaluation is achieved by confronting a model of the effects they should legitimately produce in different contexts to those observed in the field. However, this quantitative evaluation is not informative on the drifts in effectiveness, it does not help designers investigate their possible causes, increasing the time needed to resolve them. To address this problem, and assuming that models of legitimate behavior can be described by means of Input-Output Hidden Markov Models (IOHMMs), a novel generic unsupervised clustering-based IOHMM structure and parameters learning algorithm is developed. This algorithm is first used to learn a model of legitimate behavior. Then, a model of the observed behavior is learned from observations gathered in the field. A second algorithm builds a dissimilarity graph that makes clear structural and parametric differences between both models, thus providing guidance to designers to help them investigate possible causes of drift in effectiveness. The approach is validated on a real world dataset collected in a smart home.

A Generic Clustering-Based Algorithm for Approximating IOHMM Topology and Parameters

In this paper, a novel generic clustering-based algorithm for approximating the topology and the parameters of discrete state space Input/Output Hidden Markov Models (IOHMMs) with continuous observation spaces is introduced. The algorithm can accommodate any continuous space clustering method, whether incremental or not; it can easily be extended to Input/Output Hidden Semi-Markov Models (IOHSMMs) as well as standard HMMs and HSMMs. In this paper, the proposed algorithm is implemented with the Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN*) clustering algorithm. This algorithm brings numerous benefits such as the capability to learn the topology and the parameters of the model with more or less conservatism and the capability to define distributions from several frameworks of the uncertainty theory such as the probabilities, the possibilities or the imprecise probabilities. The algorithm is validated on synthetic and real-world datasets.

Prediction of students’ early dropout based on their interaction logs in online learning environment

ABSTRACTOnline learning has become more popular in higher education since it adds convenience and flexibility to students’ schedule. But, it has faced difficulties in the retention of the continuity of students and ensure continual growth in course. Dropout is a concerning factor in online course continuity. Therefore, it has sparked great interest among educators and researchers to offer many models and strategies to reduce online course students’ dropout by analyzing students’ behavior and their individual and academic information. However, online education platforms still face challenges of high dropout rate and the difficulty of accurate prediction to reduce it. The key aim of the present study is to construct a predictive model to early predict students who are at-risk of dropout. This model is useful to the course instructors to make effective and timely interventions. We have noticed that dropout prediction is basically a sequence labeling or time series prediction problem. For these reasons, we proposed two models; the Logistic Regression by adding a regularization term and the Input-Output Hidden Markov Model (IOHMM). Results showed that the proposed models achieved an accuracy of 84% compared to the baseline of Machine Learning Models for prediction of the students at-risk of dropping out.

Motion Planning With Velocity Prediction and Composite Nonlinear Feedback Tracking Control for Lane-Change Strategy of Autonomous Vehicles

The lane-change strategy of autonomous vehicles is affected by its leading vehicle. It is necessary and challenging to simultaneously conduct the lane-change maneuvers while avoiding collisions with the leading vehicle. This article proposes a lane-change strategy considering the leading vehicle by synthesizing vehicle velocity prediction, motion planning, and trajectory tracking control. A scenario-based velocity prediction method using the input-output hidden Markov model (HMM) is proposed to predict the leading vehicle velocity. Then a motion planner integrating the predicted velocity is developed to generate the optimal trajectories for the lane-change maneuvers. The generated trajectories are tracked by a trajectory tracking controller. An improved composite nonlinear feedback (CNF) control algorithm is proposed to obtain smooth transient performances and fast responses. Human driver tests on a driving simulator show the leading vehicle velocity can be predicted by the proposed method. The motion planner and the trajectory tracking controller are validated in the CarSim simulations. The collision-free optimal trajectories are generated and tracked by the motion planner and the improved CNF controller. A systematic lane-change strategy considering the leading vehicle is effectively developed in this study.

Heterodimeric DNA motif synthesis and validations.

Bound by transcription factors, DNA motifs (i.e. transcription factor binding sites) are prevalent and important for gene regulation in different tissues at different developmental stages of eukaryotes. Although considerable efforts have been made on elucidating monomeric DNA motif patterns, our knowledge on heterodimeric DNA motifs are still far from complete. Therefore, we propose to develop a computational approach to synthesize a heterodimeric DNA motif from two monomeric DNA motifs. The approach is sequentially divided into two components (Phases A and B). In Phase A, we propose to develop the inference models on how two DNA monomeric motifs can be oriented and overlapped with each other at nucleotide level. In Phase B, given the two monomeric DNA motifs oriented, we further propose to develop DNA-binding family-specific input-output hidden Markov models (IOHMMs) to synthesize a heterodimeric DNA motif. To validate the approach, we execute and cross-validate it with the experimentally verified 618 heterodimeric DNA motifs across 49 DNA-binding family combinations. We observe that our approach can even “rescue" the existing heterodimeric DNA motif pattern (i.e. HOXB2_EOMES) previously published on Nature. Lastly, we apply the proposed approach to infer previously uncharacterized heterodimeric motifs. Their motif instances are supported by DNase accessibility, gene ontology, protein-protein interactions, in vivo ChIP-seq peaks, and even structural data from PDB. A public web-server is built for open accessibility and scientific impact. Its address is listed as follows: http://motif.cs.cityu.edu.hk/custom/MotifKirin.

A Stochastic Driver Pedal Behavior Model Incorporating Road Information

Human driver's pedal behavior is difficult to model because it is the output of a very complicated virtual stochastic system. This paper provides a new way to describe driver's pedal behavior after decomposing it into a sequence of actions. By considering the vehicle and road information as the inputs and the pedal action as the output, an input–output hidden Markov model (IOHMM) is used to describe the pedal behavior. The state transition and output distribution functions are designed, and the relation between the input and the key variables of the output distributions is analyzed and modeled using statistical methods. The model parameters can be identified for individual drivers using the generalized estimation-maximization method. One-step probability prediction test shows that the proposed model can capture and distinguish each individual driver's driving style. The prediction capability of the proposed model is evaluated by comparing it with the human driver data collected on a driving simulator along with three other models. The results show that the proposed IOHMM-based driver pedal behavior model performs well in prediction horizons from 1 to 60 s.

FRAGSION: ultra-fast protein fragment library generation by IOHMM sampling.

Speed, accuracy and robustness of building protein fragment library have important implications in de novo protein structure prediction since fragment-based methods are one of the most successful approaches in template-free modeling (FM). Majority of the existing fragment detection methods rely on database-driven search strategies to identify candidate fragments, which are inherently time-consuming and often hinder the possibility to locate longer fragments due to the limited sizes of databases. Also, it is difficult to alleviate the effect of noisy sequence-based predicted features such as secondary structures on the quality of fragment. Here, we present FRAGSION, a database-free method to efficiently generate protein fragment library by sampling from an Input-Output Hidden Markov Model. FRAGSION offers some unique features compared to existing approaches in that it (i) is lightning-fast, consuming only few seconds of CPU time to generate fragment library for a protein of typical length (300 residues); (ii) can generate dynamic-size fragments of any length (even for the whole protein sequence) and (iii) offers ways to handle noise in predicted secondary structure during fragment sampling. On a FM dataset from the most recent Critical Assessment of Structure Prediction, we demonstrate that FGRAGSION provides advantages over the state-of-the-art fragment picking protocol of ROSETTA suite by speeding up computation by several orders of magnitude while achieving comparable performance in fragment quality. Source code and executable versions of FRAGSION for Linux and MacOS is freely available to non-commercial users at http://sysbio.rnet.missouri.edu/FRAGSION/ It is bundled with a manual and example data. chengji@missouri.edu Supplementary data are available at Bioinformatics online.

Context-sensitive markov models for peptide scoring and identification from tandem mass spectrometry.

Peptide and protein identification via tandem mass spectrometry (MS/MS) lies at the heart of proteomic characterization of biological samples. Several algorithms are able to search, score, and assign peptides to large MS/MS datasets. Most popular methods, however, underutilize the intensity information available in the tandem mass spectrum due to the complex nature of the peptide fragmentation process, thus contributing to loss of potential identifications. We present a novel probabilistic scoring algorithm called Context-Sensitive Peptide Identification (CSPI) based on highly flexible Input-Output Hidden Markov Models (IO-HMM) that capture the influence of peptide physicochemical properties on their observed MS/MS spectra. We use several local and global properties of peptides and their fragment ions from literature. Comparison with two popular algorithms, Crux (re-implementation of SEQUEST) and X!Tandem, on multiple datasets of varying complexity, shows that peptide identification scores from our models are able to achieve greater discrimination between true and false peptides, identifying up to ∼25% more peptides at a False Discovery Rate (FDR) of 1%. We evaluated two alternative normalization schemes for fragment ion-intensities, a global rank-based and a local window-based. Our results indicate the importance of appropriate normalization methods for learning superior models. Further, combining our scores with Crux using a state-of-the-art procedure, Percolator, we demonstrate the utility of using scoring features from intensity-based models, identifying ∼4-8 % additional identifications over Percolator at 1% FDR. IO-HMMs offer a scalable and flexible framework with several modeling choices to learn complex patterns embedded in MS/MS data.

Input-Output HMM Applied to Automatic Arrangement for Guitars

Given a relatively small selection of guitar scores for a large population of guitarists, there should be a certain demand for systems that can automatically arrange an arbitrary score for guitars. Our aim in this paper is to formulate the and in a unified framework that can be cast as a decoding problem of a hidden Markov model (HMM). The left hand forms on the fingerboard are considered as the hidden states and the note sequence of a given score as an observed sequence generated by the HMM. Finding the most likely sequence of the hidden states thus corresponds to performing fingering decision or arrangement. The manual setting of HMM parameters reflecting preference of beginner guitarists lets the framework generate natural fingerings and arrangements suitable for beginners. Some examples of fingering and arrangement produced by the proposed method are presented.

Input-Output HMM Applied to Automatic Arrangement for Guitars

Input-Output HMM Applied to Automatic Arrangement for Guitars

Training Goal Recognition Online from Low-Level Inputs in an Action-Adventure Game

A method is presented for training an Input-Output Hidden Markov Model (IOHMM) to identify a player's current goal in an action-adventure game. The goals were Explore, Fight, or Return to Town, which served as the hidden states of the IOHMM. The observation model was trained by directing the player to achieve particular goals and counting actions. When trained on first-time players, training to the specific players did not appear to provide any benefits over a model trained to the experimenter. However, models trained on these players' subsequent trials were significantly better than the models trained to the specific players the first time, and also outperformed the model trained to the experimenter. This suggests that game goal recognition systems are best trained after the players have some time to develop a style of play. Systems for probabilistic reasoning over time could help game designers make games more responsive to players' individual styles and approaches.

Online Monitoring of Batch Processes Using IOHMM Based MPLS

Online monitoring of the batch process is extremely important to the health assessment of the batch operation; it ensures making products of consistent high quality. In this paper, an integrated framework called IOHMM-MPLS is proposed to monitor the performance of a batch process. It consists of the input−output hidden Markov model (IOHMM) and multiway partial least-squares (MPLS) method. The sequence of the process variables and the product quality variables are decomposed into linear outer relations, which can be handled by MPLS, and simple inner dynamic sequence relations, which can be coped with by a set of single-input−single-output IOHMM models. MPLS is used to solve the problem of high dimensionality and collinearity, while IOHMM is used to capture the transition probability of the dynamic information. The combined IOHMM-MPLS method requires a smaller computational load, converges faster, and encounters lesser occurrences of false detection. It utilizes output quality variables and within batch pro...