Abstract
AbstractWe propose a multi-stream continuous hidden Markov model (MSCHMM) framework that can learn from multiple modalities. We assume that the feature space is partitioned into subspaces generated by different sources of information. In order to fuse the different modalities, the proposed MSCHMM introduces stream relevance weights. First, we modify the probability density function (pdf) that characterizes the standard continuous HMM to include state and component dependent stream relevance weights. The resulting pdf approximate is a linear combination of pdfs characterizing multiple modalities. Second, we formulate the CHMM objective function to allow for the simultaneous optimization of all model parameters including the relevance weights. Third, we generalize the maximum likelihood based Baum-Welch algorithm and the minimum classification error/gradient probabilistic descent (MCE/GPD) learning algorithms to include stream relevance weights. We propose two versions of the MSCHMM. The first one introduces the relevance weights at the state level while the second one introduces the weights at the component level. We illustrate the performance of the proposed MSCHMM structures using synthetic data sets. We also apply them to the problem of landmine detection using ground penetrating radar. We show that when the multiple sources of information are equally relevant across all training data, the performance of the proposed MSCHMM is comparable to the baseline CHMM. However, when the relevance of the sources varies, the MSCHMM outperforms the baseline CHMM because it can learn the optimal relevance weights. We also show that our approach outperforms existing multi-stream HMM because the latter one cannot optimize all model parameters simultaneously.
Highlights
Hidden Markov models (HMMs) have emerged as a powerful paradigm for modeling stochastic processes and pattern sequences
The multi-stream continuous hidden Markov model (MSCHMM) architectures treat the single stream sequential data as a double-stream one. In this experiment all models are trained using standard Baum-Welch, generalized Baum-Welch, standard and generalized minimum classification error/gradient probabilistic descent (MCE/generalized probabilistic descent (GPD)) algorithms, or a combination of the two (Baum-Welch followed by minimum classification error (MCE)/GPD)
This is because when both streams are relevant for the entire data, the different streams receive nearly equal weights in all states’ components and the MSCHMM reduces to the baseline CHMM
Summary
Hidden Markov models (HMMs) have emerged as a powerful paradigm for modeling stochastic processes and pattern sequences. HMMs have been applied to the domain of speech recognition, and became the dominating technology [1]. In recent years, they have attracted growing interest in automatic target detection and classification [2], computational molecular biology [3], bioinformatics [4], mine detection [5], handwritten character/word recognition [6], and other computer vision applications [7]. Continuous probability density functions have the advantage of covering the entire landscape of the feature space when dealing with continuous attributes. The discrete HMM, on the other hand, reduces the feature space to a finite set of prototypes or representatives. In this article, we focus on the continuous version of HMM for classification
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call