Non-homogeneous Hidden Markov Model Research Articles

Mapping species distributions in one dimension by non-homogeneous hidden Markov models: the case of freshwater pearl mussels in the River Dee

The investigation of species distributions in rivers involves data which are inherently sequential and unlikely to be fully independent. To take these characteristics into account, we develop a Bayesian hierarchical model for mapping the distribution of freshwater pearl mussels in the River Dee (Scotland). At the top of the hierarchy the likelihood is used to describe the sequence of sites in which mussels were observed or not. Given that false observations can occur, and that “not observed” means both that the species was not present and that it was not observed, a Markov prior is introduced at the second level of the hierarchy to represent the sequence of sites in which mussels are estimated to occur. The Markov prior allows modelling the spatial dependency between neighbouring sites. A third level in the hierarchy is given by the representation of the transition probabilities of the Markov chain in terms of site-specific explanatory variables, through a logistic regression. The selection of the explanatory variables which influence the Markov process is performed by means of a simulation-based procedure, in the complex case of association between covariates. Four features were found to be associated with reduced chance of finding a local mussel population: tributaries, bridges, dredging, and waste water treatment works. These results complement the results of a previous study, providing new evidence for the causes of the deterioration of a highly threatened species.

Non-Homogeneous Hidden Markov Model for Daily Rainfall Amount in Peninsular Malaysia

The non-homogeneous hidden Markov model (NHMM) generates the rainfall observation depends on few weather states which serve as a link between the large scale atmospheric measures. The daily rainfall at 20 stations from Peninsular Malaysia for 33 years sequences is analyzed using NHMM during the northeast monsoon season. A NHMM with six hidden states are identified. The atmospheric variable was obtained from NCEP Reanalysis Data as predictor. The gridded atmospheric fields are summarized through the principle component analysis (PCA) technique. PCA is applied to sea level pressure (SLP) to identify their principal spatial patterns co-varying with rainfall. The NHMM can accurately simulate the observed daily mean rainfall, correlations between stations for daily rainfall amounts and the quantile-quantile plots. It can be concluded that the NHMM is a useful method to simulate the daily rainfall amounts that may be used to prepare strategies and planning for the unpredicted disaster such as flood and drought.

Projection of future rainfall for the North China Plain using two statistical downscaling models and its hydrological implications

This study projected the future rainfall (2046–2065 and 2081–2100) for the North China Plain (NCP) using two stochastic statistical downscaling models, the non-homogeneous hidden Markov model and the generalized linear model for daily climate time series, conditioned by the large-scale atmospheric predictors from six general circulation models for three emission scenarios (A1B, A2 and B1). The results indicated that the annual total rainfall, the extreme daily rainfall and the maximum length of consecutive wet/dry days would decline, while the number of annual rainfall days would slightly increase (correspondingly rainfall intensity would decrease) in the NCP, in comparison with the base period (1961–2010). Moreover, the summer monsoon rainfall, which accounted for 50–75 % of the total annual rainfalls in NCP, was projected to decrease in the latter half of twenty-first century. The spatial patterns of change showed generally north–south gradients with relatively larger magnitude decrease in the northern NCP and less decrease (or even slightly increase) in the southern NCP. This could result in decline of the annual runoff by −5.5 % (A1B), −3.3 % (A2) and −4.1 % (B1) for 2046–2065 and −5.3 % (A1B), −4.6 % (A2) and −1.9 % (B1) decrease for 2081–2100. These rainfall changes, combined with the warming temperature, could lead to drier catchment soil profiles and further reduce runoff potential, would hence provide valuable references for the water availability and related climate change adaption in the NCP.

Scenario development for estimating potential climate change impacts on crop production in the North China Plain

ABSTRACTIt is important to investigate potential changes in temperature, precipitation and solar radiation for assessing the impacts of future climate change on agricultural production for specific regions. In this study, climate scenarios of precipitation, temperature and solar radiation for the North China Plain (NCP) were constructed in terms of stochastic daily weather sequences. A nonhomogeneous hidden Markov model (NHMM) was used to downscale daily precipitation projections at 32 stations during winter wheat and summer maize growing seasons for a baseline (1966–2005) and a 21st century (2080–2099) A1B scenario, using selected general circulation models (GCMs). A climatological seasonal cycle of regional‐averaged daily reanalysis precipitation was used as input to the down‐scaling for the baseline simulation; this input was then scaled by the precipitation changes from GCM projections to generate down‐scaled stochastic simulations of precipitation in the 21st century. Temperature was generated using a weakly stationary generating process, conditional on precipitation occurrence, with 21st century additive changes taken from the GCMs at the regional scale. Three hypotheses about changes in solar radiation (−20%, 0% and 20%) were made considering the large uncertainty in its future change. The down‐scaled simulations exhibit station increases in the mean daily rainfall of 13.9–69.7% in the scenarios driven by the GCM with the projected largest and multi‐model mean precipitation increase for the wheat season, with changes of 0.4–29.9% for the maize season. In the scenario driven by the GCM with the largest projected precipitation decrease, the simulated rainfall decreases at all stations, with changes ranging from −24.6 to −0.1% for the wheat and maize seasons, respectively. Temperature increases by about 3.7 °C for the wheat season and 3.6 °C for the maize season.

Sparsely correlated hidden Markov models with application to genome-wide location studies

Multiply correlated datasets have become increasingly common in genome-wide location analysis of regulatory proteins and epigenetic modifications. Their correlation can be directly incorporated into a statistical model to capture underlying biological interactions, but such modeling quickly becomes computationally intractable. We present sparsely correlated hidden Markov models (scHMM), a novel method for performing simultaneous hidden Markov model (HMM) inference for multiple genomic datasets. In scHMM, a single HMM is assumed for each series, but the transition probability in each series depends on not only its own hidden states but also the hidden states of other related series. For each series, scHMM uses penalized regression to select a subset of the other data series and estimate their effects on the odds of each transition in the given series. Following this, hidden states are inferred using a standard forward-backward algorithm, with the transition probabilities adjusted by the model at each position, which helps retain the order of computation close to fitting independent HMMs (iHMM). Hence, scHMM is a collection of inter-dependent non-homogeneous HMMs, capable of giving a close approximation to a fully multivariate HMM fit. A simulation study shows that scHMM achieves comparable sensitivity to the multivariate HMM fit at a much lower computational cost. The method was demonstrated in the joint analysis of 39 histone modifications, CTCF and RNA polymerase II in human CD4+ T cells. scHMM reported fewer high-confidence regions than iHMM in this dataset, but scHMM could recover previously characterized histone modifications in relevant genomic regions better than iHMM. In addition, the resulting combinatorial patterns from scHMM could be better mapped to the 51 states reported by the multivariate HMM method of Ernst and Kellis. The scHMM package can be freely downloaded from http://sourceforge.net/p/schmm/ and is recommended for use in a linux environment.

Nowcasting influenza epidemics using non‐homogeneous hidden Markov models

Timeliness of a public health surveillance system is one of its most important characteristics. The process of predicting the present situation using available incomplete information from surveillance systems has received the term nowcasting and has high public health interest. Generally in Europe, general practitioners' sentinel networks support the epidemiological surveillance of influenza activity, and each week's epidemiological bulletins are usually issued between Wednesday and Friday of the following week. In this work, we have developed a non-homogeneous hidden Markov model (HMM) that, on a weekly basis, uses as covariates an early observation of influenza-like illness (ILI) incidence rate and the number of ILI cases tested positive to nowcast the current week ILI rate and the probability that the influenza activity is in an epidemic state. We use Bayesian inference to find estimates of the model parameters and nowcasted quantities. The results obtained with data provided by the Portuguese influenza surveillance system show the additional value of using a non-homogeneous HMM instead of a homogeneous one. The use of a non-homogeneous HMM improves the surveillance system timeliness in 2 weeks.

Daily rainfall projections from general circulation models with a downscaling nonhomogeneous hidden Markov model (NHMM) for south‐eastern Australia

AbstractAn ensemble of stochastic daily rainfall projections has been generated for 30 stations across south‐eastern Australia using the downscaling nonhomogeneous hidden Markov model, which was driven by atmospheric predictors from four climate models for three IPCC emissions scenarios (A1B, A2, and B1) and for two periods (2046–2065 and 2081–2100). The results indicate that the annual rainfall is projected to decrease for both periods for all scenarios and climate models, with the exception of a few scenarios of no statistically significant changes. However, there is a seasonal difference: two downscaled GCMs consistently project a decline of summer rainfall, and two an increase. In contrast, all four downscaled GCMs show a decrease of winter rainfall. Because winter rainfall accounts for two‐thirds of the annual rainfall and produces the majority of streamflow for this region, this decrease in winter rainfall would cause additional water availability concerns in the southern Murray–Darling basin, given that water shortage is already a critical problem in the region. In addition, the annual maximum daily rainfall is projected to intensify in the future, particularly by the end of the 21st century; the maximum length of consecutive dry days is projected to increase, and correspondingly, the maximum length of consecutive wet days is projected to decrease. These changes in daily sequencing, combined with fewer events of reduced amount, could lead to drier catchment soil profiles and further reduce runoff potential and, hence, also have streamflow and water availability implications. Copyright © 2012 John Wiley & Sons, Ltd.

Downscaling daily precipitation over the Yellow River source region in China: a comparison of three statistical downscaling methods

Three statistical downscaling methods are compared with regard to their ability to downscale summer (June–September) daily precipitation at a network of 14 stations over the Yellow River source region from the NCEP/NCAR reanalysis data with the aim of constructing high-resolution regional precipitation scenarios for impact studies. The methods used are the Statistical Downscaling Model (SDSM), the Generalized LInear Model for daily CLIMate (GLIMCLIM), and the non-homogeneous Hidden Markov Model (NHMM). The methods are compared in terms of several statistics including spatial dependence, wet- and dry spell length distributions and inter-annual variability. In comparison with other two models, NHMM shows better performance in reproducing the spatial correlation structure, inter-annual variability and magnitude of the observed precipitation. However, it shows difficulty in reproducing observed wet- and dry spell length distributions at some stations. SDSM and GLIMCLIM showed better performance in reproducing the temporal dependence than NHMM. These models are also applied to derive future scenarios for six precipitation indices for the period 2046–2065 using the predictors from two global climate models (GCMs; CGCM3 and ECHAM5) under the IPCC SRES A2, A1B and B1scenarios. There is a strong consensus among two GCMs, three downscaling methods and three emission scenarios in the precipitation change signal. Under the future climate scenarios considered, all parts of the study region would experience increases in rainfall totals and extremes that are statistically significant at most stations. The magnitude of the projected changes is more intense for the SDSM than for other two models, which indicates that climate projection based on results from only one downscaling method should be interpreted with caution. The increase in the magnitude of rainfall totals and extremes is also accompanied by an increase in their inter-annual variability.

A comparison of three multi-site statistical downscaling models for daily rainfall in the North China Plain

Three statistical downscaling methods (conditional resampling statistical downscaling model: CR-SDSM, the generalised linear model for daily climate time series: GLIMCLIM, and the non-homogeneous hidden Markov model: NHMM) for multi-site daily rainfall were evaluated and compared in the North China Plain (NCP). The comparison focused on a range of statistics important for hydrological studies including rainfall amount, extreme rainfall, intra-annual variability, and spatial coherency. The results showed that no single model performed well over all statistics/timescales, suggesting that the user should chose appropriate methods after assessing their advantages and limitations when applying downscaling methods for particular purposes. Specifically, the CR-SDSM provided relatively robust results for annual/monthly statistics and extreme characteristics, but exhibited weakness for some daily statistics, such as daily rainfall amount, dry-spell length, and annual wet/dry days. GLIMCLIM performed well for annual dry/wet days, dry/wet spell length, and spatial coherency, but slightly overestimated the daily rainfall. Additionally, NHMM performed better for daily rainfall and annual wet/dry days, but slightly underestimated dry/wet spell length and overestimated the daily extremes. The results of this study could be applied when investigating climate change impact on hydrology and water availability for the NCP, which suffers from intense water shortages due to climate change and human activities in recent years.

A mixed non‐homogeneous hidden Markov model for categorical data, with application to alcohol consumption

Hidden Markov models (HMMs) are frequently used to analyse longitudinal data, where the same set of subjects is repeatedly observed over time. In this context, several sources of heterogeneity may arise at individual and/or time level, which affect the hidden process, that is, the transition probabilities between the hidden states. In this paper, we propose the use of a finite mixture of non-homogeneous HMMs (NH-HMMs) to face the heterogeneity problem. The non-homogeneity of the model allows us to take into account observed sources of heterogeneity by means of a proper set of covariates, time and/or individual dependent, explaining the variations in the transition probabilities. Moreover, we handle the unobserved sources of heterogeneity at the individual level, due to, for example, omitted covariates, by introducing a random term with a discrete distribution. The resulting model is a finite mixture of NH-HMM that can be used to classify individuals according to their dynamic behaviour or to estimate a mixed NH-HMM without any assumption regarding the distribution of the random term following the non-parametric maximum likelihood approach. We test the effectiveness of the proposal through a simulation study and an application to real data on alcohol abuse.

Integrating Prior Knowledge in Multiple Testing under Dependence with Applications to Detecting Differential DNA Methylation

DNA methylation has emerged as an important hallmark of epigenetics. Numerous platforms including tiling arrays and next generation sequencing, and experimental protocols are available for profiling DNA methylation. Similar to other tiling array data, DNA methylation data shares the characteristics of inherent correlation structure among nearby probes. However, unlike gene expression or protein DNA binding data, the varying CpG density which gives rise to CpG island, shore and shelf definition provides exogenous information in detecting differential methylation. This article aims to introduce a robust testing and probe ranking procedure based on a nonhomogeneous hidden Markov model that incorporates the above-mentioned features for detecting differential methylation. We revisit the seminal work of Sun and Cai (2009, Journal of the Royal Statistical Society: Series B (Statistical Methodology)71, 393-424) and propose modeling the nonnull using a nonparametric symmetric distribution in two-sided hypothesis testing. We show that this model improves probe ranking and is robust to model misspecification based on extensive simulation studies. We further illustrate that our proposed framework achieves good operating characteristics as compared to commonly used methods in real DNA methylation data that aims to detect differential methylation sites.

Linking atmospheric circulation to daily precipitation patterns over the territory of Bulgaria

An eight states non-homogeneous hidden Markov model is developed for linking daily precipitation amounts data at a network of 32 stations broadly covering the territory of Bulgaria to large-scale atmospheric patterns. At each site a 40-year record 1960–2000 of daily October–March precipitation amounts is modeled. The atmospheric data consists of daily sea-level pressure, geopotential height at 500 and 850 hPa, air temperature at 850 hPa and relative humidity at 700 and 850 hPa on a 2.5 × 2.5 grid based on NCEP-NCAR reanalysis data set covering the Europe-Atlantic sector 30W–60E, 20N–70N for the same period. The first 30 years data are used for model fitting purposes while the remaining 10 years are used for model evaluation. Detailed model validation is carried out on various aspects. The proposed model reproduces well the rainfall statistics for the observed and reserved data whereas the identified states are found to be physically interpretable in terms of regional climatology.

Asymptotic normality of M-estimators in nonhomogeneous hidden Markov models

Results on asymptotic normality for the maximum likelihood estimate in hidden Markov models are extended in two directions. The stationarity assumption is relaxed, which allows for a covariate process influencing the hidden Markov process. Furthermore, a class of estimating equations is considered instead of the maximum likelihood estimate. The basic ingredients are mixing properties of the process and a general central limit theorem for weakly dependent variables.

Asymptotic normality of M-estimators in nonhomogeneous hidden Markov models

Results on asymptotic normality for the maximum likelihood estimate in hidden Markov models are extended in two directions. The stationarity assumption is relaxed, which allows for a covariate process influencing the hidden Markov process. Furthermore, a class of estimating equations is considered instead of the maximum likelihood estimate. The basic ingredients are mixing properties of the process and a general central limit theorem for weakly dependent variables.

A comparison of multi-site daily rainfall downscaling techniques under Australian conditions

Summary Six methods of downscaling GCM simulations to multi-site daily precipitation were applied to a set of 30 rain gauges located within South-Eastern Australia. The methods were tested at reproducing a range of statistics important within hydrological studies including inter-annual variability and spatial coherency using both NCEP/NCAR reanalysis and GCM predictors, thus testing the validity of GCM downscaled predictions. The methods evaluated, all having found application in Australia previously, are: (1) the dynamical downscaling Conformal-Cubic Atmospheric Model (CCAM) of McGregor (2005) ; the historical data based (2) Scaling method applied by Chiew et al. (2009) and (3) Analogue method of Timbal (2004) ; and three stochastic methods, (4) the GLIMCLIM (Generalised Linear Model for daily Climate time series) software package ( Chandler, 2002 ), (5) the Non-homogeneous Hidden Markov Model (NHMM) of Charles et al. (1999), and (6) the modified Markov model–kernel probability density estimation (MMM–KDE) downscaling technique of Mehrotra and Sharma (2007) . The results showed that the simple Scaling approach provided relatively robust results for a range of statistics when GCM forcing data was used, and was therefore recommended for regional water availability and planning studies (subject to certain limitations as mentioned in conclusion section). The stochastic methods better capture changes to a fuller range of rainfall statistics and are recommended for detailed catchment modelling studies. In particular, the stochastic methods show better results for daily extreme rainfall (e.g. flooding/low flow) as the simulations are not based purely on temporal/spatial rainfall patterns observed in the past, and a hybrid GLIMCLIM occurrence-KDE amounts model is recommended based on performance for individual statistics. For GCM downscaled simulations, biases in annual mean and standard deviation of ±5% and −30% were observed typically, and no single model performed well over all timescales/statistics, suggesting that the user beware of model limitations when applying downscaling methods for various purposes. A brief demonstration of predictor biases is presented, highlighting that biases observed in GCM predictors can cause poorer performance during GCM validation, and that investigation of these biases should inform choice of GCMs, GCM predictors, and the downscaling methods that use them.

Statistical downscaling of rainfall data using sparse variable selection methods

In many statistical downscaling methods, atmospheric variables are chosen by using a combination of expert knowledge with empirical measures such as correlations and partial correlations. In this short communication, we describe the use of a fast, sparse variable selection method, known as RaVE, for selecting atmospheric predictors, and illustrate its use on rainfall occurrence at stations in South Australia. We show that RaVE generates parsimonious models that are both sensible and interpretable, and whose results compare favourably to those obtained by a non-homogeneous hidden Markov model ( Hughes et al., 1999).

Identity-by-descent filtering of exome sequence data for disease-gene identification in autosomal recessive disorders.

Motivation: Next-generation sequencing and exome-capture technologies are currently revolutionizing the way geneticists screen for disease-causing mutations in rare Mendelian disorders. However, the identification of causal mutations is challenging due to the sheer number of variants that are identified in individual exomes. Although databases such as dbSNP or HapMap can be used to reduce the plethora of candidate genes by filtering out common variants, the remaining set of genes still remains on the order of dozens.Results: Our algorithm uses a non-homogeneous hidden Markov model that employs local recombination rates to identify chromosomal regions that are identical by descent (IBD = 2) in children of consanguineous or non-consanguineous parents solely based on genotype data of siblings derived from high-throughput sequencing platforms. Using simulated and real exome sequence data, we show that our algorithm is able to reduce the search space for the causative disease gene to a fifth or a tenth of the entire exome.Availability: An R script and an accompanying tutorial are available at http://compbio.charite.de/index.php/ibd2.html.Contact: peter.robinson@charite.de

Evaluation of two statistical downscaling models for daily precipitation over an arid basin in China

AbstractTwo statistical downscaling (SD) models, the nonhomogeneous hidden Markov model (NHMM) and the statistical down‐scaling model (SDSM), which have been widely applied and proved skillful in terms of downscaling precipitation, were evaluated based on observed daily precipitation over the Tarim River basin, an arid basin located in China. The evaluated metrics included residual functions, correlation analyses, probability density functions (PDFs) and distributions. Overall, both models exhibited stability with little model performance difference between the calibration and validation periods. There was little difference for model performance on dry‐spell length (dsl) and wet‐spell length (wsl) between NHMM and SDSM. NHMM showed skill in simulating wet‐day precipitation amount (wpa), whilst SDSM performed relatively poorly on extreme values of wpa, especially for dry stations with annual precipitation lower than 200 mm. Both NHMM and SDSM captured the spatial distribution characteristics of precipitation for most of the stations, as 11 months had an at‐station correlation coefficient being greater than 0.9 and 0.8 for NHMM and SDSM in calibration period. NHMM showed slightly better model performance than SDSM on simulating monthly precipitation, as the former was able to model precipitation well for all months, whereas the later was well only for certain months. SDSM was able to capture the inter‐site correlation characteristics of observed series, whilst the NHMM multi‐site simulation over estimated inter‐site correlation. Both NHMM and SDSM had less skill downscaling annual series because of stochastic components in precipitation amounts modelling. Copyright © 2011 Royal Meteorological Society

Downscaling projections of Indian monsoon rainfall using a non‐homogeneous hidden Markov model

AbstractDownscaled rainfall projections for the Indian summer monsoon are generated using a non‐homogeneous hidden Markov model (NHMM) and information from both a dense observational dataset and an ensemble of general circulation models (GCMs). The projections are conditioned on two types of GCM information, corresponding approximately to dynamic and thermodynamic components of precipitation change. These have opposing effects, with a weakening circulation compensating not quite half of the regional precipitation increase that might otherwise be expected. GCM information is taken at the largest spatial scales consistent with regional physics and modelling constraints, while the NHMM produces a disaggregation consistent with the observed fine‐scale spatiotemporal variability. Projections are generated using multimodel mean predictors, with intermodel dispersion providing a measure of the uncertainty due to GCM differences. The downscaled simulations exhibit small increases in the number of dry days, in the average length of dry spells, in mean daily intensity and in the exceedance frequency of nearly all daily rainfall percentiles. Copyright © 2011 Royal Meteorological Society

Analysis of Intraseasonal and Interannual Variability of the Asian Summer Monsoon Using a Hidden Markov Model

AbstractIntraseasonal and interannual variability of Asian summer monsoon rainfall in pentad precipitation data is examined using a hidden Markov model (HMM). The spatial patterns of discrete rainfall states derived with the HMM and the associated transition probabilities between the states are shown to represent well the principal Asian summer monsoon intraseasonal oscillation (ISO), propagating eastward and northward with a period of 40–50 days. Stochastic simulations made with the HMM reasonably reproduce the canonical ISO propagation and its observed statistics such as the frequency of ISO events.The interannual modulation of the ISO associated with El Niño–Southern Oscillation (ENSO) is assessed by employing a nonhomogeneous HMM (NHMM) with summer-mean Niño-3.4 index prescribed as an input variable. ENSO influence on the ISO is found to manifest as preferences toward particular ISO phases depending on the ENSO condition, thus adding an asymmetry to the ISO. In the presence of seasonal mean anomalies, it is shown that the El Niño seasonal mean rainfall anomaly pattern is identified by the HMM as a distinct state, in addition to the ISO states, whereas the La Niña seasonal mean rainfall anomaly pattern does not appear distinct from the ISO states.