Hidden Regimes Research Articles

A comparison of three algorithms in the filtering of a Markov-modulated non-homogeneous Poisson process

ABSTRACT A Markov-modulated non-homogeneous Poisson process (MMNPP), whose intensity process is designed to capture both the cyclical and nonrecurring trends, is considered for modelling the total count of cyber incidents. Extending the Expectation-Maximisation (EM) algorithm for the current MMPP literature, we derive the filters and smoothers to support the MMNPP online parameter estimation. A scaling transformation is introduced to address the numerical issue for large data sizes whilst maintaining accuracy. The filter- and smoother-based EM algorithms are then benchmarked to the maximum likelihood-based EM algorithm at the theoretical level. The differences emerge in the E-step of the EM procedure. Both the filtering and smoothing schemes, in conjunction with the change-of-measure technique, avoid the computing complication caused by the hidden regimes. In contrast to the usual EM algorithm, the said two algorithms could be implemented given only the incident counts data without the specific times of jumps. Within the data compiled by the U.S. Department of Health and Human Services, the filter-based algorithm performs better than the algorithm involving smoothers. The benchmarked algorithm may do well in calibration under the presence of extreme incident counts with an extremely low frequency; however, overfitting may occur. For most practical applications involving 2 or 3 regimes, both algorithms are superior when it comes to efficiency, real-time update, and low computational cost. The benchmarked algorithm is better when there are more regimes under relatively closer intensities. Overall, the filter-based algorithm gives better estimation, especially if there is a low-frequency regime and the flexible binning of the data set is an important consideration.

Equity Risk and Return across Hidden Market Regimes

The key to understanding the dynamics of stock markets, particularly the mechanisms of their changes, is in the concept of the market regime. It is regarded as a regular transition from one state to another. Although the market agenda is never the same, its functioning regime allows us to reveal the logic of its development. The article employs the concept of financial turbulence to identify hidden market regimes. These are revealed through the ratio of the components, which describe single changes of correlated risks and volatility. The combinations of typical and atypical variates of correlational and magnitude components of financial turbulence allowed four hidden regimes to be revealed. These were arranged by the degree of financial turbulence, conceptually analyzed and assessed from the perspective of their duration. The empirical data demonstrated ETF day trading profits for S&P 500 sectors, covering the period of January 1998–August 2020, as well as day trade profits of the Russian blue chips within the period of October 2006–February 2021. The results show a significant difference in regard to the market performance and volatility, which depend on hidden regimes. Both sample data groups demonstrated similar contemporaneous and lagged effects, which allows the prediction of volatility jumps in the periods following atypical correlations.

A Markov Regime Switching Model for Ultra-Short-Term Wind Power Prediction Based on Toeplitz Inverse Covariance Clustering

The rapid development of wind energy has brought a lot of uncertainty to the power system. The accurate ultra-short-term wind power prediction is the key issue to ensure the stable and economical operation of the power system. It is also the foundation of the intraday and real-time electricity market. However, most researches use one prediction model for all the scenarios which cannot take the time-variant and non-stationary property of wind power time series into consideration. In this paper, a Markov regime switching method is proposed to predict the ultra-short-term wind power of multiple wind farms. In the regime switching model, the time series is divided into several regimes that represent different hidden patterns and one specific prediction model can be designed for each regime. The Toeplitz inverse covariance clustering (TICC) is utilized to divide the wind power time series into several hidden regimes and each regime describes one special spatiotemporal relationship among wind farms. To represent the operation state of the wind farms, a graph autoencoder neural network is designed to transform the high-dimensional measurement variable into a low-dimensional space which is more appropriate for the TICC method. The spatiotemporal pattern evolution of wind power time series can be described in the regime switching process. Markov chain Monte Carlo (MCMC) is used to generate the time series of several possible regime numbers. The Kullback-Leibler (KL) divergence criterion is used to determine the optimal number. Then, the spatiotemporal graph convolutional network is adopted to predict the wind power for each regime. Finally, our Markov regime switching method based on TICC is compared with the classical one-state prediction model and other Markov regime switching models. Tests on wind farms located in Northeast China verified the effectiveness of the proposed method.

Consistent estimation of the number of regimes in Markov-switching autoregressive models

Markov-switching models have become a popular tool in areas ranging from finance to electrical engineering. Determining the number of hidden regimes in such models is a key problem in applications. This paper proposes a strongly consistent estimator of the number of regimes for Markov-switching autoregressive models. By using subadditive ergodic theorem, law of iterated logarithm for martingales, together with results from information theory, we derive sufficient conditions to avoid underestimation as well as overestimation. In particular, we propose a modified information criterion, regime-switching information criterion (RSIC) which generates a simple and consistent model selection procedure. Finally, we conduct a Monte Carlo study to evaluate the efficacy of our procedure in finite sample and also compare the performance of RSIC with popular information criteria including AIC, BIC and HQC.

Стратегии прогнозирующего управления инвестиционным портфелем на финансовом рынке со скрытым переключением режимов

Стратегии прогнозирующего управления инвестиционным портфелем на финансовом рынке со скрытым переключением режимов

Hidden regimes during preheating

The Effective Field Theory (EFT) of Preheating with scalar fields, implies three types of derivative couplings between the inflaton and the reheating field. Two of these couplings lead to scales below which only one of the two species appear as the low energy modes. In this paper, the variety of low energy regimes in terms of the species they accommodate are explored by studying the scales introduced by the derivative couplings and the dispersion relations they lead to. It is noted that the EFT of two scalar fields can give rise to non-trivial sound speed for both the inflation and reheating sector even at scales where modes of both species propagate freely, suggesting the presence of additional heavy fields. The regimes where one of the species affects the dispersion relation of the other while not appearing as an effective mode itself, are named as "Hidden Regimes" during preheating.

Regime Switching Analysis of Cryptocurrencies

In this paper we test for regime changes in the price dynamics of Bitcoin, Ethereum, Litecoin and Monero, as representatives of the cryptocurrencies asset class. Data are observed daily from January, 1, 2016 to October, 15, 2019. Best specifications within Gaussian and Autoregressive Hidden Markov models for price differences are selected through the AIC and BIC information criteria by considering up to four hidden regimes. The empirical results suggest that at most three common states may be considered for the basket of cryptocurrencies under investigation; a fourth state may be relevant as an added factor to the dynamics description of the individual cryptocurrencies rather than to the whole basket. Finally, we test the out-of-sample performance of estimated regime switching models; optimal results, in terms of RMSE and correlation between predicted and real values, are obtained in the case of two common or three individual regimes.

Block Trading: Building up a Stock Position Under a Regime Switching Model

This work is intended to systematically characterize the problem of block trading. The author characterizes it as a constrained optimal purchasing (or control) problem, on behalf of the vendee. The purpose of this work is to investigate the optimal purchasing strategies and give quantitative reference information on the pricing of block trade. Noting that large block of stock trading may result in the price fluctuation and different market environment may lead to distinct investing strategies, the controlled diffusion model with regime switching is adopted, which can be used to represent distinct environment of financial market, and the model is modified to allow the price impact, by allowing the drift depend on the purchase rate. Where the switching regimes can be completely observable or unobservable (hidden). Which is a significant difference in contrast to most market models in the existing literature. The objective is to maximize the total discounted shares of the underlying stock, subject to the expected fund for building up corresponding position with an upper bound. This work mainly focuses on the completely observable case, as to the case with hidden regimes, this work just briefly talk about how to reduce the unobservable model to a completely observable one. In the completely observable case, to solve this constrained control problem, the Lagrange multiplier methods and the dynamic programming approaches are used. Though optimality is obtained, what is still lacking is an explicit solution to the optimality equation. To overcome this difficulty, the two loop approximation scheme is given. For fixed Lagrange multiplier, the inner loop is to obtain optimal strategies, which is based mainly on the finite difference approximation. While, the outer loop is a stochastic recursive approximation algorithm to get the optimal Lagrange multiplier. Convergence results are obtained for both the inner and the outer approximations. The quantitative pricing information is related to the threshold levels of the optimal purchasing strategy. Numerical examples are also provided to illustrate our results. Finally, as to the unobservable case, by using the well known Wonham filter approach, one can reduce the unobservable case to a completely observable one.

Estimating stochastic discount factor models with hidden regimes: Applications to commodity pricing

We develop new likelihood-based methods to estimate factor-based Stochastic Discount Factors (SDF) that may accommodate Hidden Markov dynamics in the factor loadings. We use these methods to investigate whether it is possible to find a SDF that jointly prices the cross-section of eight U.S. portfolios of stocks, Treasuries, corporate bonds, and commodities. In particular, we test a range of possible different specification of the SDF, including single-state and Hidden Markov models and compare their statistical and pricing performances. In addition, we assess whether and to which extent a selection of these models replicates the observed moments of the return series, and especially correlations. We report that regime-switching models clearly outperform single-state ones both in term of statistical and pricing accuracy. However, while a four-state model is selected by the information criteria, a two-state three-factor full Vector Autoregression model outperforms the others as far as the pricing accuracy is concerned.

Particle rejuvenation of Rao-Blackwellized sequential Monte Carlo smoothers for conditionally linear and Gaussian models

This paper focuses on sequential Monte Carlo approximations of smoothing distributions in conditionally linear and Gaussian state spaces. To reduce Monte Carlo variance of smoothers, it is typical in these models to use Rao-Blackwellization: particle approximation is used to sample sequences of hidden regimes while the Gaussian states are explicitly integrated conditional on the sequence of regimes and observations, using variants of the Kalman filter/smoother. The first successful attempt to use Rao-Blackwellization for smoothing extends the Bryson-Frazier smoother for Gaussian linear state space models using the generalized two-filter formula together with Kalman filters/smoothers. More recently, a forward-backward decomposition of smoothing distributions mimicking the Rauch-Tung-Striebel smoother for the regimes combined with backward Kalman updates has been introduced. This paper investigates the benefit of introducing additional rejuvenation steps in all these algorithms to sample at each time instant new regimes conditional on the forward and backward particles. This defines particle-based approximations of the smoothing distributions whose support is not restricted to the set of particles sampled in the forward or backward filter. These procedures are applied to commodity markets which are described using a two-factor model based on the spot price and a convenience yield for crude oil data.

Hidden regimes and the demand for carbon dioxide from motor-gasoline

Perhaps the most tactile source of anthropogenic carbon dioxide (CO2) emissions stem from vehicle-use. Because consumers are well aware of their price and quantity decisions it is likely that consumer demand for CO2 emissions from motor-gasoline are quite responsive to the overall state of the economy. Using a structurally identified Markov-switching demand model I find that CO2 emissions respond asymmetrically to changes in income and the price of gasoline in expansionary and contracting states of the economy. The findings of this paper indicate that flexible policy instruments have the potential to mitigate undue burden on consumers and producers compared to their static counterparts.

Linear-price term structure models

We characterize the term structure models in which the zero-coupon prices are linear functions of underlying factors. These models are called Linear-price Term Structure Models (LTSM). We provide two types of LTSM where the observable factors predict regimes which are not observed by the investor. These hidden regimes are represented by a Markov chain, which features either an exogenous, or an endogenous dynamics. We illustrate the possible term structure patterns, their evolutions, in particular their ability to stay close to a zero lower bound.

A BSDE Approach to Optimal Investment of an Insurer with Hidden Regime Switching

We discuss an optimal investment problem of an insurer in a hidden Markov, regime-switching, modeling environment using a backward stochastic differential equation (BSDE) approach. Filtering theory is used to transform the optimal investment problem into one with complete observations. Using BSDEs with jumps, we discuss the problem with complete observations.

Changepoint Analysis of Binary and Ordinal Probit Models: An Application to Bank Rate Policy Under the Interwar Gold Standard

In this paper, I introduce changepoint models for binary and ordered time series data based on Chib's hidden Markov model. The extension of the changepoint model to a binary probit model is straightforward in a Bayesian setting. However, detecting parameter breaks from ordered regression models is difficult because ordered time series data often have clustering along the break points. To address this issue, I propose an estimation method that uses the linear regression likelihood function for the sampling of hidden states of the ordinal probit changepoint model. The marginal likelihood method is used to detect the number of hidden regimes. I evaluate the performance of the introduced methods using simulated data and apply the ordinal probit changepoint model to the study of Eichengreen, Watson, and Grossman on violations of the “rules of the game” of the gold standard by the Bank of England during the interwar period.

A two-state regime switching autoregressive model with an application to river flow analysis

We propose a regime switching autoregressive model and apply it to analyze daily water discharge series of River Tisza in Hungary. The dynamics is governed by two regimes, along which both the autoregressive coefficients and the innovation distributions are altering, moreover, the hidden regime indicator process is allowed to be non-Markovian. After examining stationarity and basic properties of the model, we turn to its estimation by Markov Chain Monte Carlo (MCMC) methods and propose two algorithms. The values of the latent process serve as auxiliary parameters in the first one, while the change points of the regimes do the same in the second one in a reversible jump MCMC setting. After comparing the mixing performance of the two methods, the model is fitted to the water discharge data. Simulations show that it reproduces the important features of the water discharge series such as the highly skewed marginal distribution and the asymmetric shape of the hydrograph.

Detection of hidden regimes in stochastic cyclostationary time series

Stationary stochastic time series with nonlinear dynamics can feature a probability density function (PDF) with distinct local maxima associated with distinct regimes. For nonstationary time series, on the other hand, such regimes are not necessarily reflected in the shape of the PDF. This occurs when the duration of a regime is too short for the PDF to adjust, and such a regime is called a "hidden" regime. This paper presents an algorithm that allows one to detect hidden regimes in cyclostationary stochastic Markovian time series. The method involves analysis of an appropriately windowed time series, from which the drift and diffusion coefficients of the associated Fokker-Planck equation are estimated. The success of the algorithm is illustrated using synthetic time series with both additive and multiplicative noise.