Exact Maximum Likelihood Estimation Research Articles

Classical iterative proportional scaling of log-linear models with rational maximum likelihood estimator

In this work we investigate multipartition models, the subset of log-linear models for which one can perform the classical iterative proportional scaling (IPS) algorithm to numerically compute the maximum likelihood estimate (MLE). Multipartition models include families of models such as hierarchical models and balanced, stratified staged trees. We define a sufficient condition, called the Generalized Running Intersection Property (GRIP), on the matrix representation of a multipartition model under which the classical IPS algorithm produces the exact MLE in one cycle. In this case, the MLE is a rational function of the data. Additionally we connect the GRIP to the toric fiber product and to previous results for hierarchical models and balanced, stratified staged trees. This leads to a characterization of balanced, stratified staged trees in terms of the GRIP.

Improving financial volatility nowcasts*

ABSTRACT Recently, a simple nowcasting model for volatility has been proposed by Breitung and Hafner [2016. A Simple Model for Now-Casting Volatility Series. International Journal of Forecasting 32 (4): 1247–1255. doi:10.1016/j.ijforecast.2016.04.007]. They suggest a model in which today's volatility is not only driven by past returns, but also by the current information from the same day. Empirical results demonstrate the relevance of the current squared return for volatility nowcasting. Their model obeys an ARMA representation estimable by maximum likelihood. However, their estimation approach builds on a number of simplifications and we suggest improvements. Rather than assuming normality of the innovations in the ARMA representation for highly skewed and leptokurtic log-squared returns, we take non-normality explicitly into account. Contrary to most situations regarding volatility estimation and forecasting, the distribution actually plays a crucial role in the construction of volatility nowcasts. We devise an exact maximum likelihood estimator which offers significant improvements in estimation efficiency and volatility nowcast accuracy in finite samples. In our empirical application, we investigate five major international stock markets from 2000 to 2019 (including sub-samples relating to the Great Financial Crisis). The results suggest that our estimation approach significantly outperforms the one by Breitung and Hafner [2016. A Simple Model for Now-Casting Volatility Series. International Journal of Forecasting 32 (4): 1247–1255. doi:10.1016/j.ijforecast.2016.04.007] in all cases. Financial volatility can be nowcasted more accurately by applying our suggested estimation approach.

Modelling cycles in climate series: The fractional sinusoidal waveform process

The paper proposes a novel model for time series displaying persistent stationary cycles, the fractional sinusoidal waveform process. The underlying idea is to allow the parameters that regulate the amplitude and phase to evolve according to fractional noise processes. Its advantages with respect to popular alternative specifications, such as the Gegenbauer process, are twofold: the autocovariance function is available in closed form, which opens the way to exact maximum likelihood estimation; secondly, the model encompasses deterministic cycles, so that discrete spectra arise as a limiting case. A generalization of the process, featuring multiple components, an additive ‘red noise’ component and exogenous variables, provides the basic model for climate time series with mixed spectra. Our illustrations deal with the change in amplitude and phase of the intra-annual component of carbon dioxide concentrations in Mauna Loa, and with the estimation and the quantification of the contribution of orbital cycles to the variability of paleoclimate time series.

Recursive and Non-Recursive Generalized Least-Squares Methods for Estimation of Time Series Models with Exogenous Variables

We present in this paper two generalized least-squares (GLS) methods for estimating regression coefficients of time series models with exogenous variables. The non-recursive GLS method is a generalization of the GLS method suggested by Cochrane and Orcutt (1949). The proposed GLS method consists of a sequence of four linear regressions. A first regression is fitted and provides residuals. These residuals are modeled as an autoregressive process and are used in a second regression (or autoregression) for obtaining estimators of autoregressive coefficients. These estimators are used to generate transformed endogenous and exogenous variables. A third regression makes use of the lagged values of these transformed variables to estimate the regression coefficients. The estimators of the regression coefficients are used to determine the true residuals which are modeled as an ARMA process which is finally used for obtaining the estimators of autoregressive and moving average parameters. The second GLS method is a recursive version of the first GLS method where the estimators are updated at each time point on receipt of the additional observations. The Simulation results based on different model structures with varying numbers of observations are used to compare the performance of our methods with that of exact maximum likelihood (EML) estimates.

Application of quantile mixed-effects model in modeling CD4 count from HIV-infected patients in KwaZulu-Natal South Africa

BackgroundThe CD4 cell count signifies the health of an individual’s immune system. The use of data-driven models enables clinicians to accurately interpret potential information, examine the progression of CD4 count, and deal with patient heterogeneity due to patient-specific effects. Quantile-based regression models can be used to illustrate the entire conditional distribution of an outcome and identify various covariates effects at the respective location.MethodsThis study uses the quantile mixed-effects model that assumes an asymmetric Laplace distribution for the error term. The model also incorporated multiple random effects to consider the correlation among observations. The exact maximum likelihood estimation was implemented using the Stochastic Approximation of the Expectation–Maximization algorithm to estimate the parameters. This study used the Centre of the AIDS Programme of Research in South Africa (CAPRISA) 002 Acute Infection Study data. In this study, the response variable is the longitudinal CD4 count from HIV-infected patients who were initiated on Highly Active Antiretroviral Therapy (HAART), and the explanatory variables are relevant baseline characteristics of the patients.ResultsThe analysis obtained robust parameters estimates at various locations of the conditional distribution. For instance, our result showed that baseline BMI (at tau = 0.05: {widehat{beta }}_{4}=0.056, mathrm{p-}mathrm{value}<0.0064; mathrm{at },tau = 0.5: {widehat{beta }}_{4}=0.082, mathrm{p-}mathrm{value}<0.0025; mathrm{at},tau = 0.95: {widehat{beta }}_{4}=0.145,mathrm{p-}mathrm{value}<0.0000), baseline viral load (at tau = 0.05: {widehat{beta }}_{5}=-0.564, mathrm{p-}mathrm{value}<0.0000; mathrm{at},tau = 0. 5: {widehat{beta }}_{5}=-0.641, mathrm{p-}mathrm{value}<0.0000; mathrm{at },tau = 0.95: {widehat{beta }}_{5}=-0.739,mathrm{p-}mathrm{value}<0.0000), and post-HAART initiation (at tau = 0.05: {widehat{beta }}_{6}=1.683,mathrm{p-}mathrm{value}<0.0000; mathrm{at},tau = 0.5: {widehat{beta }}_{6}=2.560,mathrm{p-}mathrm{value}<0.0000; mathrm{at },tau =0.95: {widehat{beta }}_{6}=2.287,mathrm{p-}mathrm{value}<0.0000) were major significant factors of CD4 count across fitted quantiles.ConclusionsCD4 cell recovery in response to post-HAART initiation across all fitted quantile levels was observed. Compared to HIV-infected patients with low viral load levels at baseline, HIV-infected patients enrolled in the treatment with a high viral load level at baseline showed a significant negative effect on CD4 cell counts at upper quantiles. HIV-infected patients registered with high BMI at baseline had improved CD4 cell count after treatment, but physicians should not ignore this group of patients clinically. It is also crucial for physicians to closely monitor patients with a low BMI before and after starting HAART.

A Review of Well-Known Robust Line Search and Trust Region Numerical Optimization Algorithms for Solving Nonlinear Least-Squares Problems

The conditional, unconditional, or the exact maximum likelihood estimation and the least-squares estimation involve minimizing either the conditional or the unconditional residual sum of squares. The maximum likelihood estimation (MLE) approach and the nonlinear least squares (NLS) procedure involve an iterative search technique for obtaining global rather than local optimal estimates. Several authors have presented brief overviews of algorithms for solving NLS problems. Snezana S. Djordjevic (2019) presented a review of some unconstrained optimization methods based on the line search techniques. Mahaboob et al. (2017) proposed a different approach to estimate nonlinear regression models using numerical methods also based on the line search techniques. Mohammad, Waziri, and Santos (2019) have briefly reviewed methods for solving NLS problems, paying special attention to the structured quasi-Newton methods which are the family of the search line techniques. Ya-Xiang Yuan (2011) reviewed some recent results on numerical methods for nonlinear equations and NLS problems based on online searches and trust regions techniques, particularly on Levenberg-Marquardt type methods, quasi-Newton type methods, and trust regions algorithms. The purpose of this paper is to review some online searches and trust region's more well-known robust numerical optimization algorithms and the most used in practice for the estimation of time series models and other nonlinear regression models. The line searches algorithms considered are: Gradient algorithm, Steepest Descent (SD) algorithm, Newton-Raphson (NR) algorithm, Murray’s algorithm, Quasi-Newton (QN) algorithm, Gauss-Newton (GN) algorithm, Fletcher and Powell algorithm (FP), Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm. While the only trust-region algorithm considered is the Levenberg-Marquardt (LM) algorithm. We also give some main advantages and disadvantages of these different algorithms.

A Fast Algorithm for Exact Maximum Likelihood Estimation of Multiple Input Transfer Function Models with Auto-Correlated Errors

This paper proposes a fast algorithm for the exact maximum likelihood estimation of parameters of multiple inputs transfer function models. This algorithm is a generalization of that proposed by Mélard (1984) which is a combination of an improved version of an algorithm of Pearlman (1980) which uses an algorithm of Morf, Sidhu et Kailath (1974) and consists to replace the (matrix) Riccati-type difference equation used in the Kalman filter by a (vector) Chandrasekhar-type difference equation with the quick recursion switching suggested by Gardner, Harvey and Phillips (1980) and an algorithm of Wilson (1979). Simulations and practical examples are used to illustrate the algorithm by comparing it with the method of Poskitt (1989), the generalized least squares method suggested by Sabiti (1993), and the nonlinear least-squares method of Box and Jenkins (1976).

Analisis Faktor-Faktor yang Mempengaruhi Jumlah Kasus Tuberkulosis di Indonesia Menggunakan Model Geographically Weighted Poisson Regression

Tuberculosis is a contagious disease suffered by humans caused by mycobacterium tuberculosis bacteria. Tuberculosis in Indonesia must be eradicated both preventive and treatment. One effort that can be given to the community to reduce tuberculosis cases is by providing information on the factors that influence tuberculosis cases through Geographically Weighted Poisson Regression (GWPR) modeling. The number of tuberculosis cases in Indonesia is a count data with a small chance of occurrence so that it is suspected to have a Poisson distribution. Cases of tuberculosis are spatial data (spatial heterogeneity). The purpose of this study is to determine the GWPR model of the number of tuberculosis cases in Indonesia and determine the factors that influence tuberculosis cases in Indonesia. The research data are secondary data obtained from the Indonesian Ministry of Health. Parameter estimation method is Maximum Likelihood Estimation (MLE). Spatial weighting is calculated by using the Adaptive Gaussian weighting function and the optimum bandwidth is determined by using the Cross-Validation (CV) criteria. The research results showed that the exact Maximum Likelihood (ML) estimator could not be obtained analytically and the approximation of ML estimator was obtained by using the Newton-Raphson iterative method. Based on the results of the parameter testing of GWPR model, it was concluded that the factors affecting the number of tuberculosis cases were local and varied in 34 provinces. The factor affecting locally are the number of poor people, the percentage of houses unfit for habitation, the percentage of districts/cities that do not have a PHBS policy and the percentage of TPM not meeting health requirements, meanwhile factors influencing globally are the number of poor people.

Efficiency assessment of approximated spatial predictions for large datasets

Due to the well-known computational showstopper of the exact Maximum Likelihood Estimation (MLE) for large geospatial observations, a variety of approximation methods have been proposed in the literature, which usually require tuning certain inputs. For example, the recently developed Tile Low-Rank approximation (TLR) method involves many tuning parameters, including numerical accuracy. To properly choose the tuning parameters, it is crucial to adopt a meaningful criterion for the assessment of the prediction efficiency with different inputs. Unfortunately, the most commonly-used Mean Square Prediction Error (MSPE) criterion cannot directly assess the loss of efficiency when the spatial covariance model is approximated. Though the Kullback–Leibler Divergence criterion can provide the information loss of the approximated model, it cannot give more detailed information that one may be interested in, e.g., the accuracy of the computed MSE. In this paper, we present three other criteria, the Mean Loss of Efficiency (MLOE), Mean Misspecification of the Mean Square Error (MMOM), and Root mean square MOM (RMOM), and show numerically that, in comparison with the common MSPE criterion and the Kullback–Leibler Divergence criterion, our criteria are more informative, and thus more adequate to assess the loss of the prediction efficiency by using the approximated or misspecified covariance models. Hence, our suggested criteria are more useful for the determination of tuning parameters for sophisticated approximation methods of spatial model fitting. To illustrate this, we investigate the trade-off between the execution time, estimation accuracy, and prediction efficiency for the TLR method with extensive simulation studies and suggest proper settings of the TLR tuning parameters. We then apply the TLR method to a large spatial dataset of soil moisture in the area of the Mississippi River basin, and compare the TLR with the Gaussian predictive process and the composite likelihood method, showing that our suggested criteria can successfully be used to choose the tuning parameters that can keep the estimation or the prediction accuracy in applications.

Approximate Inferences for Nonlinear Mixed Effects Models with Scale Mixtures of Skew-Normal Distributions

Nonlinear mixed effects models have received a great deal of attention in the statistical literature in recent years because of their flexibility in handling longitudinal studies, including human immunodeficiency virus viral dynamics, pharmacokinetic analyses, and studies of growth and decay. A standard assumption in nonlinear mixed effects models for continuous responses is that the random effects and the within-subject errors are normally distributed, making the model sensitive to outliers. We present a novel class of asymmetric nonlinear mixed effects models that provides efficient parameters estimation in the analysis of longitudinal data. We assume that, marginally, the random effects follow a multivariate scale mixtures of skew--normal distribution and that the random errors follow a symmetric scale mixtures of normal distribution, providing an appealing robust alternative to the usual normal distribution. We propose an approximate method for maximum likelihood estimation based on an EM-type algorithm that produces approximate maximum likelihood estimates and significantly reduces the numerical difficulties associated with the exact maximum likelihood estimation. Techniques for prediction of future responses under this class of distributions are also briefly discussed. The methodology is illustrated through an application to Theophylline kinetics data and through some simulating studies.

A Maximum-Likelihood TDOA Localization Algorithm Using Difference-of-Convex Programming

A popular approach to estimate a source location using time difference of arrival (TDOA) measurements is to construct an objective function based on the maximum likelihood (ML) method. An iterative algorithm can be employed to minimize that objective function. The main challenge in this optimization process is the non-convexity of the objective function, which precludes the use of many standard convex optimization tools. Usually, approximations, such as convex relaxation, are applied, resulting in performance loss. In this work, we take advantage of difference-of-convex (DC) programming tools to develop an efficient solution to the ML TDOA localization problem. We show that, by using a simple trick, the objective function can be modified into an exact difference of two convex functions. Hence, tools from DC programming can be leveraged to carry out the optimization task, which guarantees convergence to a stationary point of the objective function. Simulation results show that, when initialized within the convex hull of the anchors, the proposed TDOA localization algorithm outperforms a number of benchmark methods, behaves as an exact ML estimator, and indeed achieves the Cramér-Rao lower bound.

Fast estimation of multivariate spatiotemporal Hawkes processes and network reconstruction

We present a fast, accurate estimation method for multivariate Hawkes self-exciting point processes widely used in seismology, criminology, finance and other areas. There are two major ingredients. The first is an analytic derivation of exact maximum likelihood estimates of the nonparametric triggering density. We develop this for the multivariate case and add regularization to improve stability and robustness. The second is a moment-based method for the background rate and triggering matrix estimation, which is extended here for the spatiotemporal case. Our method combines them together in an efficient way, and we prove the consistency of this new approach. Extensive numerical experiments, with synthetic data and real-world social network data, show that our method improves the accuracy, scalability and computational efficiency of prevailing estimation approaches. Moreover, it greatly boosts the performance of Hawkes process-based models on social network reconstruction and helps to understand the spatiotemporal triggering dynamics over social media.

Modeling the relation among output-gap, inflation and money supply: comparison of FIVAR and VARFI model

Purpose- It is aimed to reveal the interaction of the output-gap,inflation and money supply variables within 1987: Q2 and 2020:Q2 periods for Turkey. Methodology- The model structure between variables are determined in the scope of FIVAR and VARFI models by using the Exact Maximum Likelihood (EML), Maximum Likelihood, and Whittle estimators. Findings- Monetary and fiscal policies should be used effectively and in a coordinated manner to take under control in devastating effect of inflation on economies and ensuring stability in prices and economy are among the targets of macroeconomic policies. Central Banks uses money supply and interest rates to be effective on income, employment and prices in the economy during the crisis periods. Since changes in money supply affect investments through interest rates, they have an impact on the output-gap and sustainability. Conclusion- Although the VARFI and FIVAR models seems to be similar,there are differences in the results. While one of these models leads to series with different integration orders, the other one leads to series with the same order of integration and a relation such as cointegration. Considering the modeling structure between variables, it is determined that the best performance occurs in structures where long memory structure continuity is modeled. When the rates of return to average are examined, it is concluded that the returns to the average determined at an exponential rate is due to the improved performance of long memory models.

Time series analysis and forecasting with ECOTOOL.

This paper presents ECOTOOL, a new free MATLAB toolbox that embodies several routines for identification, validation and forecasting of dynamic models. The toolbox includes a wide range of exploratory, descriptive and diagnostic statistical tools with visual support, designed in easy-to-use Graphical User Interfaces. It also incorporates complex automatic procedures for identification, exact maximum likelihood estimation and outlier detection for many types of models available in the literature (like multi-seasonal ARIMA models, transfer functions, Exponential Smoothing, Unobserved Components, VARX). ECOTOOL is the outcome of a long period of programming effort with the aim of producing a user friendly toolkit such that, just a few lines of code written in MATLAB are able to perform a comprehensive analysis of time series. The toolbox is supplied with an in-depth documentation system and online help and is available on the internet. The paper describes the main functionalities of the toolbox, and its power is shown working on several real examples.

A note on the asymptotic and exact Fisher information matrices of a Markov switching VARMA process

We study the asymptotic and exact Fisher information (FI) matrices of Markov switching vector autoregressive moving average (MS VARMA) models. In a related paper (2017), we propose a method to derive an explicit expression in closed form for the asymptotic FI matrix of the underlying model, and use such a matrix to derive the asymptotic covariance matrix of the Gaussian maximum likelihood (ML) estimator of the parameters in the MS VARMA model. In this paper, the exact FI matrix of a Gaussian MS VARMA process is considered for a time series of length T in relation to the exact ML estimation method. Furthermore, we prove that the Gaussian exact FI matrix converges in probability to the asymptotic FI matrix when the sample size T goes to infinity.

MLML2R: an R package for maximum likelihood estimation of DNA methylation and hydroxymethylation proportions.

Accurately measuring epigenetic marks such as 5-methylcytosine (5-mC) and 5-hydroxymethylcytosine (5-hmC) at the single-nucleotide level, requires combining data from DNA processing methods including traditional (BS), oxidative (oxBS) or Tet-Assisted (TAB) bisulfite conversion. We introduce the R package MLML2R, which provides maximum likelihood estimates (MLE) of 5-mC and 5-hmC proportions. While all other available R packages provide 5-mC and 5-hmC MLEs only for the oxBS+BS combination, MLML2R also provides MLE for TAB combinations. For combinations of any two of the methods, we derived the pool-adjacent-violators algorithm (PAVA) exact constrained MLE in analytical form. For the three methods combination, we implemented both the iterative method by Qu et al. [Qu, J., M. Zhou, Q. Song, E. E. Hong and A. D. Smith (2013): "Mlml: consistent simultaneous estimates of dna methylation and hydroxymethylation," Bioinformatics, 29, 2645-2646.], and also a novel non iterative approximation using Lagrange multipliers. The newly proposed non iterative solutions greatly decrease computational time, common bottlenecks when processing high-throughput data. The MLML2R package is flexible as it takes as input both, preprocessed intensities from Infinium Methylation arrays and counts from Next Generation Sequencing technologies. The MLML2R package is freely available at https://CRAN.R-project.org/package=MLML2R.

SE-Sync: A certifiably correct algorithm for synchronization over the special Euclidean group

Many important geometric estimation problems naturally take the form of synchronization over the special Euclidean group: estimate the values of a set of unknown group elements given noisy measurements of a subset of their pairwise relative transforms . Examples of this class include the foundational problems of pose-graph simultaneous localization and mapping (SLAM) (in robotics), camera motion estimation (in computer vision), and sensor network localization (in distributed sensing), among others. This inference problem is typically formulated as a non-convex maximum-likelihood estimation that is computationally hard to solve in general. Nevertheless, in this paper we present an algorithm that is able to efficiently recover certifiably globally optimal solutions of the special Euclidean synchronization problem in a non-adversarial noise regime. The crux of our approach is the development of a semidefinite relaxation of the maximum-likelihood estimation (MLE) whose minimizer provides an exact maximum-likelihood estimate so long as the magnitude of the noise corrupting the available measurements falls below a certain critical threshold; furthermore, whenever exactness obtains, it is possible to verify this fact a posteriori, thereby certifying the optimality of the recovered estimate. We develop a specialized optimization scheme for solving large-scale instances of this semidefinite relaxation by exploiting its low-rank, geometric, and graph-theoretic structure to reduce it to an equivalent optimization problem defined on a low-dimensional Riemannian manifold, and then design a Riemannian truncated-Newton trust-region method to solve this reduction efficiently. Finally, we combine this fast optimization approach with a simple rounding procedure to produce our algorithm, SE-Sync. Experimental evaluation on a variety of simulated and real-world pose-graph SLAM datasets shows that SE-Sync is capable of recovering certifiably globally optimal solutions when the available measurements are corrupted by noise up to an order of magnitude greater than that typically encountered in robotics and computer vision applications, and does so significantly faster than the Gauss–Newton-based approach that forms the basis of current state-of-the-art techniques.

Analytic computation of nonparametric Marsan–Lengliné estimates for Hawkes point processes

ABSTRACTIn 2008, Marsan and Lengliné presented a nonparametric way to estimate the triggering function of a Hawkes process. Their method requires an iterative and computationally intensive procedure which ultimately produces only approximate maximum likelihood estimates (MLEs) whose asymptotic properties are poorly understood. Here, we note a mathematical curiosity that allows one to compute, directly and extremely rapidly, exact MLEs of the nonparametric triggering function. The method here requires that the number q of intervals on which the nonparametric estimate is sought equals the number n of observed points. The resulting estimates have very high variance but may be smoothed to form more stable estimates. The performance and computational efficiency of the proposed method is verified in two disparate, highly challenging simulation scenarios: first to estimate the triggering functions, with simulation-based 95% confidence bands, for earthquakes and their aftershocks in Loma Prieta, California, and second, to characterise triggering in confirmed cases of plague in the United States over the last century. In both cases, the proposed estimator can be used to describe the rate of contagion of the processes in detail, and the computational efficiency of the estimator facilitates the construction of simulation-based confidence intervals.

The impact of inflation rate on stock market returns: evidence from Kenya

This study examined the stochastic properties of inflation rate, stock market returns and their cointegrating residuals using monthly data for the period 1993 to 2015. The Autoregressive Fractionally Integrated Moving Average (ARFIMA)-based exact maximum likelihood estimation was employed to determine the integration orders of the individual variables as well as the cointegrating residuals. Results from the ARFIMA model indicate that the month-on-month inflation rate, year-on-year inflation rate and stock market returns have non-integer orders of integration. This is inconsistent with the stationary/nonstationary results often obtained from the conventional unit root tests and implies that any shocks to the variables are highly persistent but eventually disappear. The results also reveal that the cointegrating residuals have non-integer orders of integration, suggesting that deviations from the long run equilibrium are prolonged, contrary to the assumption held under the conventional cointegration framework. The Fractionally Integrated Error Correction Model (FIECM) reveals that the year-on-year inflation rate positively granger causes stock market returns. This supports Fisher Effect and implies that stock market returns in Kenya provide shelter against inflationary pressures. This is the first study to empirically examine fractional cointegration and ARFIMA-based Granger Causality between inflation rate and stock market returns in Kenya.

Multivariate normal estimation: the case (n < p)

ABSTRACTEstimation in the multivariate context when the number of observations available is less than the number of variables is a classical theoretical problem. In order to ensure estimability, one has to assume certain constraints on the parameters. A method for maximum likelihood estimation under constraints is proposed to solve this problem. Even in the extreme case where only a single multivariate observation is available, this may provide a feasible solution. It simultaneously provides a simple, straightforward methodology to allow for specific structures within and between covariance matrices of several populations. This methodology yields exact maximum likelihood estimates.