Constrained Maximum Likelihood Estimation Research Articles

Lgrgtest: Lagrange multiplier test after constrained maximum-likelihood estimation

Besides the Wald and likelihood-ratio tests, the Lagrange multiplier test (Rao, 1948, Mathematical Proceedings of the Cambridge Philosophical Society 44: 50–57; Aitchison and Silvey, 1958, Annals of Mathematical Statistics 29: 813–828; Silvey, 1959, Annals of Mathematical Statistics 30: 389–407) is the third canonical approach to testing hypotheses after maximum likelihood estimation. While the Stata commands test and lrtest implement the first two, Stata does not have an official command for implementing the third. The community-contributed boottest package (Roodman et al., 2019, Stata Journal 19: 4–60) focuses on methods of bootstrap inference and also implements the Lagrange multiplier test functionality. In this article, I introduce the new community-contributed postestimation command lgrgtest, which allows for straightforwardly using the Lagrange multiplier test after constrained maximum-likelihood estimation. lgrgtest is intended to be compatible with all Stata estimation commands that use maximum likelihood and allow for the options constraints(), iterate(), and from(). lgrgtest can also be used after cnsreg.

Nonparametric regression with nonignorable missing covariates and outcomes using bounded inverse weighting

We consider nonparametric regression where the covariate and the outcome variable are both subject to missingness. Previous work only discussed one of the variables that may be missing, but not both. Since missing at random is not an appropriate assumption in such a nonmonotone missing data context, we shall assume a missing not at random mechanism. We construct an inverse probability weighting local polynomial estimator based on a recently developed nonmonotone missing data model. It is well known that if the inverse probability weighting is too large at some fully observed cases, the resulting estimator would be deteriorated. To overcome this issue, we introduce a constrained maximum likelihood estimation and an estimating equations method to ensure that the resulting weighting is bounded. We prove the asymptotically normal result for the resulting regression estimator. Simulation results show good practical performance of our method. A real data example is also presented.

Estimation of the trajectory and attitude of railway vehicles using inertial sensors with application to track geometry measurement

This paper describes a novel method for the estimation of the trajectory and orientation of a rigid body moving along a railway track. Compared to other recent developments in the literature, the presented approach has the significant advantage of using inertial sensors only, excluding global position and orientation sensors. The excluded sensors are compensated with an odometry system and previous knowledge of the design track geometry. The procedure is based on a kinematic model of the relative motion of the body with respect to the track, together with a Kalman filter algorithm. Two different approaches are used and compared for the estimation of the noise covariance matrices in the Kalman filter. One is based on the use of experimental results with a known output. The other one relies upon constrained maximum likelihood estimation. The calculated trajectory and orientation are applied in this research to the problem of track geometry measurement. A scale track is used for experimental validation, showing that results are sufficiently accurate for this application. The obtained results also reveal that the constrained maximum likelihood estimation performs similarly to the known-output method. This is very convenient because it allows a straightforward application of the algorithm in different scenarios.

Tensor Robust Principal Component Analysis From Multilevel Quantized Observations

We consider Quantized Tensor Robust Principal Component Analysis (Q-TRPCA), which aims to recover a low-rank tensor and a sparse tensor from noisy, quantized, and sparsely corrupted measurements. A nonconvex constrained maximum likelihood (ML) estimation method is proposed for Q-TRPCA. We provide an upper bound on the Frobenius norm of tensor estimation error under this method. Making use of tools in information theory, we derive a theoretical lower bound on the best achievable estimation error from unquantized measurements. Compared with the lower bound, the upper bound on the estimation error is nearly order-optimal. We further develop an efficient convex ML estimation scheme for Q-TRPCA based on the tensor nuclear norm (TNN) constraint. This method is more robust to sparse noises than the latter nonconvex ML estimation approach. Conducting experiments on both synthetic data and real-world data, we show the effectiveness of the proposed methods.

A new hybrid algorithm for maximum likelihood estimation in a model of accident frequencies

In this paper, we are interested in the numerical computation of the constrained maximum likelihood estimator (MLE) of the parameter vector of a discrete statistical model used in statistics applied to road safety. The parameter vector is divided into two blocks: one block with the parameter of interest and the second block with secondary parameters. The MLE is the solution to a system of non-linear implicit equations difficult to solve in closed-form. To overcome this difficulty, we propose a hybrid algorithm (HA) mixing the use of a one-dimensional Newton-Raphson (NR) algorithm for the first equation of the system and a fixed-point strategy for the remaining equations. Our proposed algorithm involves no matrix inversion but it partially enjoys the quadratic convergence rate of the one-dimensional NR algorithm. We illustrate its performance on simulated data and we compare it to Newton-Raphson (NR) and quasi-Newton algorithms which are two of the most used optimization algorithms. The results suggest that our HA outperforms NR and quasi-Newton algorithms. It is accurate and converges quickly for all the starting values.

The “wrong skewness” problem: Moment constrained maximum likelihood estimation of the stochastic frontier model

Wrong skewness in the stochastic frontier model leads to many empirical and conceptual problems. A variety of approaches have been proposed to deal with this issue. Here we offer a solution based on moment constraints of the assumed density. • Offers alternative approach to wrong skewness. • Moment constrained maximum likelihood estimation. • Find new approach outperforms standard methods. • Application reveals important empirical differences.

MLE of Jointly Constrained Mean-Covariance of Multivariate Normal Distributions

Estimating the unconstrained mean and covariance matrix is a popular topic in statistics. However, estimation of the parameters of $N_p(\mu,\Sigma)$ under joint constraints such as $\Sigma\mu = \mu$ has not received much attention. It can be viewed as a multivariate counterpart of the classical estimation problem in the $N(\theta,\theta^2)$ distribution. In addition to the usual inference challenges under such non-linear constraints among the parameters (curved exponential family), one has to deal with the basic requirements of symmetry and positive definiteness when estimating a covariance matrix. We derive the non-linear likelihood equations for the constrained maximum likelihood estimator of $(\mu,\Sigma)$ and solve them using iterative methods. Generally, the MLE of covariance matrices computed using iterative methods do not satisfy the constraints. We propose a novel algorithm to modify such (infeasible) estimators or any other (reasonable) estimator. The key step is to re-align the mean vector along the eigenvectors of the covariance matrix using the idea of regression. In using the Lagrangian function for constrained MLE (Aitchison et al. 1958), the Lagrange multiplier entangles with the parameters of interest and presents another computational challenge. We handle this by either iterative or explicit calculation of the Lagrange multiplier. The existence and nature of location of the constrained MLE are explored within a data-dependent convex set using recent results from random matrix theory. A simulation study illustrates our methodology and shows that the modified estimators perform better than the initial estimators from the iterative methods.

A constrained maximum likelihood estimation for skew normal mixtures

A constrained maximum likelihood estimation for skew normal mixtures

Single-cell generalized trend model (scGTM): a flexible and interpretable model of gene expression trend along cell pseudotime.

Modeling single-cell gene expression trends along cell pseudotime is a crucial analysis for exploring biological processes. Most existing methods rely on nonparametric regression models for their flexibility; however, nonparametric models often provide trends too complex to interpret. Other existing methods use interpretable but restrictive models. Since model interpretability and flexibility are both indispensable for understanding biological processes, the single-cell field needs a model that improves the interpretability and largely maintains the flexibility of nonparametric regression models. Here, we propose the single-cell generalized trend model (scGTM) for capturing a gene's expression trend, which may be monotone, hill-shaped or valley-shaped, along cell pseudotime. The scGTM has three advantages: (i) it can capture non-monotonic trends that are easy to interpret, (ii) its parameters are biologically interpretable and trend informative, and (iii) it can flexibly accommodate common distributions for modeling gene expression counts. To tackle the complex optimization problems, we use the particle swarm optimization algorithm to find the constrained maximum likelihood estimates for the scGTM parameters. As an application, we analyze several single-cell gene expression datasets using the scGTM and show that scGTM can capture interpretable gene expression trends along cell pseudotime and reveal molecular insights underlying biological processes. The Python package scGTM is open-access and available at https://github.com/ElvisCuiHan/scGTM. Supplementary data are available at Bioinformatics online.

Non-Bayesian Parametric Missing-Mass Estimation

We consider the classical problem of missing-mass estimation, which deals with estimating the total probability of unseen elements in a sample. The missing-mass estimation problem has various applications in machine learning, statistics, language processing, ecology, sensor networks, and others. The naive, constrained maximum likelihood (CML) estimator is inappropriate for this problem since it tends to overestimate the probability of the observed elements. Similarly, the conventional constrained Cramer-Rao bound (CCRB), which is a lower bound on the mean-squared-error (MSE) of unbiased estimators, does not provide a relevant bound on the performance for this problem. In this paper, we introduce a frequentist, non-Bayesian parametric model of the problem of missing-mass estimation. We introduce the concept of missing-mass unbiasedness by using the Lehmann unbiasedness definition. We derive a non-Bayesian CCRB-type lower bound on the missing-mass MSE (mmMSE), named the missing-mass CCRB (mmCCRB), based on the missing-mass unbiasedness. The missing-mass unbiasedness and the proposed mmCCRB can be used to evaluate the performance of existing estimators. Based on the new mmCCRB, we propose a new method to improve existing estimators by an iterative missing-mass Fisher scoring method. Finally, we demonstrate via numerical simulations that the proposed mmCCRB is a valid and informative lower bound on the mmMSE of state-of-the-art estimators for this problem: the CML, the Good-Turing, and Laplace estimators. We also show that the performance of the Laplace estimator is improved by using the new Fisher-scoring method.

Intelligence-Aware Batch Processing for TMA with Bearings-Only Measurements.

This paper develops a framework to track the trajectory of a target in 2D by considering a moving ownship able to measure bearing measurements. Notably, the framework allows one to incorporate additional information (e.g., obtained via intelligence) such as knowledge on the fact the target’s trajectory is contained in the intersection of some sets or the fact it lies outside the union of other sets. The approach is formally characterized by providing a constrained maximum likelihood estimation (MLE) formulation and by extending the definition of the Cramér–Rao lower bound (CRLB) matrix to the case of MLE problems with inequality constraints, relying on the concept of generalized Jacobian matrix. Moreover, based on the additional information, the ownship motion is chosen by mimicking the Artificial Potential Fields technique that is typically used by mobile robots to aim at a goal (in this case, the region where the target is assumed to be) while avoiding obstacles (i.e., the region that is assumed not to intersect the target’s trajectory). In order to show the effectiveness of the proposed approach, the paper is complemented by a simulation campaign where the MLE computations are carried out via an evolutionary ant colony optimization software, namely, mixed-integer distributed ant colony optimization solver (MIDACO-SOLVER). As a result, the proposed framework exhibits remarkably better performance, and in particular, we observe that the solution is less likely to remain stuck in unsatisfactory local minima during the MLE computation.

Maximum likelihood estimation of two-sample population proportions under constraint on their difference

We derive the maximum likelihood estimate (MLE) of a population proportion when it differs from the same of a second population by a known value. This constrained MLE (CMLE) has a closed form in limited scenarios, which are completely characterized. These include the cases when the CMLE takes a boundary value in the parameter space. The existence of solution is established in the other cases and numerical methods are adopted in R and Excel to obtain the estimates solving a nonlinear equation. The standard error of the CMLE is estimated via bootstrap which also yields a confidence interval estimate; this is compared with a second method based on asymptotic distribution. The CMLE is of particular importance in the two sample testing of hypothesis of proportions based on independent samples, when these parameters differ by a non-zero value under the null hypothesis. Numerical computation establishes that the test statistic using the standard error based on this CMLE leads to a more reliable decision than the existing alternatives when the sample sizes are moderate to large.

Automatic Spline Knot Selection in Modeling Mortgage Loan Default Using Shape Constrained Regression

In mortgage default modeling, many of the key variables, such as loan age, FICO score, Debt-to-Income ratio (DTI), and Loan-to-House-Value ratio (LTV), have nonlinear relationships with the target default rates. Experienced modelers generally apply a spline transformation with knots to the individual variables. In this article, we introduce the Quantile-based Shape Constrained Maximum Likelihood Estimator (QSC-MLE), which features an automatic spline knot selection in a mortgage default model. QSC-MLE is an enhanced variant of SC-MLE (Chen and Samworth 2016) used in combination with a quantile-based knots set, to effectively process large datasets. QSC-MLE requires generic shape information of the inputs, for example, the monotonicity or convexity of the FICO score, DTI, and LTV, to capture any nonlinear effects. We show that the new default model considerably improves the accuracy of the out-of-sample prediction in comparison with the logistic regression and the Cox proportional hazards model. Moreover, the model conveniently generates component-wise spline functions, which facilitates the interpretation of the default rate response to the input variables. <b>Key Findings</b> ▪ A mortgage default model using the Quantile-based Shape Constrained Maximum Likelihood Estimator (QSC-MLE), which features automatic spline knot selection. ▪ QSC-MLE constructs shape-constrained spline functions to capture nonlinear effects of model inputs. ▪ The new default model considerably improves the accuracy of the out-of-sample prediction.

Generative adversarial network based regularized image reconstruction for PET

Positron emission tomography (PET) is an ill-posed inverse problem and suffers high noise due to limited number of detected events. Prior information can be used to improve the quality of reconstructed PET images. Deep neural networks have also been applied to regularized image reconstruction. One method is to use a pretrained denoising neural network to represent the PET image and to perform a constrained maximum likelihood estimation. In this work, we propose to use a generative adversarial network (GAN) to further improve the network performance. We also modify the objective function to include a data-matching term on the network input. Experimental studies using computer-based Monte Carlo simulations and real patient datasets demonstrate that the proposed method leads to noticeable improvements over the kernel-based and U-net-based regularization methods in terms of lesion contrast recovery versus background noise trade-offs.

Warranty Claims Forecasting for New Products Sold with a Two-Dimensional Warranty

Warranty claims forecasting plays an increasingly important role not only for preparing financial plans but also for optimizing warranty policy and improving after-sale services. In the case of new products, an important feature is that the new generation of products often has a close connection with the previous generations of products it replaces. Thus, the warranty claims data of the previous generations of products can be used for extracting reliability information of new products. In this context, we propose a warranty claims forecasting model considering usage rate for new products sold with a two-dimensional warranty. The accelerate failure time model is introduced to investigate the effect of usage rate on product degradation. The non-homogeneous Poisson process is used to model failure counts of repairable products and the constrained maximum likelihood estimation method is used to estimate model parameters. The results of data experiments based on both simulation and real data collected from an automobile manufacturer in China show that the proposed model considering the varying usage rate outperforms the traditional models in forecasting the number of warranty claims.

A Financial Market Model for the Netherlands: A Methodological Refinement

The Committee Parameters (Langejan et al. (2014)) advises to use the KNW-model (after Koijen et al. (2010)) to generate a representative scenario set for feasibility studies of pension funds. The scenario set enables a stochastic analysis of such feasibility studies. The underlying KNW-model is based on an affine factor model for the term structure. Stock returns, bond returns, interest rates, and inflation depend on observed factors and two latent factors. As such, the model contains relations between key financial risk factors of pension funds. CPB’s task is to estimate the model on Dutch data and, if appropriate, to calibrate some parameters in order to fit the recommendations of the Committee Parameters. Draper (2014) describes the current methods for this estimation and calibration.The calibration aims to adjust the Ultimate Forward Rate (UFR) and certain long-term expectations and covariances of the variables in the model. However, this calibration process introduces some arbitrariness. More specifically, the resulting parameter set may deviate substantially from the maximum likelihood set, even when taking the restrictions of the calibration into account. Instead of calibrating the model, we show how to impose restrictions in a continuous-time affine term structure model. In this way, the parameters correspond to the optimum of a constrained maximum likelihood estimation. The results suggest that the method in Draper (2014) provides suboptimal parameter estimates.The main result of this paper is the derivation of closed-form expressions for the long-term (unconditional) expectations, covariances, and the term structure. The expressions are required for the constrained likelihood optimization, and replace simulations for a long-run analysis of parameter sets. Our results apply to a wide range of continuous-time affine term strucure models with the Markov property, including the models in Dai and Singleton (2002) and Koijen et al. (2010).The model is outlined in Section 2. Section 3 provides expressions for the mean and covariance of possibly transformed variables in a VAR(1)-model. Section 4 presents closed-form expressions for some characteristics of the term structure in terms of the parameters. The estimation results are in Section 5. We draw conclusions in Section 6.

Inference for the mode of a log-concave density

We study a likelihood ratio test for the location of the mode of a log-concave density. Our test is based on comparison of the log-likelihoods corresponding to the unconstrained maximum likelihood estimator of a log-concave density and the constrained maximum likelihood estimator where the constraint is that the mode of the density is fixed, say at $m$. The constrained estimation problem is studied in detail in Doss and Wellner (2018). Here, the results of that paper are used to show that, under the null hypothesis (and strict curvature of $-\log f$ at the mode), the likelihood ratio statistic is asymptotically pivotal: that is, it converges in distribution to a limiting distribution which is free of nuisance parameters, thus playing the role of the $\chi_{1}^{2}$ distribution in classical parametric statistical problems. By inverting this family of tests, we obtain new (likelihood ratio based) confidence intervals for the mode of a log-concave density $f$. These new intervals do not depend on any smoothing parameters. We study the new confidence intervals via Monte Carlo methods and illustrate them with two real data sets. The new intervals seem to have several advantages over existing procedures. Software implementing the test and confidence intervals is available in the R package \verb+logcondens.mode+.

Efficient Likelihood Ratio Confidence Intervals using Constrained Optimization

Using constrained optimization, we develop a simple, efficient approach (applicable in both unconstrained and constrained maximum-likelihood estimation problems) to computing profile-likelihood confidence intervals. In contrast to Wald-type or score-based inference, the likelihood ratio confidence intervals use all the information encoded in the likelihood function concerning the parameters, which leads to improved statistical properties. In addition, the method does no suffer from the computational burdens inherent in the bootstrap. In an application to Rust's (1987) bus-engine replacement problem, our approach does better than either the Wald or the bootstrap methods, delivering very accurate estimates of the confidence intervals quickly and efficiently. An extensive Monte Carlo study reveals that in small samples, only likelihood ratio confidence intervals yield reasonable coverage properties, while at the same time discriminating implausible values.

Cramér-Rao Bound Under Norm Constraint

The constrained Cramer-Rao bound (CCRB) is a benchmark for constrained parameter estimation. However, the CCRB unbiasedness conditions are too strict and thus, the CCRB may not be a lower bound for estimators under constraints. The recently developed Lehmann-unbiased-CCRB (LU-CCRB) was shown to be a lower bound for the commonly used constrained maximum likelihood (CML) estimator performance in cases where the CCRB is not. In constrained parameter estimation, the estimator is usually required to satisfy the constraints. However, the LU-CCRB is a lower bound for Lehmann-unbiased estimators that do not necessarily satisfy the constraints. In this letter, we consider the norm constraint and derive a novel bound, called norm-constrained CCRB (NC-CCRB), which is a lower bound on the mean-squared-error matrix trace of Lehmann-unbiased estimators that satisfy the norm constraint. The NC-CCRB is shown to be tighter than the LU-CCRB. In the simulations, we consider a linear estimation problem under norm constraint in which the proposed NC-CCRB better predicts the performance of the CML estimator than the CCRB trace and the LU-CCRB.

Synthetic data method to incorporate external information into a current study.

We consider the situation where there is a known regression model that can be used to predict an outcome, Y, from a set of predictor variables X. A new variable B is expected to enhance the prediction of Y. A dataset of size n containing Y, X and B is available, and the challenge is to build an improved model for Y|X,B that uses both the available individual level data and some summary information obtained from the known model for Y|X. We propose a synthetic data approach, which consists of creating m additional synthetic data observations, and then analyzing the combined dataset of size n+m to estimate the parameters of the Y|X, B model. This combined dataset of size n+m now has missing values of B form of the observations, and is analyzed using methods that can handle missing data (e.g. multiple imputation). We present simulation studies and illustrate the method using data from the Prostate Cancer Prevention Trial. Though the synthetic data method is applicable to a general regression context, to provide some justification, we show in two special cases that the asymptotic variance of the parameter estimates in the Y|X, B model are identical to those from an alternative constrained maximum likelihood estimation approach. This correspondence in special cases and the method's broad applicability makes it appealing for use across diverse scenarios.