Invariant Quadratic Estimators Research Articles

Estimating variances in time series kriging using convex optimization and empirical BLUPs

We propose a two-stage estimation method of variance components in time series models known as FDSLRMs, whose observations can be described by a linear mixed model (LMM). We based estimating variances, fundamental quantities in a time series forecasting approach called kriging, on the empirical (plug-in) best linear unbiased predictions of unobservable random components in FDSLRM. The method, providing invariant non-negative quadratic estimators, can be used for any absolutely continuous probability distribution of time series data. As a result of applying the convex optimization and the LMM methodology, we resolved two problems $-$ theoretical existence and equivalence between least squares estimators, non-negative (M)DOOLSE, and maximum likelihood estimators, (RE)MLE, as possible starting points of our method and a practical lack of computational implementation for FDSLRM. As for computing (RE)MLE in the case of $ n $ observed time series values, we also discovered a new algorithm of order $\mathcal{O}(n)$, which at the default precision is $10^7$ times more accurate and $n^2$ times faster than the best current Python(or R)-based computational packages, namely CVXPY, CVXR, nlme, sommer and mixed. We illustrate our results on three real data sets $-$ electricity consumption, tourism and cyber security $-$ which are easily available, reproducible, sharable and modifiable in the form of interactive Jupyter notebooks.

Road detection based on illuminant invariance and quadratic estimation

Road extraction is an important part of the intelligent vehicle systems for automatic driving, navigation, and traffic warning. For the complicated road scene, we present a road detection method based on illumination invariant image and quadratic estimation. The algorithm firstly extracts the illumination invariant image, and a priori triangular road region is used as the color sample to analyze the illumination invariant image and obtain the probability maps. Next, based on the histogram analysis, the combined probability map is significantly resettled, and the road region is estimated for the first time. Then gradient images of the illumination invariant image and the probability map are extracted, and the gradient image is analyzed by the estimated road region. Finally, the effective road boundary is extracted, and the more accurate road region is obtained. The experimental results show that our method can adapt to the road image in a variety of environments. Compared with other algorithms, our algorithm is more stable, and the computational efficiency is improved obviously.

Quadratic subspaces and construction of Bayes invariant quadratic estimators of variance components in mixed linear models

Gnot et al. (J Statist Plann Inference 30(1):223–236, 1992) have presented the formulae for computing Bayes invariant quadratic estimators of variance components in normal mixed linear models of the form \(\mathcal{N}\{y,X\beta,\sum_{i=1}^{k} \sigma_{i}^{2} V_{i} \},\) where the matrices Vi, 1 ≤ i ≤ k − 1, are symmetric and nonnegative definite and Vk is an identity matrix. These formulae involve a basis of a quadratic subspace containing MV1M,...,MVk-1M,M, where M is an orthogonal projector on the null space of X′. In the paper we discuss methods of construction of such a basis. We survey Malley’s algorithms for finding the smallest quadratic subspace including a given set of symmetric matrices of the same order and propose some modifications of these algorithms. We also consider a class of matrices sharing some of the symmetries common to MV1M,...,MVk-1M,M. We show that the matrices from this class constitute a quadratic subspace and describe its explicit basis, which can be directly used for computing Bayes invariant quadratic estimators of variance components. This basis can be also used for improving the efficiency of Malley’s algorithms when applied to finding a basis of the smallest quadratic subspace containing the matrices MV1M,...,MVk-1M,M. Finally, we present the results of a numerical experiment which confirm the potential usefulness of the proposed methods.

Unbiased invariant minimum norm estimation in generalized growth curve model

This paper considers the generalized growth curve model Y = ∑ i = 1 m X i B i Z i ′ + U E subject to R ( X m ) ⊆ R ( X m - 1 ) ⊆ ⋯ ⊆ R ( X 1 ) , where B i are the matrices of unknown regression coefficients, X i , Z i and U are known covariate matrices, i = 1 , 2 , … , m , and E splits into a number of independently and identically distributed subvectors with mean zero and unknown covariance matrix Σ . An unbiased invariant minimum norm quadratic estimator ( MINQE ( U , I ) ) of tr ( C Σ ) is derived and the conditions for its optimality under the minimum variance criterion are investigated. The necessary and sufficient conditions for MINQE ( U , I ) of tr ( C Σ ) to be a uniformly minimum variance invariant quadratic unbiased estimator ( UMVIQUE) are obtained. An unbiased invariant minimum norm quadratic plus linear estimator ( MINQLE ( U , I ) ) of tr ( C Σ ) + ∑ i = 1 m tr ( D i ′ B i ) is also given. To compare with the existing maximum likelihood estimator ( MLE) of tr ( C Σ ) , we conduct some simulation studies which show that our proposed estimator performs very well.

Estimation of variances in orthogonal finite discrete spectrum linear regression models

The Invariant Quadratic Estimators, the Maximum Likelihood Estimator (MLE) and Restricted Maximum Likelihood Estimator (REML) of variances in an orthogonal Finite Discrete Spectrum Linear Regression Model (FDSLRM) are derived and the problems of unbiasedness and consistency of these estimators are investigated.

On geometry of the set of admissible invariant quadratic estimators in balanced two variance components model

Two variance components model for which each invariant quadratic admissible estimator of a linear function of variance components (under quadratic loss function) is a linear combination of two quadratic forms,Z1,Z2, say, is considered. A setD={(d1,d2)′:d1Z1+d2Z2 is admissible} is described by giving formulae on the boundary ofD. Different forms of the setD are presented on figures.

On estimation of variance components with constraints

The so-called linear approach to estimating linear functions of variance and/or covariance components clearly shows the well-known connection between quadratic, invariant and unbiased, estimation and the classical linear theory methods, LSE and BLUE. This approach allows us to consider and to use further results of the linear theory for variance-covariance components estimation. In the present paper the minimax invariant quadratic estimators of linear functions of variance-covariance components under full or partial restrictions on the parameter set are investigated.

A multivariate approach for estimating the random effects variance component in one-way random effects model

It is well known that the ANOVA estimator of the random effects variance component in one-way random effects model can assume negative values. It is also well known that nonnegative quadratic unbiased estimators do not exist for estimating the random effects variance component (LaMotte, 1973). LaMotte (1985) indicated the possibility that nonnegative invariant quadratic estimator of the random effects variance component uniformly better than the ANOVA estimator may exist for the balanced one-way random effects model. Mathew et al. (1992a) have shown that such estimator exists only when the number of treatments is 9 or less. As noted in Herbach (1959) (see also Thompson, 1962), a simple truncation of the ANOVA estimator at zero yields uniform improvement over the ANOVA estimator. The estimators suggested by Herbach and Thompson are, in fact, restricted maximum likelihood estimators, and they are nonquadratic by nature. In this paper, we discuss a multivariate technique which always yields positive estimate of the random effects variance component in one-way random effects model. The multivariate approach exploits the estimates of the eigenvalues of the covariance matrix of the model in estimating the variance components including the error variance. The resulting estimates are nonquadratic. The success of this multivariate approach depends on the precise estimation of the eigenvalues. Since there does not exist any unbiased estimation procedure in the small sample case for the estimation of the eigenvalues, we use a delete-d jackknife procedure to estimate them. This delete-d jackknife based multivariate approach yields better estimates (in terms of mean squared error) for the random effects variance component than the restricted maximum likelihood estimation as well as Chow and Shao's (1988) nonquadratic estimation approaches, which is shown through a simulation study for the cases with number of treatments up to 20.

On the nonexistence of a global nonnegative minimum bias invariant quadratic estimator of variance components

We consider nonnegative invariant quadratic estimation of variance components and prove that if there does not exist a nonnegative quadratic unbiased estimator of variance components, there does not exist a nonnegative invariant quadratic estimator with a global minimum bias. The result demonstrates that any nonnegative invariant quadratic estimator can only achieve local minimum bias and explains existing numerical results.

Best estimation of variance components with arbitrary kurtosis in two-way layouts mixed models

Under the assumptions that a linear model corresponding to two-way layouts is the mixed model without interaction, necessary and sufficient conditions are derived under which the minimum norm invariant quadratic unbiased estimators have uniformly minimum variance among all unbiased invariant quadratic estimators for data with arbitrary kurtosis. The conditions are expressed in terms of the incidence matrix N and C-matrix of the design. Under the model considered in this paper, the results obtained here strengthen those by Anderson et al. (1984), where data are required to satisfy a special balance condition.

Improvement of ANOVA estimators of variance components in block designs with random treatments

This paper concerns the problem of estimating variance components in the experiments with block designs with random treatments. In this paper we present conditions under which the nonnegative invariant quadratic estimator, presented as a multiple of a treatments sum of squares from the analysis of variance, dominates the ANOVA estimator of treatment variance in the sense that it has uniformly smaller mean squared error.

Tests Based on Admissible Estimators in Two Variance Components Models

In the paper a new class of tests for variance components in two variance components model is presented. The tests are based on nonnegative admissible invariant quadratic estimators of variance components. A comparison of the tests with the known ones are given. Two-way classification models corresponding to block designs are considered separately. Examples of these models are given, for which the power functions of the tests are computed by using Imhof’s procedure, and compared with the attainable upper bound obtained by using the Neyman–Pearson Lemma.

Improved Estimators of Variance Components with Smaller Probability of Negativity

SUMMARY A linear model with two variance components is considered, one variance component (say, σ 1 2 ≥ 0) corresponding to a random effect, and a second variance component (say, σ 2 > 0) corresponding to the experimental errors. A class of invariant quadratic estimators (IQEs) is characterized, having uniformly smaller mean-squared error (MSE), and uniformly smaller probability of negativity, compared with the analysis-of-variance (ANOVA) estimator of σ 1 2. It turns out that for balanced models, among IQEs of σ 1 2 with uniformly smaller MSE than its ANOVA estimator, there exists an IQE with the smallest probability of being negative, uniformly in the parameters. The results are applied to the balanced one-way classification model. Numerical computations show that the MSE improvement achievable through the use of the proposed estimators is quite significant. It is noted that, for the nonnegative estimation of σ 1 2, truncation of one of the proposed estimators at 0 provides a satisfactory solution. Extension of the results to the unbalanced case, and to the general mixed model with balanced data, is also indicated.

Admissible Estimators of Variance Components Obtained Via Submodels

Simultaneous estimation of the vector of variance components under unbalanced mixed models w.r.t. the ordinary quadratic loss function is considered. A new method of constructing invariant quadratic admissible estimators, both with and without the condition of unbiasedness, is presented. Using this method, admissible estimators for the factorial models with imbalance at the last stage and the unbalanced $(p - 1)$-way nested factors design are constructed.

Nonnegative estimation of variance components in unbalanced mixed models with two variance components

An unbalanced mixed linear model with two variance components is considered, one variance component (say σ 1 2 ≥ 0) corresponding to a random effect (treatments) and a second variance component (say σ 2 > 0) corresponding to the experimental errors. Sufficient conditions are obtained under which there will exist a nonnegative invariant quadratic estimator (IQE) having a uniformly smaller mean squared error (MSE) than every unbiased IQE of σ 1 2. In particular, for the one-way unbalanced ANOVA model, necessary and sufficient conditions are also obtained for a multiple of the usual treatment sum of squares to uniformly dominate the ANOVA estimator of σ 1 2. For estimating σ 2, it is shown that the best multiple of the residual sum of squares can be improved by using nonquadratic estimators. One such estimator is obtained using the idea of a testimator ( Stein, 1964, Ann. Inst. Statist. Math. 16 155–160). A second estimator is obtained following the approach in Strawderman (1974, Ann. Statist. 2 190–198). Numerical results regarding the performance of the proposed estimators of σ 1 2 are also reported.

Bayes invariant quadratic estimation in general linear regression models

In the paper the general linear regression model E( y )= Xβ , Cov( y)= Σ k i=1 σ 2 i V i ; V i ≥0, i=1,2,…, k−1, V k = I , is considered. For this model explicit formulae for the Bayes invariant quadratic unbiased estimators and the Bayes invariant quadratic estimators are given. These estimators are expressed in terms of a base of a Jordan algebra generated by MV 1 M , MV 2 M ,…, MV k−1 M , M , where M is the orthogonal projector on the null space of X ′. Two-way classification random models corresponding to orthogonal and partially balanced block designs are considered as examples.

Quadratic unbiased estimation without invariance and its application in the unbalanced one-way random model

Quadratic unbiased estimation in a general mixed model is considered. It was recently proved in Stępniak (1987, Ann. Inst. Statist. Math. 39, 563–573) that any admissible estimator in the problem is a limit of unique locally best quadratics in a wide sense. We focus on the usual locally best quadratic unbiased estimators (LBQUE's). It is shown, that any locally best invariant quadratic unbiased estimator (or MINQUE with invariance) is a limit of LBQUE's. Explicit formulae for LBQUE's and their variances are given. The unbalanced one-way random model is considered in detail.

Consistency of linear and quadratic least squares estimators in regression models with covariance stationary errors

The least squres invariant quadratic estimator of an unknown covariance function of a stochastic process is defined and a sufficient condition for consistency of this estimator is derived. The mean value of the observed process is assumed to fulfil a linear regresion model. A sufficient condition for consistency of the least squares estimator of the regression parameters is derived, too.

Characterizations of two-way layouts from the point of view of variance component estimation in the corresponding mixed linear models

Under the assumption that a linear model corresponding to a two-way layout is the mixed model without interaction, complete characterizations are derived of the layouts that assure the invariant unbiased estimability of every linear function of two variance components involved and of designs that assure the existence of the uniformly best invariant quadratic unbiased estimator for every estimable function of variance components.

Limiting admissible estimators for variance components

Simultaneous estimation of the vector of the variance components for mixed and random models under the quadratic loss function is considered. For a large class of such models there are identified classes of admissible biased invariant quadratic estimators that are better than some admissible unbiased estimators. Numerous numerical results presented in the paper show that for many of the commonly used balanced models the improvements in the quadratic risk may be considerable over a large set of the parameter space.