Least-squares Estimation Research Articles

Finite Sample Identification of Low-Order LTI Systems via Nuclear Norm Regularization

This paper studies the problem of identifying low-order linear time-invariant systems via Hankel nuclear norm (HNN) regularization. This regularization encourages the Hankel matrix to be low-rank, which corresponds to the dynamical system being of low order. We provide novel statistical analysis for this regularization, and contrast it with the unregularized ordinary least-squares (OLS) estimator. Our analysis leads to new finite-sample error bounds on estimating the impulse response and the Hankel matrix associated with the linear system using HNN regularization. We design a suitable input excitation, and show that we can recover the system using a number of observations that scales optimally with the true system order and achieves strong statistical estimation rates. Complementing these, we also demonstrate that the input design indeed matters by proving that intuitive choices, such as i.i.d. Gaussian input, lead to sub-optimal sample complexity. To better understand the benefits of regularization, we also revisit the OLS estimator. Besides refining existing bounds, we experimentally identify when HNN regularization improves over OLS: (1) For loworder systems with slow impulse-response decay, OLS method performs poorly in terms of sample complexity, (2) the Hankel matrix returned by regularization has a more clear singular value gap that makes determining the system order easier, (3) HNN regularization is less sensitive to hyperparameter choice. To choose the regularization parameter, we also outline a simple joint train-validation procedure

Learning Causal Estimates of Linear Operators from Noisy Data

This paper studies the identification problem of linear systems from a set of noisy input-output trajectories. The problem is formulated and solved as a least-square regularized estimate on a suitable function space of finite-bandwidth operators. This abstract setting is well suited to represent a broad class of finite- and infinite-dimensional linear systems. We determine the value of the regularization parameter as a function of the amount of noise on the learning trajectories and we show how to obtain recursive and causal estimates for the case of linear dynamical systems.

Bias-Bounded Estimation of AmbiguiTy: A Method for Radio Interferometric Positioning

Carrier phase signals form the basis of various interferometric measurement models and estimation techniques that have parameters of which some are integer. In this paper, integer least-squares estimation theory is extended and applied to bias-bounded mixed-integer models. This extension accommodates the presence of bounded real-valued parameters in mixed-integer models through incorporating prior knowledge of a set, in which the parameters reside, into the estimation process. This enables one to jointly estimate the ambiguous phase cycles and the parameters of interest. To compute such mixed-integer estimates, a fast search strategy is developed that makes use of a dual-ellipsoid encompassing region. The volume of the stated region is quantified and its links to existing ellipsoidal search spaces are highlighted. Simulated and real-world data are employed to illustrate the theory. It is then, for the first time, shown that the proposed method makes <i>single-epoch</i>, <i>phase-only</i> positioning feasible with Global Navigation Satellite Systems (GNSS).

The more data, the better? Demystifying deletion-based methods in linear regression with missing data.

We compare two deletion-based methods for dealing with the problem of missing observations in linear regression analysis. One is the complete-case analysis (CC, or listwise deletion) that discards all incomplete observations and only uses common samples for ordinary least-squares estimation. The other is the available-case analysis (AC, or pairwise deletion) that utilizes all available data to estimate the covariance matrices and applies these matrices to construct the normal equation. We show that the estimates from both methods are asymptotically unbiased under missing completely at random (MCAR) and further compare their asymptotic variances in some typical situations. Surprisingly, using more data (i.e., AC) does not necessarily lead to better asymptotic efficiency in many scenarios. Missing patterns, covariance structure and true regression coefficient values all play a role in determining which is better. We further conduct simulation studies to corroborate the findings and demystify what has been missed or misinterpreted in the literature. Some detailed proofs and simulation results are available in the online supplemental materials.

Comparison of IDEAL-IQ and IVIM-DWI for Differentiating between Alpha Fetoprotein-Negative Hepatocellular Carcinoma and Focal Nodular Hyperplasia

Background: To compare the differential diagnostic value of iterative decomposition of water and fat with the echo asymmetrical and least-squares estimation quantitation sequence (IDEAL-IQ) with that of intravoxel incoherent motion diffusion-weighted imaging (IVIM DWI) in differentiating between alpha fetoprotein (AFP)-negative hepatocellular carcinoma (HCC) and focal nodular hyperplasia (FNH). Materials and Methods: A total of 28 AFP-negative HCC cases and 15 FNH cases were scanned using the IDEAL-IQ and IVIM-DWI magnetic resonance imaging (MRI) protocols. Two radiologists independently assessed the fat fraction (FF) and the iron level surrogate (R2*) derived from the IDEAL-IQ images and the apparent diffusion coefficient (ADC), pure diffusion coefficient (D), pseudo-diffusion coefficient (D*), and perfusion fraction (f) derived from the IVIM-DWI images. The intraclass correlation coefficients (ICC) were estimated to evaluate the agreement repeatability between the two readers. The area under the curve (AUC) of a receiver operating characteristics curve was used to compare the diagnostic efficiency of the parameters extracted from the two techniques. Results: The lesions in the HCC group had significantly higher FF than the FNH group (8.284 ± 5.756 vs. 2.559 ± 1.247, P < 0.05). ADC and D values were lower in the HCC lesions when compared with the FNH lesions (1.310 ± 0.253 and 0.909 ± 0.192, respectively, vs. 1.624 ± 0.304 and 1.230 ± 0.314; ×10−3 mm2 /s). The FF parameter had the highest AUC (0.923) followed by D (0.864) and ADC (0.854). Conclusions: FF derived from IDEAL-IQ, and ADC and D derived from IVIM-DWI were able to differentiate AFP-negative HCC from FNH. IDEAL-IQ showed better performance for the differentiation of FNH from HCC than the IVIM-DWI-derived parameters.

Modeling of organizing function management in Vietnam’s public construction works

Organizing function is the significant component of management guaranteeing the success of construction works. Many studies have been emphasizing on the topic of critical success factors (CSFs) within construction works, yet the results have rarely emphasized examining organizing behaviors within public construction work management; these less researched topics were the aims of this study. To fulfil this research aim, a regression analysis design was employed. Data were collected using questionnaires conducted from 139 professionals involved in public construction works management in Vietnam. The structural equation modelling (SEM) technique with partial least-squares estimation (PLS) was utilized to analyze the data. The results revealed 6 behavioral dimensions (i.e., structure organizing (OR1), authorization organizing (OR2), coordination organizing (OR4), human resource organizing (OR3), job organizing (OR5), responsibility organizing (OR6) to assess organizing function in terms of public construction work management. The study also reveals that structure organizing (OR1), authorization organizing (OR2), coordination organizing (OR4) have significant effects on management effectiveness (ME). In addition, coordination organizing (OR4) acts as the mediator of Human resource organizing (OR3), job organizing (OR5); while responsibility organizing (OR6) indicates an indirect influence through the mediator of OR5. The success of this approach is expected to reinforce the contribution of organizing function and suggest a useful tool for supporting the professionals in enhancing public construction work management.

Varying coefficient linear discriminant analysis for dynamic data

Linear discriminant analysis (LDA) is an important classification tool in statistics and machine learning. This paper investigates the varying coefficient LDA model for dynamic data, with Bayes’ discriminant direction being a function of some exposure variable to address the heterogeneity. We propose a new least-square estimation method based on the B-spline approximation. The data-driven discriminant procedure is more computationally efficient than the dynamic linear programming rule [21]. We also establish the convergence rates for the corresponding estimation error bound and the excess misclassification risk. The estimation error in L2 distance is optimal for the low-dimensional regime and is near optimal for the high-dimensional regime. Numerical experiments on synthetic data and real data both corroborate the superiority of our proposed classification method.

Robust Mediation Analysis: The R Package robmed

Mediation analysis is one of the most widely used statistical techniques in the social, behavioral, and medical sciences. Mediation models allow to study how an independent variable affects a dependent variable indirectly through one or more intervening variables, which are called mediators. The analysis is often carried out via a series of linear regressions, in which case the indirect effects can be computed as products of coefficients from those regressions. Statistical significance of the indirect effects is typically assessed via a bootstrap test based on ordinary least-squares estimates. However, this test is sensitive to outliers or other deviations from normality assumptions, which poses a serious threat to empirical testing of theory about mediation mechanisms. The R package robmed implements a robust procedure for mediation analysis based on the fast-and-robust bootstrap methodology for robust regression estimators, which yields reliable results even when the data deviate from the usual normality assumptions. Various other procedures for mediation analysis are included in package robmed as well. Moreover, robmed introduces a new formula interface that allows to specify mediation models with a single formula, and provides various plots for diagnostics or visual representation of the results.

Estimating Single-Epoch Integrated Atmospheric Refractivity From InSAR for Assimilation in Numerical Weather Models

Numerical weather prediction (NWP) models are used to predict the weather based on current observations in combination with physical and mathematical models. Yet, they are limited by the spatial density and the accuracy of the available observations. Satellite radar interferometry (InSAR) is known to be extremely sensitive to the 3D atmospheric refractivity distribution, and has a high spatial resolution, providing information that can be used for assimilation in NWP models. However, due to the inherent superposition of two or more atmospheric states, only biased and temporally differenced signals can be retrieved, that can also be contaminated by deformation signals and decorrelation. Here we present a method to estimate single-epoch absolute atmospheric delays by combining InSAR time series with prior NWP model prediction time series, using a constrained least-squares estimation. We show that this leads to a solution that reliably extracts the single-epoch relative delays from InSAR data and uses prior NWP model data to find the absolute reference for these delays, while mitigating long-term deformation and decorrelation signal. This approach leads to repetitive delay updates with a spatial resolution of 500 m, that can be directly assimilated into numerical weather models.

Lindley Generalized Inverted Exponential Distribution: Model and Applications

In this article, we have proposed three parameters distribution using lindley family of distribution and Generalized inverted Exponential distribution called Lindley Generalized Inverted Exponential Distribution: Model and Applications(LGIE). The statistical properties as well as different characteristics like the hazard rate function (HRF), the probability density function (PDF), the cumulative distribution function (CDF), quantile function , skewness, and kurtosis of the proposed distribution are discussed. The parameters of the proposed distribution are estimated by using the maximum likelihood estimation (MLE), least-square estimation (LSE) and Cramer-Von-Mises (CVM) methods. All the computations are performed in R programming software. To assess the application of the proposed distribution, a real lifetime data set is analyzed and performed the goodness-of-fit.

PR-LTTE: Link travel time estimation based on path recovery from large-scale incomplete trip data

The widespread use of positioning devices has generated large-scale trip data, boosting the study of traffic modeling. For instance, New York City Taxi & Limousine Commission regularly releases over 165 million taxi trip records containing the end-point information of each trip every year. Such big datasets provide us potential new perspectives to tackle traditional issues in traffic modeling. In this paper, we propose to use such vehicle trip data, which do not contain information on the intermediate vehicle positions other than the end points, to study the link (road segment) travel time estimation problem. A method named PR-LTTE is developed to estimate the mean travel time for interested links in a road network based on path recovery, jointly modeling trip distance and trip travel time. The key idea is to iteratively alternate between 1) inferring the most likely path for a given trip by minimizing a loss function that considers both trip distance and trip travel time, and 2) computing the link travel time via the least-squares estimation with the inferred path. Our experiments on two large-scale trip datasets (including New York City taxi trip data and Chengdu DiDi trip data) show very promising results. PR-LTTE improves the accuracy of travel time estimation by more than 50% in most cases over state-of-the-art methods.

Privacy-preserving indoor localization based on inner product encryption in a cloud environment

How to utilize the resource-rich cloud to provide scalable indoor localization service to solve the problem of limited user equipment resources while preventing privacy leakage has become a hot topic. A novel privacy-preserving indoor localization scheme based on Inner Product Encryption in a cloud environment is proposed to address this issue. By using the computability of Inner Product Encryption on ciphertext without exposing plaintext information, an honest-but-curious cloud server estimates the users location without obtaining any private information. The least-squares estimation algorithm of the Time Difference location system is decomposed into the basic form of the inner product of two sets of vectors. One is the anchor information encrypted with Inner Product Encryption during the offline phase, the other is the users distance difference information encrypted and sent to the cloud server during the online phase. To hide the users real location, the users real distance difference information and several decoy information are encrypted and included in a request to achieve k-Anonymity. The location estimation algorithm operates on the cloud to locate the user from the ciphertext. In addition, three optimization methods applied to cloud server location estimation algorithm were designed to further improve the performance. The experimental results and theoretical analysis show that the proposed scheme consumes less time, and has a lower communication overhead, while maintaining positioning accuracy. At the same time, the scheme protects the users locational and data privacy of the indoor positioning service provider when migrating the indoor positioning services to the cloud environment.

Symmetry Analysis of the Uncertain Alternative Box-Cox Regression Model

The asymmetry of residuals about the origin is a severe issue in estimating a Box-Cox transformed model. In the framework of uncertainty theory, there are such theoretical issues regarding the least-squares estimation (LSE) and maximum likelihood estimation (MLE) of the linear models after the Box-Cox transformation on the response variables. Heretofore, only weighting methods for least-squares analysis have been available. This article proposes an uncertain alternative Box-Cox model to alleviate the asymmetry of residuals and avoid λ tending to negative infinity for uncertain LSE or uncertain MLE. Such symmetry of residuals about the origin is reasonable in applications of experts’ experimental data. The parameter estimation method was given via a theorem, and the performance of our model was supported via numerical simulations. According to the numerical simulations, our proposed ‘alternative Box-Cox model’ can overcome the problems of a grossly underestimated lambda and the asymmetry of residuals. The estimated residuals neither deviated from zero nor changed unevenly, in clear contrast to the LSE and MLE for the uncertain Box-Cox model downward biased residuals. Thus, though the LSE and MLE are not applicable on the uncertain Box-Cox model, they fit the uncertain alternative Box-Cox model. Compared with the uncertain Box-Cox model, the issue of a systematically underestimated λ is not likely to occur in our uncertain alternative Box-Cox model. Both the LSE and MLE can be used directly without constructing a weighted estimation method, offering better performance in the asymmetry of residuals.

A Correction Method of Height Variation Error Based on One SNR Arc Applied in GNSS–IR Sea-Level Retrieval

Sea-level monitoring is important for the safety of coastal cities and analysis of ocean and climate. Sea levels can be estimated based using the global navigation satellite system–interferometry reflectometry (GNSS–IR). The frequency in a signal-to-noise ratio (SNR) arc has been found to be related to the height between the GNSS antenna and reflecting surface, which is called reflector height (RH, h). The height variation of the reflecting surface causes an error, and this error is the most significant error in the GNSS–IR sea-level retrieval. The key to the correction of height variation error lies in the determination of the RH variation rate h˙. The classical correction method determines h˙ based on tide analysis of a coarse RH series over a longer time period. Therefore, h˙ inherits errors in coarse RH series, which contains significant bias during a storm surge, and correcting this requires data accumulation. This study proposes a correction method of height variation error based on just one SNR arc based on wavelet analysis and least-square estimation. First, using wavelet analysis, instantaneous frequencies are extracted in one SNR arc; these frequencies are then converted to RH series. Second, using least-square estimation, h and h˙ are conjointly solved based on the RH series from wavelet analysis. Data of GNSS site HKQT located in Hong Kong, China, during a period of time that includes Typhoon Hato were used. The root-mean-square errors (RMSEs) of retrievals were 21.5 cm for L1, 9.5 cm for L2P, 9.3 cm for L2C, and 7.6 cm for L5 of GPS; 16.8 cm for L1C, 14.1 cm for L1P, 12.6 cm for L2C, and 10.7 cm for L2P of GLONASS; 15.7 cm for L1, 11.2 cm for L5, 12.2 cm for L7, and 9.6 cm for L8 of Galileo. Results showed this method can correct the height variation error based on just one SNR arc, can avoid the inheritance of errors, and can be used during periods of storm surge.

Estimation of the slope parameter in a linear regression model under a bounded loss function

The estimation of the slope parameter of a simple linear regression model in the presence of nonsample prior information under the reflected normal loss function is considered. Usually, the traditional estimation methods such as the least squared (LS) error are used to estimate the slope parameter. Sometimes the researcher has information about the unknown slope parameter from experience as a point guess, the nonsample prior information. In this paper, the shrinkage pretest estimators are introduced and their risk functions are derived under the reflected normal loss function. Several methods of finding distrust coefficient of the shrinkage pretest estimators are proposed. The behavior of the estimators are compared using a simulation study. The results show that the shrinkage pretest estimator outperforms the LS estimator when nonsample prior information is close to the real value. A real data set is analyzed for illustrating the results.

A Comparison of Model-Assisted Estimators, With and Without Data-Driven Transformations of Auxiliary Variables, With Application to Forest Inventory

Forest information is requested at many levels and for many purposes. Sampling-based national forest inventories (NFIs) can provide reliable estimates on national and regional levels. By combining expensive field plot data with different sources of remotely sensed information, from airplanes and/or satellite platforms, the precision in estimators of forest variables can be improved. This paper focuses on the design-based model-assisted approach to using NFI data together with remotely sensed data to estimate forest variables for small areas, where the variables studied are total growing stock volume, volume of Norway spruce (Picea abies), and volume of broad-leaved trees. Remote sensing variables may be highly correlated with one another and some may have poor predictive ability for target forest variables, and therefore model selection and/or coefficient shrinkage may be appropriate to improve the efficiency of model-assisted estimators of forest variables. For this purpose, one can use modern shrinkage estimators based on lasso, ridge, and elastic net regression methods. In a simulation study using real NFI data, Sentinel 2 remote-sensing data, and a national airborne laser scanning (ALS) campaign, we show that shrinkage estimators offer advantages over the (weighted) ordinary least-squares (OLS) estimator in a model-assisted setting. For example, for a sample size n of about 900 and with 72 auxiliary variables, the RMSE was up to 41% larger when based on OLS. We propose a data-driven method for finding suitable transformations of auxiliary variables, and show that it can improve estimators of forest variables. For example, when estimating volume of Norway spruce, using a smaller expert selection of auxiliary variables, transformations reduced the RMSE by up to 10%. The overall best results in terms of RMSE were obtained using shrinkage estimators and a larger set of 72 auxiliary variables. However, for this larger set of variables, the use of transformations yielded at most small improvements of RMSE, and at worst large increases of RMSE, except in combination with ridge and elastic net regression.

A Spatial-Enhanced LSE-SFIM Algorithm for Hyperspectral and Multispectral Images Fusion

The fusion of a hyperspectral image (HSI) and multispectral image (MSI) can significantly improve the ability of ground target recognition and identification. The quality of spatial information and the fidelity of spectral information are normally contradictory. However, these two properties are non-negligible indicators for multi-source remote-sensing images fusion. The smoothing filter-based intensity modulation (SFIM) method is a simple yet effective model for image fusion, which can improve the spatial texture details of the image well, and maintain the spectral characteristics of the image significantly. However, traditional SFIM has a poor effect for edge information sharpening, leading to a bad overall fusion result. In order to obtain better spatial information, a spatial filter-based improved LSE-SFIM algorithm is proposed in this paper. Firstly, the least square estimation (LSE) algorithm is combined with SFIM, which can effectively improve the spatial information quality of the fused image. At the same time, in order to better maintain the spatial information, four spatial filters (mean, median, nearest and bilinear) are used for the simulated MSI image to extract fine spatial information. Six quality indexes are used to compare the performance of different algorithms, and the experimental results demonstrate that the LSE-SFIM based on bilinear (LES-SFIM-B) performs significantly better than the traditional SFIM algorithm and other spatially enhanced LSE-SFIM algorithms proposed in this paper. Furthermore, LSE-SFIM-B could also obtain similar performance compared with three state-of-the-art HSI-MSI fusion algorithms (CNMF, HySure, and FUSE), while the computing time is much shorter.

A Note on Confusion between Linear and Affine Functions and the Generalized Forms of Gradient

Background: Arguably the most frequently used term in science, particularly in mathematics and statistics, is linear. However, confusion arises from the various meanings of linearity instructed in different levels of mathematical courses. The definition of linearity taught in high school is less correct than the one learned in a linear algebra class. The correlation coefficient of two quantitative variables is a numerical measure of the affinity, not only linearity, of two variables. However, every statistics book loosely says it is a measure of linear relationship. This clearly show that there is some confusion between use of the terms the linear function and affine function.&#x0D; Objective: This article aims at clarifying the confusion between use of the terms linear function and affine function. It also provides more generalized forms of the gradient in different branches of mathematics and show their equivalency.&#x0D; Materials and Methods: We have used the pure analytical deductive methods to proof the statements. &#x0D; Results: We have clearly presented that gradient is the measure of affinity, not just linearity. It becomes a special case of the derivative in calculus, of the least-squares estimate of the regression coefficient in statistics and matrix theory. The gradient can ­­­­be seen in terms of the inverse of the informative matrix in the most general setting of the linear model estimation.&#x0D; Conclusion: The article has been clearly written to show the distinction between the linear and affine functions in a concise and unambiguous manner. We hope that readers will clearly see various generalizations of the gradient and article itself would be a simple exposition, enlightening, and fun to read.

On Channel Estimation for Rician Fading with the Phase-Shift in Cell-Free Massive MIMO System

Channel estimation (CE) is a crucial phase in wireless communication systems, especially in cell-free (CF) massive multiple input multiple output (M-MIMO) since it is a dynamic wireless network. Therefore, this work is introduced to study CE for the CF M-MIMO system in the uplink phase, wherein the performance of different estimators are evaluated, discussed, and compared in various situations. We assume the scenario in which each access point has prior knowledge of the channel statistics. The phase-aware-minimum mean square error (PA-MMSE) estimator, the non-phase-aware-MMSE (NPA-MMSE) estimator, and the least-squares estimator are the three estimators which are exploited in this work. Besides, we consider the Rician fading channel in which the line-of-sight path is realized with a phase-shift that models the users’ mobility where the considered phase-shift follows a uniform distribution. On the other hand, the mean-squared error metric is employed in order to evaluate the performance of each estimator, where an analytical and simulated result is provided for the PA-MMSE estimator and the NPA-MMSE estimator in order to assert our numerical results.

Sample Complexity of Block-Sparse System Identification Problem

In this article, we study the system identification problem for sparse linear time-invariant systems. We propose a sparsity promoting block-regularized estimator to identify the dynamics of the system with only a limited number of input-state data samples. We characterize the properties of this estimator under high-dimensional scaling, where the growth rate of the system dimension is comparable to or even faster than that of the number of available sample trajectories. In particular, using contemporary results on high-dimensional statistics, we show that the proposed estimator results in a small elementwise error, provided that the number of sample trajectories is above a threshold. This threshold depends polynomially on the size of each block and the number of nonzero elements at different rows of input and state matrices, but only logarithmically on the system dimension. A by product of this result is that the number of sample trajectories required for sparse system identification is significantly smaller than the dimension of the system. Furthermore, we show that, unlike the recently celebrated least-squares estimators for system identification problems, the method developed in this work is capable of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">exact recovery</i> of the underlying sparsity structure of the system with the aforementioned number of data samples. Extensive case studies on switching networks and power systems are offered to demonstrate the effectiveness of the proposed method.