Simple Linear Regression Problem Research Articles

Improving the Robustness of the Theil-Sen Estimator Using a Simple Heuristic-Based Modification

One of the most widely used robust regression methods for solving simple linear regression problems is the Theil-Sen (TS) estimator. This estimator has some notable advantages; however, it does not belong to the most robust estimation methods (called high-breakdown estimators) and is prone to outliers whose distribution is highly asymmetric with respect to the correct data points. This paper presents a modification of the TS estimator, the Robustified Theil-Sen (RTS) estimator. The new method uses a heuristic-based selection procedure to reduce the number of initial estimates of the regression function parameters computed with at least one outlier, thereby improving the regression results. The use of this heuristic procedure only slightly increases the computational time required for using the RTS estimator compared to the TS estimator. Preliminary results of two numerical experiments presented in the paper show that the RTS estimator outperforms other comparable estimators, i.e., the TS estimator and the repeated median estimator, in terms of robustness. The results presented also suggest that the breakpoint value (which is a measure of the robustness of estimators) of the RTS estimator is higher than the breakpoint value of the TS estimator and equal to the breakpoint value of the high-breakpoint estimators.

Stable and compact face recognition via unlabeled data driven sparse representation-based classification

Sparse representation-based classification (SRC) has attracted much attention by casting the recognition problem as simple linear regression problem. SRC methods, however, still is limited to enough labeled samples per category, insufficient use of unlabeled samples, and instability of representation. For tackling these problems, an unlabeled data driven inverse projection pseudo-full-space representation-based classification model is proposed with low-rank sparse constraints. The proposed model aims to mine the hidden semantic information and intrinsic structure information of all available data, which is suitable for few labeled samples and proportion imbalance between labeled samples and unlabeled samples problems in frontal face recognition. The mixed Gauss–Seidel and Jacobian ADMM algorithm is introduced to solve the model. The convergence, representation capability and stability of the model are analyzed. Experiments on three public datasets show that the proposed LR-S-PFSRC model achieves stable results, especially for proportion imbalance of samples.

Rule-based Bayesian regression

We introduce a novel rule-based approach for handling regression problems. The new methodology carries elements from two frameworks: (i) it provides information about the uncertainty of the parameters of interest using Bayesian inference, and (ii) it allows the incorporation of expert knowledge through rule-based systems. The blending of those two different frameworks can be particularly beneficial for various domains (e.g., engineering), where even though the significance of uncertainty quantification motivates a Bayesian approach, there is no simple way to incorporate researcher intuition into the model. We validate our models by applying them to synthetic applications: a simple linear regression problem and two more complex structures based on partial differential equations, and we illustrate their use through two cases derived from real data. Finally, we review the advantages of our methodology, which include the simplicity of the implementation, the uncertainty reduction due to the added information and, in some occasions, the derivation of better point predictions, and we outline limitations, mainly from the computational complexity perspective, such as the difficulty in choosing an appropriate algorithm and the added computational burden.

Comparing nested data sets and objectively determining financial bubbles’ inceptions

Motivated by the question of identifying the start time τ of financial bubbles, we propose an improved calibration approach for time series in which the inception of the latest regime of interest is unknown. By taking into account the tendency of a given model to overfit data, we introduce the Lagrange regularisation of the normalised sum of the squared residuals, χnp2(Φ), to endogenously detect the optimal fitting window size ≔w∗∈[τ:t̄2] that should be used for calibration, assuming a fixed pseudo present time t̄2. The Lagrange regularisation of χnp2(Φ) defines the Lagrange regularised sum of the squared residuals, χλ2(Φ). Its performance is exemplified on a simple Linear Regression problem with a change point and compared against the performances of the Residual Sum of Squares (RSS) ≔χ2(Φ) and RSS/(N-p) ≔χnp2(Φ), where N is the sample size, p is the number of degrees of freedom and Φ is the parameter vector. Applied to synthetic models of financial bubbles with a well-defined transition regime and to a number of financial time series (US S&P500, Brazil IBovespa and China SSEC Indices), χλ2(Φ) is found to provide well-defined reasonable determinations of the starting times for major bubbles such as the bubbles ending with the 1987 Black-Monday, the 2008 Sub-prime crisis and minor speculative bubbles on other Indexes, without any further exogenous information. The application of the method thus allows one to endogenise the determination of the starting time of bubbles, a problem that has yet not received a systematic objective solution. Moreover, the technique appears as a practical solution for comparing goodness-of-fit across unbalanced sample sizes.

Lagrange Regularisation Approach to Compare Nested Data Sets and Determine Objectively Financial Bubbles' Inceptions

Inspired by the question of identifying the start time τ of financial bubbles, we address the calibration of time series in which the inception of the latest regime of interest is unknown. By taking into account the tendency of a given model to overfit data, we introduce the Lagrange regularisation of the normalised sum of the squared residuals, χ2np(Φ), to endogenously detect the optimal fitting window size := w∗ ∈ [τ : t2] that should be used for calibration purposes for a fixed pseudo present time t2. The performance of the Lagrange regularisation of χnp(Φ) defined as χ2λ(Φ) is exemplified on a simple Linear Regression problem with a change point and compared against the Residual Sum of Squares (RSS) := χ2 (Φ) and RSS/(N-p):= χ2np (Φ), where N is the sample size and p is the number of degrees of freedom. Applied to synthetic models of financial bubbles with a well-defined transition regime and to a number of financial time series (US S&P500, Brazil IBovespa and China SSEC Indices), the Lagrange regularisation of χ2λ(Φ) is found to provide well-defined reasonable determinations of the starting times for major bubbles such as the bubbles ending with the 1987 Black-Monday, the 2008 Sub-prime crisis and minor speculative bubbles on other Indexes, without any further exogenous information. It thus allows one to endogenise the determination of the beginning time of bubbles, a problem that had not received previously a systematic objective solution.

Convergence of a Fixed-Point Minimum Error Entropy Algorithm

The minimum error entropy (MEE) criterion is an important learning criterion in information theoretical learning (ITL). However, the MEE solution cannot be obtained in closed form even for a simple linear regression problem, and one has to search it, usually, in an iterative manner. The fixed-point iteration is an efficient way to solve the MEE solution. In this work, we study a fixed-point MEE algorithm for linear regression, and our focus is mainly on the convergence issue. We provide a sufficient condition (although a little loose) that guarantees the convergence of the fixed-point MEE algorithm. An illustrative example is also presented.

Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It

We empirically show that Bayesian inference can be inconsistent under misspecification in simple linear regression problems, both in a model averaging/selection and in a Bayesian ridge regression setting. We use the standard linear model, which assumes homoskedasticity, whereas the data are heteroskedastic (though, significantly, there are no outliers). As sample size increases, the posterior puts its mass on worse and worse models of ever higher dimension. This is caused by hypercompression, the phenomenon that the posterior puts its mass on distributions that have much larger KL divergence from the ground truth than their average, i.e. the Bayes predictive distribution. To remedy the problem, we equip the likelihood in Bayes' theorem with an exponent called the learning rate, and we propose the SafeBayesian method to learn the learning rate from the data. SafeBayes tends to select small learning rates, and regularizes more, as soon as hypercompression takes place. Its results on our data are quite encouraging.

Convex vs non-convex estimators for regression and sparse estimation

We study a simple linear regression problem for grouped variables; we are interested in methods which jointly perform estimation and variable selection, that is, that automatically set to zero grou...

Diagnosis of inhomogeneous insulation degradation in electric cables by distributed shunt conductance estimation

For the diagnosis of inhomogeneous insulation degradation in electric cables, the estimation of distributed shunt conductance is studied in this paper. Gradual growth of the shunt conductance is a consequence of degradation of the dielectric properties of the insulator. The proposed estimation method is based on voltage and current measurements at a single end of the cable. After the linearization of the bilinear term of the telegrapher's equations through a perturbation approach, the Kalman filter is applied to transform the problem of dynamic system parameter estimation to a simple linear regression problem. Results of numerical simulations are presented to demonstrate the feasibility of the proposed method. In particular, it is shown that the weak sensitivity of the available measurements to the shunt conductance can be compensated by long time data samples.

Diagnosis of inhomogeneous insulation degradation in electric cables by distributed shunt conductance estimation

For the diagnosis of inhomogeneous insulation degradation in electrical cables, the estimation of the distributed shunt conductance is studied in this paper. Gradual growth of the shunt conductance is a consequence of degradation of the dielectric properties of the insulator. The proposed estimation method is based on voltage and current measurements at a single end of the cable. After the linearization of the bilinear term of the telegrapher's equations through a perturbation approach, the Kalman filter is applied to transform the problem of dynamic system parameter estimation to a simple linear regression problem. Results of numerical simulations are presented to demonstrate the feasibility of the proposed method. In particular, it is shown that the weak sensitivity of the available measurements to the shunt conductance can be compensated by long time data samples.

How to Find all Buckley-James Estimates Instead of Just One?

Consider the simple linear regression problem with right-censored data. The Buckley-James estimator (BJE) (1979) has been considered superior over the other available procedures. So far the existing algorithms fail to present all possible BJE's. In this note, we present a feasible non-iterative algorithm which can find all BJE's. We use the Stanford heart transplant data to show that if we do not find all BJE's (6 BJE's for the data), then the analysis may be misleading as is the case in Buckley and James (1979, p. 435) and in Miller (1981, p. 156). Extension to multiple linear regression is also discussed.

On Asymmetric Properties of the Correlation Coeffcient in the Regression Setting

Although correlation is a symmetric concept of two variables, this is not the case for regression where we distinguish a response from an explanatory variable. This article presents several ways of expressing the correlation coefficient as an asymmetric formula of the two variables involved in the regression setting. Contrary to some well-known results, those are not necessarily preserved in the sample when the model is wrong. As a consequence, they may be used for model checking or model selection. In particular, we propose a criterion for choosing the response variable in a simple linear regression problem. An example from the domain of finance illustrates our purpose. We find evidence under our model that the U.S. dollar influenced other currencies during the period of consideration.

Robust estimation for the coefficient of a first order autoregressive process

Nonparametric methods, Theil's method and Hussain's method have been applied to simple linear regression problems for estimating the slope of the regression line.We extend these methods and propose a robust estimator to estimate the coefficient of a first order autoregressive process under various distribution shapes, A simulation study to compare Theil's estimator, Hus-sain's estimator, the least squares estimator, and the proposed estimator is also presented.

Nonrandom sequence of slope-intercept estimates in longitudinal gompertzian analysis suggests biological relevance

The significance of intersections in age-specific mortality rate distributions could be attributed to a fundamental statistical relationship between estimates of slope and intercept. A strong negative correlation between estimates of slope and intercept is often observed in simple linear regression problems. The net result is that the family of lines generated by repetitive estimates of slope and intercept in a static experimental situation will tend to intersect at a common point. This statistical relationship between slope and intercept, however, should be random with respect to the time-ordered sequence of slope and intercept estimates. Annual paired slope and intercept estimates derived using the method of longitudinal Gompertzian analysis of age-specific mortality rates for men and women in the United States from 1900 to 1988 were analyzed to determine if they varied randomly. The probability that the observed sequence in these annual paired slope-intercept estimates was random is less than 10 −50 for both men and women. This finding essentially excludes the possibility that intersections in age-specific mortality rate distributions reflect a fundamental statistical relationship between slope and intercept and further suggests biological relevance for the method of longitudinal Gompertzian analysis.

Teaching Regression in a Nonparametric Statistics Course

Abstract Various estimators of slope, intercept, and mean response in the simple linear regression problem are compared in terms of unbiasedness, efficiency, breakdown, and mean squared error. Theil's estimator of slope and two intercept estimators based on Theil's estimator are recommended for inclusion in nonparametrics courses as robust, efficient, and easy-to-calculate alternatives to least squares.

Algorithms for unconstrained L1 simple linear regression

Many algorithms have been proposed for L 1 simple linear regression computations, and several codes implementing these algorithms are now available. Some of these algorithms are based on a linear programming formulation while others propose a descent approach. We report on a computational study comparing L 1 computer programs for solving the simple linear regression problem.

A comparison of robust estimators in simple linear regression

The mean squared errors of various estimators of slope, intercept, and mean response in the simple linear regression problem are compared in a simulation study. A weighted median estimator of slope proposed by Sievers (1978) and Scholz (1978) and two intercept estimators based upon it are found to perform well for most error distributions studied. Theil's (1950) estimator of slope and two intercept estimators based on it are preferable for certain heavily contaminated error distributions.

L1for the simple linear regression model

By modifying the direct method to solve the overdetermined linear system we are able to present an algorithm for L1 estimation which appears to be superior computationally to any other known algorithm for the simple linear regression problem.

A Strategy for Efficient Sample Selection in Simple Linear Regression Problems with Unequal per unit Sampling Costs

Simple linear regression is widely used in forestry, but often only a vaguely defined strategy for selecting sampling units is followed. Trial and error methods exist for aiding efficient sample allocation for simple linear regression purposes. These methods are computationally tedious and often impractical without the aid of a computer. This paper briefly describes a computerized iterative search procedure that can provide an efficient design for sample allocation in simple linear regression problems with equal or unequal sampling costs and balanced or unbalanced prediction intervals. Potential savings gained by employing an efficient design over other more easily derived but less efficient designs are illustrated by an example. Key words: Simple linear regression, optimal sampling design, iterative search procedure.

A linked list data structure for a simple linear regression algorithm

This paper presents a special purpose linear programming algorithm to solve simple linear regression problems with least absolute value criterion. A special data structure is implemented and discussed in detail. Computational results are given.