Linear Regression Problem Research Articles

Fast and accurate pseudoinverse with sparse matrix reordering and incremental approach

How can we compute the pseudoinverse of a sparse feature matrix efficiently and accurately for solving optimization problems? A pseudoinverse is a generalization of a matrix inverse, which has been extensively utilized as a fundamental building block for solving linear systems in machine learning. However, an approximate computation, let alone an exact computation, of pseudoinverse is very time-consuming due to its demanding time complexity, which limits it from being applied to large data. In this paper, we propose FastPI (Fast PseudoInverse), a novel incremental singular value decomposition (SVD) based pseudoinverse method for sparse matrices. Based on the observation that many real-world feature matrices are sparse and highly skewed, FastPI reorders and divides the feature matrix and incrementally computes low-rank SVD from the divided components. To show the efficacy of proposed FastPI, we apply them in real-world multi-label linear regression problems. Through extensive experiments, we demonstrate that FastPI computes the pseudoinverse faster than other approximate methods without loss of accuracy. Results imply that our method efficiently computes the low-rank pseudoinverse of a large and sparse matrix that other existing methods cannot handle with limited time and space.

High-dimensional dynamic systems identification with additional constraints

This note presents a unified analysis of the identification of dynamical systems with low-rank constraints under high-dimensional scaling. This identification problem for dynamic systems is challenging due to the intrinsic dependency of the data. To alleviate this problem, we first formulate this identification problem into a multivariate linear regression problem with a row-sub-Gaussian measurement matrix using the more general input designs and the independently repeated sampling schemes. We then propose a nuclear norm heuristic method that estimates the parameter matrix of a dynamic system from a few input-state data samples. Based on this, we can extend the existing results. In this article, we consider two scenarios. (i) In the noiseless scenario, nuclear-norm minimization is introduced for promoting low-rank. We define the notion of weak restricted isometry property, which is weaker than the ordinary restricted isometry property, and show it holds with high probability for the row-sub-Gaussian measurement matrix. Thereby, the rank-minimization matrix can be exactly recovered from a finite number of data samples. (ii) In the noisy scenario, a regularized framework involving the nuclear norm penalty is established. We give the notion of operator norm curvature condition for the loss function and show it holds for the row-sub-Gaussian measurement matrix with high probability. Consequently, when specifying the suitable choice of the regularization parameter, the operator norm error of the optimal solution of this program has a sharp bound given a finite amount of data samples. This operator norm error bound is always smaller than the ordinary Frobenius norm error bound obtained in the existing work.

Block-Based Refitting in $$\ell _{12}$$ Sparse Regularization

In many linear regression problems, including ill-posed inverse problems in image restoration, the data exhibit some sparse structures that can be used to regularize the inversion. To this end, a classical path is to use $$\ell _{12}$$ block-based regularization. While efficient at retrieving the inherent sparsity patterns of the data—the support—the estimated solutions are known to suffer from a systematical bias. We propose a general framework for removing this artifact by refitting the solution toward the data while preserving key features of its structure such as the support. This is done through the use of refitting block penalties that only act on the support of the estimated solution. Based on an analysis of related works in the literature, we introduce a new penalty that is well suited for refitting purposes. We also present a new algorithm to obtain the refitted solution along with the original (biased) solution for any convex refitting block penalty. Experiments illustrate the good behavior of the proposed block penalty for refitting solutions of total variation and total generalized variation models.

Sparse linear regression from perturbed data

The problem of sparse linear regression is relevant in the context of linear system identification from large datasets. When data are collected from real-world experiments, measurements are always affected by perturbations or low-precision representations. However, the problem of sparse linear regression from fully-perturbed data is scarcely studied in the literature, due to its mathematical complexity. In this paper, we show that, by assuming bounded perturbations, this problem can be tackled by solving low-complex ℓ2 and ℓ1 minimization problems. Both theoretical guarantees and numerical results are illustrated.

Jump or kink: on super-efficiency in segmented linear regression breakpoint estimation

Summary We consider the problem of segmented linear regression with a single breakpoint, with the focus on estimating the location of the breakpoint. If $n$ is the sample size, we show that the global minimax convergence rate for this problem in terms of the mean absolute error is $O(n^{-1/3})$. On the other hand, we demonstrate the construction of a super-efficient estimator that achieves the pointwise convergence rate of either $O(n^{-1})$ or $O(n^{-1/2})$ for every fixed parameter value, depending on whether the structural change is a jump or a kink. The implications of this example and a potential remedy are discussed.

Recovery guarantees for polynomial coefficients from weakly dependent data with outliers

Learning non-linear systems from noisy, limited, and/or dependent data is an important task across various scientific fields including statistics, engineering, computer science, mathematics, and many more. In general, this learning task is ill-posed; however, additional information about the data’s structure or on the behavior of the unknown function can make the task well-posed. In this work, we study the problem of learning nonlinear functions from corrupted and weakly dependent data. The learning problem is recast as a sparse robust linear regression problem where we incorporate both the unknown coefficients and the corruptions in a basis pursuit framework. The main contribution of our paper is to provide a reconstruction guarantee for the associated ℓ1-optimization problem where the sampling matrix is formed from weakly dependent data. Specifically, we prove that the sampling matrix satisfies the null space property and the stable null space property, provided that the data is compact and satisfies a suitable concentration inequality. We show that our recovery results are applicable to various types of weakly dependent data such as exponentially strongly α-mixing data, geometrically C-mixing data, and uniformly ergodic Markov chain. Our theoretical results are verified via several numerical simulations.

Distributed Online Linear Regressions

We study online linear regression problems in a distributed setting, where the data is spread over a network. In each round, each network node proposes a linear predictor, with the objective of fitting the network-wide data. It then updates its predictor for the next round according to the received local feedback and information received from neighboring nodes. The predictions made at a given node are assessed through the notion of regret, defined as the difference between their cumulative network-wide square errors and those of the best off-line network-wide linear predictor. Various scenarios are investigated, depending on the nature of the local feedback (full information or bandit feedback), on the set of available predictors (the decision set), and the way data is generated (by an oblivious or adaptive adversary). We propose simple and natural distributed regression algorithms, involving, at each node and in each round, a local gradient descent step and a communication and averaging step where nodes aim at aligning their predictors to those of their neighbors. We establish regret upper bounds typically in O(T <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3/4</sup> ) when the decision set is unbounded and in O(√T) in case of bounded decision set.

Extreme Wavelet Fast Learning Machine for Evaluation of the Default Profile on Financial Transactions

Extreme learning machines enable multilayered neural networks to perform activities to facilitate the process and business dynamics. It acts in pattern classification, linear regression problems, and time series prediction. The financial area needs efficient models that can perform businesses in a short time. Credit card fraud and debits occur regularly, and effective decision making can avoid significant obstacles for both clients and financial companies. This paper proposes a training model for multilayer networks where the weights of the training algorithm are defined by the nature and characteristics of the dataset using the concepts of the wavelet transform. The traditional algorithm of weights’ definition of the output layer is changed to a regularized method that acts more quickly in the description of the weights of the output layer. Finally, several activation functions are applied to the model to verify its efficiency in several scenarios. This model was subjected to an extensive dataset and comparing to different machine learning approaches. Its answers were satisfactory in a short-time execution, proving that the Extreme Learning Machine works efficiently to identify possible profiles of defaulters in payments in the financial relationships involving a credit card.

Randomized numerical linear algebra: Foundations and algorithms

This survey describes probabilistic algorithms for linear algebraic computations, such as factorizing matrices and solving linear systems. It focuses on techniques that have a proven track record for real-world problems. The paper treats both the theoretical foundations of the subject and practical computational issues.Topics include norm estimation, matrix approximation by sampling, structured and unstructured random embeddings, linear regression problems, low-rank approximation, subspace iteration and Krylov methods, error estimation and adaptivity, interpolatory and CUR factorizations, Nyström approximation of positive semidefinite matrices, single-view (‘streaming’) algorithms, full rank-revealing factorizations, solvers for linear systems, and approximation of kernel matrices that arise in machine learning and in scientific computing.

Distributed Mirror Descent for Online Composite Optimization

In this article, we consider an online distributed composite optimization problem over a time-varying multiagent network that consists of multiple interacting nodes, where the objective function of each node consists of two parts: a loss function that changes over time and a regularization function. This problem naturally arises in many real-world applications ranging from wireless sensor networks to signal processing. We propose a class of online distributed optimization algorithms that are based on approximate mirror descent, which utilizes the Bregman divergence as a distance-measuring function that includes the Euclidean distances as a special case. We consider two standard information feedback models when designing the algorithms, that is, full-information feedback and bandit feedback. For the full-information feedback model, the first algorithm attains an average regularized regret of order O(1/√T) with the total number of rounds T. The second algorithm, which only requires the information of the values of the loss function at two predicted points instead of the gradient information, achieves the same average regularized regret as that of the first algorithm. Simulation results of a distributed online regularized linear regression problem are provided to illustrate the performance of the proposed algorithms.

Benign overfitting in linear regression

The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accurate prediction. We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size. By studying examples of data covariance properties that this characterization shows are required for benign overfitting, we find an important role for finite-dimensional data: the accuracy of the minimum norm interpolating prediction rule approaches the best possible accuracy for a much narrower range of properties of the data distribution when the data lie in an infinite-dimensional space vs. when the data lie in a finite-dimensional space with dimension that grows faster than the sample size.

Identical fixed points in state evolutions of AMP and VAMP

AMP and VAMP are two different algorithms that solve high-dimensional standard linear regression problems effectively and efficiently. In previous studies, they were long observed to have identical fixed points in their state evolutions (SEs), however, a formal justification is still missing. This issue is addressed in the paper. Instead of looking directly into the problem, we consider a scope more ambitious which puts aside all assumptions underlying the proven SEs. We then show that: 1) Having the spectrum of a Marchenko-Pastur law is a necessary and sufficient condition for the AMP-style and VAMP-style SEs to share an identical fixed point; 2) As the zero-mean i.i.d. Gaussian ensemble have a Marchenko-Pastur spectrum and satisfies all conditions required by the proven AMP SE and VAMP SE, this particular ensemble is the first example of having an identical fixed point. After that, we conjecture the proven VAMP SE might be extended to cover ensembles that break the right rotational invariance, among which the i.i.d. zero-mean sub-Gaussian ensemble could be a second example to yield identical fixed points. This conjecture is supported by experiments from wireless communications and compressive sampling.

Method of frequency-spatial selection of radio emissions using a triorthogonal antenna system

Introduction: An unresolved problem in radio monitoring is the separation of overlapping radio spectrum from different sourceswhen restricted dimensions of the antenna system do not provide the necessary spatial resolution. Purpose: Developing models of radioemissions and procedures for their processing in order to select signals from different sources out of their additive mixture at the antennasystem input. Methods: Point linear regression for the decomposition of the resulting field at the triorthogonal antenna system inputinto basic functions which are consistent with the frequency parameters of the radio emissions. Results: On the geometric basis, using acoordinate transformation matrix, due to the reference system rotation, radio emission models with different polarization are proposedfor a plane electromagnetic wave with a given arrival direction. The developed models have the amplitude and phase parameters split intoseparate factors. Procedures have been developed for separating the additive mixture of radio emissions in the triorthogonal antennasystem into components related to different sources. The novelty of the presented solution lies in the use of geometric interpretationof the point linear regression problem, when the resulting vector of the electric field of the sum of two radio emissions is decomposedinto linearly independent vectors composed of basic functions whose choice is determined by the frequency parameters of the signals.In addition to the actual selection of the signals, it is possible to analyze the spatial and polarization parameters of the radio emissions.Practical relevance: The method can be implemented in radio monitoring equipment with restrictions on the weight and dimensions, forthe use in complex signal environments.

Variational Bayesian weighted complex network reconstruction

Complex network reconstruction is a hot topic in many fields. Currently, the most popular data-driven reconstruction framework is based on lasso. However, it is found that, in the presence of noise, lasso loses efficiency for weighted networks. This paper builds a new framework to cope with this problem. The key idea is to employ a series of linear regression problems to model the relationship between network nodes, and then to use an efficient variational Bayesian algorithm to infer the unknown coefficients. The numerical experiments conducted on both synthetic and real data demonstrate that the new method outperforms lasso with regard to both reconstruction accuracy and running speed.

Decision-Level Data Fusion in Quality Control and Predictive Maintenance

Data fusion integrates data from multiple sources to improve prediction performance. While significant research has been conducted to develop data-level and feature-level fusion methods, very few studies are performed to develop more effective decision-level data fusion methods. This research aims at developing a decision-level data fusion approach that transforms low-dimensional decisions (i.e., predictions) made based on individual sensor data such as temperature and vibration to high-dimensional decisions. Integration of these high-dimensional decisions is formulated as a convex optimization problem rather than a traditional multivariate linear regression problem. The proposed decision-level data fusion approach is demonstrated in two cases: 1) quality control in additive manufacturing and 2) predictive maintenance in aircraft engines. Experimental results have shown that the proposed decision-level fusion method can reduce prediction variance by at least 30% as well as increase prediction accuracy by 45%.

Pseudo estimation and variable selection in regression

Often a problem in linear regression either for correlated data or high-dimensional data is the singularity of the sample covariance matrix. While dealing with an ill-conditioned covariance matrix has been a longstanding challenge in the statistical literature, recent proposals relying on adding noises to the original data have found their successes in obtaining reliable estimates and making predictions. It has been shown that perturbing data with noises has a close relationship to the well-known ridge estimator. In this paper, we propose to add noises to the predictors with a known covariance structure, and call the estimator obtained in such a way a “pseudo estimator”. A new variable selection procedure based on the concept of pseudo confidence interval which works for both correlated predictors and “large p small n” problem is proposed. We study the theoretical properties of the proposed pseudo estimator and variable selection procedure. Furthermore, we use an ensemble step to stabilize the pseudo estimation (variable selection) results. The advantages of the proposed method are demonstrated by both simulation studies and real data analyses.

Robust regression based on shrinkage with application to Living Environment Deprivation

A robust estimator is proposed for the parameters that characterize the linear regression problem. It is based on the notion of shrinkages, often used in Finance and previously studied for outlier detection in multivariate data. A thorough simulation study is conducted to investigate: the efficiency with normal and heavy-tailed errors, the robustness under contamination, the computational times, the affine equivariance and breakdown value of the regression estimator. Two classical data-sets often used in the literature and a real socio-economic data-set about the Living Environment Deprivation of areas in Liverpool (UK), are studied. The results from the simulations and the real data examples show the advantages of the proposed robust estimator in regression.

Horseshoe Regularisation for Machine Learning in Complex and Deep Models1

SummarySince the advent of the horseshoe priors for regularisation, global–local shrinkage methods have proved to be a fertile ground for the development of Bayesian methodology in machine learning, specifically for high‐dimensional regression and classification problems. They have achieved remarkable success in computation and enjoy strong theoretical support. Most of the existing literature has focused on the linear Gaussian case; for which systematic surveys are available. The purpose of the current article is to demonstrate that the horseshoe regularisation is useful far more broadly, by reviewing both methodological and computational developments in complex models that are more relevant to machine learning applications. Specifically, we focus on methodological challenges in horseshoe regularisation in non‐linear and non‐Gaussian models, multivariate models and deep neural networks. We also outline the recent computational developments in horseshoe shrinkage for complex models along with a list of available software implementations that allows one to venture out beyond the comfort zone of the canonical linear regression problems.

A Fuzzy Transformation of the Classic Stream Sediment Transport Formula of Yang

The objective of this study is to transform the arithmetic coefficients of the total sediment transport rate formula of Yang into fuzzy numbers, and thus create a fuzzy relationship that will provide a fuzzy band of in-stream sediment concentration. A very large set of experimental data, in flumes, was used for the fuzzy regression analysis. In a first stage, the arithmetic coefficients of the original equation were recalculated, by means of multiple regression, in an effort to verify the quality of data, by testing the closeness between the original and the calculated coefficients. Subsequently, the fuzzy relationship was built up, utilizing the fuzzy linear regression model of Tanaka. According to Tanaka’s fuzzy regression model, all the data must be included within the produced fuzzy band and the non-linear regression can be concluded to a linear regression problem when auxiliary variables are used. The results were deemed satisfactory for both the classic and fuzzy regression-derived equations. In addition, the linear dependence between the logarithmized total sediment concentration and the logarithmized subtraction of the critical unit stream power from the exerted unit stream power is presented. Ultimately, a fuzzy counterpart of Yang’s stream sediment transport formula is constructed and made available to the readership.

Perturbation Method in Problems of Linear Matrix Regression

Within the frameworks of linear regression this work studied estimates linear by observations, in particular the unbiased ones that leads to unbiased equations, among solutions of which the ones minimum in norm are distinguished, that allows us to minimize the root mean square error at uncorrelated observation errors with equal variances. Preliminary the problem of linear regression analysis is presented in the form of linear operator in a space of independent rectangular matrices connected with the equation of linear functions unbiasedness from matrix parameters. It is assumed that for this operator in the unperturbed version its SVD representation is known as well as SVD representation of the pseudo inverse to it operator. Since it is necessary to determine the singular set of the perturbed operator for determining eigenvalues and eigenvectors of the special symmetric matrix we employ the perturbation method, according to the general theory of operators in Euclidean space we determine the eigenmatrices of conjugate perturbed operator. Assuming that the problem of the linear regression analysis at the presence of perturbation of observation matrices can be solved in real time conditions, the resulting formulas are given in the first approximation of the small parameter. We present the test example which besides a small parameter includes parameters of random perturbations.