Ridge Regression Problem Research Articles

An efficient Newton-type matrix splitting algorithm for solving generalized absolute value equations with application to ridge regression problems

A generalized Newton-based matrix splitting (GNMS) method is proposed for solving the generalized absolute value equations (GAVEs). Under mild conditions, the GNMS method converges to the unique solution of GAVEs. Moreover, we can obtain a few weaker convergence conditions for some existing methods. Numerical results verify the effectiveness of the proposed method.

Condition numbers of the generalized ridge regression and its statistical estimation

<abstract><p>In this paper, we considered the condition number theory of a new generalized ridge regression model. The explicit expressions of different types of condition numbers were derived to measure the ill-conditionness of the generalized ridge regression problem with respect to different circumstances. To overcome the computational difficulty of computing the exact value of the condition number, we employed the statistical condition estimation theory to design efficient condition number estimators, and the numerical examples were also given to illustrate its efficiency.</p></abstract>

Recursive ridge regression using second-order stochastic algorithms

Recursive second-order stochastic algorithms are presented for solving ridge regression problems in the linear and binary logistic case. The proposed algorithms allow to update the estimates of ridge solution when the data arrive in continuous flow. Some guarantees on the almost sure behavior of the proposed algorithms are established. Numerical experiments on simulated and real-world data show the advantages of our algorithms compared to alternative methods.

Ridge-Regression-Induced Robust Graph Relational Network.

Graph convolutional networks (GCNs) have attracted increasing research attention, which merits in its strong ability to handle graph data, such as the citation network or social network. Existing models typically use first-order neighborhood information to design specific convolution operations, which aggregate the features of all adjacent nodes. However, such models ignore the high-order spatial relationship among neighboring nodes in noisy data due to its modeling complexity. In this article, we propose a novel robust graph relational network to address this issue toward modeling high-order relationships in noisy data for graph convolution. Our key innovation lies in designing a generic relation network layer, which is used to infer the underlying relations among adjacent noisy nodes. Specifically, a fixed number of adjacent nodes for each node is chosen by solving the ridge regression problem, in which the regression coefficients are used to rank the adjacent nodes of each node in a graph. Furthermore, to mine the rich features, we extract high-order information from the nodes to significantly enhance the representation ability of the GCNs for extensive applications. We conduct extensive semisupervised node classification experiments on the noisy benchmark datasets, which clearly show that our model is superior to the existing methods and can achieve state-of-the-art performance.

A Piecewise Linear Regression and Classification Algorithm With Application to Learning and Model Predictive Control of Hybrid Systems

This paper proposes an algorithm for solving multivariate regression and classification problems using piecewise linear predictors over a polyhedral partition of the feature space. The resulting algorithm that we call PARC (Piecewise Affine Regression and Classification) alternates between (<i>i</i>) solving ridge regression problems for numeric targets, softmax regression problems for categorical targets, and either softmax regression or cluster centroid computation for piecewise linear separation, and (<i>ii</i>) assigning the training points to different clusters on the basis of a criterion that balances prediction accuracy and piecewise-linear separability. We prove that PARC is a block-coordinate descent algorithm that minimizes a suitably constructed objective function and that it converges in a finite number of steps. The algorithm is used to learn hybrid numerical/categorical (HYNC) dynamical models from data that contain real and discrete labeled values. The resulting model has a piecewise linear structure that is particularly useful to formulate model predictive control problems and solve them by mixed-integer programming. A Python implementation of PARC is available at <uri>http://cse.lab.imtlucca.it/~bemporad/parc</uri>.

Active Broad-Transfer Learning Algorithm for Class-Imbalanced Fault Diagnosis

Knowledge transfer with class-imbalanced data is a challenge in predictive maintenance and fault diagnosis. Deep learning algorithms have provided promising results in fault diagnosis. However, their prediction performance is affected by class-imbalanced data in cross-domain tasks. Broad learning algorithms present promising performance in handling class-imbalanced domain-adaptation (CIDA) problems. In the presence of a domain shift, active broad transfer for class-imbalanced learning (ABTCI), an active broad-transfer learning algorithm for CIDA, is proposed. First, the ABTCI algorithm extracts the time–frequency features and feeds them into a recurrent cell to capture spatial–temporal features. Subsequently, it augments the feature space using a sparse autoencoder and an orthogonal mapping projector. By solving the ridge regression problem, the classifier is initialized. Next, the algorithm samples the target data with reliable pseudo-labels and synthesizes new data using random intraclass interpolation among the minor classes containing source and target knowledge. Finally, the classifier is updated using an incremental continuous learning strategy. The performance of the ABTCI algorithm is validated using three datasets, which include 20 class-balanced and 27 class-imbalanced transfer tasks. The performance of the proposed algorithm benchmarked against other deep-learning algorithms is promising.

RidgeSketch: A Fast Sketching Based Solver for Large Scale Ridge Regression

We propose new variants of the sketch-and-project method for solving large scale ridge regression problems. First, we propose a new momentum alternative and provide a theorem showing it can speed up the convergence of sketch-and-project, through a fast sublinear convergence rate. We carefully delimit under what settings this new sublinear rate is faster than the previously known linear rate of convergence of sketch-and-project without momentum. Second, we consider combining the sketch-and-project method with new modern sketching methods such as Count sketch, SubCount sketch (a new method we propose), and subsampled Hadamard transforms. We show experimentally that when combined with the sketch-and-project method, the (Sub)Count sketch is very effective on sparse data and the standard Subsample sketch is effective on dense data. Indeed, we show that these sketching methods, combined with our new momentum scheme, result in methods that are competitive even when compared to the conjugate gradient method on real large scale data. On the contrary, we show the subsampled Hadamard transform does not perform well in this setting, despite the use of fast Hadamard transforms, and nor do recently proposed acceleration schemes work well in practice. To support all of our experimental findings, and invite the community to validate and extend our results, with this paper we are also releasing an open source software package: RidgeSketch. We designed this object-oriented package in Python for testing sketch-and-project methods and benchmarking ridge regression solvers. RidgeSketch is highly modular, and new sketching methods can easily be added as subclasses. We provide code snippets of our package in the appendix.

Fast and Accurate Gaussian Kernel Ridge Regression Using Matrix Decompositions for Preconditioning

This paper presents a preconditioner-based method for solving a kernel ridge regression problem. In contrast to other methods, which utilize either fast matrix-vector multiplication or a preconditioner, the suggested approach uses randomized matrix decompositions for building a preconditioner with a special structure that can also utilize fast matrix-vector multiplications. This hybrid approach is efficient in reducing the condition number, exact, and computationally efficient, enabling the processing of large datasets with computational complexity linear to the number of data points. Also, a theoretical upper bound for the condition number is provided. For Gaussian kernels, we show that given a desired condition number, the rank of the needed preconditioner can be determined directly from the dataset.

Predictive Modeling With an Adaptive Unsupervised Broad Transfer Algorithm

Deep-learning algorithms have produced promising results, however, domain adaptation remains a challenge. In addition, excessive training time and computing resource requirements need to be addressed. Deep-learning algorithms face a domain adaptation issue when the data distribution of a target domain differs from that of the source domain. The emerging concept of broad learning shows potential in addressing the domain adaptation and training time issues. An adaptive unsupervised broad transfer learning (AUBTL) algorithm is proposed to tackle the cross-domain problems. The proposed algorithm utilizes a sparse autoencoder and random orthogonal mapping to extract and augment the feature space. Then, it initializes the weights of a classifier by solving a ridge regression problem. The logit ranking strategy is applied to develop a transfer estimator to evaluate and sample data in the target domain for an adaptive transfer. Based on the sampled data, AUBTL optimizes the hyperparameter space. The performance of the AUBTL algorithm is validated with three benchmark datasets, including 20 transfer tasks. The computational results demonstrated the efficiency and accuracy of the proposed algorithm over other deep-learning algorithms considered in this research.

Autonomous Lane Keeping System: Lane Detection, Tracking and Control on Embedded System

Autonomous lane keeping system is the key technique to autonomous driving. It includes lane detection, lane tracking and control. It has been developed enormously, but it is still a challenge work due to different factors such as illumination, general hyper-parameters setting for different road condition and lane boundary correction. In addition, due to imbalance on accuracy and processing time, it is hard to conduct in embedding system. In this study, an autonomous lane keeping system is developed based on deep learning. First, a lane detection and tracking system is designed, which is robust to lane boundary correction. Especially for lane detection, a light-weight network named as LaneFCNet is proposed, which can balance accuracy and processing time. Then, lane tracking was followed by detector to improve the detection performance and create autonomous driving trajectory. Finally, to brief lane fitting problem, it was treated as ridge regression problem, which can enhance the effectiveness to the whole system. Experimental results show that our integrated lane detection and tracking system can trade off accuracy and processing time and the whole line keeping system is robust enough to autonomous driving.

Robust visual tracking using self-adaptive strategy

Discriminative correlation filter-based algorithms have recently demonstrated prominent advantages in the community of computer visual tracking, due to their ability to convert ridge regression problems in the frequency domain for creating solutions efficiently, which has attracted a great deal of attention and spurred new research. High precision and robustness have always been the goals of visual tracking. However, during the tracking process, target objects often encounter sophisticated scenarios such as fast motion and occlusion. During this period, erroneous tracking information will be generated and delivered to the next frame for updating; the information will seriously deteriorate the overall tracking model. To address the problem mentioned above, in this paper, we propose an accurate model self-adaptive update method based on a discriminative correlation filter framework. The proposed tracking method is achieved by utilizing the peak score of a response map generated by the discriminative correlation filter as a dynamic threshold with comparisons to its PSR (peak side-lobe ratio) scores, and then the comparative results are used as the differentiated condition for updating the translation filter and scale filter model. In addition, multiple hand-crafted features such as HOG (histogram of oriented gradient), CN (color names), and HOI (histogram of local intensities) are fused self-adaptively for comprehensive feature representation, which further improve tracking performance. We evaluate the performance of the proposed tracker on OTB benchmark datasets; the experimental results demonstrate that the proposed tracker performs favorably against most state-of-the-art discriminative correlation filter-based trackers including some methods follow deep learning paradigm, and the effectiveness of updating the model self-adaptive is verified.

A Sharper Generalization Bound for Divide-and-Conquer Ridge Regression

We study the distributed machine learning problem where the n feature-response pairs are partitioned among m machines uniformly at random. The goal is to approximately solve an empirical risk minimization (ERM) problem with the minimum amount of communication. The divide-and-conquer (DC) method, which was proposed several years ago, lets every worker machine independently solve the same ERM problem using its local feature-response pairs and the driver machine combine the solutions. This approach is in one-shot and thereby extremely communication-efficient. Although the DC method has been studied by many prior works, reasonable generalization bound has not been established before this work.For the ridge regression problem, we show that the prediction error of the DC method on unseen test samples is at most ε times larger than the optimal. There have been constantfactor bounds in the prior works, their sample complexities have a quadratic dependence on d, which does not match the setting of most real-world problems. In contrast, our bounds are much stronger. First, our 1 + ε error bound is much better than their constant-factor bounds. Second, our sample complexity is merely linear with d.

Tracking Network Evolution and Their Applications in Structural Network Analysis

Structural network analysis, including node ranking, community detection, and link prediction, has received a lot of attention lately. In the literature, most works focused on the structural analysis of a single network. In this paper, we are particularly interested in how the network structure evolves over time. For this, we propose a general framework to track, model, and predict the dynamic network structures. Unlike some recent works that directly tracks the adjacency matrices of the networks, our framework utilizes the spectral graph theory to track the latent feature vectors obtained by a low-rank eigendecomposition of the Laplacian matrices of the networks. We then use the Finite Impulse Response (FIR) filter to model the evolution of the latent feature vector of each node. By solving a ridge regression problem, the parameters of the FIR filter can be learned and used for predicting the future network structures, including node ranking, community detection, and link prediction. To test the effectiveness of our framework, we perform various experiments based on our synthetic datasets and three real-world datasets. Our experimental results show that our framework is very effective in tracking latent feature vectors and predicting future network structures.

Robust [formula omitted]Hypergraph and its applications

Hypergraph, an important learning tool to modulate high-order data correlations, has a wide range of applications in machine learning and computer vision. The key issue of the hypergraph-based applications is to construct an informative hypergraph, in which the hyperedges effectively represent the high-order data correlations. In practice, the real-world data is usually sampled from a union of non-linear manifolds. Due to the issues of noise and data corruptions, many data samples deviate from the underlying data manifolds. To construct an informative hypergraph that represents real-world data distribution well, we propose a hypergraph model (ℓ2-Hypergraph). Our model generates each hyperedge by solving an affine subspace ridge regression problem, where the samples with non-zero representation coefficients are used for hyperege generation. Specifically, to be robust to sparse noise and corruptions, a sparse constraint is imposed on data errors. We have conducted image clustering and classification experiments on real-world datasets. The experimental results demonstrate that our hypergraph model is superior to the existing hypergraph construction methods in both accuracy and robustness to sparse noise.

Learning Scale-Adaptive Tight Correlation Filter for Object Tracking.

In this paper, we propose a novel tracking method by formulating tracking as a correlation filtering as well as a ridge regression problem. First, we develop a tight correlation filter-based tracking framework from the signal detection perspective. In this formulation, the correlation filter is set as the same size as the target, which can make full use of the relations of the adjacent image patches and effectively exclude the influence of the background. Specifically, we point out that the novel correlation filter model can be regarded as the ridge regression model which takes into account the different importance of the samples and has the consistent objective with tracking. Second, we focus on the scale variation problem in tracking. By making use of the spatial structure of the correlation filter, the multiscale filter banks can be generated via interpolation to handle the scale estimation problem easily. Third, we present a novel distance importance-based confidence calculation model to determine the final tracking result, which not only makes use of the fine discriminability of the correlation filter but also takes the distance importance of the candidate samples into account to alleviate the impact of similar distractors. Experimental results demonstrate that our method is superior to several state-of-the-art trackers and many other correlation filter-based methods in the benchmark datasets.

The complexity of primal-dual fixed point methods for ridge regression

We study the ridge regression (L2 regularized least squares) problem and its dual, which is also a ridge regression problem. We observe that the optimality conditions describing the primal and dual optimal solutions can be formulated in several different but equivalent ways. The optimality conditions we identify form a linear system involving a structured matrix depending on a single relaxation parameter which we introduce for regularization purposes. This leads to the idea of studying and comparing, in theory and practice, the performance of the fixed point method applied to these reformulations. We compute the optimal relaxation parameters and uncover interesting connections between the complexity bounds of the variants of the fixed point scheme we consider. These connections follow from a close link between the spectral properties of the associated matrices. For instance, some reformulations involve purely imaginary eigenvalues; some involve real eigenvalues and others have all eigenvalues on the complex circle. We show that the deterministic Quartz method—which is a special case of the randomized dual coordinate ascent method with arbitrary sampling recently developed by Qu, Richtárik and Zhang—can be cast in our framework, and achieves the best rate in theory and in numerical experiments among the fixed point methods we study. Remarkably, the method achieves an accelerated convergence rate. Numerical experiments indicate that our main algorithm is competitive with the conjugate gradient method.

Robust kernelised correlation filter tracking with least absolute deviations regression

Kernelised correlation filter (KCF) has demonstrated its superior performance in visual tracking. The strength of the approach stems from its ability to efficiently learn the object appearance variations over time. A fundamental drawback to KCF, however, is that the objective function is modelled as a ridge regression problem that is sensitive to outliers, thereby resulting in suboptimal performance when the target appearance during tracking suffers from severe variations due to illumination variation, partial occlusion, background clutter, and so on. To address this issue, the KCF as a least absolute deviations regression (LADR) that is robust to outliers is reformulated, and then an efficient and effective approach to solve it via the alternating direction method of multipliers (ADMM) is presented. The ADMM splits the LADR into a ridge regression problem as the conventional KCF and a soft-thresholding one, and hence the proposed approach is still extremely efficient as the conventional KCF. Evaluations are conducted on the CVPR2013 tracking benchmark dataset with 50 sequences, which demonstrate the superior performance of the proposed method over the state-of-the-art trackers.

Stochastic L-BFGS: Improved Convergence Rates and Practical Acceleration Strategies

We revisit the stochastic limited-memory BFGS (L-BFGS) algorithm. By proposing a new framework for the convergence analysis, we prove improved convergence rates and computational complexities of the stochastic L-BFGS algorithms compared to previous works. In addition, we propose several practical acceleration strategies to speed up the empirical performance of such algorithms. We also provide theoretical analyses for most of the strategies. Experiments on large-scale logistic and ridge regression problems demonstrate that our proposed strategies yield significant improvements vis-\`a-vis competing state-of-the-art algorithms.

A new accelerated algorithm for ill-conditioned ridge regression problems

In this work, we study the primal and dual formulations of the regularized least squares problem, in special norm $$L_2$$ , named ridge regression. We state the Gradient method for the corresponding primal-dual problem through a fixed point approach. For this formulation, we present an original convergence analysis involving the spectral radius of a suitable iteration matrix. The main contribution of this work is to accelerate the studied method by means of a memory-like strategy. Some connections with the spectral properties of the associated iteration matrices were established to prove the convergence of the accelerated method. Preliminary experiments showed the good performance of the proposed algorithm when applied to solve ill-conditioned problems, providing better results than the conjugate gradient method for this class of problems.

Sketched ridge regression

We address the statistical and optimization impacts of the classical sketch and Hessian sketch used to approximately solve the Matrix Ridge Regression (MRR) problem. Prior research has quantified t...