Large-scale Regression Problems Research Articles

New Efficient Approach to Solve Big Data Systems Using Parallel Gauss–Seidel Algorithms

In order to perform big-data analytics, regression involving large matrices is often necessary. In particular, large scale regression problems are encountered when one wishes to extract semantic patterns for knowledge discovery and data mining. When a large matrix can be processed in its factorized form, advantages arise in terms of computation, implementation, and data-compression. In this work, we propose two new parallel iterative algorithms as extensions of the Gauss–Seidel algorithm (GSA) to solve regression problems involving many variables. The convergence study in terms of error-bounds of the proposed iterative algorithms is also performed, and the required computation resources, namely time- and memory-complexities, are evaluated to benchmark the efficiency of the proposed new algorithms. Finally, the numerical results from both Monte Carlo simulations and real-world datasets are presented to demonstrate the striking effectiveness of our proposed new methods.

StereoSpike: Depth Learning With a Spiking Neural Network

Depth estimation is an important computer vision task, useful in particular for navigation in autonomous vehicles, or for object manipulation in robotics. Here, we propose to solve it using <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">StereoSpike</i> , an end-to-end neuromorphic approach, combining two event-based cameras and a Spiking Neural Network (SNN) with a modified U-Net-like encoder-decoder architecture. More specifically, we used the Multi Vehicle Stereo Event Camera Dataset (MVSEC). It provides a depth ground-truth, which was used to train StereoSpike in a supervised manner, using surrogate gradient descent. We propose a novel readout paradigm to obtain a dense analog prediction –the depth of each pixel– from the spikes of the decoder. We demonstrate that this architecture generalizes very well, even better than its non-spiking counterparts, leading to near state-of-the-art test accuracy. To the best of our knowledge, it is the first time that such a large-scale regression problem is solved by a fully spiking neural network. Finally, we show that very low firing rates (<5%) can be obtained via regularization, with a minimal cost in accuracy. This means that StereoSpike could be efficiently implemented on neuromorphic chips, opening the door for low power and real time embedded systems.

Tree-aggregated predictive modeling of microbiome data

Modern high-throughput sequencing technologies provide low-cost microbiome survey data across all habitats of life at unprecedented scale. At the most granular level, the primary data consist of sparse counts of amplicon sequence variants or operational taxonomic units that are associated with taxonomic and phylogenetic group information. In this contribution, we leverage the hierarchical structure of amplicon data and propose a data-driven and scalable tree-guided aggregation framework to associate microbial subcompositions with response variables of interest. The excess number of zero or low count measurements at the read level forces traditional microbiome data analysis workflows to remove rare sequencing variants or group them by a fixed taxonomic rank, such as genus or phylum, or by phylogenetic similarity. By contrast, our framework, which we call trac (tree-aggregation of compositional data), learns data-adaptive taxon aggregation levels for predictive modeling, greatly reducing the need for user-defined aggregation in preprocessing while simultaneously integrating seamlessly into the compositional data analysis framework. We illustrate the versatility of our framework in the context of large-scale regression problems in human gut, soil, and marine microbial ecosystems. We posit that the inferred aggregation levels provide highly interpretable taxon groupings that can help microbiome researchers gain insights into the structure and functioning of the underlying ecosystem of interest.

Efficient and Scalable Multi-Task Regression on Massive Number of Tasks

Many real-world large-scale regression problems can be formulated as Multi-task Learning (MTL) problems with a massive number of tasks, as in retail and transportation domains. However, existing MTL methods still fail to offer both the generalization performance and the scalability for such problems. Scaling up MTL methods to problems with a tremendous number of tasks is a big challenge. Here, we propose a novel algorithm, named Convex Clustering Multi-Task regression Learning (CCMTL), which integrates with convex clustering on the k-nearest neighbor graph of the prediction models. Further, CCMTL efficiently solves the underlying convex problem with a newly proposed optimization method. CCMTL is accurate, efficient to train, and empirically scales linearly in the number of tasks. On both synthetic and real-world datasets, the proposed CCMTL outperforms seven state-of-the-art (SoA) multi-task learning methods in terms of prediction accuracy as well as computational efficiency. On a real-world retail dataset with 23,812 tasks, CCMTL requires only around 30 seconds to train on a single thread, while the SoA methods need up to hours or even days.

Sparse network estimation for dynamical spatio-temporal array models

Neural field models represent neuronal communication on a population level via synaptic weight functions. Using voltage sensitive dye (VSD) imaging it is possible to obtain measurements of neural fields with a relatively high spatial and temporal resolution. The synaptic weight functions represent functional connectivity in the brain and give rise to a spatio-temporal dependence structure. We present a stochastic functional differential equation for modeling neural fields, which leads to a vector autoregressive model of the data via basis expansions of the synaptic weight functions and time and space discretization. Fitting the model to data is a practical challenge as this represents a large scale regression problem. By using a 1-norm penalty in combination with localized basis functions it is possible to learn a sparse network representation of the functional connectivity of the brain, but still, the explicit construction of a design matrix can be computationally prohibitive. We demonstrate that by using tensor product basis expansions, the computation of the penalized estimator via a proximal gradient algorithm becomes feasible. It is crucial for the computations that the data is organized in an array as is the case for the three dimensional VSD imaging data. This allows for the use of array arithmetic that is both memory and time efficient. Theproposed method is implemented and showcased in the R package dynamo available from CRAN.

Large-Scale Regression: A Partition Analysis of the Least Squares Multisplitting

Indirect measurements of physical parameters of interest require a mathematical model in which these parameters are estimated from the gathered measurements. Within the least squares (LS) estimation, the parameters are estimated through a regression problem. The presence of dynamics, multiple sensors, and high sampling rates leads to high-dimensional regression matrices. This paper deals with solving such large-scale regression problems time efficiently. We revisit Renaut’s least squares multisplitting (LSMS) technique aimed at solving the ordinary LS problem in parallel. The LSMS decomposes the design matrix column-wise into several blocks. The global LS solution is subsequently replaced by an equivalent set of local LS problems that are to be solved in parallel. We study how the user should configure the partition of the multisplitting. We propose a partition design based on a clustering analysis and prove the consistency of this approach. The method is illustrated with dedicated numerical simulations for a highly scalable LS-based problem within engineering: frequency response function (FRF) estimation in the presence of missing output samples. Finally, its practical utility is shown with a laboratory measurement application.

Hypergraph Learning and Reweighted $\ell _1$-Norm Minimization for Hyperspectral Unmixing

Hyperspectral image unmixing techniques are developing to tackle the problem of mixed pixels caused by low spatial resolutions. In sparse unmixing, the redundant spectral library of materials is provided beforehand as a priori information to find the optimal representation by sparse linear regression. In order to improve the estimation of abundance distributions, the spatial correlation is taken into account and hypergraph learning is introduced to make full use of the underlying spatial-contextual information. Specifically, we find $K$ -nearest pixels of each pixel in spectral domains from a local region and construct a hypergraph to exploit the fact that spatial neighboring pixels have a high probability of sharing similar spectral information. Furthermore, a reweighted $\ell _1$ -norm minimization scheme is adopted instead to enhance the sparsity of estimated fractional abundances. The complicated large-scale regression problem is decomposed into subproblems to obtain the optimal solution within the framework of alternating direction method of multipliers. Experimental results on both synthetic and real-world datasets demonstrate the effectiveness of the proposed algorithm.

Approximate large-scale Bayesian spatial modeling with application to quantitative magnetic resonance imaging

We consider the Bayesian inference of nonlinear, large-scale regression problems in which the parameters model the spatial distribution of some property. A homoscedastic Gaussian sampling distribution is supposed as well as certain assumptions about the regression function. Propriety of the posterior and the existence of its moments are explored when using improper prior distributions expressing different levels of prior knowledge, ranging from a purely noninformative prior over intrinsic Gaussian Markov random field priors to a partition prior. The considered class of problems includes magnetic resonance fingerprinting (MRF). We apply an approximate Bayesian inference to this particular application and demonstrate its practicability in dimensions up to $$10^5$$ or larger. The benefit of incorporating substantial prior knowledge is illustrated. By analyzing simulated realistic MRF data, it is shown that MAP estimates can significantly improve the results achieved with maximum likelihood estimation.

A Highly Efficient Semismooth Newton Augmented Lagrangian Method for Solving Lasso Problems

We develop a fast and robust algorithm for solving large scale convex composite optimization models with an emphasis on the $\ell_1$-regularized least squares regression (Lasso) problems. Despite the fact that there exist a large number of solvers in the literature for the Lasso problems, we found that no solver can efficiently handle difficult large scale regression problems with real data. By leveraging on available error bound results to realize the asymptotic superlinear convergence property of the augmented Lagrangian algorithm, and by exploiting the second order sparsity of the problem through the semismooth Newton method, we are able to propose an algorithm, called {\sc Ssnal}, to efficiently solve the aforementioned difficult problems. Under very mild conditions, which hold automatically for Lasso problems, both the primal and the dual iteration sequences generated by {\sc Ssnal} possess a fast linear convergence rate, which can even be superlinear asymptotically. Numerical comparisons between our approach and a number of state-of-the-art solvers, on real data sets, are presented to demonstrate the high efficiency and robustness of our proposed algorithm in solving difficult large scale Lasso problems.

METSK-HDe: A multiobjective evolutionary algorithm to learn accurate TSK-fuzzy systems in high-dimensional and large-scale regression problems

In this contribution, we propose a two-stage method for Accurate Fuzzy Modeling in High-Dimensional Regression Problems using Approximate Takagi–Sugeno–Kang Fuzzy Rule-Based Systems. In the first stage, an evolutionary data base learning is performed (involving variables, granularities and slight fuzzy partition displacements) together with an inductive rule base learning within the same process. The second stage is a post-processing process to perform a rule selection and a scatter-based tuning of the membership functions for further refinement of the learned solutions. Moreover, the second stage incorporates an efficient Kalman filter to learn the coefficients of the consequent polynomial function in the Takagi–Sugeno–Kang rules. Both stages include mechanisms that significantly improve the accuracy of the model and ensure a fast convergence in high-dimensional and large-scale regression datasets.We tested our approach on 28 real-world datasets with different numbers of variables and instances. Five well-known methods have been executed as references. We compared the different approaches by applying non-parametric statistical tests for pair-wise and multiple comparisons. The results confirm the effectiveness of the proposed method, showing better results in accuracy within a reasonable computing time.

Communication-efficient algorithms for statistical optimization

We analyze two communication-efficient algorithms for distributed optimization in statistical settings involving large-scale data sets. The first algorithm is a standard averaging method that distributes the N data samples evenly to m machines, performs separate minimization on each subset, and then averages the estimates. We provide a sharp analysis of this average mixture algorithm, showing that under a reasonable set of conditions, the combined parameter achieves mean-squared error (MSE) that decays as O(N-1 +(N/m)-2). Whenever m ≤ √N, this guarantee matches the best possible rate achievable by a centralized algorithm having access to all N samples. The second algorithm is a novel method, based on an appropriate form of bootstrap subsampling. Requiring only a single round of communication, it has mean-squared error that decays as O(N-1 + (N/m)-3), and so is more robust to the amount of parallelization. In addition, we show that a stochastic gradient-based method attains mean-squared error decaying as O(N-1 + (N/m)-3/2), easing computation at the expense of a potentially slower MSE rate. We also provide an experimental evaluation of our methods, investigating their performance both on simulated data and on a large-scale regression problem from the internet search domain. In particular, we show that our methods can be used to efficiently solve an advertisement prediction problem from the Chinese SoSo Search Engine, which involves logistic regression with N ≈ 2.4×108 samples and d ≈ 740,000 covariates.

Bayesian variable selection regression for genome-wide association studies and other large-scale problems

We consider applying Bayesian Variable Selection Regression, or BVSR, to genome-wide association studies and similar large-scale regression problems. Currently, typical genome-wide association studies measure hundreds of thousands, or millions, of genetic variants (SNPs), in thousands or tens of thousands of individuals, and attempt to identify regions harboring SNPs that affect some phenotype or outcome of interest. This goal can naturally be cast as a variable selection regression problem, with the SNPs as the covariates in the regression. Characteristic features of genome-wide association studies include the following: (i) a focus primarily on identifying relevant variables, rather than on prediction; and (ii) many relevant covariates may have tiny effects, making it effectively impossible to confidently identify the complete "correct" subset of variables. Taken together, these factors put a premium on having interpretable measures of confidence for individual covariates being included in the model, which we argue is a strength of BVSR compared with alternatives such as penalized regression methods. Here we focus primarily on analysis of quantitative phenotypes, and on appropriate prior specification for BVSR in this setting, emphasizing the idea of considering what the priors imply about the total proportion of variance in outcome explained by relevant covariates. We also emphasize the potential for BVSR to estimate this proportion of variance explained, and hence shed light on the issue of "missing heritability" in genome-wide association studies.

Fixed-size Least Squares Support Vector Machines: A Large Scale Application in Electrical Load Forecasting

Based on the Nystrom approximation and the primal-dual formulation of the least squares support vector machines, it becomes possible to apply a nonlinear model to a large scale regression problem. This is done by using a sparse approximation of the nonlinear mapping induced by the kernel matrix, with an active selection of support vectors based on quadratic Renyi entropy criteria. The methodology is applied to the case of load forecasting as an example of a real-life large scale problem in industry. The forecasting performance, over ten different load series, shows satisfactory results when the sparse representation is built with less than 3% of the available sample.

10.1162/15324430152733142

Support Vector Machines (SVMs) for regression problems are trained by solving a quadratic optimization problem which needs on the order of l square memory and time resources to solve, where l is the number of training examples. In this paper, we propose a decomposition algorithm, SVMTorch (available at http://www.idiap.ch/learning/SVMTorch.html ), which is similar to SVM-Light proposed by Joachims (1999) for classification problems, but adapted to regression problems. With this algorithm, one can now efficiently solve large-scale regression problems (more than 20000 examples). Comparisons with Nodelib, another publicly available SVM algorithm for large-scale regression problems from Flake and Lawrence (2000) yielded significant time improvements. Finally, based on a recent paper from Lin (2000), we show that a convergence proof exists for our algorithm.