Linear Regression Task Research Articles

Prediction of segmental motor outcomes in traumatic spinal cord injury: Advances beyond sum scores

Background and objectivesNeurological and functional recovery after traumatic spinal cord injury (SCI) is highly challenged by the level of the lesion and the high heterogeneity in severity (different degrees of in/complete SCI) and spinal cord syndromes (hemi-, ant-, central-, and posterior cord). So far outcome predictions in clinical trials are limited in targeting sum motor scores of the upper (UEMS) and lower limb (LEMS) while neglecting that the distribution of motor function is essential for functional outcomes. The development of data-driven prediction models of detailed segmental motor recovery for all spinal segments from the level of lesion towards the lowest motor segments will improve the design of rehabilitation programs and the sensitivity of clinical trials. MethodsThis study used acute-phase International Standards for Neurological Classification of SCI exams to forecast 6-month recovery of segmental motor scores as the primary evaluation endpoint. Secondary endpoints included severity grade improvement, independent walking, and self-care ability. Different similarity metrics were explored for k-nearest neighbor (kNN) matching within 1267 patients from the European Multicenter Study about Spinal Cord Injury before validation in 411 patients from the Sygen trial. The kNN performance was compared to linear and logistic regression models. ResultsWe obtained a population-wide root-mean-squared error (RMSE) in motor score sequence of 0.76(0.14, 2.77) and competitive functional score predictions (AUCwalker = 0.92, AUCself-carer = 0.83) for the kNN algorithm, improving beyond the linear regression task (RMSElinear = 0.98(0.22, 2.57)). The validation cohort showed comparable results (RMSE = 0.75(0.13, 2.57), AUCwalker = 0.92). We deploy the final historic control model as a web tool for easy user interaction (https://hicsci.ethz.ch/). DiscussionOur approach is the first to provide predictions across all motor segments independent of the level and severity of SCI. We provide a machine learning concept that is highly interpretable, i.e. the prediction formation process is transparent, that has been validated across European and American data sets, and provides reliable and validated algorithms to incorporate external control data to increase sensitivity and feasibility of multinational clinical trials.

Regression-Based Hyperparameter Learning for Support Vector Machines.

Unification of classification and regression is a major challenge in machine learning and has attracted increasing attentions from researchers. In this article, we present a new idea for this challenge, where we convert the classification problem into a regression problem, and then use the methods in regression to solve the problem in classification. To this end, we leverage the widely used maximum margin classification algorithm and its typical representative, support vector machine (SVM). More specifically, we convert SVM into a piecewise linear regression task and propose a regression-based SVM (RBSVM) hyperparameter learning algorithm, where regression methods are used to solve several key problems in classification, such as learning of hyperparameters, calculation of prediction probabilities, and measurement of model uncertainty. To analyze the uncertainty of the model, we propose a new concept of model entropy, where the leave-one-out prediction probability of each sample is converted into entropy, and then used to quantify the uncertainty of the model. The model entropy is different from the classification margin, in the sense that it considers the distribution of all samples, not just the support vectors. Therefore, it can assess the uncertainty of the model more accurately than the classification margin. In the case of the same classification margin, the farther the sample distribution is from the classification hyperplane, the lower the model entropy. Experiments show that our algorithm (RBSVM) provides higher prediction accuracy and lower model uncertainty, when compared with state-of-the-art algorithms, such as Bayesian hyperparameter search and gradient-based hyperparameter learning algorithms.

Hierarchical Aggregation for Numerical Data under Local Differential Privacy.

The proposal of local differential privacy solves the problem that the data collector must be trusted in centralized differential privacy models. The statistical analysis of numerical data under local differential privacy has been widely studied by many scholars. However, in real-world scenarios, numerical data from the same category but in different ranges frequently require different levels of privacy protection. We propose a hierarchical aggregation framework for numerical data under local differential privacy. In this framework, the privacy data in different ranges are assigned different privacy levels and then disturbed hierarchically and locally. After receiving users' data, the aggregator perturbs the privacy data again to convert the low-level data into high-level data to increase the privacy data at each privacy level so as to improve the accuracy of the statistical analysis. Through theoretical analysis, it was proved that this framework meets the requirements of local differential privacy and that its final mean estimation result is unbiased. The proposed framework is combined with mini-batch stochastic gradient descent to complete the linear regression task. Sufficient experiments both on synthetic datasets and real datasets show that the framework has a higher accuracy than the existing methods in both mean estimation and mini-batch stochastic gradient descent experiments.

Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks

We study the transfer learning process between two linear regression problems. An important and timely special case is when the regressors are overparameterized and perfectly interpolate their training data. We examine a parameter transfer mechanism whereby a subset of the parameters of the target task solution are constrained to the values learned for a related source task. We analytically characterize the generalization error of the target task in terms of the salient factors in the transfer learning architecture, i.e., the number of examples available, the number of (free) parameters in each of the tasks, the number of parameters transferred from the source to target task, and the correlation between the two tasks. Our non-asymptotic analysis shows that the generalization error of the target task follows a two-dimensional double descent trend (with respect to the number of free parameters in each of the tasks) that is controlled by the transfer learning factors. Our analysis points to specific cases where the transfer of parameters is beneficial.

CLR: Cloud Linear Regression Environment as a More Effective Resource-Task Scheduling Environment (State-of-the-Art)

The cloud paradigm has swiftly developed, and it is now well known as one of the emerging technologies that will have a significant influence on technology and society in the next few years. Cloud computing also has several benefits, including lower operating costs, server consolidation, flexible system setup, and elastic resource supply. However, there are still technological hurdles to overcome, particularly with real-time applications by providing resources. Resources allocation management most charming part of cloud computing; therefore, several authors have worked in the area of resource usage. This study introduces an innovative cloud machine learning framework-based linear regression approach called cloud linear regression (CLR), which entails both cloud technology and machine learning concept. CLR using machine learning yielded good prediction results for resource allocation management, as appeared with many researching, and still seek, research to raise optimal solutions to the resources' allocation problem as the aim of this study. This study discusses the relation between cloud resource allocation management and machine learning techniques by illustrating the role of linear regression methods, resource distribution, and task scheduling. The analytical analysis shows that the CLR promises to present an effective solution for resources (scheduling, provisioning, allocation, and availability).

Neural network learning using non-ideal resistive memory devices

We demonstrate a modified stochastic gradient (Tiki-Taka v2 or TTv2) algorithm for deep learning network training in a cross-bar array architecture based on ReRAM cells. There have been limited discussions on cross-bar arrays for training applications due to the challenges in the switching behavior of nonvolatile memory materials. TTv2 algorithm is known to overcome the device non-idealities for deep learning training. We demonstrate the feasibility of the algorithm for a linear regression task using 1R and 1T1R ReRAM devices. Using the measured device properties, we project the performance of a long short-term memory (LSTM) network with 78 K parameters. We show that TTv2 algorithm relaxes the criteria for symmetric device update response. In addition, further optimization of the algorithm increases noise robustness and significantly reduces the required number of states, thereby drastically improving the model accuracy even with non-ideal devices and achieving the test error close to that of the conventional learning algorithm with an ideal device.

Coded Distributed Computing With Partial Recovery

Coded computation techniques provide robustness against <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">straggling</i> workers in distributed computing. However, most of the existing schemes require exact provisioning of the straggling behavior and ignore the computations carried out by straggling workers. Moreover, these schemes are typically designed to recover the desired computation results accurately, while in many machine learning and iterative optimization algorithms, faster approximate solutions are known to result in an improvement in the overall convergence time. In this paper, we first introduce a novel coded matrix-vector multiplication scheme, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">coded computation with partial recovery (CCPR)</i> , which benefits from the advantages of both coded and uncoded computation schemes, and reduces both the computation time and the decoding complexity by allowing a trade-off between the accuracy and the speed of computation. We then extend this approach to distributed implementation of more general computation tasks by proposing a coded communication scheme with partial recovery, where the results of subtasks computed by the workers are coded before being communicated. Numerical simulations on a large linear regression task confirm the benefits of the proposed scheme in terms of the trade-off between the computation accuracy and latency.

Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning

In this article, we propose a communication-efficient decentralized machine learning (ML) algorithm, coined quantized group ADMM (Q-GADMM). To reduce the number of communication links, every worker in Q-GADMM communicates only with two neighbors, while updating its model via the group alternating direction method of multipliers (GADMM). Moreover, each worker transmits the quantized difference between its current model and its previously quantized model, thereby decreasing the communication payload size. However, due to the lack of centralized entity in decentralized ML, the spatial sparsity and payload compression may incur error propagation, hindering model training convergence. To overcome this, we develop a novel stochastic quantization method to adaptively adjust model quantization levels and their probabilities, while proving the convergence of Q-GADMM for convex objective functions. Furthermore, to demonstrate the feasibility of Q-GADMM for non-convex and stochastic problems, we propose quantized stochastic GADMM (Q-SGADMM) that incorporates deep neural network architectures and stochastic sampling. Simulation results corroborate that Q-GADMM significantly outperforms GADMM in terms of communication efficiency while achieving the same accuracy and convergence speed for a linear regression task. Similarly, for an image classification task using DNN, Q-SGADMM achieves significantly less total communication cost with identical accuracy and convergence speed compared to its counterpart without quantization, i.e., stochastic GADMM (SGADMM).

An Algebraic-Geometric Approach for Linear Regression Without Correspondences

Linear regression without correspondences is the problem of performing a linear regression fit to a dataset for which the correspondences between the independent samples and the observations are unknown. Such a problem naturally arises in diverse domains such as computer vision, data mining, communications and biology. In its simplest form, it is tantamount to solving a linear system of equations, for which the entries of the right hand side vector have been permuted. This type of data corruption renders the linear regression task considerably harder, even in the absence of other corruptions, such as noise, outliers or missing entries. Existing methods are either applicable only to noiseless data or they are very sensitive to initialization or they work only for partially shuffled data. In this paper we address these issues via an algebraic geometric approach, which uses symmetric polynomials to extract permutation-invariant constraints that the parameters $\xi ^{*} \in \mathbb {R} ^{\text {n}}$ of the linear regression model must satisfy. This naturally leads to a polynomial system of n equations in n unknowns, which contains $\xi ^{*}$ in its root locus. Using the machinery of algebraic geometry we prove that as long as the independent samples are generic, this polynomial system is always consistent with at most n ! complex roots, regardless of any type of corruption inflicted on the observations. The algorithmic implication of this fact is that one can always solve this polynomial system and use its most suitable root as initialization to the Expectation Maximization algorithm. To the best of our knowledge, the resulting method is the first working solution for small values of n able to handle thousands of fully shuffled noisy observations in milliseconds.

A bio-inspired quaternion local phase CNN layer with contrast invariance and linear sensitivity to rotation angles

Deep learning models have been particularly successful with image recognition using Convolutional Neural Networks (CNN). However, the learning of a contrast invariance and rotation equivariance response may fail even with very deep CNNs or by large data augmentations in training.We were inspired by the V1 visual features of the mammalian visual system to emulate as much as possible the early visual system and add more invariant capacities to the CNN. We present a new quaternion local phase convolutional neural network layer encoding three local phases. We present two experimental setups: An image classification task with four contrast levels, and a linear regression task that predicts the rotation angle of an image. In sum, we obtain new patterns and feature representations for deep learning, which capture illumination invariance and a linear response to rotation angles.

Analysis of Approximate Message Passing With Non-Separable Denoisers and Markov Random Field Priors

Approximate message passing (AMP) is a class of low-complexity, scalable algorithms for solving high-dimensional linear regression tasks where one wishes to recover an unknown signal from noisy, linear measurements. AMP is an iterative algorithm that performs estimation by updating an estimate of the unknown signal at each iteration and the performance of AMP (quantified, for example, by the mean squared error of its estimates) depends on the choice of a “denoiser” function that is used to produce these signal estimates at each iteration. An attractive feature of AMP is that its performance can be tracked by a scalar recursion referred to as state evolution. Previous theoretical analysis of the accuracy of the state evolution predictions has been limited to the use of only separable denoisers or block-separable denoisers, a class of denoisers that underperform when sophisticated dependencies exist between signal entries. Since signals with entrywise dependencies are common in image/video-processing applications, in this work we study the high-dimensional linear regression task when the dependence structure of the input signal is modeled by a Markov random field prior distribution. We provide a rigorous analysis of the performance of AMP, demonstrating the accuracy of the state evolution predictions, when a class of non-separable sliding-window denoisers is applied. Moreover, we provide numerical examples where AMP with sliding-window denoisers can successfully capture local dependencies in images.

Conditional Linear Regression

Previous work in machine learning and statistics commonly focuses on building models that capture the vast majority of data, possibly ignoring a segment of the population as outliers. By contrast, we may be interested in finding a segment of the population for which we can find a linear rule capable of achieving more accurate predictions. We give an efficient algorithm for the conditional linear regression task, which is the joint task of identifying a significant segment of the population, described by a k-DNF, along with its linear regression fit.

A tutorial on Bayesian Normal linear regression

Regression is a common task in metrology and often applied to calibrate instruments, evaluate inter-laboratory comparisons or determine fundamental constants, for example. Yet, a regression model cannot be uniquely formulated as a measurement function, and consequently the Guide to the Expression of Uncertainty in Measurement (GUM) and its supplements are not applicable directly. Bayesian inference, however, is well suited to regression tasks, and has the advantage of accounting for additional a priori information, which typically robustifies analyses. Furthermore, it is anticipated that future revisions of the GUM shall also embrace the Bayesian view.Guidance on Bayesian inference for regression tasks is largely lacking in metrology. For linear regression models with Gaussian measurement errors this tutorial gives explicit guidance. Divided into three steps, the tutorial first illustrates how a priori knowledge, which is available from previous experiments, can be translated into prior distributions from a specific class. These prior distributions have the advantage of yielding analytical, closed form results, thus avoiding the need to apply numerical methods such as Markov Chain Monte Carlo. Secondly, formulas for the posterior results are given, explained and illustrated, and software implementations are provided. In the third step, Bayesian tools are used to assess the assumptions behind the suggested approach.These three steps (prior elicitation, posterior calculation, and robustness to prior uncertainty and model adequacy) are critical to Bayesian inference. The general guidance given here for Normal linear regression tasks is accompanied by a simple, but real-world, metrological example. The calibration of a flow device serves as a running example and illustrates the three steps. It is shown that prior knowledge from previous calibrations of the same sonic nozzle enables robust predictions even for extrapolations.

Collective Support Recovery for Multi-Design Multi-Response Linear Regression

The multi-design multi-response linear regression problem is investigated, in which design matrices are Gaussian with covariance matrices Σ (1:K) = (Σ (1) , ... , 1 (K) ) for K linear regression tasks. Design matrices across tasks are assumed to be independent. The support union of K p-dimensional regression vectors (collected as columns of matrix B*) is recovered using l 1 /l 2 -regularized lasso. Sufficient and necessary conditions on sample complexity are characterized as a sharp threshold to guarantee successful recovery of the support union. This model has been previously studied via l 1 /l ∞ -regularized lassoand via l 1 /l 1 + l 1 /l ∞ -regularized lasso, in which sharp threshold on sample complexity is characterized only for K = 2 and under special conditions. In this paper, using l 1 /l 2 -regularized lasso, sharp threshold on sample complexity is characterized under standard regularization conditions. Namely, if n > c p1 ψ(B*, Σ (1:K) ) log(p - s) where c p1 is a constant, and s is the size of the support set, then l 1 /l 2 -regularized lasso correctly recovers the support union; and if n <; c p2 ψ(B*, Σ (1:K) )log(p - s) where c p2 is a constant, then l 1 /l 2 -regularized lasso fails to recover the support union. In particular, the function ψ(B*, Σ (1:K) ) captures the impact of the sparsity of K regression vectors and the statistical properties of the design matrices on the threshold on sample complexity. Therefore, such threshold function also demonstrates the advantages of joint support union recovery using multitask lasso over individual support recovery using single-task lasso.

Distance algorithm based procedure for non‐negative least squares

In chemistry and many other scientific disciplines, non‐negativity‐constrained estimation of models is of practical importance. The time required for estimating true least squares non‐negativity‐constrained models is typically many times longer than that for estimating unconstrained models. That is why it is necessary to find faster and faster non‐negative least squares (NNLS) algorithms. Very recently, the distance algorithm has been developed, and this algorithm can be adapted to solve NNLS regression task faster (in some cases) than the conventional algorithms. Based on some simulated investigation, DA_NNLS was the fastest for small‐sized and medium‐sized linear regression tasks. The visualization (geometry) of the NNLS task being solved by our new algorithm is discussed as well. Besides linear algebra, convex geometrical concepts and tools are suggested to investigate, to use, and to develop in chemometrics for exploiting the geometry of chemometry. Copyright © 2014 John Wiley &amp; Sons, Ltd.

Robust physics-based analysis of thermal and visual imagery

Extensions to an existing physics-based approach for intersensory perception in which thermal and visual imagery of outdoor scenes is analyzed simultaneously for object recognition are discussed. The existing approach uses a model that is based on the principle of the conservation of energy at the surface of the imaged object. The model permits the computation of physically meaningful features that may be used for object classification. Two significant extensions are discussed. First, the model is used to analyze a temporal sequence of spatially registered thermal and visual imagery. Second, the energy-exchange model is used to formulate a linear-regression task in which the physical properties of the imaged object are the unknown parameters that are estimated. A statistically robust scheme is presented for this task. The robust technique minimizes sensitivity to outliers caused by segmentation errors and misregistration, which are endemic to multisensor fusion. Thus reliable physics-based features are made available by this approach.