Simple Regression Problem Research Articles

A physics-aware learning architecture with input transfer networks for predictive modeling

Hybrid modeling architectures seek to combine a machine learning model with a computationally efficient (simplified or partial) physics model to predict the behavior of physical systems. Existing sequential or parallel approaches to hybrid modeling do not typically exploit the potential relationship between the input or latent features of the partial and full physics. In addition, very few existing architectures take advantage of the provision to generously or on-demand sample the partial physics model. To address these gaps, we have developed a novel neural network-based hybrid architecture called “Opportunistic Physics-mining Transfer Mapping Architecture” or OPTMA. The goal of the OPTMA architecture is to facilitate greater exploitation of input space correlations between partial and full physics where they exist. To this end, a transfer neural network is used to transform the original inputs into modified inputs or latent features, where the partial physics operates on these artificially transformed features to produce the final prediction. An extended back-propagation approach and a Particle Swarm Optimization (to deal with multimodal loss functions) are used to train the network weights. The new architecture is first tested on a simple regression problem for analysis. It is then used to predict the behavior of more complex dynamic systems — an Inverted pendulum and the motion of an unmanned aerial vehicle, both under wind effects. Subsequent tests on unseen samples demonstrate OPTMA’s competitive performance compared to pure ANN and sequential hybrid models and provide empirical validation of the transfer concepts underlying OPTMA.

Assessing Bayes Factor Surfaces Using Interactive Visualization and Computer Surrogate Modeling

Bayesian model selection provides a natural alternative to classical hypothesis testing based on p-values. While many articles mention that Bayesian model selection can be sensitive to prior specification on parameters, there are few practical strategies to assess and report this sensitivity. This article has two goals. First, we aim to educate the broader statistical community about the extent of potential sensitivity through visualization of the Bayes factor surface. The Bayes factor surface shows the value a Bayes factor takes as a function of user-specified hyperparameters. Second, we suggest surrogate modeling via Gaussian processes to visualize the Bayes factor surface in situations where computation is expensive. We provide three examples including an interactive R shiny application that explores a simple regression problem, a hierarchical linear model selection exercise, and finally surrogate modeling via Gaussian processes to a study of the influence of outliers in empirical finance. We suggest Bayes factor surfaces are valuable for scientific reporting since they (i) increase transparency by making instability in Bayes factors easy to visualize, (ii) generalize to simple and complicated examples, and (iii) provide a path for researchers to assess the impact of prior choice on modeling decisions in a wide variety of research areas. Supplementary materials for this article are available online.

A Limitation of Gradient Descent Learning

Over decades, gradient descent has been applied to develop learning algorithm to train a neural network (NN). In this brief, a limitation of applying such algorithm to train an NN with persistent weight noise is revealed. Let V(w) be the performance measure of an ideal NN. V(w) is applied to develop the gradient descent learning (GDL). With weight noise, the desired performance measure (denoted as J(w) ) is E[V(~w)|w] , where ~w is the noisy weight vector. Applying GDL to train an NN with weight noise, the actual learning objective is clearly not V(w) but another scalar function L(w) . For decades, there is a misconception that L(w) = J(w) , and hence, the actual model attained by the GDL is the desired model. However, we show that it might not: 1) with persistent additive weight noise, the actual model attained is the desired model as L(w) = J(w) ; and 2) with persistent multiplicative weight noise, the actual model attained is unlikely the desired model as L(w) ≠ J(w) . Accordingly, the properties of the models attained as compared with the desired models are analyzed and the learning curves are sketched. Simulation results on 1) a simple regression problem and 2) the MNIST handwritten digit recognition are presented to support our claims.

A tutorial on Gaussian process regression: Modelling, exploring, and exploiting functions

This tutorial introduces the reader to Gaussian process regression as an expressive tool to model, actively explore and exploit unknown functions. Gaussian process regression is a powerful, non-parametric Bayesian approach towards regression problems that can be utilized in exploration and exploitation scenarios. This tutorial aims to provide an accessible introduction to these techniques. We will introduce Gaussian processes which generate distributions over functions used for Bayesian non-parametric regression, and demonstrate their use in applications and didactic examples including simple regression problems, a demonstration of kernel-encoded prior assumptions and compositions, a pure exploration scenario within an optimal design framework, and a bandit-like exploration–exploitation scenario where the goal is to recommend movies. Beyond that, we describe a situation modelling risk-averse exploration in which an additional constraint (not to sample below a certain threshold) needs to be accounted for. Lastly, we summarize recent psychological experiments utilizing Gaussian processes. Software and literature pointers are also provided.

Multivariate Ensemble Sensitivity with Localization

Abstract Ensemble sensitivities have proven a useful alternative to adjoint sensitivities for large-scale dynamics, but their performance in multiscale flows has not been thoroughly examined. When computing sensitivities, the analysis covariance is usually approximated with the corresponding diagonal matrix, leading to a simple univariate regression problem rather than a more general multivariate regression problem. Sensitivity estimates are affected by sampling error arising from a finite ensemble and can lead to an overestimated response to an analysis perturbation. When forecasts depend on many details of an analysis, it is reasonable to expect that the diagonal approximation is too severe. Because spurious covariances are more likely when correlations are weak, computing the sensitivity with a multivariate regression that retains the full analysis covariance may increase the need for sampling error mitigation. The purpose of this work is to clarify the effects of the diagonal approximation, and investigate the need for mitigating spurious covariances arising from sampling error. A two-scale model with realistic spatial covariances is the basis for experimentation. For most problems, an efficient matrix inversion is possible by finding a minimum-norm solution, and employing appropriate matrix factorization. A published hierarchical approach for estimating regression factors for tapering (localizing) covariances is used to measure the effects of sampling error. Compared to univariate regressions in the diagonal approximation, skill in predicting a nonlinear response from the linear sensitivities is superior when localized multivariate sensitivities are used, particularly when fast scales are present, model error is present, and the observing network is sparse.

Bayesian feature selection for high-dimensional linear regression via the Ising approximation with applications to genomics

Feature selection, identifying a subset of variables that are relevant for predicting a response, is an important and challenging component of many methods in statistics and machine learning. Feature selection is especially difficult and computationally intensive when the number of variables approaches or exceeds the number of samples, as is often the case for many genomic datasets. Here, we introduce a new approach--the Bayesian Ising Approximation (BIA)-to rapidly calculate posterior probabilities for feature relevance in L2 penalized linear regression. In the regime where the regression problem is strongly regularized by the prior, we show that computing the marginal posterior probabilities for features is equivalent to computing the magnetizations of an Ising model with weak couplings. Using a mean field approximation, we show it is possible to rapidly compute the feature selection path described by the posterior probabilities as a function of the L2 penalty. We present simulations and analytical results illustrating the accuracy of the BIA on some simple regression problems. Finally, we demonstrate the applicability of the BIA to high-dimensional regression by analyzing a gene expression dataset with nearly 30 000 features. These results also highlight the impact of correlations between features on Bayesian feature selection. An implementation of the BIA in C++, along with data for reproducing our gene expression analyses, are freely available at http://physics.bu.edu/∼pankajm/BIACode.

Gaussian-Process-Based Real-Time Ground Segmentation for Autonomous Land Vehicles

Ground segmentation is a key component for Autonomous Land Vehicle (ALV) navigation in an outdoor environment. This paper presents a novel algorithm for real-time segmenting three-dimensional scans of various terrains. An individual terrain scan is represented as a circular polar grid map that is divided into a number of segments. A one-dimensional Gaussian Process (GP) regression with a non-stationary covariance function is used to distinguish the ground points or obstacles in each segment. The proposed approach splits a large-scale ground segmentation problem into many simple GP regression problems with lower complexity, and can then get a real-time performance while yielding acceptable ground segmentation results. In order to verify the effectiveness of our approach, experiments have been carried out both on a public dataset and the data collected by our own ALV in different outdoor scenes. Our approach has been compared with two previous ground segmentation techniques. The results show that our approach can get a better trade-off between computational time and accuracy. Thus, it can lead to successive object classification and local path planning in real time. Our approach has been successfully applied to our ALV, which won the championship in the 2011 Chinese Future Challenge in the city of Ordos.

Investigating the performance of different estimation techniques for crop yield data analysis in crop insurance applications

AbstractWe investigate the performance of the ordinary least squares (OLS)‐, M‐, MM‐, and the Theil–Sen (TS)‐estimator for crop yield data analysis in crop insurance applications using Monte Carlo simulations. More specifically, the performance is assessed with respect to trend estimation, prediction of future yield levels, and the estimation of expected indemnity payments. In agreement with earlier findings, other estimators are found to be superior to OLS in simple regression problems if yield distributions are outlier contaminated and heteroscedastic. While this conclusion is also valid for subsequent applications such as yield prediction and the estimation of expected indemnity payments, the difference between the considered estimators becomes less distinct. For these applications, we find particularly the M‐estimator to be a good compromise between high‐breakdown (very robust) estimators and the very efficient OLS‐estimator. Because no regression technique dominates all others in all applications and scenarios for error term distributions, our results underline that the choice of the estimation technique should be dependent on the purpose of the crop yield data analysis. However, alternative estimators such as M‐, MM‐, and TS‐estimator can reduce (and bound) the risk of unreliable or inefficient crop yield data analysis in crop insurance applications.

A New Method for Simplifying Algebraic Expressions in Genetic Programming Called Equivalent Decision Simplification

Symbolic Regression is one of the most important applications of Genetic Programming, but suffers from one of the key issues in Genetic Programming, bloat. For a variety of reasons, reliable techniques to remove bloat are highly desirable. This paper introduces a novel approach of removing bloat, Equivalent Decision Simplification, in which subtrees are evaluated over the set of regression points. The effectiveness of the proposed method is confirmed by computer simulation taking simple Symbolic Regression problems as examples.

Orthogonal Polynomials and Regression-Based Symmetric Derivatives

We demonstrate how certain types of symmetric derivatives originate from a simple least-squares regression problem involving discrete Chebyshev polynomials. As the number of data points used in this regression tends to infinity, the resulting integrals, which involve Legendre polynomials, lead to Lanczos derivatives, a result that demonstrates how this latter entity is merely a continuous version of the symmetric derivative.

A Version of Quadratic Regression with Interpretable Parameters

The quadratic regression model is popular and effective in describing a wide variety of data, but it is based on a function whose parameters are not easy to interpret. We suggest an alternative form of the quadratic model that has the same expectation function, but also has the useful feature that its parameters are interpretable. Examples are provided of a simple regression problem and also of a nonlinear mixed-effects model. The models can be estimated with available software.

ON PHILLIPS–PERRON-TYPE TESTS FOR SEASONAL UNIT ROOTS

In this paper we consider a semiparametric version of the test for seasonal unit roots suggested by Hylleberg, Engle, Granger, and Yoo (1990, Journal of Econometrics 44, 215–238). The asymptotic theory is based on the analysis of a simple regression problem, and the results apply to tests at any given frequency in the range (0,π]. Monte Carlo simulations suggest that the test may have more power than the parametric test of Hylleberg et al. (1990). On the other hand, the semiparametric version suffers from severe size distortions in some situations.

Uncertainty and the Assessment of Extinction Probabilities

A proper assessment of the probability of early collapse or extinction of a population requires consideration of our uncertainty about crucial parameters and processes. Simple simulation approaches to assessment consider only a single set of parameter values, but what is required is consideration of all more or less plausible combinations of parameters. Bayesian decision theory is an appropriate tool for such assessment. I contrast standard (frequentist) and Bayesian approaches to a simple regression problem. I use these results to calculate the probability of early population collapse for three data sets relating to the Palila, Laysan Finch, and Snow Goose. The Bayesian results imply much higher risk of early collapse than maximum likelihood methods. This difference is due to large probabilities of early collapse for certain parameter values that are plausible in light of the data. Because of simplifying assumptions, these results are not directly applicable to assessment. Nevertheless they imply that maximum likelihood and similar methods based upon point parameter estimates will grossly underestimate the risk of early collapse.

PC-based differential model fitting as a support for clinical research.

A PC-based minimisation software written in C-language is described, which solves numerically both simple non-linear regression problems and problems expressed as systems of (unsolved) initial-value ordinary or partial differential equations. The software uses second-order iterated Runge-Kutta algorithm to approximate numerically the solution curves. It uses a quasi-Newton algorithm to minimize either sums of squares (weighted or unweighted) or NONMEM loss functions. Inverse Hessian approximation to the parameter dispersion and Monte Carlo generation of artificial samples are offered to test the robustness of the parameter values obtained. A real test problem is described, involving the hydrolysation of plasma Medium Chain Triglycerides to Free Fatty Acids and the uptake of these from plasma. Two competing models were evaluated, one involving linear terms for each transfer and one involving carrier-mediated, rate-limited hydrolysis and tissue absorption steps. The simpler linear model was found to be more robust and eventually used to describe the experimental data.

Exact likelihood function for a regression model with MA(1) errors

Balestra (1980) derives analytical expressions for the transformation that can be used to transform a generalized regression problem into a simple regression problem. This note solves this problem using the Kalman filter. It is shown that the exact likelihood function can be obtained very easily.

Solving algebraic equations on a microcomputer.

Equation-solving programs for microcomputers make the numerical solution of algebraic equations an easy task. It is no longer necessary to learn or to program algorithms for the solution of many different types of equations. A single equation or a set of simultaneous equations may simply be entered into the computer and numerically solved for unknowns without concern as to whether the equations are linear or non-linear. Several examples of possible applications of equation-solving programs are discussed. Solution times for these examples are given for SEQS on the Apple II and Macintosh computers. The example sets of equations, which include chemical equilibrium and enzyme kinetics problems, have been chosen to demonstrate important aspects of the uses and limitations of equation solving. The four examples discussed are: a two-compartment pharmacokinetic model, citric acid ionization in aqueous solution, an enzyme inhibition model, and an example of the application of an equation-solving program in doing a simple non-linear regression problem.

L1Norm estimates for the simple regression problem

L₁Norm estimates for the simple regression problem

A note on the exact transformation associated with the first-order moving average process

Exact analytical expressions for the transformation that can be used to transform a generalized regression problem into a simple regression problem are available for a variety of models. Such is the case, for instance, for purely heteroscedastic models, for the first-order Markov process and for error components models. For the first-order moving average process, on the other hand, the exact transformation has not yet been produced. This gap is filled in the present note. Implications for estimation and prediction are also considered.

Probability and Statistical Inference.

1. Empirical and Probability Distributions. Basic Concepts. The Mean, Variance, and Standard Deviation. Continuous-Type Data. Exploratory Data Analysis. Graphical Comparisons of Data Sets. Time Sequences. Probability Density and Mass Functions. 2. Probability. Properties of Probability. Methods of Enumeration. Conditional Probability. Independent Events. Bayes' Theorem. 3. Discrete Distributions. Random Variables of the Discrete Type. Mathematical Expectation. Bernoulli Trials and the Binomial Distribution. The Moment-Generating Function. The Poisson Distribution. 4. Continuous Distributions. Random Variables of the Continuous Type. The Uniform and Exponential Distributions. The Gamma and Chi-Square Distributions. The Normal Distribution. Distributions of Functions of a Random Variable. Mixed Distributions and Censoring. 5. Multivariable Distributions. Distributions of Two Random Variables. The Correlation Coefficient. Conditional Distributions. The Bivariate Normal Distribution. Transformations of Random Variables. Order Statistics. 6. Sampling Distribution Theory. Independent Random Variables. Distributions of Sums of Independent Random Variables. Random Functions Associated with Normal Distributions. The Central Limit Theorem. Approximations for Discrete Distributions. The t and F Distributions. Limiting Moment-Generating Functions. Chebyshev's Inequality and Convergence in Probability. Importance of Understanding Variability. 7. Estimation. Point Estimation. Confidence Intervals for Means. Confidence Intervals for Difference of Two Means. Confidence Intervals for Variances. Confidence Intervals for Proportions. Sample Size. Distribution-Free Confidence Intervals for Percentiles. A Simple Regression Problem. More Regression. 8. Tests of Statistical Hypotheses. Tests about Proportions. Tests about One Mean and One Variance. Tests of the Equality of Two Normal Distributions. Chi-Square Goodness of Fit Test. Contingency Tables. Tests of the Equality of Several Means. Two-Factor Analysis of Variance. Tests Concerning Regression and Correlation. The Wilcoxon Tests. Kolmogorov-Smirnov Goodness of Fit Test. Resampling Methods. Run Test and Test for Randomness. 9. Theory of Statistical Inference. Sufficient Statistics. Power of a Statistical Test. Best Critical Regions. Likelihood Ratio Tests. Bayesian Estimation. Asymptotic Distributions of Maximum Likelihood Estimators. 10. Quality Improvement through Statistical Methods. Statistical Quality Control. General Factorial and 2k Factorial Designs. More on Design of Experiments. Epilogue.Appendix A. Review of Selected Mathematical Techniques. Algebra of Sets. Mathematical Tools for the Hypergeometric Distribution. Limits. Infinite Series. Integration. Multivariate Calculus. Appendix B. References.Appendix C. Tables.Appendix D. Answers to Odd-Numbered Exercises.Index.

Nonparametric testing for simple regression under progressive censoring with stagering entry and random withdrawal

In the context of (multi-center) clinical trials and life testins problems, a general model incorporating both the staggering entry and random withdrawal and pertaining to a simple regression problem (including the two-sample location problem as a special case) is conceived, and, within this framework, a scheme allowing progressive censoring (continuous monitoring of experimentation from the beginning) is developed along with the proposal for and study of some nonparametric testing procedures, The proposed tests rest on the construction of certain two-dimensional time-parameter stochastic processes from a triangular array of progressively censored linear rank statistics and their weak convergence to appropriate Gaussian functions. Asymptotic properties of these procedures are studied. A computer program pertaining to the numerical computations and practical administrations of these testing procedures is also provided at the end.