True Regression Function Research Articles

Almost complete convergence of the functional local linear estimator of the expectile regression for data missing at random

In this article, the authors investigate the properties of a local linear estimator for the expectile regression function in scenarios where the scalar response variable is missing at random, and the covariate is a functional variable. The study addresses the challenge of estimating the expectile regression function when the response variable is not fully observed, a common issue in statistical analysis where missing data can significantly impact the accuracy of predictions and inferences. The key contribution of the paper lies in the establishment of the almost complete convergence of the proposed estimator. This convergence resultis derived under a set of appropriate conditions, ensuring that the estimator approaches the true expectile regression function with a probability tending to wards one as the sample size increases. The authors provide a rigorous theoretical foundation for this convergence, including assumptions about the smoothness of the functional covariate and the randomness of the missing data mechanism. To demonstrate the practical applicability of their method, the authors conduct a comprehensive simulation study. This study compares the performance of their local linear estimator with other existing methods in terms of bias, variance, and convergence speed. The simulation resultshighlight the effectiveness of the proposed approach, especially in settings where the response variable is partially missing and the covariates are functional in nature.

Posterior contraction and testing for multivariate isotonic regression

We consider the nonparametric regression problem with multiple predictors and an additive error, where the regression function is assumed to be coordinatewise nondecreasing. We propose a Bayesian approach to make inferences on the multivariate monotone regression function, obtain the posterior contraction rate, and construct a universally consistent Bayesian testing procedure for multivariate monotonicity. To facilitate posterior analysis, we temporarily set aside the shape restrictions, and endow a prior on blockwise constant regression functions with independently normally distributed heights. The unknown variance of the error term is either estimated by the marginal maximum likelihood estimate, or is equipped with an inverse-gamma prior. Then the unrestricted block-heights are a posteriori also independently normally distributed given the error variance, by conjugacy. To comply with the shape restrictions, we project samples from the unrestricted posterior onto the class of multivariate monotone functions, inducing the “projection-posterior distribution”, to be used for making an inference. Under an L1-metric, we show that the projection-posterior based on n independent samples contracts around the true monotone regression function at the optimal rate n−1∕(2+d). Then we construct a Bayesian test for multivariate monotonicity based on the posterior probability of a shrinking neighborhood of the class of multivariate monotone functions. We show that the test is universally consistent, that is, the level of the Bayesian test goes to zero, and the power at any fixed alternative goes to one. Moreover, we show that for a smooth alternative function, power goes to one as long as its distance to the class of multivariate monotone functions is at least of the order of the estimation error for a smooth function. To the best of our knowledge, no other test for multivariate monotonicity is available in the Bayesian or the frequentist literature.

Simulating the behavior of a kernel M-estimator for left-truncated and associated model

This paper investigates theoretical and numerical asymptotic properties of a nonparametric M-estimator’s regression function when the data are left truncated and associated. Using specific tools to this weak dependence, we study the strong uniform consistency of the proposed estimator and give its rate of convergence. A large Monte Carlo simulation study is carried out to comfort the good behavior of the M-estimator for different sample sizes and different truncation rates: First we show that the proposed estimator performs better than the Nadaraya-Watson estimator by looking at their respective closeness to the true univariate and bivariate regression function. Thereafter and to highlight the robustness of our estimator, we consider a regression model with a heavy-tailed error distribution and we compare the global mean square error of the two estimators. Finally, we study the behavior of the two estimators in the presence of outliers using the influence function to show the superiority of the M-estimator.

Nonparametric, Tuning-Free Estimation of S-Shaped Functions

AbstractWe consider the nonparametric estimation of an S-shaped regression function. The least squares estimator provides a very natural, tuning-free approach, but results in a non-convex optimization problem, since the inflection point is unknown. We show that the estimator may nevertheless be regarded as a projection onto a finite union of convex cones, which allows us to propose a mixed primal-dual bases algorithm for its efficient, sequential computation. After developing a projection framework that demonstrates the consistency and robustness to misspecification of the estimator, our main theoretical results provide sharp oracle inequalities that yield worst-case and adaptive risk bounds for the estimation of the regression function, as well as a rate of convergence for the estimation of the inflection point. These results reveal not only that the estimator achieves the minimax optimal rate of convergence for both the estimation of the regression function and its inflection point (up to a logarithmic factor in the latter case), but also that it is able to achieve an almost-parametric rate when the true regression function is piecewise affine with not too many affine pieces. Simulations and a real data application to air pollution modelling also confirm the desirable finite-sample properties of the estimator, and our algorithm is implemented in the R package Sshaped.

The One Standard Error Rule for Model Selection: Does It Work?

Previous research provided a lot of discussion on the selection of regularization parameters when it comes to the application of regularization methods for high-dimensional regression. The popular “One Standard Error Rule” (1se rule) used with cross validation (CV) is to select the most parsimonious model whose prediction error is not much worse than the minimum CV error. This paper examines the validity of the 1se rule from a theoretical angle and also studies its estimation accuracy and performances in applications of regression estimation and variable selection, particularly for Lasso in a regression framework. Our theoretical result shows that when a regression procedure produces the regression estimator converging relatively fast to the true regression function, the standard error estimation formula in the 1se rule is justified asymptotically. The numerical results show the following: 1. the 1se rule in general does not necessarily provide a good estimation for the intended standard deviation of the cross validation error. The estimation bias can be 50–100% upwards or downwards in various situations; 2. the results tend to support that 1se rule usually outperforms the regular CV in sparse variable selection and alleviates the over-selection tendency of Lasso; 3. in regression estimation or prediction, the 1se rule often performs worse. In addition, comparisons are made over two real data sets: Boston Housing Prices (large sample size n, small/moderate number of variables p) and Bardet–Biedl data (large p, small n). Data guided simulations are done to provide insight on the relative performances of the 1se rule and the regular CV.

Variable selection consistency of Gaussian process regression

Bayesian nonparametric regression under a rescaled Gaussian process prior offers smoothness-adaptive function estimation with near minimax-optimal error rates. Hierarchical extensions of this approach, equipped with stochastic variable selection, are known to also adapt to the unknown intrinsic dimension of a sparse true regression function. But it remains unclear if such extensions offer variable selection consistency, that is, if the true subset of important variables could be consistently learned from the data. It is shown here that variable consistency may indeed be achieved with such models at least when the true regression function has finite smoothness to induce a polynomially larger penalty on inclusion of false positive predictors. Our result covers the high-dimensional asymptotic setting where the predictor dimension is allowed to grow with the sample size. The proof utilizes Schwartz theory to establish that the posterior probability of wrong selection vanishes asymptotically. A necessary and challenging technical development involves providing sharp upper and lower bounds to small ball probabilities at all rescaling levels of the Gaussian process prior, a result that could be of independent interest.

Confidence intervals for multiple isotonic regression and other monotone models

We consider the problem of constructing pointwise confidence intervals in the multiple isotonic regression model. Recently, Han and Zhang (2020) obtained a pointwise limit distribution theory for the so-called block max–min and min–max estimators (Fokianos, Leucht and Neumann (2020); Deng and Zhang (2020)) in this model, but inference remains a difficult problem due to the nuisance parameter in the limit distribution that involves multiple unknown partial derivatives of the true regression function. In this paper, we show that this difficult nuisance parameter can be effectively eliminated by taking advantage of information beyond point estimates in the block max–min and min–max estimators. Formally, let uˆ(x0) (resp. vˆ(x0)) be the maximizing lower-left (resp. minimizing upper-right) vertex in the block max–min (resp. min–max) estimator, and fˆn be the average of the block max–min and min–max estimators. If all (first-order) partial derivatives of f0 are nonvanishing at x0, then the following pivotal limit distribution theory holds: nuˆ,vˆ(x0)(fˆn(x0)−f0(x0))⇝σ·L1d. Here nuˆ,vˆ(x0) is the number of design points in the block [uˆ(x0),vˆ(x0)], σ is the standard deviation of the errors, and L1d is a universal limit distribution free of nuisance parameters. This immediately yields confidence intervals for f0(x0) with asymptotically exact confidence level and oracle length. Notably, the construction of the confidence intervals, even new in the univariate setting, requires no more efforts than performing an isotonic regression once using the block max–min and min–max estimators, and can be easily adapted to other common monotone models including, for example, (i) monotone density estimation, (ii) interval censoring model with current status data, (iii) counting process model with panel count data, and (iv) generalized linear models. Extensive simulations are carried out to support our theory.

“Regression anytime” with brute-force SVD truncation

We propose a new least-squares Monte Carlo algorithm for the approximation of conditional expectations in the presence of stochastic derivative weights. The algorithm can serve as a building block for solving dynamic programming equations, which arise, for example, in nonlinear option pricing problems or in probabilistic discretization schemes for fully nonlinear parabolic partial differential equations. Our algorithm can be generically applied when the underlying dynamics stem from an Euler approximation to a stochastic differential equation. A built-in variance reduction ensures that the convergence in the number of samples to the true regression function takes place at an arbitrarily fast polynomial rate, if the problem under consideration is smooth enough.

Predicting longitudinal dispersion coefficient in natural streams using minimax probability machine regression and multivariate adaptive regression spline

This article employs minimax probability machine regression (MPMR) and multivariate adaptive regression spline (MARS) for prediction of longitudinal dispersion coefficient in natural streams. The variables of hydraulic features such as channel width (B), flow depth (H), flow velocity (U), shear velocity (u*) and geometric features such as channel sinuosity (σ) and channel shape parameter (β) were taken as the input. The dispersion coefficient Kx was the decision parameter for the proposed machine learning models. MARS does not assume any functional relationship between inputs and output. The MARS model is a non-parametric regression model that splits the data and fits each interval into a basis function. MPMR is a probabilistic model which maximises the minimum probability of predicted output. MPMR also provides output within some bound of the true regression function. The proposed study gives an equation for prediction of longitudinal dispersion coefficient based on the developed MARS. The developed MARS has been compared with proposed MPMR. Finally, the performances of the models have been measured by different performance metrics.

Predicting longitudinal dispersion coefficient in natural streams using minimax probability machine regression and multivariate adaptive regression spline

This article employs minimax probability machine regression (MPMR) and multivariate adaptive regression spline (MARS) for prediction of longitudinal dispersion coefficient in natural streams. The variables of hydraulic features such as channel width (B), flow depth (H), flow velocity (U), shear velocity (u*) and geometric features such as channel sinuosity (σ) and channel shape parameter (β) were taken as the input. The dispersion coefficient Kx was the decision parameter for the proposed machine learning models. MARS does not assume any functional relationship between inputs and output. The MARS model is a non-parametric regression model that splits the data and fits each interval into a basis function. MPMR is a probabilistic model which maximises the minimum probability of predicted output. MPMR also provides output within some bound of the true regression function. The proposed study gives an equation for prediction of longitudinal dispersion coefficient based on the developed MARS. The developed MARS has been compared with proposed MPMR. Finally, the performances of the models have been measured by different performance metrics.

Консистентнiсть оцiнки найменших квадратiв параметрiв тригонометричної моделi регресiї у присутностi лiнiйного випадкового шуму

Регресiйний аналiз є iстотною частиною математичної та прикладної статистики. Нелiнiйний регресiйний аналiз є значним розширенням та ускладненням класичного лiнiйного регресiйного аналiзу, завдяки використанню нелiнiйних або частково нелiнiйних за параметрами моделей, якi адекватнiше описують, нiж лiнiйнi моделi, явища, що потребують статистичного аналiзу. Велика кiлькiсть прикладних проблем у численних наукових, технiчних та гуманiтарних галузях знань дають поштовх розвитку нелiнiйного регресiйного аналiзу. Задача оцiнювання векторного параметра сигналу в моделях спостереження «сигнал + шум» є добре вiдомою проблемою статистики випадкових процесiв, та у випадку нелiнiйного параметра сигналу – задачею нелiнiйного регресiйного аналiзу. Серед рiзноманiтностi задач нелiнiйного регресiйного аналiзу оцiнювання амплiтуд та кутових частот суми гармонiчних коливань, що спостерiгається на фонi випадкового шуму, займає значне мiсце, завдяки її численним застосуванням. Статистичнi моделi такого типу називаються тригонометричними моделями регресiї, а проблема статистичного оцiнювання її параметрiв називається задачею виявлення прихованих перiодичностей. Статтю присвячено вивченню тригонометричної моделi регресiї, в якiй випадковий шум є лiнiйним Левi-керованим стацiонарним четвертого порядку випадковим процесом iз нульовим середнiм, iнтегровную та iнтегровную з квадратом iмпульсною перехiдною функцiєю. Це припущення призводить до iнтегровностi коварiацiйної функцiї та кумулянтної функцiї 4-го порядку. Для оцiнювання амплiтуд та кутових частот такої тригонометричної моделi ми використовуємо оцiнки найменших квадратiв у сенсi Уолкера, тобто розглянуто спецiальну множину параметрiв, щоб розрiзнити належним чином рiзнi кутовi частоти в сумi гармонiчних коливань. У статтi доведено теорему про сильну консистентнiсть оцiнки найменших квадратiв за описаними вище припущеннями щодо випадкового шуму. Для отримання такого результату було доведено дуже важливу лему про рiвномiрну збiжнiсть майже напевно середнього значення перетворення Фурьє лiнiйного Левi-керованого випадкового процеса. Ця лема є головним iнструментом доведення теореми про сильну консистентнiсть. Для доведення теореми, по-перше, знаходимо деякi представлення оцiнок найменших квадратiв амплiтуд через вiдповiднi оцiнки кутових частот. По-друге, ми пiдставляємо цi формули у функцiонал методу найменших квадратiв. Останнiй крок доведення полягає у перетвореннi L2-норми рiзницi мiж емпiричною тригонометричною функцiєю регресiї та iстиною функцiєю регресiї таким чином, що ця норма прямує до нуля майже напевно тодi i тiльки тодi, коли оцiнки є сильно консистентними.

Adaptive confidence sets in shape restricted regression

A simple construction of adaptive confidence sets is proposed in isotonic, convex and unimodal regression. In univariate isotonic regression, the proposed confidence set enjoys uniform coverage over all non-decreasing regression functions. Furthermore, the diameter of the proposed confidence set automatically adapts to the unknown number of pieces of the true parameter, in the sense that the diameter is bounded from above by the minimax risk over the class of $k$-piecewise constant functions. The diameter of the confidence set is a simple increasing function of the number of jumps of the isotonic least-squares estimate. A similar construction is proposed in convex regression where the true regression function is convex and piecewise affine. Here, the confidence set enjoys uniform coverage and its diameter automatically adapts to the number of affine pieces of the true regression function. The diameter of the confidence set is an increasing function of the number of affine pieces of the convex least-squares estimate. We explain how to extend this technique to a non-convex set by proposing a similar adaptive confidence set in unimodal regression. The confidence set automatically adapts to the number of jumps of the true unimodal regression function and its diameter is an increasing function of the number of jumps of the unimodal least-squares estimate.

On data-driven choice of [formula omitted] in nonparametric Gaussian regression via Propagation–Separation approach

A new procedure is proposed for selecting the input value for the adaptation bandwidth λ in nonparametric Gaussian regression via Propagation–Separation approach. Since λ stands at the bias–variance trade-off of this estimation technique, it is its key tuning parameter. So far, however, apart from theoretical concepts, there exists no approach for a data-driven choice of its input value. This complicates the practical application of the Propagation–Separation method despite its highly desirable statistical properties — it is dimension-free and very well suited for estimation problems where the underlying regression function has sharp discontinuities and large homogeneous regions. The proposed selection method is based on the idea of cross-validation. Therefore, it allows to choose λ in a data-driven way. Its performance is evaluated via simulations. The results are very convincing: Cross-validation is a very transparent selection procedure that provides for each sample a “tailor-made” input value for λ and allows a successful identification of the underlying regression function. As the sample size increases, the accuracy of estimation improves in the sense that the estimates approach the true regression function. In addition, cross-validation accounts for the weighting functions and other parameter values used within the Propagation–Separation method by adjusting λ accordingly, which results in very robust estimates.

Posterior concentration for Bayesian regression trees and forests

Since their inception in the 1980s, regression trees have been one of the more widely used nonparametric prediction methods. Tree-structured methods yield a histogram reconstruction of the regression surface, where the bins correspond to terminal nodes of recursive partitioning. Trees are powerful, yet susceptible to overfitting. Strategies against overfitting have traditionally relied on pruning greedily grown trees. The Bayesian framework offers an alternative remedy against overfitting through priors. Roughly speaking, a good prior charges smaller trees where overfitting does not occur. While the consistency of random histograms, trees and their ensembles has been studied quite extensively, the theoretical understanding of the Bayesian counterparts has been missing. In this paper, we take a step toward understanding why/when do Bayesian trees and forests not overfit. To address this question, we study the speed at which the posterior concentrates around the true smooth regression function. We propose a spike-and-tree variant of the popular Bayesian CART prior and establish new theoretical results showing that regression trees (and forests) (a) are capable of recovering smooth regression surfaces (with smoothness not exceeding one), achieving optimal rates up to a log factor, (b) can adapt to the unknown level of smoothness and (c) can perform effective dimension reduction when $p>n$. These results provide a piece of missing theoretical evidence explaining why Bayesian trees (and additive variants thereof?) have worked so well in practice.

GRID: A variable selection and structure discovery method for high dimensional nonparametric regression

We consider nonparametric regression in high dimensions where only a relatively small subset of a large number of variables are relevant and may have nonlinear effects on the response. We develop methods for variable selection, structure discovery and estimation of the true low-dimensional regression function, allowing any degree of interactions among the relevant variables that need not be specified a-priori. The proposed method, called the GRID, combines empirical likelihood based marginal testing with the local linear estimation machinery in a novel way to select the relevant variables. Further, it provides a simple graphical tool for identifying the low dimensional nonlinear structure of the regression function. Theoretical results establish consistency of variable selection and structure discovery, and also Oracle risk property of the GRID estimator of the regression function, allowing the dimension $d$ of the covariates to grow with the sample size $n$ at the rate $d=O(n^{a})$ for any $a\in(0,\infty)$ and the number of relevant covariates $r$ to grow at a rate $r=O(n^{\gamma})$ for some $\gamma\in(0,1)$ under some regularity conditions that, in particular, require finiteness of certain absolute moments of the error variables depending on $a$. Finite sample properties of the GRID are investigated in a moderately large simulation study.

Machine learning in the estimation of causal effects: targeted minimum loss-based estimation and double/debiased machine learning.

In recent decades, the fields of statistical and machine learning have seen a revolution in the development of data-adaptive regression methods that have optimal performance under flexible, sometimes minimal, assumptions on the true regression functions. These developments have impacted all areas of applied and theoretical statistics and have allowed data analysts to avoid the biases incurred under the pervasive practice of parametric model misspecification. In this commentary, I discuss issues around the use of data-adaptive regression in estimation of causal inference parameters. To ground ideas, I focus on two estimation approaches with roots in semi-parametric estimation theory: targeted minimum loss-based estimation (TMLE; van der Laan and Rubin, 2006) and double/debiased machine learning (DML; Chernozhukov and others, 2018). This commentary is not comprehensive, the literature on these topics is rich, and there are many subtleties and developments which I do not address. These two frameworks represent only a small fraction of an increasingly large number of methods for causal inference using machine learning. To my knowledge, they are the only methods grounded in statistical semi-parametric theory that also allow unrestricted use of data-adaptive regression techniques.

Adaptive Bayesian credible bands in regression with a Gaussian process prior

A credible band is the set of all functions between a lower and an upper bound that are constructed so that the set has prescribed mass under the posterior distribution. In a Bayesian analysis such a band is used to quantify the remaining uncertainty on the unknown function in a similar manner as a confidence band. We investigate the validity of a credible band in the nonparametric regression model with the prior distribution on the function given by a Gaussian process. We show that there are many true regression functions for which the credible band has the correct order of magnitude to be used as a confidence set. We also exhibit functions for which the credible band is misleading.

Doubly penalized estimation in additive regression with high-dimensional data

Additive regression provides an extension of linear regression by modeling the signal of a response as a sum of functions of covariates of relatively low complexity. We study penalized estimation in high-dimensional nonparametric additive regression where functional semi-norms are used to induce smoothness of component functions and the empirical $L_{2}$ norm is used to induce sparsity. The functional semi-norms can be of Sobolev or bounded variation types and are allowed to be different amongst individual component functions. We establish oracle inequalities for the predictive performance of such methods under three simple technical conditions: a sub-Gaussian condition on the noise, a compatibility condition on the design and the functional classes under consideration and an entropy condition on the functional classes. For random designs, the sample compatibility condition can be replaced by its population version under an additional condition to ensure suitable convergence of empirical norms. In homogeneous settings where the complexities of the component functions are of the same order, our results provide a spectrum of minimax convergence rates, from the so-called slow rate without requiring the compatibility condition to the fast rate under the hard sparsity or certain $L_{q}$ sparsity to allow many small components in the true regression function. These results significantly broaden and sharpen existing ones in the literature.

Isotonic regression in general dimensions

We study the least squares regression function estimator over the class of real-valued functions on $[0,1]^{d}$ that are increasing in each coordinate. For uniformly bounded signals and with a fixed, cubic lattice design, we establish that the estimator achieves the minimax rate of order $n^{-\min\{2/(d+2),1/d\}}$ in the empirical $L_{2}$ loss, up to polylogarithmic factors. Further, we prove a sharp oracle inequality, which reveals in particular that when the true regression function is piecewise constant on $k$ hyperrectangles, the least squares estimator enjoys a faster, adaptive rate of convergence of $(k/n)^{\min(1,2/d)}$, again up to polylogarithmic factors. Previous results are confined to the case $d\leq2$. Finally, we establish corresponding bounds (which are new even in the case $d=2$) in the more challenging random design setting. There are two surprising features of these results: first, they demonstrate that it is possible for a global empirical risk minimisation procedure to be rate optimal up to polylogarithmic factors even when the corresponding entropy integral for the function class diverges rapidly; second, they indicate that the adaptation rate for shape-constrained estimators can be strictly worse than the parametric rate.

An oracle property of the Nadaraya–Watson kernel estimator for high‐dimensional nonparametric regression

AbstractThe Nadaraya–Watson estimator is among the most studied nonparametric regression methods. A classical result is that its convergence rate depends on the number of covariates and deteriorates quickly as the dimension grows. This underscores the “curse of dimensionality” and has limited its use in high‐dimensional settings. In this paper, however, we show that the Nadaraya–Watson estimator has an oracle property such that when the true regression function is single‐ or multi‐index, it discovers the low‐rank dependence structure between the response and the covariates, mitigating the curse of dimensionality. Specifically, we prove that, using K‐fold cross‐validation and a positive‐semidefinite bandwidth matrix, the Nadaraya–Watson estimator has a convergence rate that depends on the number of indices rather than on the number of covariates. This result follows by allowing the bandwidths to diverge to infinity rather than restricting them all to converge to zero at certain rates, as in previous theoretical studies.