Predictor Space Research Articles

Reliability Measures for Local Nodes Assessment in Classification Trees

Most of the modern developments with classification trees are aimed at improving their predictive capacity.This article considers a curiously neglected aspect of classification trees, namely the reliability of predictions that come from a given classification tree. In the sense that a node of a tree represents a point in the predictor space in the limit, the aim of this article is the development of localized assessment of the reliability of prediction rules. A classification tree may be used either to provide a probability forecast, where for each node the membership probabilities for each class constitutes the prediction, or a true classification where each new observation is predictively assigned to a unique class. Correspondingly, two types of reliability measure will be derived—namely, prediction reliability and classification reliability. We use bootstrapping methods as the main tool to construct these measures. We also provide a suite of graphical displays by which they may be easily appreciated. In addition to providing some estimate of the reliability of specific forecasts of each type, these measures can also be used to guide future data collection to improve the effectiveness of the tree model. The motivating example we give has a binary response, namely the presence or absence of a species of Eucalypt, Eucalyptus cloeziana, at a given sampling location in response to a suite of environmental covariates, (although the methods are not restricted to binary response data).

Bayes Model Averaging with Selection of Regressors

SummaryWhen a number of distinct models contend for use in prediction, the choice of a single model can offer rather unstable predictions. In regression, stochastic search variable selection with Bayesian model averaging offers a cure for this robustness issue but at the expense of requiring very many predictors. Here we look at Bayes model averaging incorporating variable selection for prediction. This offers similar mean-square errors of prediction but with a vastly reduced predictor space. This can greatly aid the interpretation of the model. It also reduces the cost if measured variables have costs. The development here uses decision theory in the context of the multivariate general linear model. In passing, this reduced predictor space Bayes model averaging is contrasted with single-model approximations. A fast algorithm for updating regressions in the Markov chain Monte Carlo searches for posterior inference is developed, allowing many more variables than observations to be contemplated. We discuss the merits of absolute rather than proportionate shrinkage in regression, especially when there are more variables than observations. The methodology is illustrated on a set of spectroscopic data used for measuring the amounts of different sugars in an aqueous solution.

SPARTAN

While a variety of lossy compression schemes have been developed for certain forms of digital data (e.g., images, audio, video), the area of lossy compression techniques for arbitrary data tables has been left relatively unexplored. Nevertheless, such techniques are clearly motivated by the ever-increasing data collection rates of modern enterprises and the need for effective, guaranteed-quality approximate answers to queries over massive relational data sets.In this paper, we propose SPARTAN, a system that takes advantage of attribute semantics and data-mining models to perform lossy compression of massive data tables. SPARTAN is based on the novel idea of exploiting predictive data correlations and prescribed error-tolerance constraints for individual attributes to construct concise and accurate Classification and Regression Tree (CaRT) models for entire columns of a table. More precisely, SPARTAN selects a certain subset of attributes (referred to as predicted attributes) for which no values are explicitly stored in the compressed table; instead, concise error-constrained CaRTs that predict these values (within the prescribed error tolerances) are maintained. To restrict the huge search space of possible CaRT predictors, SPARTAN uses a Bayesian network structure to guide the selection of CaRT models that minimize the overall storage requirement, based on the prediction and materialization costs for each attribute. SPARTAN 's CaRT-building algorithms employ novel integrated pruning strategies that take advantage of the given error constraints on individual attributes to minimize the computational effort involved. Our experimentation with several real-life data sets offers convincing evidence of the effectiveness of SPARTAN 's model-based approach --- SPARTAN is able to consistently yield substantially better compression ratios than existing semantic or syntactic compression tools (e.g., gzip) while utilizing only small samples of the data for model inference.

Regional estimation of flood quantiles: Parametric versus nonparametric regression models

A recent trend in regional frequency analysis is to consider floating regions where only basins that are sufficiently similar to the design site are considered for information transfer. Similarity is measured in some suitable metric of catchment characteristics. This paper discusses the analogy between this idea and nonparametric regression. Some of the techniques developed recently in the area of nonparametric regression are employed to develop improved regional flood estimators. The additive model used here to a large extent overcomes the curse of dimensionality often associated with nonparametric regression on multivariate predictor space. The application of the proposed methodology to selected areas of the United States suggests that there can be substantial gains over the traditional log linear models currently employed.

Partitioning and peeling for constructing prognostic groups.

We investigate methods for identifying groups of patients with different prognosis. Our focus is on procedures that yield interpretable descriptions for groups of patients and corresponding regions of the predictor space. These strategies include tree-based methods and techniques which construct a single prognostic group through 'peeling' or refining of a larger group patients. The peeling methods are relatively new and are developed in detail. Simulations and examples for describing groups of patients with poor outcomes using data from a clinical trial for patients with multiple myeloma are presented.

Tree-structured Generalized Autoregressive Conditional Heteroscedastic Models

Summary We propose a new generalized autoregressive conditional heteroscedastic (GARCH) model with tree-structured multiple thresholds for the estimation of volatility in financial time series. The approach relies on the idea of a binary tree where every terminal node parameterizes a (local) GARCH model for a partition cell of the predictor space. The fitting of such trees is constructed within the likelihood framework for non-Gaussian observations: it is very different from the well-known regression tree procedure which is based on residual sums of squares. Our strategy includes the classical GARCH model as a special case and allows us to increase model complexity in a systematic and flexible way. We derive a consistency result and conclude from simulation and real data analysis that the new method has better predictive potential than other approaches.

State‐space Models with Finite Dimensional Dependence

We consider nonlinear state‐space models, where the state variable (ζt) is Markov, stationary and features finite dimensional dependence (FDD), i.e. admits a transition function of the type: π(ζt|ζt−1) =π(ζt)a′(ζt)b(ζt−1), where π(ζt) denotes the marginal distribution of ζt, with a finite number of cross‐effects between the present and past values. We discuss various characterizations of the FDD condition in terms of the predictor space and nonlinear canonical decomposition. The FDD models are shown to admit explicit recursive formulas for filtering and smoothing of the observable process, that arise as an extension of the Kitagawa approach. The filtering and smoothing algorithms are given in the paper.JEL. C4.

Multiple Comparisons for a Large Number of Parameters

When there are many parameters of interest (finitely large or infinite), standard multiple comparison procedures for a finite number of parameters (called discrete-domain approaches) may lead to a simultaneous confidence region (SCR) too conservative to be useful. Such cases often arise in locating disease genes, detecting changes in image data and examining shapes and patterns in growth curves; or generally, in quantifying uncertainty in an estimate of a regression function (as one entity). In these cases, procedures designed for a continuous domain must be used. Scheffe's method is a classical example of continuous-domain approaches. It provides an SCR for a regression function when errors are iid Gaussian and the predictor space is unconstrained, i.e. the domain of interest is the q dimensional Euclidean space. In practice, however, functions defined on finite intervals or other constrained domains are often of interest and data may not be Gaussian. Thus, Scheffe's SCR becomes either too conservative or inadequate. In this paper, we introduce and survey a modern-type continuous-domain approach, and explore a connection between some discrete- and continuous-domain multiple comparison procedures. We show that, in some cases, even for a small number of parameters, it is still better to use a continuous-domain multiple comparison procedure. The main ideas behind the continuous-domain procedures are shown. A new procedure for comparing a finite number of contrasts about k regression curves is developed. Relevant software is provided.

SPARTAN

While a variety of lossy compression schemes have been developed for certain forms of digital data (e.g., images, audio, video), the area of lossy compression techniques for arbitrary data tables has been left relatively unexplored. Nevertheless, such techniques are clearly motivated by the ever-increasing data collection rates of modern enterprises and the need for effective, guaranteed-quality approximate answers to queries over massive relational data sets. In this paper, we propose SPARTAN , a system that takes advantage of attribute semantics and data-mining models to perform lossy compression of massive data tables. SPARTAN is based on the novel idea of exploiting predictive data correlations and prescribed error tolerances for individual attributes to construct concise and accurate Classification and Regression Tree (CaRT) models for entire columns of a table. More precisely, SPARTAN selects a certain subset of attributes for which no values are explicitly stored in the compressed table; instead, concise CaRTs that predict these values (within the prescribed error bounds) are maintained. To restrict the huge search space and construction cost of possible CaRT predictors, SPARTAN employs sophisticated learning techniques and novel combinatorial optimization algorithms. Our experimentation with several real-life data sets offers convincing evidence of the effectiveness of SPARTAN 's model-based approach — SPARTAN is able to consistently yield substantially better compression ratios than existing semantic or syntactic compression tools (e.g., gzip) while utilizing only small data samples for model inference.

Prediction‐based estimating functions

A generalization of martingale estimating functions is presented which is useful when there are no natural or easily calculated martingales that can be used to construct a class of martingale estimating functions. An estimating function of the new type, which is based on linear predictors, is called a prediction‐based estimating functions. Special attention is given to classes of prediction‐based estimating functions given by a finite‐dimensional space of predictors. It is demonstrated that such a class of estimating functions has most of the attractive properties of martingale estimating functions. In particular, a simple expression is found for the optimal estimating function. This type of prediction‐based estimating functions only involve unconditional moments, in contrast to the martingale estimating functions where conditional moments are required. Thus, for applications to discretely observed continuous time models, a considerably smaller amount of simulation is, in general, needed for these than for martingale estimating functions. This is also true of the optimal prediction‐based estimating functions. Conditions are given that ensure the existence, consistency and asymptotic normality of the corresponding estimators. The new method is applied to inference for sums of Ornstein–Uhlenbeck‐type processes and stochastic volatility models. Stochastic volatility models are studied in considerable detail. It is demonstrated that for inference about models by Hull and White and Chesney and Scott, an explicit optimal prediction‐based estimating function can be found so that no simulations are needed.

Analysis of oxygen rate time series in a strongly polluted lagoon using a regression tree method

This article presents results on daily forecasts of dissolved oxygen rates in a lagoon, the ‘Etang de Berre’, situated close to the French Mediterranean sea coast and submitted to particular meteorological conditions and freshwater inputs. To provide estimates of the response variable y (dissolved oxygen), a set of jointed decision rules is constructed by successive binary splitting of the predictor space composed of wind speed and freshwater flow data. The prediction model is displayed in the shape of a binary decision tree. We propose a transfer procedure in order to improve the prediction error of the tree model. Results obtained on the ‘Etang de Berre’ data set allow to describe and precise the effects of the environmental variables on the dissolved oxygen dynamics. The transfer procedure applied after the tree building process gives gains of prediction accuracy of about 17%.

Improving bias-robustness of regression estimates through projections

We define a robust procedure to “correct” a regression estimate β ̂ 0 along the directions in predictor space where the fit is worse. When β ̂ 0 is the least median of squares estimate, the “corrected estimate” has a smaller maximum asymptotic bias under contamination, and a much better finite-sample behavior than β ̂ 0.

Prediction via Orthogonalized Model Mixing

Abstract We introduce an approach and algorithms for model mixing in large prediction problems with correlated predictors. We focus on the choice of predictors in linear models, and mix over possible subsets of candidate predictors. Our approach is based on expressing the space of models in terms of an orthogonalization of the design matrix. Advantages are both statistical and computational. Statistically, orthogonalization often leads to a reduction in the number of competing models by eliminating correlations. Computationally, large model spaces cannot be enumerated; recent approaches are based on sampling models with high posterior probability via Markov chains. Based on orthogonalization of the space of candidate predictors, we can approximate the posterior probabilities of models by products of predictor-specific terms. This leads to an importance sampling function for sampling directly from the joint distribution over the model space, without resorting to Markov chains. Compared to the latter, ortho...

Simultaneous Confidence Bands for Linear Regression with Heteroscedastic Errors

Abstract The Scheffé method may be used to construct simultaneous confidence bands for a regression surface for the whole predictor space. When the bands need only hold for a subset of that space, previous authors have described how the bands can be appropriately narrowed while still maintaining the desired level of confidence. Data with heteroscedastic errors occur often, and unless some transformation is feasible, there is no obvious way to construct bands using the current methods. This article shows how to construct approximate simultaneous confidence bands when the errors are heteroscedastic and symmetric. The method works when the weights are known or unknown and have to be estimated. The region in which the bands must hold can be quite general and will work for any linear unbiased estimate of the regression surface. The method can even be extended to linear estimates with a small amount of bias such as nonparametric kernel regression smoothers.

Simultaneous Confidence Bands for Linear Regression with Heteroscedastic Errors

Abstract The Scheffe method may be used to construct simultaneous confidence bands for a regression surface for the whole predictor space. When the bands need only hold for a subset of that space, previous authors have described how the bands can be appropriately narrowed while still maintaining the desired level of confidence. Data with heteroscedastic errors occur often, and unless some transformation is feasible, there is no obvious way to construct bands using the current methods. This article shows how to construct approximate simultaneous confidence bands when the errors are heteroscedastic and symmetric. The method works when the weights are known or unknown and have to be estimated. The region in which the bands must hold can be quite general and will work for any linear unbiased estimate of the regression surface. The method can even be extended to linear estimates with a small amount of bias such as nonparametric kernel regression smoothers.

The binary decision tree: The growing algorithm and application to thunderstorm forecasting

The paper deals with the probabilistic prediction of event occurrence with use of the binary decision tree which is grown from the learning sample. The tree growing algorithm consists in recursive partition of the predictor space by either single-predictor-based (SP) splits or by hyperplanes perpendicular to the best linear discriminant function (BLDF), and is intended to maximally effectively discriminate the elements of the learning sample with event occurrence from the elements without event occurrence.

Adapting for the Missing Link

We consider the fitting of generalized linear models in which the link function is assumed to be unknown, and propose the following computational method: First, estimate regression coefficients using the canonical link. Then, estimate the link via a kernel smoother, treating the direction in the predictor space determined by the regression coefficients as known. Then reestimate the direction using the estimated link and alternate between these two steps. We show that under fairly general conditions, $n^{1/2}$-consistent estimates of the direction are obtained. A small Monte Carlo study is presented.

A note on slope rotatability over all directions

Slope rotatability of a response surface, averaged over all directions at any point in the predictor space, is considered. A measure of this averaged slope rotatability for any design in proposed. The measure takes advantage of the fact that the contours of the averaged variance of the slope are ellipsoids. This measure can be used to add points to any design to make it ‘more’ slope rotatable.

Local Regression: Automatic Kernel Carpentry

A kernel smoother is an intuitive estimate of a regression function or conditional expectation; at each point $x_0$ the estimate of $E(Y\mid x_0)$ is a weighted mean of the sample $Y_i$, with observations close to $x_0$ receiving the largest weights. Unfortunately this simplicity has flaws. At the boundary of the predictor space, the kernel neighborhood is asymmetric and the estimate may have substantial bias. Bias can be a problem in the interior as well if the predictors are nonuniform or if the regression function has substantial curvature. These problems are particularly severe when the predictors are multidimensional. A variety of kernel modifications have been proposed to provide approximate and asymptotic adjustment for these biases. Such methods generally place substantial restrictions on the regression problems that can be considered; in unfavorable situations, they can perform very poorly. Moreover, the necessary modifications are very difficult to implement in the multidimensional case. Local regression smoothers fit lower-order polynomials in $x$ locally at $x_0$, and the estimate of $f(x_0)$ is taken from the fitted polynomial at $x_0$. They automatically, intuitively and simultaneously adjust for both the biases above to the given order and generalize naturally to the multidimensional case. They also provide natural estimates for the derivatives of $f$, an approach more attractive than using higher-order kernel functions for the same purpose.

Tables of the casella and strawderman (1980) critical points for use in multiple comparisons with a control in response surface methodology

Casella and Strawderman (1980) provide a method to find the critical points for constructing exact (1−α)100% confidence bands for a regression function over certain subsets of the predictor space. They provide a table of squared upper−α percentage points, c2, under known σ2. For the case of unknown variance they provide expressions for coverage probability, but no tables. Both theorems in their section 5 on unknown variance have minor errors, but they would affect the solutions without doubt. In this paper, the correct version of both of the theorems used to find the critical pointsfor unknown σ2 are provided, along with tables of the squared upper-α percentage points c2 for use in a specific application, multiple