Automatic Model Selection Research Articles

Twenty-five years of progress, problems, and conflicting evidence in econometric forecasting. What about the next 25 years?

In the early 1940s, the Cowles Commission for Research (later, the Cowles Foundation) fostered the development of statistical methodology for application in economics and paved the way for large-scale econometric models to be used for both structural estimation and forecasting. This approach stood for decades. Vector autoregression (VAR), appearing in the 1980s, was a clear improvement over early Cowles Foundation models, primarily because it paid attention to dynamic structure. As a way of imposing long-run equilibrium restrictions on sets of variables, cointegration and error-correction modeling (ECM) gained popularity in the 1980s and 1990s, though ECMs have so far failed to deliver on their early promise. ARCH and GARCH modeling have been used with great success in specialized financial areas to model dynamic heteroscedasticity, though in mainstream econometrics, evidence of their value is limited and conflicting. Concerning misspecification tests, any model will inevitably fail some of them for the simple reason that there are many possible tests. Which failures matter? The root of the difficulty regarding all issues related to modeling is that we can never know the true data generating process. In the next 25 years, what new avenues will open up? With ever greater computational capacity, more complex models with larger data sets seem the way to the future. Will they require the automatic model selection methods that have recently been introduced? Preliminary evidence suggests that these methods can do well. The quality of aggregate data is no better than it was. Will greater use of more disaggregated data be sufficient to provide better forecasts? That remains an open question.

Evaluating PcGets and RETINA as Automatic Model Selection Algorithms*

AbstractThe paper describes two automatic model selection algorithms, RETINA and PcGets, briefly discussing how the algorithms work and what their performance claims are. RETINA's Matlab implementation of the code is explained, then the program is compared with PcGets on the data in Perez‐Amaral, Gallo and White (2005, Econometric Theory, Vol. 21, pp. 262–277), ‘A Comparison of Complementary Automatic Modelling Methods: RETINA and PcGets’, and Hoover and Perez (1999, Econometrics Journal, Vol. 2, pp. 167–191), ‘Data Mining Reconsidered: Encompassing and the General‐to‐specific Approach to Specification Search’. Monte Carlo simulation results assess the null and non‐null rejection frequencies of the RETINA and PcGets model selection algorithms in the presence of nonlinear functions.

Leading Indicators for Euro‐area Inflation and GDP Growth*

AbstractIn this paper, we evaluate the role of a set of variables as leading indicators for Euro‐area inflation and GDP growth. Our leading indicators are taken from the variables in the European Central Bank's (ECB) Euro‐area‐wide model database, plus a set of similar variables for the US. We compare the forecasting performance of each indicator ex post with that of purely autoregressive models. We also analyse three different approaches to combining the information from several indicators. First, ex post, we discuss the use as indicators of the estimated factors from a dynamic factor model for all the indicators. Secondly, within an ex ante framework, an automated model selection procedure is applied to models with a large set of indicators. No future information is used, future values of the regressors are forecast, and the choice of the indicators is based on their past forecasting records. Finally, we consider the forecasting performance of groups of indicators and factors and methods of pooling the ex ante single‐indicator or factor‐based forecasts. Some sensitivity analyses are also undertaken for different forecasting horizons and weighting schemes of forecasts to assess the robustness of the results.

Mining epidemiological time series: an approach based on dynamic regression

In epidemiology, time-series regression models are specially suitable for evaluating short-term effects of time-varying exposures to pollution. To summarize findings from different studies on different cities, the techniques of designed meta-analyses have been employed. In this context, city-specific findings are summarized by an ‘effect size’ measured on a common scale. Such effects are then pooled together on a second hierarchy of analysis. The objective of this article is to exploit exploratory analysis of city-specific time series. In fact, when dealing with many sources of data, that is, many cities, an exploratory analysis becomes almost unaffordable. Our idea is to explore the time series by fitting complete dynamic regression models. These models are easier to fit than models usually employed and allow implementation of very fast automated model selection algorithms. The idea is to highlight the common features across cities through this analysis, which might then be used to design the meta-analysis. The proposal is illustrated by analysing data on the relationship between daily nonaccidental deaths and air pollution in the 20 US largest cities.

Constructing a speculative kernel machine for pattern classification

We propose and investigate the performance of a new geometry-based algorithm designed to identify potentially informative data points for classification. An incremental QR update scheme is used to build a classifier using a subset of these points as radial basis function centers. The minimum descriptive length and the leave-one-out error criteria are employed for automatic model selection. The proposed scheme is shown to generate parsimonious models, which perform generalization comparable to the state-of-the-art support and relevance vector machines.

Are there any reliable leading indicators for US inflation and GDP growth?

In this paper, we evaluate the relative merits of three alternative approaches to extracting information from a large data set for forecasting, namely, the use of an automated model selection procedure, the adoption of a factor model, and the adoption of single-indicator-based forecast pooling. The comparison is conducted using a large set of indicators for forecasting US inflation and GDP growth. We also compare our large set of leading indicators with purely autoregressive models, using an evaluation procedure that is particularly relevant for policy making. The evaluation is conducted both ex post and in a pseudo-real-time context, for several forecast horizons, and using both recursive and rolling estimation. The results indicate a preference for simple forecasting tools, with a good relative performance of pure autoregressive models, and substantial instability in the characteristics of the leading indicators. A pseudo real-time analysis provides a useful guide to the selection of the best leading indicator, in particular for GDP growth.

Are there any Reliable Leading Indicators for U.S. Inflation and GDP Growth?

In this paper we evaluate the relative merits of three approaches to information extraction from a large data set for forecasting, namely, the use of an automated model selection procedure, the adoption of a factor model, and single-indicator-based forecast pooling. The comparison is conducted using a large set of indicators for forecasting US inflation and GDP growth. We also compare our large set of leading indicators with purely autoregressive models, using an evaluation procedure that is particularly relevant for policy making. The evaluation is conducted both ex-post and in a pseudo real time context, for several forecast horizons, and using both recursive and rolling estimation. The results indicate a preference for simple forecasting tools, with a good relative performance of pure autoregressive models, and substantial instability in the leading characteristics of the indicators.

Maximum weighted likelihood via rival penalized EM for density mixture clustering with automatic model selection

Expectation-maximization (EM) algorithm (A.P. Dempster et al., 1977) has been extensively used in density mixture clustering problems, but it is unable to-perform model selection automatically. This paper, therefore, proposes to learn the model parameters via maximizing a weighted likelihood. Under a specific weight design, we give out a rival penalized expectation-maximization (RPEM) algorithm, which makes the components in a density mixture compete each other at each time step. Not only are the associated parameters of the winner updated to adapt to an input, but also all rivals' parameters are penalized with the strength proportional to the corresponding posterior density probabilities. Compared to the EM algorithm (A.P. Dempster et al., 1977), the RPEM is able to fade out the redundant densities from a density mixture during the learning process. Hence, it can automatically select an appropriate number of densities in density mixture clustering. We experimentally demonstrate its outstanding performance on Gaussian mixtures and color image segmentation problem. Moreover, a simplified version of RPEM generalizes our recently proposed RPCCL algorithm (Y.M. Cheung, 2002) so that it is applicable to elliptical clusters as well with any input proportion. Compared to the existing heuristic RPCL (L. Xu et al., 1993) and its variants, this generalized RPCCL (G-RPCCL) circumvents the difficult preselection of the so-called delearning rate. Additionally, a special setting of the G-RPCCL not only degenerates to RPCL and its Type A variant, but also gives a guidance to choose an appropriate delearning rate for them. Subsequently, we propose a stochastic version of RPCL and its type A variant, respectively, in which the difficult selection problem of delearning rate has been novelly circumvented. The experiments show the promising results of this stochastic implementation.

Automatic model selection for the optimization of SVM kernels

This approach aims to optimize the kernel parameters and to efficiently reduce the number of support vectors, so that the generalization error can be reduced drastically. The proposed methodology suggests the use of a new model selection criterion based on the estimation of the probability of error of the SVM classifier. For comparison, we considered two more model selection criteria: GACV (‘Generalized Approximate Cross-Validation’) and VC (‘Vapnik-Chernovenkis’) dimension. These criteria are algebraic estimates of upper bounds of the expected error. For the former, we also propose a new minimization scheme. The experiments conducted on a bi-class problem show that we can adequately choose the SVM hyper-parameters using the empirical error criterion. Moreover, it turns out that the criterion produces a less complex model with fewer support vectors. For multi-class data, the optimization strategy is adapted to the one-against-one data partitioning. The approach is then evaluated on images of handwritten digits from the USPS database.

Integrating fault detection and isolation with model predictive control

AbstractThis paper illustrates how the application of partial least squares (PLS) can be extended to provide an integrated solution to fault detection and isolation, inferential estimation and model predictive control. It is shown that if PLS is used to identify a dynamic model of a plant then the latent variables of the model can identify the suitability of using this model under current conditions. This functionality enables automated model switching in piecewise linear systems. A further advantage of the proposed technique is that the inner structure of the model can be used to provide fault detection and isolation capabilities. By extending the approach to control systems and integrating a dynamic model, identified using the PLS algorithm, within a model predictive controller, similar benefits, such as automatic model selection can be achieved for the control system. The proposed approach is illustrated through its application to the Tennessee Eastman challenge process. Copyright © 2004 John Wiley & Sons, Ltd.

An environmental decision‐support system for the management of water pollution in a tidal river network

This paper is about the development of a decision‐support system for water‐pollution management and environmental planning. More specifically, the paper first presents the overall concept and the system architecture of a generic environmental decision‐support system (EDSS) and then develops an EDSS especially for analysing the tidal flow pattern and water quality of China's Pearl River Delta. The EDSS developed here employs the object‐oriented approach to design the environmental database and utilizes the system integration technology to develop the overall user‐friendly system that operates in the Windows environment. Furthermore, the system can be expanded to facilitate automated model selection and analysis. The EDSS should be of value for managing water quality of river networks with complicated flow patterns, such as that found in the Pearl River Delta.

ENTROPY PENALIZED AUTOMATED MODEL SELECTION ON GAUSSIAN MIXTURE

Gaussian mixture modeling is a powerful approach for data analysis and the determination of the number of Gaussians, or clusters, is actually the problem of Gaussian mixture model selection which has been investigated from several respects. This paper proposes a new kind of automated model selection algorithm for Gaussian mixture modeling via an entropy penalized maximum-likelihood estimation. It is demonstrated by the experiments that the proposed algorithm can make model selection automatically during the parameter estimation, with the mixing proportions of the extra Gaussians attenuating to zero. As compared with the BYY automated model selection algorithms, it converges more stably and accurately as the number of samples becomes large.

ON DIFFERENT MODEL SELECTION CRITERIA IN A FORWARD AND BACKWARD REGRESSION HYBRID NETWORK

An assessment of the performance of a hybrid network with different model selection criteria is considered. These criteria are used in an automatic model selection algorithm for constructing a hybrid network of radial and perceptron hidden units for regression. A forward step builds the full hybrid network; a model selection criterion is used for controlling the network-size and another criterion is used for choosing the appropriate hidden unit for different regions of input space. This is followed by a conservative pruning step using Likelihood Ratio Test, Bayesian or Minimum Description Length, which leads to robust estimators with low variance. The result is a small architecture that performs well on difficult, realistic, benchmark data-sets of high dimensionality and small training size. Best results are obtained by using the Bayesian approach for the model selection.

Automatic model selection: a new instrument for social science

The properties of automatic model selection are discussed, focusing on PcGets. We explain the background concepts and why automatic methods can perform well. Criticisms of model selection procedures are noted and rebutted. The algorithm is sketched, distinguishing between costs of search and costs of inference: the latter are unavoidable in any statistical science, whereas the costs of searching seem small in comparison. The choice of a ‘search strategy’ and the actual simulation performance of the approach are discussed. We outline a number of developments that will improve the behavior, and generalize the scope, of such programs, and tackle hitherto intractable problems.

Bootstrap Methods for Developing Predictive Models

Researchers frequently use automated model selection methods such as backwards elimination to identify variables that are independent predictors of an outcome under consideration. We propose using bootstrap resampling in conjunction with automated variable selection methods to develop parsimonious prediction models. Using data on patients admitted to hospital with a heart attack, we demonstrate that selecting those variables that were identified as independent predictors of mortality in at least 60%% of the bootstrap samples resulted in a parsimonious model with excellent predictive ability.

Automatic Inference of the Contemporaneous Causal Order of a System of Equations

David Hendry and Hans-Martin Krolzig have demonstrated that PCGets, an automatic model selection algorithm that implements general-to-specific search procedures, can be successfully applied to the individual equations of vector autoregressions (VARs), provided that the contemporaneous causal order is known so that the covariance matrix of the VAR is diagonal. Graph-theoretic causal-search algorithms provide a means of determining the contemporaneous causal order of a VAR needed before general-to-specific search procedures can be applied. This paper explains the basis for those algorithms with special reference to VARs.

Integrated Spectral and Spatial Information Mining in Remote Sensing Imagery

Most existing remote sensing image retrieval systems allow only simple queries based on sensor, location, and date of image capture. This approach does not permit the efficient retrieval of useful hidden information from large image databases. This paper presents an integrated approach to retrieving spectral and spatial patterns from remotely sensed imagery using state-of-the-art data mining and advanced database technologies. Land cover information corresponding to spectral characteristics is identified by supervised classification based on support vector machines with automatic model selection, while textural features characterizing spatial information are extracted using Gabor wavelet coefficients. Within identified land cover categories, textural features are clustered to acquire search-efficient space in an object-oriented database with associated images in an image database. Interesting patterns are then retrieved using a query-by-example approach. The evaluation of the study results using coverage and novelty measures validates the effectiveness of the proposed remote sensing image information mining framework, which is potentially useful for applications such as agricultural and environmental monitoring.

A gradient BYY harmony learning rule on Gaussian mixture with automated model selection

One important feature of Bayesian Ying–Yang (BYY) harmony learning is that model selection can be made automatically during parametric learning. In this paper, BYY harmony learning with a bi-directional architecture is studied for Gaussian mixture modelling via a gradient learning rule. It has been demonstrated by simulation experiments that the number of Gaussians can be determined automatically during learning the parameters of the Gaussian mixture.

A Finite Sample Comparison of Automatic Model Selection Methods

Three automatic model selection procedures are compared: The first one is the pem subroutine implemented in the MATLAB toolbox system identification. The second procedure is based on the CCA subspace procedure. The third procedure finally is the ARMAsel procedure. The three procedures have heen studied using a simulation experiment. The accuracy of the subspace algorithms depends heavily on the choice of the integers f and p. Simple strategies for choosing f and p can yield very inaccurate models. For one simulation example, it is shown that a more complex strategy can yield a more accurate result. The ARMAsel algorithm provided an accurate model for all processes that have been considered.

A Bayesian neural network approach for modelling censored data with an application to prognosis after surgery for breast cancer

A Bayesian framework is introduced to carry out Automatic Relevance Determination (ARD) in feedforward neural networks to model censored data. A procedure to identify and interpret the prognostic group allocation is also described. These methodologies are applied to 1616 records routinely collected at Christie Hospital, in a monthly cohort study with 5-year follow-up. Two cohort studies are presented, for low- and high-risk patients allocated by standard clinical staging. The results of contrasting the Partial Logistic Artificial Neural Network (PLANN)–ARD model with the proportional hazards model are that the two are consistent, but the neural network may be more specific in the allocation of patients into prognostic groups. With automatic model selection, the regularised neural network is more conservative than the default stepwise forward selection procedure implemented by SPSS with the Akaike Information Criterion.