Automatic Model Selection Research Articles

Automatic model selection for high-dimensional survival analysis

Many different models for the analysis of high-dimensional survival data have been developed over the past years. While some of the models and implementations come with an internal parameter tuning automatism, others require the user to accurately adjust defaults, which often feels like a guessing game. Exhaustively trying out all model and parameter combinations will quickly become tedious or infeasible in computationally intensive settings, even if parallelization is employed. Therefore, we propose to use modern algorithm configuration techniques, e.g. iterated F-racing, to efficiently move through the model hypothesis space and to simultaneously configure algorithm classes and their respective hyperparameters. In our application we study four lung cancer microarray data sets. For these we configure a predictor based on five survival analysis algorithms in combination with eight feature selection filters. We parallelize the optimization and all comparison experiments with the BatchJobs and BatchExperiments R packages.

On intermittent demand model optimisation and selection

Intermittent demand time series involve items that are requested infrequently, resulting in sporadic demand. Croston׳s method and its variants have been proposed in the literature to address this forecasting problem. Recently other novel methods have appeared. Although the literature provides guidance on the suggested range for model parameters, a consistent and valid optimisation methodology is lacking. Growing evidence in the literature points against the use of conventional accuracy error metrics for model evaluation for intermittent demand time series. Consequently these may be inappropriate for parameter or model selection. This paper contributes to the discussion by evaluating a series of conventional time series error metrics, along with two novel ones for parameter optimisation for intermittent demand methods. The proposed metrics are found to not only perform best, but also provide consistent parameters with the literature, in contrast to conventional metrics. Furthermore, this work validates that employing different parameters for smoothing the non-zero demand and the inter-demand intervals of Croston׳s method and its variants is beneficial. The evaluated error metrics are considered for automatic model selection for each time series. Although they are found to perform similar to theory driven model selection schemes, they fail to outperform single models substantially. These findings are validated using both out-of-sample forecast evaluation and inventory simulations.

Explaining commodity prices by a cointegrated time series-cross section model

This paper analyzes idiosyncratic and common determinants of commodity prices by adopting a time series-cross section approach. Different from the previous empirical literature, we allow for long-run and short-run effects, testing for cointegration and accounting for cross-dependence among different commodities. An automatic model selection algorithm is used to obtain a dominant congruent econometric model, selecting among many potential explanatory variables. As the effects of commodity prices’ determinants may vary over time and across commodities and we are estimating single conditional models, poolability and exogeneity issues are also evaluated. Our results indicate that commodity price formation in the long run is determined by supply and demand factors, apart from the US exchange rate. We find significant effects of individual commodity production and the demand-pull of China’s economy. The economic growth of both emerging and developed countries as well as the dollar’s depreciation, inventory changes, and easier monetary policies also have significant short-run effects on real commodity prices.

Learning local factor analysis versus mixture of factor analyzers with automatic model selection

Considering Factor Analysis (FA) for each component of Gaussian Mixture Model (GMM), clustering and local dimensionality reduction can be addressed simultaneously by Mixture of Factor Analyzers (MFA) and Local Factor Analysis (LFA), which correspond to two FA parameterizations, respectively. This paper investigates the performance of Variational Bayes (VB) and Bayesian Ying-Yang (BYY) harmony learning on MFA/LFA for the problem of automatically determining the component number and the local hidden dimensionalities (i.e., the number of factors of FA in each component). Similar to the existing VB learning algorithm on MFA, we develop an alternative VB algorithm on LFA with a similar conjugate Dirichlet–Normal–Gamma (DNG) prior on all parameters of LFA. Also, the corresponding BYY algorithms are developed for MFA and LFA. A wide range of synthetic experiments shows that LFA is superior to MFA in model selection under either VB or BYY, while BYY outperforms VB reliably on both MFA and LFA. These empirical findings are consistently observed from real applications on not only face and handwritten digit images clustering, but also unsupervised image segmentation.

Fully automated shape model positioning for bone segmentation in whole-body CT scans

Analysing osteolytic and osteoblastic bone lesions in systematically affected skeletons, e.g. in multiple myeloma or bone metastasis, is a complex task. Quantification of the degree of bone destruction needs segmentation of all lesions but cannot be managed manually. Automatic bone lesion detection is necessary. Our future objective is comparing modified bones with healthy shape models. For applying model based strategies successfully, identification and position information of single bones is necessary. A solution to these requirements based on bone medullary cavities is presented in this paper.Medullary cavities are useful for shape model positioning since they have similar position and orientation as the bone itself but can be separated more easily. Skeleton segmentation is done by simple thresholding. Inside the skeleton medullary cavities are segmented by a flood filling algorithm. The filled regions are considered as medullary cavity objects. To provide automatic shape model selection, medullary cavity objects are assigned to bone structures with pattern recognition. To get a good starting position for shape models, principal component analysis of medullary cavities is performed. Bone identification was tested on 14 whole-body low-dose CT scans of multiple myeloma patients.Random forest classification assigns medullary cavities of long bones to the corresponding bone (overall accuracy 90%). Centroid and first principal component of medullary cavity are sufficiently similar to those of bone (mean centroid difference 21.7 mm, mean difference angle 1.54° for all long bones of one example patient) and therefore suitable for shape model initialization.This method enables locating long bone structures in whole-body CT scans and provides useful information for a reasonable shape model initialization.

Learning binary factor analysis with automatic model selection

Binary Factor Analysis (BFA) uncovers the independent binary information sources from observations with wide applications. BFA learning hierarchically nests three levels of inverse problems, i.e., inference of binary code for each observation, parameter estimation and model selection. Under Bayesian Ying-Yang (BYY) framework, the first level becomes an intractable Binary Quadratic Programming (BQP) problem, while model selection can be conducted automatically during parameter learning. We conduct extensive experiments to reveal that the performance order of four BQP methods is reversed from making BQP optimization to making BYY automatic model selection, which implies that learning is not merely optimization. Moreover, the BFA learning algorithm is further developed with priors over parameters to improve the performance. Finally, based on BFA, we empirically compare BYY with Variational Bayes (VB) and Bayesian information criterion (BIC).

Link and Route Travel Time Prediction Including the Corresponding Reliability in an Urban Network Based on Taxi Floating Car Data

This study addressed the modeling of route travel times (including their associated uncertainty) in urban networks based on taxi floating car data. The model decomposes observed link travel speeds into the expected speed (modeled with daily and seasonal profiles) and deviations thereof. The latter were shown to be strongly heteroscedastic by providing an explicit model for the time variance. Temporal and spatial correlations were considered with a vector autoregression framework. Modeling was supported by automatic model selection methods for identifying the relevant effects and providing one-step-ahead predictions. The potential of the proposed model was investigated with taxi floating car data from a real-world test site near the city core of Vienna, Austria. Various specifications of the vector autoregression model were tested and compared. Taxi floating car measurements of local speeds were found to be strongly heteroscedastic, a factor that must be considered in estimation of models for expected travel speeds. The modeling of the mean suggested no remaining daily or weekly patterns, and it was superior to simple models explaining travel speeds as a linear function of the travel speeds in the last time period. The variance model successfully captured heteroscedasticity. More complex models for link travel speeds, including temporal and spatial correlation, do not increase prediction accuracy consistently; the lack indicates that a sampling frequency of 15 min for floating car data in urban settings is too low for use of temporal dependencies for prediction. An introduced method for computing route travel time uncertainty showed variability over the day for a highly frequented route.

Hzar: hybrid zone analysis using an R software package

We present a new software package (HZAR) that provides functions for fitting molecular genetic and morphological data from hybrid zones to classic equilibrium cline models using the Metropolis-Hastings Markov chain Monte Carlo (MCMC) algorithm. The software applies likelihood functions appropriate for different types of data, including diploid and haploid genetic markers and quantitative morphological traits. The modular design allows flexibility in fitting cline models of varying complexity. To facilitate hypothesis testing, an autofit function is included that allows automated model selection from a set of nested cline models. Cline parameter values, such as cline centre and cline width, are estimated and may be compared statistically across clines. The package is written in the R language and is available through the Comprehensive R Archive Network (CRAN; http://cran.r-project.org/). Here, we describe HZAR and demonstrate its use with a sample data set from a well-studied hybrid zone in western Panama between white-collared (Manacus candei) and golden-collared manakins (M. vitellinus). Comparisons of our results with previously published results for this hybrid zone validate the hzar software. We extend analysis of this hybrid zone by fitting additional models to molecular data where appropriate.

Hedge fund replication: putting the pieces together

In this paper, we aim to bring together into one common framework various advances in factor-based hedge fund replication. Our replication methodology relies on a set of investable dynamic risk factors extracted from futures contract prices and on an automatic variable and model selection procedure. The methodology is then validated by creating out-of-sample replicating portfolios for the monthly returns of about 7000 hedge funds from 2006 to 2012 and under the assumption of transaction costs. Our results suggest that hedge fund replication is on average possible and works best for liquid strategies.

Simultaneous sparse model selection and coefficient estimation for heavy-tailed autoregressive processes

We propose a sparse coefficient estimation and automated model selection procedure for autoregressive processes with heavy-tailed innovations based on penalized conditional maximum likelihood. Under mild moment conditions on the innovation processes, the penalized conditional maximum likelihood estimator satisfies a strong consistency, OP(N−1/2) consistency, and the oracle properties, where N is the sample size. We have the freedom in choosing penalty functions based on the weak conditions on them. Two penalty functions, least absolute shrinkage and selection operator and smoothly clipped average deviation, are compared. The proposed method provides a distribution-based penalized inference to AR models, which is especially useful when the other estimation methods fail or under perform for AR processes with heavy-tailed innovations [Feigin, Resnick. Pitfalls of fitting autoregressive models for heavy-tailed time series. Extremes. 1999;1:391–422]. A simulation study confirms our theoretical results. At the end, we apply our method to a historical price data of the US Industrial Production Index for consumer goods, and obtain very promising results.

Genetic algorithm guided population pharmacokinetic model development for simvastatin, concurrently or non-concurrently co-administered with amlodipine

An automated model development was performed for simvastatin, co-administered with amlodipine concurrently or non-concurrently (i.e., 4 hours later) in 17 patients with coexisting hyperlipidemia and hypertension. The single objective hybrid genetic algorithm (SOHGA) was implemented in the NONMEM software by defining the search space for structural, statistical and covariate models. Candidate models obtained from the SOHGA runs were further assessed for biological plausibility and the precision of parameter estimates, followed by traditional backward elimination process for model refinement. The final population pharmacokinetic model shows that the elimination rate constant for simvastatin acid, the active form by hydrolysis of its lactone prodrug (i.e., simvastatin), is only 44% in the concurrent amlodipine administration group compared with the non-concurrent group. The application of SOHGA for automated model selection, combined with traditional model selection strategies, appears to save time for model development, which also can generate new hypotheses that are biologically more plausible.

Autonomous Neural Models for the Classification of Events in Power Distribution Networks

This paper presents a method for automatic classification of faults and transients in power distribution networks, based on voltage oscillographies of the distribution networks feeders. For signal preprocessing, the discrete wavelet transform was used with the performances of several families of wavelet functions being compared. In the classification stage, three neural models were assessed: multi-layer perceptrons, radial basis function networks, and support vector machines. The models were trained autonomously, i.e., using automatic model selection and complexity control. Promising results were obtained using a set of simulations generated using the Alternative Transients Program (ATP). Initial results obtained for real data acquired from a set of oscillograph loggers installed in a distribution network are also presented.

Spectrum-Based Kernel Length Estimation for Gaussian Process Classification

Recent studies have shown that Gaussian process (GP) classification, a discriminative supervised learning approach, has achieved competitive performance in real applications compared with most state-of-the-art supervised learning methods. However, the problem of automatic model selection in GP classification, involving the kernel function form and the corresponding parameter values (which are unknown in advance), remains a challenge. To make GP classification a more practical tool, this paper presents a novel spectrum analysis-based approach for model selection by refining the GP kernel function to match the given input data. Specifically, we target the problem of GP kernel length scale estimation. Spectrums are first calculated analytically from the kernel function itself using the autocorrelation theorem as well as being estimated numerically from the training data themselves. Then, the kernel length scale is automatically estimated by equating the two spectrum values, i.e., the kernel function spectrum equals to the estimated training data spectrum. Compared with the classical Bayesian method for kernel length scale estimation via maximizing the marginal likelihood (which is time consuming and could suffer from multiple local optima), extensive experimental results on various data sets show that our proposed method is both efficient and accurate.

Macroeconomic shocks and the probability of being employed

Macroeconomic theories take polar views on the importance of choice versus chance. At the micro level, it seems realistic to assume that both dimensions play a role for individual employment outcomes, although it might be difficult to separate these two effects. Nevertheless the choice and chance dimension are seldom treated symmetrically in models that use micro data. We estimate a logistic model of the probability of being employed among married or cohabitating women that are in the labor force. Besides variables that measure individual characteristics (choice), we allow a full set of indicator variables for observation periods that represent potential effects of aggregate shocks (chance) on job probabilities. To reduce the number of redundant indicator variables automatic model selection is used, and we assess the economic interpretation of the statistically significant indicator variables with reference to a theoretical framework that allows for friction in the Norwegian labor market. In addition, we also estimate models that use female and male unemployment rates as ‘sufficient’ variables for the chance element in individual employment outcomes. Data are for Norway for the period 1988q2–2008q4.

Improving Predictions Using Linear Combination Of Multiple Extreme Learning Machines

This work presents several effective approaches for linear combination of multiple artificial neural networks based on the extreme learning machine (ELM) algorithm. Given a learning task, a large set of neural networks are firstly trained by ELM. Then, these trained machines are efficiently ranked and the useless models are effectively discarded in order to provide an ensemble system with better generalization performance. The ensemble system is constructed using an automatic and fast forward model selection by minimizing the leave-one-out error, without user intervention. Experiments on an artificial regression dataset and three real-world engineering problems are discussed. According to the obtained results, the weighted linear combination of ELMs improves predictions by exploiting model diversity in the ensemble system with fast learning speed. DOI: http://dx.doi.org/10.5755/j01.itc.42.1.1667

Automatic selection of a spatial weight matrix in spatial econometrics: Application to a spatial hedonic approach

The recent progress of spatial econometrics has developed a new technique called the “spatial hedonic approach,” which considers the elements of spatial autocorrelation among property values and geographically distributed attributes. The practical difficulties in applying spatial econometric models include the specification of the spatial weight matrix (SWM), which affects the final analysis results. Some simulation studies suggest that information criteria such as AIC are useful for the SWM's selection, but if many model candidates exist (e.g., when the selections of explanatory variables are performed simultaneously), then the computational burden of calculating such criteria for each model is large. The present study develops an automatic model selection algorithm using the technique of reversible jump MCMC combined with simulated annealing; termed trans-dimensional simulated annealing (TDSA). The performance of the TDSA algorithm is verified using the well-known Boston housing dataset, and it is applied empirically to a Japanese real estate dataset. The obtained results suggest a two-step strategy for model selection, with SWM (W) first, followed by the explanatory variables (X and WX), will result in local optima, and therefore these variables should be selected simultaneously. The TDSA algorithm can find the significant variables that are “hidden” because of multicollinearity in the unrestricted model, and can attain the minimum AIC automatically.

Hedge Fund Replication: Putting the Pieces Together

In this paper, we aim to bring together into one common framework various advances in factor-based hedge fund replication. Our replication methodology relies on a set of investable dynamic risk factors extracted from futures contract prices and on an automatic variable and model selection procedure. The methodology is then validated by creating out-of-sample replicating portfolios for the monthly returns of more than 7,000 hedge funds ranging from 2006 to 2012 and under the assumption of transaction costs. Our results suggest that hedge fund replication is on average possible and works best for liquid strategies.

Model Selection of Hammerstein System Nonlinearity under Heavy Noise

Abstract The paper deals with the problem of automatic model selection of the nonlinear characteristic in a block-oriented dynamic system. We look for the parametric model of Hammerstein system nonlinearity. From the finite set of candidate classes of parametric models we select the best one on the basis of the input-output measurement data, using the concept of nearest neighbour borrowed from pattern recognitions techniques. The algorithm uses the pattern of the true characteristic generated by its nonparametric estimates on the grid of fixed (e.g. equidistant) points. Each class generates parametrized learning sequence through the values on the same grid of points. Next, for each class, the optimal parameters are computed by the least squares method. Finally, the nearest neighbour approach is applied for the selection of the best model in the mean square sense. The idea is presented on the exemplary competition between polynomial, exponential and piece-wise linear models of the same complexity (i.e. number of parameters needed to be stored in memory). For all classes, the upper bounds of the integrated approximation errors of the true characteristic are computed and compared.

Derivative estimation with local polynomial fitting

We present a fully automated framework to estimate derivatives nonparametrically without estimating the regression function. Derivative estimation plays an important role in the exploration of structures in curves (jump detection and discontinuities), comparison of regression curves, analysis of human growth data, etc. Hence, the study of estimating derivatives is equally important as regression estimation itself. Via empirical derivatives we approximate the qth order derivative and create a new data set which can be smoothed by any nonparametric regression estimator. We derive L1 and L2 rates and establish consistency of the estimator. The new data sets created by this technique are no longer independent and identically distributed (i.i.d.) random variables anymore. As a consequence, automated model selection criteria (data-driven procedures) break down. Therefore, we propose a simple factor method, based on bimodal kernels, to effectively deal with correlated data in the local polynomial regression framework.

Bayesian Selection of Nucleotide Substitution Models and Their Site Assignments

Probabilistic inference of a phylogenetic tree from molecular sequence data is predicated on a substitution model describing the relative rates of change between character states along the tree for each site in the multiple sequence alignment. Commonly, one assumes that the substitution model is homogeneous across sites within large partitions of the alignment, assigns these partitions a priori, and then fixes their underlying substitution model to the best-fitting model from a hierarchy of named models. Here, we introduce an automatic model selection and model averaging approach within a Bayesian framework that simultaneously estimates the number of partitions, the assignment of sites to partitions, the substitution model for each partition, and the uncertainty in these selections. This new approach is implemented as an add-on to the BEAST 2 software platform. We find that this approach dramatically improves the fit of the nucleotide substitution model compared with existing approaches, and we show, using a number of example data sets, that as many as nine partitions are required to explain the heterogeneity in nucleotide substitution process across sites in a single gene analysis. In some instances, this improved modeling of the substitution process can have a measurable effect on downstream inference, including the estimated phylogeny, relative divergence times, and effective population size histories.