Model Selection Issues Research Articles

Weibull mixture estimation based on censored data with applications to clustering in reliability engineering

AbstractIt is the purpose of this paper to propose a novel clustering technique tailored to randomly censored data in reliability/survival analysis. It is based on an underlying mixture model of Weibull distributions and consists in estimating its parameters by means of a variant of the Expectation–Maximization method in the presence of random censorship. Beyond the description of the algorithm, model selection issues are addressed and we investigate its performance from an empirical perspective by applying it to possibly strongly censored (synthetic and real) survival data. The experiments carried out provide strong empirical evidence that our algorithm performs better than alternative methods standing as natural competitors in this framework.

Bayesian Forecasting of Mortality Rates for Small Areas Using Spatiotemporal Models

Estimation and prediction of subnational mortality rates for small areas are essential planning tools for studying health inequalities. Standard methods do not perform well when data are noisy, a typical behavior of subnational datasets. Thus, reliable estimates are difficult to obtain. I present a Bayesian hierarchical model framework for prediction of mortality rates at a small or subnational level. By combining ideas from demography and epidemiology, the classical mortality modeling framework is extended to include an additional spatial component capturing regional heterogeneity. Information is pooled across neighboring regions and smoothed over time and age. To make predictions more robust and address the issue of model selection, a Bayesian version of stacking is considered using leave-future-out validation. I apply this method to forecast mortality rates for 96 regions in Bavaria, Germany, disaggregated by age and sex. Uncertainty surrounding the forecasts is provided in terms of prediction intervals. Using posterior predictive checks, I show that the models capture the essential features and are suitable to forecast the data at hand. On held-out data, my predictions outperform those of standard models lacking a regional component.

Forecasting GDP growth: The economic impact of COVID‐19 pandemic

AbstractThe primary goal of this study is to effectively measure the impact of a severe random shock, such as the COVID‐19 pandemic on aggregate economic activity in Greece, seven other euro area economies, three Scandinavian countries, and the United States. The class of linear and quantile predictive regression models is proposed for the analysis of real gross domestic product (GDP) growth, and a Bayesian approach for model selection is developed, by using a computationally flexible Markov chain Monte Carlo stochastic search algorithm that explores the posterior distribution of linear and quantile models, and identifies the relevant predictor variables. Penalized likelihood regression models are also implemented to tackle the issue of model selection. The model confidence set approach is applied and verifies that the selected models identified by the stochastic search algorithm belong to the set of superior models. Our analysis confirms that the outbreak of the pandemic had a profound effect on the economies under study, and reveals that different predictor variables are able to explain different quantiles of the underlying real GDP growth distribution for analyzed countries, suggesting that the quantile modeling approach improves the ability to adequately explain real GDP series compared with the standard conditional mean approach that explains only the average of the relationship between real GDP growth and several predictor variables.

Brainomaly: Unsupervised Neurologic Disease Detection Utilizing Unannotated T1-weighted Brain MR Images.

Harnessing the power of deep neural networks in the medical imaging domain is challenging due to the difficulties in acquiring large annotated datasets, especially for rare diseases, which involve high costs, time, and effort for annotation. Unsupervised disease detection methods, such as anomaly detection, can significantly reduce human effort in these scenarios. While anomaly detection typically focuses on learning from images of healthy subjects only, real-world situations often present unannotated datasets with a mixture of healthy and diseased subjects. Recent studies have demonstrated that utilizing such unannotated images can improve unsupervised disease and anomaly detection. However, these methods do not utilize knowledge specific to registered neuroimages, resulting in a subpar performance in neurologic disease detection. To address this limitation, we propose Brainomaly, a GAN-based image-to-image translation method specifically designed for neurologic disease detection. Brainomaly not only offers tailored image-to-image translation suitable for neuroimages but also leverages unannotated mixed images to achieve superior neurologic disease detection. Additionally, we address the issue of model selection for inference without annotated samples by proposing a pseudo-AUC metric, further enhancing Brainomaly's detection performance. Extensive experiments and ablation studies demonstrate that Brainomaly outperforms existing state-of-the-art unsupervised disease and anomaly detection methods by significant margins in Alzheimer's disease detection using a publicly available dataset and headache detection using an institutional dataset. The code is available from https://github.com/mahfuzmohammad/Brainomaly.

An integrated system to significant wave height prediction: Combining feature engineering, multi-criteria decision making, and hybrid kernel density estimation

Accurate prediction of significant wave height is paramount for the effective design, operation, and maintenance of wave energy converters. However, current research falls short in achieving precise and stable point predictions, along with comprehensive uncertainty analysis of significant wave height. To address this gap, this study presents a comprehensive significant wave height combined prediction system. This integrated system encompasses outlier detection utilizing Autoencoders, sophisticated feature engineering, a multi-criteria decision-based model selection methodology, a multi-objective homogeneous nuclear molecular optimization algorithm, and a hybrid kernel density estimation technique. To tackle the critical issue of model selection within ensemble prediction, we introduce a multi-criteria compromise solution ranking algorithm known as VIKOR for the selection of sub-models. Additionally, a novel multi-objective homogeneous nuclear molecular optimization algorithm is proposed, which incorporates joint opposing selection and an elite retention strategy to effectively manage multiple objectives simultaneously, yielding Pareto optimal solutions for combining weights. Furthermore, a hybrid kernel density estimation approach is developed, surpassing previous methods reliant on a single kernel function and fixed bandwidth, thereby achieving a more precise fit to the distribution of wave height data. The effectiveness of the proposed model is rigorously evaluated using wave height datasets from three distinct locations. The experimental results convincingly demonstrate that the significant wave height combined prediction system outperforms existing solutions, excelling in both point and interval predictions of significant wave height.

Description length guided nonlinear unified Granger causality analysis.

Most Granger causality analysis (GCA) methods still remain a two-stage scheme guided by different mathematical theories; both can actually be viewed as the same generalized model selection issues. Adhering to Occam's razor, we present a unified GCA (uGCA) based on the minimum description length principle. In this research, considering the common existence of nonlinearity in functional brain networks, we incorporated the nonlinear modeling procedure into the proposed uGCA method, in which an approximate representation of Taylor's expansion was adopted. Through synthetic data experiments, we revealed that nonlinear uGCA was obviously superior to its linear representation and the conventional GCA. Meanwhile, the nonlinear characteristics of high-order terms and cross-terms would be successively drowned out as noise levels increased. Then, in real fMRI data involving mental arithmetic tasks, we further illustrated that these nonlinear characteristics in fMRI data may indeed be drowned out at a high noise level, and hence a linear causal analysis procedure may be sufficient. Next, involving autism spectrum disorder patients data, compared with conventional GCA, the network property of causal connections obtained by uGCA methods appeared to be more consistent with clinical symptoms.

Technique for Automatic Algorithm Selection for Computational Intelligence in Cloud-Based Computing Environments

This research presents an Auto-Algorithm selection technique for Computational Intelligence in Cloud-Based computing environment. By leveraging the capabilities of a network information processing platform, users can effortlessly and intelligently build Computational Intelligence models tailored to their specific problems without the need for manually configuring the Computational Intelligence environment, selecting algorithms, or adjusting complex Computational Intelligence functions and parameters. The proposed procedure allows users to simply upload sample data through a web interface, freeing Computational Intelligence applications from environmental constraints and taking advantage of the network information processing platform's capabilities. This approach transparently handles the model building process, significantly reducing the barriers to entry for utilizing Computational Intelligence. By addressing the issues of unpredictable model selection, manual parameter adjustment, and the challenges faced by common users, this auto selection procedure empowers the practical application of Computational Intelligence in various domains.

A hybrid framework for sequential data prediction with end-to-end optimization

We investigate nonlinear prediction in an online setting and introduce a hybrid model that effectively mitigates, via an end-to-end architecture, the need for hand-designed features and manual model selection issues of conventional nonlinear prediction/regression methods. In particular, we use recursive structures to extract features from sequential signals, while preserving the state information, i.e., the history, and boosted decision trees to produce the final output. The connection is in an end-to-end fashion and we jointly optimize the whole architecture using stochastic gradient descent, for which we also provide the backward pass update equations. In particular, we employ a recurrent neural network (LSTM) for adaptive feature extraction from sequential data and a gradient boosting machinery (soft GBDT) for effective supervised regression. Our framework is generic so that one can use other deep learning architectures for feature extraction (such as RNNs and GRUs) and machine learning algorithms for decision making as long as they are differentiable. We demonstrate the learning behavior of our algorithm on synthetic data and the significant performance improvements over the conventional methods over various real life datasets. Furthermore, we openly share the source code of the proposed method to facilitate further research.

Estimation of Error Variance in Regularized Regression Models via Adaptive Lasso

Estimation of error variance in a regression model is a fundamental problem in statistical modeling and inference. In high-dimensional linear models, variance estimation is a difficult problem, due to the issue of model selection. In this paper, we propose a novel approach for variance estimation that combines the reparameterization technique and the adaptive lasso, which is called the natural adaptive lasso. This method can, simultaneously, select and estimate the regression and variance parameters. Moreover, we show that the natural adaptive lasso, for regression parameters, is equivalent to the adaptive lasso. We establish the asymptotic properties of the natural adaptive lasso, for regression parameters, and derive the mean squared error bound for the variance estimator. Our theoretical results show that under appropriate regularity conditions, the natural adaptive lasso for error variance is closer to the so-called oracle estimator than some other existing methods. Finally, Monte Carlo simulations are presented, to demonstrate the superiority of the proposed method.

Model selection in time series analysis: using information criteria as an alternative to hypothesis testing

PurposeThe issue of model selection in applied research is of vital importance. Since the true model in such research is not known, which model should be used from among various potential ones is an empirical question. There might exist several competitive models. A typical approach to dealing with this is classic hypothesis testing using an arbitrarily chosen significance level based on the underlying assumption that a true null hypothesis exists. In this paper, the authors investigate how successful the traditional hypothesis testing approach is in determining the correct model for different data generating processes using time series data. An alternative approach based on more formal model selection techniques using an information criterion or cross-validation is also investigated.Design/methodology/approachMonte Carlo simulation experiments on various generating processes are used to look at the response surfaces resulting from hypothesis testing and response surfaces resulting from model selection based on minimizing an information criterion or the leave-one-out cross-validation prediction error.FindingsThe authors find that the minimization of an information criterion can work well for model selection in a time series environment, often performing better than hypothesis-testing strategies. In such an environment, the use of an information criterion can help reduce the number of models for consideration, but the authors recommend the use of other methods also, including hypothesis testing, to determine the appropriateness of a model.Originality/valueThis paper provides an alternative approach for selecting the best potential model among many for time series data. It demonstrates how minimizing an information criterion can be useful for model selection in a time-series environment in comparison to some standard hypothesis testing strategies.

On the Quantification of Model Uncertainty: A Bayesian Perspective.

Issues of model selection have dominated the theoretical and applied statistical literature for decades. Model selection methods such as ridge regression, the lasso, and the elastic net have replaced ad hoc methods such as stepwise regression as a means of model selection. In the end, however, these methods lead to a single final model that is often taken to be the model considered ahead of time, thus ignoring the uncertainty inherent in the search for a final model. One method that has enjoyed a long history of theoretical developments and substantive applications, and that accounts directly for uncertainty in model selection, is Bayesian model averaging (BMA). BMA addresses the problem of model selection by not selecting a final model, but rather by averaging over a space of possible models that could have generated the data. The purpose of this paper is to provide a detailed and up-to-date review of BMA with a focus on its foundations in Bayesian decision theory and Bayesian predictive modeling. We consider the selection of parameter and model priors as well as methods for evaluating predictions based on BMA. We also consider important assumptions regarding BMA and extensions of model averaging methods to address these assumptions, particularly the method of Bayesian stacking. Simple empirical examples are provided and directions for future research relevant to psychometrics are discussed.

Decline in body condition in the Antarctic minke whale (Balaenoptera bonaerensis) in the Southern Ocean during the 1990s

Changes in the body condition of Antarctic minke whales (Balaenoptera bonaerensis) have been investigated in a number of studies, but remain contested. Here we provide a new analysis of body condition measurements, with particularly careful attention to the statistical model building and to model selection issues. We analyse body condition data for a large number (4704) of minke whales caught between 1987 and 2005. The data consist of five different variables related to body condition (fat weight, blubber thickness and girth) and a number of temporal, spatial and biological covariates. The body condition variables are analysed using linear mixed-effects models, for which we provide sound biological motivation. Further, we conduct model selection with the focused information criterion (FIC), reflecting the fact that we have a clearly specified research question, which leads us to a clear focus parameter of particular interest. We find that there has been a substantial decline in body condition over the study period (the net declines are estimated to 10% for fat weight, 7% for blubber thickness and 3% for the girth). Interestingly, there seems to be some differences in body condition trends between males and females and in different regions of the Antarctic. The decline in body condition could indicate major changes in the Antarctic ecosystem, in particular, increased competition from some larger krill-eating whale species.

Realising transfer learning through convolutional neural network and support vector machine for mental task classification

Electroencephalogram (EEG) measures brainwaves that have been the widely used modality for brain–computer interface (BCI) applications. EEG signal with machine learning has gained substantial success in the BCI. However, the availability of limited training data, appropriate model selection and high false-positive rates are the challenges that need immediate attention. Therefore, in this Letter, the authors present a mental task classification model based on the notion of transfer learning that addresses the issue of data scarcity, model selection and misclassification ratio. In the framework, the proposed model uses pre-trained network for the extraction of diverse feature and classify using support vector machine. The authors employed four pre-trained networks to identify the optimal network for the proposed framework: Vgg16, Vgg19, Resnet18 and Resnet50. The highest classification accuracy of 86.85% (using Resnet50) was achieved using transfer learning. Comparison results showed that convolutional neural network-based approach outperformed conventional machine learning approaches and hence it can be concluded that the EEG-based classification of the mental task using transfer learning model could be used in developing a superior model despite the limited data availability.

Galaxy Zoo Builder: Four-component Photometric Decomposition of Spiral Galaxies Guided by Citizen Science

Abstract Multicomponent modeling of galaxies is a valuable tool in the effort to quantitatively understand galaxy evolution, yet the use of the technique is plagued by issues of convergence, model selection, and parameter degeneracies. These issues limit its application over large samples to the simplest models, with complex models being applied only to very small samples. We attempt to resolve this dilemma of “quantity or quality” by developing a novel framework, built inside the Zooniverse citizen-science platform, to enable the crowdsourcing of model creation for Sloan Digital Sky Survey galaxies. We have applied the method, including a final algorithmic optimization step, on a test sample of 198 galaxies, and examine the robustness of this new method. We also compare it to automated fitting pipelines, demonstrating that it is possible to consistently recover accurate models that either show good agreement with, or improve on, prior work. We conclude that citizen science is a promising technique for modeling images of complex galaxies, and release our catalog of models.

Focused model selection for linear mixed models with an application to whale ecology

A central point of disagreement, in certain long-standing discussions about a particular whaling dataset in the Scientific Committee of the International Whaling Commission, has directly involved model selection issues for linear mixed effect models. The biological question under discussion is associated with a clearly defined parameter of primary interest, a focus parameter, which makes model selection with the Focused Information Criterion (FIC) more appropriate than other selection methods. Since the existing FIC methodology has not covered the case of linear mixed effects models, this article sets up the required framework and develops the necessary formulae for the relevant FIC. Our new criterion requires the asymptotic distribution of estimators derived for a given candidate linear mixed model but with behaviour examined under a wider linear mixed model. These results, needed here to build our FIC, also have independent interest.

New model averaging approach in predicting mortality rate of intensive care unit patients

Model Averaging (MA) is one of the well-known statistical modelling approach to produce a fitted model in applied research. Even though it was proposed to overcome underestimation of parameter estimates issues in Model Selection (MS), the final best model of MA includes insignificant variables. The goal of this research is to propose an New Model Averaging (NMA) method which is based on MA approach with elimination of insignificant variables. Data of Intensive Care Unit (ICU) was studied to highlight the most influential factor of mortality rate. ICU is commonly associated with a high mortality rate due of its complexity of treatments. The guidelines of NMA method on ICU patient’s data were presented and the models obtain were compared using 10-Fold Cross-Validation. The results reveals that the performance of NMA is slightly better than MA. The most significant factors for mortality of ICU patients were concluded to be patient’s age, SAPS II score discharge and whether or not the patients use ventilation machine. In conclusion, the study showed that the elderly patients have a greater risk of mortality after discharge from the hospital and SAPS II score provide a good indication in predicting hospital mortality.

Microeconometric dynamic panel data methods: Model specification and selection issues

A motivated strategy is presented to find step by step an adequate model specification and a matching set of instrumental variables by applying the programming tools provided by the Stata package Xtabond2. The aim is to implement generalized method of moment techniques such that useful and reasonably accurate inferences can be extracted from an observational panel data set on a single microeconometric structural presumably dynamic behavioral relationship. In the suggested specification search three comprehensive heavily interconnected goals are pursued: (i) to include all the relevant appropriately transformed possibly lagged regressors, as well as any interactions between these, if it is required to relax the otherwise very strict homogeneity restrictions on the dynamic impacts of the explanatories in standard linear panel data models; (ii) to correctly classify all regressors as either endogenous, predetermined or exogenous, as well as being either effect-stationary or effect-nonstationary, implying which internal variables could represent valid and relatively strong instruments; (iii) to enhance the accuracy of inference in finite samples by omitting redundant regressors and by profitably reducing the space spanned by the full set of available internal instruments. For the various tests which trigger the decisions to be made in the sequential selection process the relevant considerations are spelled out to interpret the magnitude of p-values. Also the complexities to estimate and interpret the ultimately established dynamic impacts are explained. Finally the developed strategy is applied to a classic data set and is shown to yield new insights.

Compulsory Schooling and Returns to Education: A Re-Examination

This paper re-examines the instrumental variable (IV) approach to estimating returns to education by use of compulsory school law (CSL) in the US. We show that the IV-approach amounts to a change in model specification by changing the causal status of the variable of interest. From this perspective, the IV-OLS (ordinary least square) choice becomes a model selection issue between non-nested models and is hence testable using cross validation methods. It also enables us to unravel several logic flaws in the conceptualisation of IV-based models. Using the causal chain model specification approach, we overcome these flaws by carefully distinguishing returns to education from the treatment effect of CSL. We find relatively robust estimates for the first effect, while estimates for the second effect are hindered by measurement errors in the CSL indicators. We find reassurance of our approach from fundamental theories in statistical learning.

An Empirical Study on Distributed Bayesian Approximation Inference of Piecewise Sparse Linear Models

The importance of interpretability of machine learning models has been increasing due to emerging enterprise predictive analytics. Piecewise linear models have been actively studied to achieve both accuracy and interpretability. They often produce competitive accuracy against state-of-the-art non-linear methods. In addition, their representations (i.e., rule-based segmentation plus sparse linear formula) are often preferred by domain experts. A disadvantage of such models, however, is high computational cost for simultaneous determinations of the number of “pieces” and cardinality of each linear predictor, which has restricted their applicability to middle-scale data sets. This paper discusses an empirical study on the derivation of a distributed factorized asymptotic Bayesian (FAB) inference of learning piece-wise sparse linear models on distributed memory architectures from the original FAB inference algorithm. The distributed FAB inference solves the simultaneous model selection issue without communicating O(N) data where N is the number of training samples and achieves linear scale-out against the number of CPU cores. Experimental results demonstrate that the distributed FAB inference achieves high prediction accuracy and performance scalability with both synthetic and public benchmark data.

Observational constraints in nonlocal gravity: the Deser-Woodard case

We study the cosmology of a specific class of nonlocal model of modified gravity, the so-called Deser-Woodard (DW) model, modifying the Einstein-Hilbert action by a term ∼ R f(−1R), where f is a free function. Choosing f so as to reproduce the ΛCDM cosmological background expansion history within the nonlocal model, we implement the model in a cosmological linear Einstein-Boltzmann solver and study the deviations to GR the model induces in the scalar and tensor perturbations. We observe that the DW nonlocal model describes a modified propagation for the gravitational waves, as well as a lower linear growth rate and a stronger lensing power as compared to ΛCDM, up to several percents. Such prominent growth and lensing features lead to the inference of a significantly smaller value of σ8 with respect to the one in ΛCDM, given Planck CMB+lensing data. The prediction for the linear growth rate fσ8 within the DW model is therefore significantly smaller than the one in ΛCDM and the addition of growth rate data fσ8 from Redshift-space distortion measurements to Planck CMB+lensing, opens a (dominant) tension between Redshift-space distortion data and the reconstructed Planck CMB lensing potential. However, model selection issues only result in “weak” evidences for ΛCDM against the DW model given the data. Such a fact shows that the joined datasets we consider are not constraining enough for distinguishing between the models on firm grounds. As we discuss, the addition of galaxy WL data or the consideration of cosmological constraints from future galaxy clustering, weak lensing surveys, but also third generation gravitational wave interferometers, prove to be useful for discriminating modified gravity models such as the DW one from ΛCDM, within the close future.