Model Selection Problem Research Articles

PanIC: Consistent information criteria for general model selection problems

PanIC: Consistent information criteria for general model selection problems

SVOH: Rigorous Selection Approach for Optimal Hyperparameter Values

The problem we address in this paper is a model selection problem. We consider the k-fold cross-validation (KCV) technique, applied to the Gaussian support vector machine (SVM) classification algorithm. In the cross-vali- dation process, the value of k for the number of subsets is generally chosen and set aprioristically (without any ex- periment). However, the value of k affects the choice of the best compromise between the estimation error and the ap- proximation error of the model. In this way, the k value of the number of subsets can severely influence the optimal values of the SVM classifier's hyperparameters and conse- quently affect the performance of the selected model and its ability to generalize. In this work, we propose a rigorous approach for finding the values of the hyperparameters of the Gaussian SVM known as SVOH (Selection of Optimal Hyperparam- eter Values) in a context of protein-protein interaction (PPI) prediction, where it is necessary to classify the pairs of pro- teins that interact together and those that do not interact together. The proposed approach considers the k value of the number of subsets as an influential parameter of the model and therefore performs learning to find an optimal value of k.

Variable selection for uncertain regression models based on elastic net method

On one hand, determining the exact number of explanatory variables in uncertain regression analysis is often challenging, as real-world scenarios are interconnected. On the other hand, it’s worth noting that even when the number of variables is determined, some variables may have minimal impact on the response variable, and retaining them would only increase the complexity of the model. Therefore, variable selection methods can help eliminate variables with minimal impact to enhance the simplicity and interpretability of the model. In light of this, this paper first introduces a novel variable selection strategy based on the elastic net method to address the variable selection problem in uncertain regression models. This method not only offers the flexibility to simplify uncertain regression analysis into Lasso estimation or Ridge estimation but also exhibits extremely high efficiency in variable selection. Secondly, to address the issue of determining tuning parameters, we introduce the cross-validation method to ensure optimal model performance. Furthermore, we propose an uncertain hypothesis testing method based on imprecise observational data, which can help us assess the suitability of estimated parameters. Finally, through a series of numerical examples, we validate the feasibility and effectiveness of this method.

Split-and-merge model selection of mixtures of Gaussian processes with RJMCMC

The mixture of Gaussian processes is a powerful statistical learning model that can be effectively applied to curve clustering and prediction. However, the corresponding model selection problem, that is, selecting an appropriate number of components in the mixture, is rather difficult to solve. In our previous work, we established the split-and-merge automatic model selection algorithm for mixtures of Gaussian processes along the output space under the framework of Reversible Jump Markov Chain Monte Carlo (RJMCMC), which can not only determine the number of actual Gaussian processes but also dynamically adjust the Gaussian process components to avoid dependence on parameter initialization and initial partitioning of the dataset during the parameter learning on a given dataset. In this study, we propose two algorithms: Penalized Likelihood RJMCMC and Penalized Prior RJMCMC. The former integrates a penalized term into the likelihood, while the latter incorporates a penalized term into the prior and operates within the full Bayesian inference framework, both aiming to focus more sharply on determining the number of components in the convergence process. Furthermore, we prove the geometric ergodicity of the RJMCMC algorithm for the mixture of Gaussian processes model, ensuring convergence of the posterior distribution with sufficient iterations. The experimental results further demonstrate the robustness of our PP-RJMCMC algorithm in model selection, showing superior performance compared to traditional approaches in curve classification and clustering. Additionally, the prediction performance is comparable to the EM algorithm. Although not directly explored in this study, the RJMCMC results can be used to initialize the EM algorithm, which could potentially improve prediction accuracy and accelerate computation.

Integrated deviance information criterion for spatial autoregressive models with heteroskedasticity

In this study, we introduce the integrated deviance information criterion (DIC) for nested and non-nested model selection problems in heteroskedastic spatial autoregressive models. In a Bayesian estimation setting, we assume that the idiosyncratic error terms of our spatial autoregressive model have a scale mixture of normal distributions, where the scale mixture variables are latent variables that induce heteroskedasticity. We first derive the integrated likelihood function by analytically integrating out the scale mixture variables from the complete-data likelihood function. We then use the integrated likelihood function to formulate the integrated DIC measure. We investigate the finite sample performance of the integrated DIC in selecting the true model in a simulation study. The simulation results show that the integrated DIC performs satisfactorily and can be useful for selecting the correct model in specification search exercises. Finally, in a spatially augmented economic growth model, we use the integrated DIC to choose the spatial weights matrix that leads to better predictive accuracy.

Rank-based sequential feature selection for high-dimensional accelerated failure time models with main and interaction effects

High-dimensional accelerated failure time (AFT) models are commonly used regression models in survival analysis. Feature selection problem in high-dimensional AFT models is addressed, considering scenarios involving solely main effects or encompassing both main and interaction effects. A rank-based sequential feature selection (RankSFS) method is proposed, the selection consistency is established and illustrated by comparing it with existing methods through extensive numerical simulations. The results show that RankSFS achieves a higher Positive Discovery Rate (PDR) and lower False Discovery Rate (FDR). Additionally, RankSFS is applied to analyze the data on Breast Cancer Relapse. With a remarkable short computational time, RankSFS successfully identifies two crucial genes.

An Improved Ensemble of Land Surface Air Temperatures Since 1880 Using Revised Pair-Wise Homogenization Algorithms Accounting for Autocorrelation

Abstract Land surface air temperatures (LSAT) inferred from weather station data differ among major research groups. The estimate by NOAA’s monthly Global Historical Climatology Network (GHCNm) averages 0.02°C cooler between 1880 and 1940 than Berkeley Earth’s and 0.14°C cooler than the Climate Research Unit estimates. Such systematic offsets can arise from differences in how poorly documented changes in measurement characteristics are detected and adjusted. Building upon an existing pairwise homogenization algorithm used in generating the fourth version of NOAA’s GHCNm(V4), PHA0, we propose two revisions to account for autocorrelation in climate variables. One version, PHA1, makes minimal modification to PHA0 by extending the threshold used in breakpoint detection to be a function of LSAT autocorrelation. The other version, PHA2, uses penalized likelihood to detect breakpoints through optimizing a model-selection problem globally. To facilitate efficient optimization for series with more than 1000 time steps, a multiparent genetic algorithm is proposed for PHA2. Tests on synthetic data generated by adding breakpoints to CMIP6 simulations and realizations from a Gaussian process indicate that PHA1 and PHA2 both similarly outperform PHA0 in recovering accurate climatic trends. Applied to unhomogenized GHCNmV4, both revised algorithms detect breakpoints that correspond with available station metadata. Uncertainties are estimated by perturbing algorithmic parameters, and an ensemble is constructed by pooling 50 PHA1- and 50 PHA2-based members. The continental-mean warming in this new ensemble is consistent with that of Berkeley Earth, despite using different homogenization approaches. Relative to unhomogenized data, our homogenization increases the 1880–2022 trend by 0.16 [0.12, 0.19]°C century−1 (95% confidence interval), leading to continental-mean warming of 1.65 [1.62, 1.69]°C over 2010–22 relative to 1880–1900. Significance Statement Accurately correcting for systematic errors in observational records of land surface air temperature (LSAT) is critical for quantifying historical warming. Existing LSAT estimates are subject to systematic offsets associated with processes including changes in instrumentation and station movement. This study improves a pairwise homogenization algorithm by accounting for the fact that climate signals are correlated over time. The revised algorithms outperform the original in identifying discontinuities and recovering accurate warming trends. Applied to monthly station temperatures, the revised algorithms adjust trends in continental mean LSAT since the 1880s to be 0.16°C century−1 greater relative to raw data. Our estimate is most consistent with that from Berkeley Earth and indicates lesser and greater warming than estimates from NOAA and the Met Office, respectively.

Green Edge Intelligence Scheme for Mobile Keyboard Emoji Prediction

Emoji prediction has been widely adopted in most mobile keyboards to improve the quality of user experience. Considering the resource constraints of smartphones, it is promising to deploy well-trained prediction models on edge servers, with which smartphones can carry out emoji prediction in an online fashion. However, a key issue in such a scenario lies in how the smartphone should select a subset of models to achieve high-accuracy and real-time emoji prediction with energy efficiency ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a.k.a.</i> the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">model selection</i> problem). Moreover, part of the system dynamics such as the prediction accuracy and the inference latency of each model are usually unknown <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> in practice, further complicating the problem. In this paper, with an effective integration of history-aware online learning and online control, we propose the first green edge intelligence scheme to solve the model selection problem for mobile keyboard emoji prediction. Our theoretical analysis and simulation results verify the effectiveness of our proposed scheme in achieving a sub-linear round-averaged regret bound and energy efficiency with a high prediction accuracy and a low latency.

Forest Fire Risk Prediction Based on Stacking Ensemble Learning for Yunnan Province of China

Forest fire risk prediction is essential for building a forest fire defense system. Ensemble learning methods can avoid the problem of difficult model selection for disaster susceptibility prediction and can significantly improve modeling accuracy. This study introduces a stacking ensemble learning model for predicting forest fire risks in Yunnan Province by integrating various data types, such as meteorological, topographic, vegetation, and human activity factors. A total of 70,274 fire points and an equal number of randomly selected nonfire points were used to develop the model, with 70% of the data allocated for training and the remaining 30% for testing. The stacking model combined four diverse machine learning methods: random forest (RF), extreme gradient boosting (XGBoost), light gradient boosting machine (LightGBM), and multilayer perceptron (MLP). We evaluated the model’s predictive performance using metrics like accuracy, area under the characteristic curve (AUC), and fire density (FD). The results demonstrated that the stacking fusion model exhibited remarkable accuracy with an AUC of 0.970 on the test set, significantly surpassing the performance of individual machine learning models, which had AUC values ranging from 0.935 to 0.953. Furthermore, the stacking fusion model effectively captured the maximum fire density in extremely high susceptibility areas, demonstrating enhanced generalization capabilities.

Analysis of the criteria selection problem in diversification models

The digitalization of the economy reduces the cost of doing business by automating the relevant processes, but any transformation creates new risks and economic instability. Economic instability leads to a drop in the standard of living and, as a result, negatively affects the activities of trade enterprises. Small and medium businesses are especially sensitive to any changes. The decrease in demand for most everyday goods has a painful effect on the activities of small and medium-sized businesses and leads to the emergence of new risks. These risks have a significant impact on reducing the profitability of enterprises. Therefore, it is important for each enterprise to diversify the activities of the enterprise, which includes the expansion of the product range, the reorientation of sales markets and the optimal distribution of goods between divisions of one enterprise. The subject of the article is multi-criteria models of a diversified portfolio that minimize the risks that arise in the era of the digital economy when managing retail chains. To formalize the problem, five models are proposed that differ in vector objective functions, both in the quantity and quality of the selected criteria. The aim of the work is to analyze the problem of choosing criteria in the corresponding multicriteria or vector diversification problems. The article examines the advantages of introducing an additional criterion of entropy maximization into the criteria of the classical two-criteria model of portfolio theory, which characterizes the degree of diversity of the portfolio composition. A complex combination of methods of classical portfolio theory and multicriteria optimization is applied. The results include a comparison of three methods for solving the following problems: criteria convolution, successive concessions, and computer simulation of the Pareto set. Conclusions: the results obtained will be useful for automating the risk management of retail chains. The practical value is that the obtained results of real data for the network have demonstrated the possibility of using the developed tool for automatic allocation of resources in the form of pareto-optimal portfolios in order to minimize risks.

Estimation of l0 norm penalized models: A statistical treatment

Fitting penalized models for the purpose of merging the estimation and model selection problem has become commonplace in statistical practice. Of the various regularization strategies that can be leveraged to this end, the use of the l0 norm to penalize parameter estimation poses the most daunting model fitting task. In fact, this particular strategy requires an end user to solve a non-convex NP-hard optimization problem irregardless of the underlying data model. For this reason, the use of the l0 norm as a regularization strategy has been woefully under utilized. To obviate this difficulty, a strategy to solve such problems that is generally accessible by the statistical community is developed. The approach can be adopted to solve l0 norm penalized problems across a very broad class of models, can be implemented using existing software, and is computationally efficient. The performance of the method is demonstrated through in-depth numerical experiments and through using it to analyze several prototypical data sets.

Are party families in Europe ideologically coherent today?

AbstractResearchers classify political parties into families by their shared cleavage origins. However, as parties have drifted from the original ideological commitments, it is unclear to what extent party families today can function as effective heuristics for shared positions. We propose an alternative way of classifying parties based solely on their ideological positions as one solution to this challenge. We use model‐based clustering to recast common subjective decisions involved in the process of creating party groups as problems of model selection, thus, providing non‐subjective criteria to define ideological clusters. By comparing canonical families to our ideological clusters, we show that while party families on the right are often too similar to justify categorizing them into different clusters, left‐wing families are weakly internally cohesive. Moreover, we identify two clusters predominantly composed of parties in Eastern Europe, questioning the degree to which categories originally designed to describe Western Europe can generalize to other regions.

Multi‐aspect permutation tests for model selection

AbstractIn typical machine learning frameworks, model selection is of fundamental importance: commonly, multiple models have to be trained and compared in order to identify the one with the best predictive performances. The aim of this study is to provide a new tool to improve the model selection process, allowing the user to identify the algorithm which significantly outperforms the other candidates. It proposes a robust model selection procedure based on a multi‐aspect permutation test which makes it possible to detect differences in both location and variability between two paired samples of prediction errors. A new extension of the nonparametric combination (NPC) methodology is therefore introduced and is integrated with an appropriate ranking procedure in order to deal with the comparison of candidate models. A simulation study is conducted to evaluate the performances of this testing procedure in 2‐sample and ‐sample problems, by generating data from various well‐known distributions and simulating several possible null and alternative scenarios. The adoption of the proposed technique in machine learning model selection problems is then discussed by means of multiple real data applications. These applications confirm what emerges from the simulation study: the introduced NPC‐based approach performs well under several different scenarios and represents a valuable tool for robust machine learning model selection.

Choice of the co-opetition model for a new energy vehicle supply chain under government subsidies

We study the co-opetition model selection problem for a new energy vehicle (NEV) supply chain comprising a dominant manufacturer and a subordinate supplier. Given the impact of R&D subsidies on suppliers’ technological innovation, we consider two co-opetition models, namely wholesale co-opetition and patent licensing co-opetition, derive the optimal decisions of the models, and analyze the optimal selections for supply chain members. We also conduct extensive numerical studies to verify the research findings and generate practical insights. We find that patent licensing co-opetition is not necessarily beneficial to supply chain members. However, under certain conditions, patent licensing co-opetition can create a “win-win” situation for the supply chain. Furthermore, as the degree of product substitution increases under patent licensing co-opetition, the optimal retail prices of the manufacturer and supplier both decrease first and then increase, but their thresholds are different. In addition, the R&D cost coefficient positively affects the fixed cost of patent usage, but it does not affect the unit patent licensing fee. This study provides theoretical guidance for supply chain members to choose the optimal co-opetition models and has important practical significance for promoting the sustainable development of the NEV industry.

Model Selection in Generalized Linear Models

The problem of model selection in regression analysis through the use of forward selection, backward elimination, and stepwise selection has been well explored in the literature. The main assumption in this, of course, is that the data are normally distributed and the main tool used here is either a t test or an F test. However, the properties of these model selection procedures are not well-known. The purpose of this paper is to study the properties of these procedures within generalized linear regression models, considering the normal linear regression model as a special case. The main tool that is being used is the score test. However, the F test and other large sample tests, such as the likelihood ratio and the Wald test, the AIC, and the BIC, are included for the comparison. A systematic study, through simulations, of the properties of this procedure was conducted, in terms of level and power, for symmetric and asymmetric distributions, such as normal, Poisson, and binomial regression models. Extensions for skewed distributions, over-dispersed Poisson (the negative binomial), and over-dispersed binomial (the beta-binomial) regression models, are also given and evaluated. The methods are applied to analyze two health datasets.

Eryn: a multipurpose sampler for Bayesian inference

ABSTRACTIn recent years, methods for Bayesian inference have been widely used in many different problems in physics where detection and characterization are necessary. Data analysis in gravitational-wave astronomy is a prime example of such a case. Bayesian inference has been very successful because this technique provides a representation of the parameters as a posterior probability distribution, with uncertainties informed by the precision of the experimental measurements. During the last couple of decades, many specific advances have been proposed and employed in order to solve a large variety of different problems. In this work, we present a Markov Chain Monte Carlo (MCMC) algorithm that integrates many of those concepts into a single MCMC package. For this purpose, we have built Eryn, a user-friendly and multipurpose toolbox for Bayesian inference, which can be utilized for solving parameter estimation and model selection problems, ranging from simple inference questions, to those with large-scale model variation requiring trans-dimensional MCMC methods, like the Laser Interferometer Space Antenna Global Fit problem. In this paper, we describe this sampler package and illustrate its capabilities on a variety of use cases.

Online Simulator-Based Experimental Design for Cognitive Model Selection

The problem of model selection with a limited number of experimental trials has received considerable attention in cognitive science, where the role of experiments is to discriminate between theories expressed as computational models. Research on this subject has mostly been restricted to optimal experiment design with analytically tractable models. However, cognitive models of increasing complexity with intractable likelihoods are becoming more commonplace. In this paper, we propose BOSMOS, an approach to experimental design that can select between computational models without tractable likelihoods. It does so in a data-efficient manner by sequentially and adaptively generating informative experiments. In contrast to previous approaches, we introduce a novel simulator-based utility objective for design selection and a new approximation of the model likelihood for model selection. In simulated experiments, we demonstrate that the proposed BOSMOS technique can accurately select models in up to two orders of magnitude less time than existing LFI alternatives for three cognitive science tasks: memory retention, sequential signal detection, and risky choice.

Model selection via reweighted partial sparse recovery

In this work, we propose an iterative reweighted algorithm for model selection with a-priori information on some possible terms present in the unknown dynamics, which is able to recover the governing equations from noisy time-series data and has strong adaptability. In addition, we prove that the algorithm converges to a local minimizer with in a few steps. Through several examples, we show that the proposed algorithm is more robust to noise and less dependent on hyperparameters, compared to its non-reweighted counterpart. We also apply the algorithm to the model selection problems.

Information criteria for matrix exponential spatial specifications

In this study, we suggest using information criteria for nested and non-nested model selection problems for the matrix exponential spatial specifications (MESS) under both homoskedasticity and heteroskedasticity. To this end, we consider the deviance information criterion, the Akaike information criterion and the Bayesian information criterion in a Bayesian setting. In the heteroskedastic case, we assume that the error terms have a scale mixture of normal distributions, where the scale mixture variables are latent variables that lead to different distributions. We demonstrate how the integrated likelihood function can be obtained analytically by integrating out the scale mixture variables from the complete-data likelihood function, and how this integrated likelihood function can be used to formulate the information criteria. We investigate the finite sample performance of these criteria in selecting the true model in a simulation study. The results show that these criteria perform satisfactorily and can be useful for selecting the correct model in specification search exercises. Finally, we apply the proposed information criteria to a spatially augmented growth model and a carbon emission model to show their usefulness for both nested and non-nested model selection problems.

Multiple-hypothesis testing rules for high-dimensional model selection and sparse-parameter estimation

We consider the problem of model selection for high-dimensional sparse linear regression models. We pose the model selection problem as a multiple-hypothesis testing problem and employ the methods of false discovery rate (FDR) and familywise error rate (FER) to solve it. We also present the reformulation of the FDR/FER-based approaches as criterion-based model selection rules and establish their relation to the extended Bayesian Information Criterion (EBIC), which is a state-of-the-art high-dimensional model selection rule. We use numerical simulations to show that the proposed FDR/FER method is well suited for high-dimensional model selection and performs better than EBIC.