Model Selection Techniques Research Articles

Deep learning based network similarity for model selection

Capturing data in the form of networks is becoming an increasingly popular approach for modeling, analyzing and visualising complex phenomena, to understand the important properties of the underlying complex processes. Access to many large-scale network datasets is restricted due to the privacy and security concerns. Also for several applications (such as functional connectivity networks), generating large scale real data is expensive. For these reasons, there is a growing need for advanced mathematical and statistical models (also called generative models) that can account for the structure of these large-scale networks, without having to materialize them in the real world. The objective is to provide a comprehensible description of the network properties and to be able to infer previously unobserved properties. Various models have been developed by researchers, which generate synthetic networks that adhere to the structural properties of real networks. However, the selection of the appropriate generative model for a given real-world network remains an important challenge. In this paper, we investigate this problem and provide a novel technique (named as TripletFit) for model selection (or network classification) and estimation of structural similarities of the complex networks. The goal of network model selection is to select a generative model that is able to generate a structurally similar synthetic network for a given real-world (target) network. We consider six outstanding generative models as the candidate models. The existing model selection methods mostly suffer from sensitivity to network perturbations, dependency on the size of the networks, and low accuracy. To overcome these limitations, we considered a broad array of network features, with the aim of representing different structural aspects of the network and employed deep learning techniques such as deep triplet network architecture and simple feed-forward network for model selection and estimation of structural similarities of the complex networks. Our proposed method, outperforms existing methods with respect to accuracy, noise-tolerance, and size independence on a number of gold standard data set used in previous studies.

Explaining Predictive Model Performance: An Experimental Study of Data Preparation and Model Choice.

Although confirmatory modeling has dominated much of applied research in medical, business, and behavioral sciences, modeling large data sets with the goal of accurate prediction has become more widely accepted. The current practice for fitting predictive models is guided by heuristic-based modeling frameworks that lead researchers to make a series of often isolated decisions regarding data preparation and cleaning that may result in substandard predictive performance. In this article, we use an experimental design to evaluate the impact of six factors related to data preparation and model selection (techniques for numerical imputation, categorical imputation, encoding, subsampling for unbalanced data, feature selection, and machine learning algorithm) and their interactions on the predictive accuracy of models applied to a large, publicly available heart transplantation database. Our factorial experiment includes 10,800 models evaluated on 5 independent test partitions of the data. Results confirm that some decisions made early in the modeling process interact with later decisions to affect predictive performance; therefore, the current practice of making these decisions independently can negatively affect predictive outcomes. A key result of this case study is to highlight the need for improved rigor in applied predictive research. By using the scientific method to inform predictive modeling, we can work toward a framework for applied predictive modeling and a standard for reproducibility in predictive research.

Rapid model comparison of equations of state from gravitational wave observation of binary neutron star coalescences

The discovery of the coalescence of binary neutron star GW170817 was a watershed moment in the field of gravitational wave astronomy. Among the rich variety of information that we were able to uncover from this discovery was the first non-electromagnetic measurement of the neutron star radius, and the cold nuclear equation of state. It also led to a large equation of state model-selection study from gravitational-wave data. In those studies Bayesian nested sampling runs were conducted for each candidate equation of state model to compute their evidence in the gravitational-wave data. Such studies, though invaluable, are computationally expensive and require repeated, redundant, computation for any new models. We present a novel technique to conduct model-selection of equation of state in an extremely rapid fashion (~minutes) on any arbitrary model. We test this technique against the results of a nested-sampling model-selection technique published earlier by the LIGO/Virgo collaboration, and show that the results are in good agreement with a median fractional error in Bayes factor of about 10%, where we assume that the true Bayes factor is calculated in the aforementioned nested sampling runs. We found that the highest fractional error occurs for equation of state models that have very little support in the posterior distribution, thus resulting in large statistical uncertainty. We then used this method to combine multiple binary neutron star mergers to compute a joint-Bayes factor between equation of state models. This is achieved by stacking the evidence of the individual events and computing the Bayes factor from these stacked evidences for each pairs of equation of state.

Hyperparameter Tuning and Pipeline Optimization via Grid Search Method and Tree-Based AutoML in Breast Cancer Prediction.

Automated machine learning (AutoML) has been recognized as a powerful tool to build a system that automates the design and optimizes the model selection machine learning (ML) pipelines. In this study, we present a tree-based pipeline optimization tool (TPOT) as a method for determining ML models with significant performance and less complex breast cancer diagnostic pipelines. Some features of pre-processors and ML models are defined as expression trees and optimal gene programming (GP) pipelines, a stochastic search system. Features of radiomics have been presented as a guide for the ML pipeline selection from the breast cancer data set based on TPOT. Breast cancer data were used in a comparative analysis of the TPOT-generated ML pipelines with the selected ML classifiers, optimized by a grid search approach. The principal component analysis (PCA) random forest (RF) classification was proven to be the most reliable pipeline with the lowest complexity. The TPOT model selection technique exceeded the performance of grid search (GS) optimization. The RF classifier showed an outstanding outcome amongst the models in combination with only two pre-processors, with a precision of 0.83. The grid search optimized for support vector machine (SVM) classifiers generated a difference of 12% in comparison, while the other two classifiers, naïve Bayes (NB) and artificial neural network—multilayer perceptron (ANN-MLP), generated a difference of almost 39%. The method’s performance was based on sensitivity, specificity, accuracy, precision, and receiver operating curve (ROC) analysis.

Behavioral model summarisation for other agents under uncertainty

Modelling other agents is one of the fundamental research challenges in multiagent decision making under uncertainty. A subject agent expects to optimise its own decisions by correctly modelling what other agents are going to act when they share a common environment and their actions have a joint impact in the transition of environmental states and their individual rewards. However, a true model of the other agent is often unknown due to limited prior knowledge, which particularly occurs to a competitive agent setting. Consequently, the subject agent needs to consider a large number of potential models (the number is infinite in theory) for the other agent, which turns to be computationally hard in its decision making process. A feasible solution is to select a subset of potential models that are representative of all the models for the other agent. In this article, we investigate such a model selection given some known behavioral stereotypes ascribed to the other agents. We propose a new measurement to diversify the selected models under the constraint of the number of models (K models). We formulate the top-K model selection as a sub-modular function optimisation problem and develop a greedy algorithm to obtain high-quality solutions. We develop the proposed model selection techniques in a well-recognised multiagent decision model, namely interactive dynamic influence diagrams, that represents individual agents’ decision making process with the consideration of other agents. By doing this, we can test the performance of the new model selection techniques over two problem domains and provide experimental results in support. Hence, our research in this article contributes to a new model selection technique for modelling other agents therefore improving decision quality of a subject agent interacting with the other agents. It also inspires a creative usage of sub-modular function optimisation in multiagent decision making.

Model selection in time series analysis: using information criteria as an alternative to hypothesis testing

PurposeThe issue of model selection in applied research is of vital importance. Since the true model in such research is not known, which model should be used from among various potential ones is an empirical question. There might exist several competitive models. A typical approach to dealing with this is classic hypothesis testing using an arbitrarily chosen significance level based on the underlying assumption that a true null hypothesis exists. In this paper, the authors investigate how successful the traditional hypothesis testing approach is in determining the correct model for different data generating processes using time series data. An alternative approach based on more formal model selection techniques using an information criterion or cross-validation is also investigated.Design/methodology/approachMonte Carlo simulation experiments on various generating processes are used to look at the response surfaces resulting from hypothesis testing and response surfaces resulting from model selection based on minimizing an information criterion or the leave-one-out cross-validation prediction error.FindingsThe authors find that the minimization of an information criterion can work well for model selection in a time series environment, often performing better than hypothesis-testing strategies. In such an environment, the use of an information criterion can help reduce the number of models for consideration, but the authors recommend the use of other methods also, including hypothesis testing, to determine the appropriateness of a model.Originality/valueThis paper provides an alternative approach for selecting the best potential model among many for time series data. It demonstrates how minimizing an information criterion can be useful for model selection in a time-series environment in comparison to some standard hypothesis testing strategies.

Selection of uncertain differential equations using cross validation

Uncertain differential equations have been widely applied to modelling dynamic phenomena with noises. Facing with the same dynamic phenomenon, different scholars may adopt different uncertain differential equations. Naturally, there is a crucial question about which of these equations is most suitable to describe the given dynamic phenomenon. A desired uncertain differential equation should have a good prediction ability, that is, it performs well in a new dataset. As a model selection technique, cross validation assesses the model’s ability to predict new data in order to avoid overfitting. This paper applies k fold cross validation to the selection of uncertain differential equations. Observations in training sets and testing sets are used to calculate generalized moment estimations of unknown parameters and testing errors for uncertain differential equations, respectively. Then the uncertain differential equation with the smallest total testing error is selected. A numerical example and a real data example using closing prices of Industrial and Commercial Bank of China (ICBC) stock illustrate our methods in detail.

Deviations from the covered interest parity: The role of fundamentals, financial and political turmoil, and market frictions

Recent studies of mature markets on covered interest parity suggest that deviations are mean-reverting, but persistent, particularly after the 2008 crisis (Du et al., 2018). Our study contributes to the literature by modeling the deviations from covered interest rate parity (CIP) of an important emerging-market economy. We focus on Brazilian data, given the importance of its derivative market. One of the strengths of our study is the use of an agnostic approach, based on an automatic model-selection technique that is robust to structural change, the Autometrics algorithm (Hendry and Doornik, 2014), to unveil the possible determinants of CIP deviations from a wide information data set. We show that CIP deviations are highly sensitive to changes in Brazilian federal government total debt, level of reserves, inflation, and degree of trade openness. We also document the existence of instability in the model due to financial and political turmoil. We reach these conclusions based on the algorithm’s intercept correction, which can be seen as a byproduct of our methodology. Finally, we find evidence that, even after correction for fundamentals and instability points, CIP deviations still have persistence, suggesting that market frictions play an important role in the dynamics of CIP deviations.

Implied volatility directional forecasting: a machine learning approach

This study investigates whether the direction of U.S. implied volatility, VIX index, can be forecast. Multiple forecasts are generated based on standard econometric models, but, more importantly, on several machine learning techniques. Their statistical significance is assessed by a plethora of performance evaluation measures, while real-time investment strategies are devised to appraise the investment implications of the underlying modeling approaches. The main conclusion of the analysis is that the implementation of machine learning techniques in implied volatility forecasting can be more effective compared to mainstream econometric models and model selection techniques, as they are superior both in a statistical and an economic evaluation setting.

On multivariate copula modeling of dependent degradation processes

Multivariate degradation processes have been observed in many engineering systems. Most existing multivariate degradation modeling techniques, such as multivariate general path models or multivariate Wiener process models, assume an underlying Gaussian dependence structure. Unfortunately, in reality, the dependencies among degradation processes are often nonlinear, asymmetric and greatly tail-skewed, and thus limit the usefulness of the conventional modeling techniques in practice. To overcome these limitations, in this paper, we develop a copula-based multivariate modeling framework. Three fundamental copula classes are applied to model the complex dependence structure among correlated degradation processes. Statistical inference and model selection techniques, including two graphical diagnostic tools, a test of independence and a goodness-of-fit test, are employed to identify the best model. The advantages of the proposed modeling framework are demonstrated through simulation studies. And we also discuss the effect of ignoring tail dependence on system failure probability assessment. Finally, the applications of the copula-based multivariate degradation models on both system reliability evaluation and remaining useful life prediction are provided. The proposed methodology is illustrated using a numerical example.

A decomposition-based algorithm for learning the structure of multivariate regression chain graphs

We extend the decomposition approach for learning Bayesian networks (BNs) proposed by Xie et al. (2006) [54] to learning multivariate regression chain graphs (MVR CGs), which include BNs as a special case. The same advantages of this decomposition approach hold in the more general setting: reduced complexity and increased power of computational independence tests. Moreover, latent (hidden) variables can be represented in MVR CGs by using bidirected edges, and our algorithm correctly recovers any independence structure that is faithful to an MVR CG, thus greatly extending the range of applications of decomposition-based model selection techniques. Simulations under a variety of settings demonstrate the competitive performance of our method in comparison with the PC-like algorithm (Sonntag and Peña, 2012) [43]. In fact, the decomposition-based algorithm usually outperforms the PC-like algorithm except in running time. The performance of both algorithms is much better when the underlying graph is sparse.

Do firm characteristics of concentrated ownership firms affect dividend payout beyond traditional motivations?

Purpose This study aims to analyze the influence of firm characteristics in dividend payout in a concentrated ownership setting. Design/methodology/approach This study is probably the first to use the lasso technique for model selection and error prediction in the study of dividend payout in India. The lasso method comprises subsampling the available data set and performing reiterative regressions on those samples to generate the model with the best fit. This study incorporates four different ways of performing lasso treatment to get the best fit among them. Findings This study analyzes the influence of firm characteristics on dividend payout in the Indian context and asserts that firms with growth potential and earnings volatility do not hesitate to cut dividends. This study does not find evidence for signaling, agency cost and life cycle theories in a concentrated ownership setting. Earnings is the single most important factor to have a positive influence on dividend, while excessively leveraged firms are restrictive of dividend payout. Taxation has a prominent role in altering the way firms pay dividend. Research limitations/implications The recent changes in buyback taxation offer another opportunity to test the reactive behavior of firms. Also, given the disregard for traditional motivations, further research needs to be done to determine if dividend adjustments (on the lower side) help enhance firm value or not. Practical implications This study may help investors view dividends in a proper perspective. Firms give importance to investments over dividends and thus investors need not dwell on dividend changes if firms fulfill their growth potential. Social implications It lends perspective to investors about dividend changes and its importance. Originality/value The methodology used for analysis is absolutely original in the literature pertaining to dividend policy in the Indian context. The literature is abundant with theories advocating or opposing the eminence of dividend payout; however, this study takes a holistic view of all influential dividend determinants in literature to understand dividend payout.

A practical guide to selecting models for exploration, inference, and prediction in ecology.

Selecting among competing statistical models is a core challenge in science. However, the many possible approaches and techniques for model selection, and the conflicting recommendations for their use, can be confusing. We contend that much confusion surrounding statistical model selection results from failing to first clearly specify the purpose of the analysis. We argue that there are three distinct goals for statistical modeling in ecology: data exploration, inference, and prediction. Once the modeling goal is clearly articulated, an appropriate model selection procedure is easier to identify. We review model selection approaches and highlight their strengths and weaknesses relative to each of the three modeling goals. We then present examples of modeling for exploration, inference, and prediction using a time series of butterfly population counts. These show how a model selection approach flows naturally from the modeling goal, leading to different models selected for different purposes, even with exactly the same data set. This review illustrates best practices for ecologists and should serve as a reminder that statistical recipes cannot substitute for critical thinking or for the use of independent data to test hypotheses and validate predictions.

General Circulation Model Selection Technique for Downscaling: Exemplary Application to East Africa

AbstractDownscaling is widely used in studies of local and/or regional climate as it yields a greater spatial resolution than general circulation models (GCMs) can provide. It utilizes GCM output or reanalysis data, which is transformed using mathematical relationships or used to force the lateral boundaries of a regional climate model. However, there is no set selection technique to determine which GCM realization(s) to employ. Here, a comprehensive yet easily applicable model selection technique for studies requiring GCM data as a constraint was developed. The technique evaluates, with respect to a reanalysis product and/or observational data, the ability of GCM realizations to reconstruct the mean state of the climate and the space‐time climatic anomalies for the atmospheric state variables at three distinct pressure levels. It was applied to the region of East Africa, where GISS‐E2‐H r6i1p3 was found to perform the strongest. The top ranked realizations were found to better capture processes when evaluated for the example of the Indian Ocean Dipole. Furthermore, the surface air temperature and precipitation from three 10‐year regional climate model simulations, one forced by the Modern‐Era Retrospective Analysis for Research and Applications version 2 reanalysis, one forced by the top ranked GCM, and one by the lowest ranked one, were compared to gridded observations. Results show that using a top ranked GCM for the boundary conditions leads to a better dynamical downscaling simulation than a low‐ranked GCM, suggesting the potential of the proposed technique for future downscaling techniques.

An Enhanced Deep Learning Model to Network Attack Detection, by using Parameter Tuning, Hidden Markov Model and Neural Network

In recent years, Deep Learning has become a critical success factor for Machine Learning. In the present study, we introduced a Deep Learning model to network attack detection, by using Hidden Markov Model and Artificial Neural Networks. We used a model aggregation technique to find a single consolidated Deep Learning model for better data fitting. The model selection technique is applied to optimize the bias-variance trade-off of the expected prediction. We demonstrate its ability to reduce the convergence, reach the optimal solution and obtain more cluttered decision boundaries. Experimental studies conducted on attack detection indicate that our proposed model outperformed existing Deep Learning models and gives an enhanced generalization.

A foci cohort analysis to monitor successful and persistent foci under Thailand\u2019s Malaria Elimination Strategy

BackgroundThailand’s success in reducing malaria burden is built on the efficient “1-3-7” strategy applied to the surveillance system. The strategy is based on rapid case notification within 1 day, case investigation within 3 days, and targeted foci response to reduce the spread of Plasmodium spp. within 7 days. Autochthonous transmission is still occurring in the country, threatening the goal of reaching malaria-free status by 2024. This study aimed to assess the effectiveness of the 1-3-7 strategy and identify factors associated with presence of active foci.MethodsData from the national malaria information system were extracted from fiscal years 2013 to 2019; after data cleaning, the final dataset included 81,012 foci. A Cox’s proportional hazards model was built to investigate factors linked with the probability of becoming an active focus from 2015 to 2019 among foci that changed status from non-active to active focus during the study period. We performed a model selection technique based on the Akaike Information Criteria (AIC).ResultsThe number of yearly active foci decreased from 2227 to 2013 to 700 in 2019 (68.5 %), and the number of autochthonous cases declined from 17,553 to 3,787 (78.4 %). The best Cox’s hazard model showed that foci in which vector control interventions were required were 18 % more likely to become an active focus. Increasing compliance with the 1-3-7 strategy had a protective effect, with a 22 % risk reduction among foci with over 80 % adherence to 1-3-7 timeliness protocols. Other factors associated with likelihood to become or remain an active focus include previous classification as an active focus, presence of Plasmodium falciparum infections, level of forest disturbance, and location in border provinces.ConclusionsThese results identified factors that favored regression of non-active foci to active foci during the study period. The model and relative risk map align with the national malaria program’s district stratification and shows strong spatial heterogeneity, with high probability to record active foci in border provinces. The results of the study may be useful for honing Thailand’s program to eliminate malaria and for other countries aiming to accelerate malaria elimination.

Precision Analysis of Drug-Induced Conformational Changes of P-Glycoprotein Observed in AFM

P-glycoprotein (Pgp) is an ATP-Binding Cassette transporter commonly overexpressed in cancer cells. This protein effluxes drugs out of the cell, leading to multi-drug resistance. Significant research interest surrounds the determination of the mechanism of action. A powerful single-molecule tool is the Atomic Force Microscope (AFM), which can directly image biomolecules in physiological buffer and temperature. Pgp reconstituted in supported lipid bilayers is imaged in the presence of cancer drugs and ATP-analogs to probe conformational changes. The high spatial resolution of the AFM must be matched by the precision of the post-imaging analysis. Previously, we have introduced the Hessian blob algorithm as a method of particle detection that minimizes user input [B.P. Marsh et al. Scientific Reports 8, 978 (2018)]. With the addition of scale-space Laplace interpolation to detect background levels, we increase the precision of particle heights, revealing previously unresolved populations of Pgp. Model selection techniques such as the Bayesian information criterion add to the objectivity of data interpretation. Together, these analyses refine our ability to deduce ligand-induced conformational shifts of Pgp in an unbiased manner.

Drivers of distributions and niches of North American cold‐adapted amphibians: evaluating both climate and land use

Species distribution estimates are often used to understand the niche of a species; however, these are often based solely on climatic predictors. When the influences of biotic factors are ignored, erroneous inferences about range and niche may be made. We aimed to integrate climate data with a unique set of available land cover and land use data for the six cold-adapted amphibians of North America (Ambystoma macrodactylum, Anaxyrus hemiophrys, Anaxyrus boreas, Pseudacris maculata, Rana sylvatica, Rana luteiventris) to determine the relative importance of climate and non-climate drivers through the use of ecological niche models for present-day range estimates. We compared climate-only, land use-only, and combination models of climate and land use, derived from two different model selection techniques, to determine which was most likely to drive current distributions of cold-adapted amphibian species. Land use layers included land cover type, human population, vegetation type, ecoregion, and the overall human footprint. The most supported models included both climate and land use, with climate and human footprint variables having the highest permutation importance and percent contribution. Models that incorporated climate and land use data performed best as measured with AIC and AUC, although qualitatively most underestimated the northern range edge, implying potential sampling bias or locations of reduced habitat quality for these species in the northern area of the ranges. There were small differences in overall combination models dependent on the method of model selection. The overall effect sizes of landscape factors within the combination models were small except for one landscape feature: human footprint, which incorporated multiple aspects of anthropogenic change on the landscape, including human population density, travel access, and agricultural impact. This aspect of the landscape was just as important as climate, and counter to what we expected, the association was mostly positive, with a negative response only occurring at very high levels. This highlights the importance of moving beyond climate only species range estimates as land cover, specifically human impact, may be driving the patterns of species' ranges.

Demand Estimation Using Managerial Responses to Automated Price Recommendations

We provide a new framework to identify demand elasticities in markets where managers rely on algorithmic recommendations for price setting, and apply it to a dataset containing bookings for a sample of mid-sized hotels in Europe. Using non-binding algorithmic price recommendations and observed delay in price adjustments by decision makers, we demonstrate that a control-function approach, combined with state-of-the-art model selection techniques, can be used to isolate exogenous price variation and identify demand elasticities across hotel room types and over time. We confirm these elasticity estimates with a difference-in-differences approach that leverages the same delays in price adjustments by decision makers. However, the difference-in-differences estimates are more noisy and only yield consistent estimates if data is pooled across hotels. We then apply our control-function approach to two classic questions in the dynamic pricing literature: the evolution of price elasticity of demand over time as well as the effects of a transitory price change on future demand due to the presence of strategic buyers. Finally, we discuss how our empirical framework can be applied directly to other decision-making situations in which recommendation systems are used.

How Statistical Learning Can Help to Estimate the Number of Modes in Switched System Identification?

Abstract This paper deals with hybrid dynamical system identification, and focuses more particularly on the estimation of the number of modes. An evaluation of a recent method based on model selection techniques from statistical learning is proposed, together with its comparison with more standard approaches based on algebraic arguments. Overall, three methods are benchmarked in various settings, including different noise conditions and data set sizes. The results provide insights into the respective advantages and weaknesses of the methods, thus yielding a set of guidelines on the choice of the most suitable method in a given situation for the practitioner.