Model Selection Techniques Research Articles

A theoretical approach based on machine learning for estimation of physical properties of LLDPE in moulding process

This study explores the prediction of mechanical characteristics of linear polyethylene based on oven residence time, employing various regression models and hyper-parameter tuning through the Whale Optimization Algorithm. The dataset comprises one input variable (oven residence time) and three output parameters (Tensile Strength, Impact Strength, and Flexure Strength). The models investigated include Multilayer Perceptron, K-Nearest Neighbors, Support Vector Regression, Polynomial Regression, and Theil–Sen Regression. The results showcased distinct performances across the models for each output parameter. The Polynomial Regression (WOA-PR) method has been identified as the most suitable option for predicting Tensile Strength due to its ability to achieve the lowest errors in terms of Mean Absolute Error, Root Mean Square Error, and Average Absolute Relative Deviation. K-Nearest Neighbors (WOA-KNN) outperforms other models in predicting Impact Strength due to its superior accuracy and reliability. Additionally, Support Vector Regression (WOA-SVR) emerges as the best model for predicting Flexure Strength, showcasing notable performance in minimizing prediction errors. These findings underscore the significance of model selection and optimization techniques in accurately predicting the mechanical properties of polymers, paving the way for enhanced manufacturing processes and material design.

The future of cosmological likelihood-based inference: accelerated high-dimensional parameter estimation and model comparison

We advocate for a new paradigm of cosmological likelihood-based inference, leveraging recent developments in machine learning and its underlying technology, to accelerate Bayesian inference in high-dimensional settings. Specifically, we combine (i) emulation, where a machine learning model is trained to mimic cosmological observables, e.g. CosmoPower-JAX; (ii) differentiable and probabilistic programming, e.g. JAX and NumPyro, respectively; (iii) scalable Markov chain Monte Carlo (MCMC) sampling techniques that exploit gradients, e.g. Hamiltonian Monte Carlo; and (iv) decoupled and scalable Bayesian model selection techniques that compute the Bayesian evidence purely from posterior samples, e.g. the learned harmonic mean implemented in harmonic. This paradigm allows us to carry out a complete Bayesian analysis, including both parameter estimation and model selection, in a fraction of the time of traditional approaches. First, we demonstrate the application of this paradigm on a simulated cosmic shear analysis for a Stage IV survey in 37- and 39-dimensional parameter spaces, comparing ΛCDM and a dynamical dark energy model ( w0waCDM). We recover posterior contours and evidence estimates that are in excellent agreement with those computed by the traditional nested sampling approach while reducing the computational cost from 8 months on 48 CPU cores to 2 days on 12 GPUs. Second, we consider a joint analysis between three simulated next-generation surveys, each performing a 3x2pt analysis, resulting in 157- and 159-dimensional parameter spaces. Standard nested sampling techniques are simply unlikely to be feasible in this high-dimensional setting, requiring a projected 12 years of compute time on 48 CPU cores; on the other hand, the proposed approach only requires 8 days of compute time on 24 GPUs. All packages used in our analyses are publicly available.

Assessing model selection techniques for distributions use in hydrological extremes in the presence of trimming and subsampling

A suitable probability distribution is required to quantify and estimate hydraulic structure design for risk evaluation and management. The inability of model selection criteria to differentiate, in some cases, among candidate distributions used in the analysis of hydrological extremes is often criticised. This study verifies, with the aid of model selection techniques, the potential utility of trimming and subsampling in distinguishing between candidate distributions, which might not be feasible using the traditional goodness of fit method alone, when samples available are small. The performance of the proposed method is evaluated through its application to real and simulated yearly peak rainfall datasets. The proposed approach is then compared with several standard model selection techniques. Results show that the model selection techniques with the aid of subsampling are effective in identifying the true parent distribution for the untrimmed samples given a two-parameter distribution; contrarily, they are inefficient where a distribution with a three-parameter is the parent distribution. However, as trimming is introduced, all model selection methods recognise the true parent distribution for a three-parameter distribution. Overall, utilising trimming and subsampling with the aid of model selection methods yields promising outcomes in the analysis of hydrological extreme frequencies. Drawing from the results of numerical simulation and examination of observed data, the use of trimming and subsampling can be a viable tool in differentiating among candidate distributions used in the investigation of hydrological extreme frequencies.

Plant silicon content as a proxy for understanding plant community properties and ecosystem structure

AbstractSilicon (Si) content in plant tissues is considered a functional trait that can provide multiple morpho‐physiological benefits to plant individuals. However, it is still unclear whether and how these individual benefits extend to plant community processes and ecosystem functioning. Here we investigated how plant Si content is associated with plant community properties and the ecosystem structure of herbaceous communities in Israel. We sampled 15 sites across the Mediterranean and desert ecosystems and built models to evaluate how plant silicon content (community‐weighted mean and standard variation) is associated with variables such as species richness, biomass production, plant cover, and functional diversity. Finally, we used model selection techniques to test whether models depicting plant Si content perform better than models using data on soil Si instead. Sites with lower susceptibility to drought had significantly more Si‐accumulating grass species and higher soils Si content. Models with plant Si content instead of soil Si, always performed better, although those considering Si content variation had overall stronger associations with community properties than only mean Si content. For instance, up to 51% of plant Si content variation was explained by climate, biomass production, and species richness, combined. Still, mean plant Si content and plant cover combined explained up to 42% of plant functional diversity. Our results suggest the that plant Si content serves as a proxy for understanding the ecological properties and functioning of arid and Mediterranean ecosystems. Nevertheless, the significance of Si has not been fully explored in other ecosystem types, where its influence may be less pronounced compared with the ecosystems examined in this study. In light of various global change scenarios, enhancing our understanding of Si as a plant functional trait could help bridge existing knowledge gaps and improve ecological modeling, thus enabling more accurate forecasts of changes in plant distributions.

Model selection techniques for seafloor scattering statistics in synthetic aperture sonar images of complex seafloors

AbstractIn quantitative analysis of seafloor scattering measurements, it is common to model the single‐point probability density function of the scattered intensity or amplitude. For more complex seafloors, the pixel amplitude distribution has previously been modelled with a mixture model consisting of two K distributions, but the environment may have more identifiable scattering mechanisms. Choosing the number of components of a mixture model is a decision that must be made, using a priori information, or using a data driven approach. Several common model selection techniques from the statistics literature are explored (the Akaike, Bayesian, deviance, and Watanabe‐Akaike information criteria) and compared to the authors' choice. Examples are given for synthetic aperture sonar data collected by an autonomous underwater vehicle in a rocky environment off the coast of Bergen, Norway, using the HISAS‐1032 synthetic aperture sonar system. The Bayesian information criterion aligned most closely with the interpretation of both the acoustic images and the plots of the probability of false alarm.

Statistical modelling of height growth in urban forestry plantations

ABSTRACT Urban plantation dynamics in different topographical and climatic conditions in Odisha were evaluated using linear model selection and regularisation techniques. The main objective was to evaluate how and to what extent the urban plantations respond to various climatic and edaphic conditions. The relationship between vegetation growth and climatic and soil parameters was studied using four statistical learning tools, subset selection, ridge regression, lasso, and partial least squares regression, and their performance was compared to a multiple regression model. The test MSE for the subset selection, ridge regression, lasso, and partial least squares regression models was evaluated to be 16,261.54, 12245.11, 16263.79 and 14,317.21, respectively. Results proved that statistical learning methods, namely subset selection, lasso, ridge regressions and partial least squares regression, were more accurate than multiple linear regression. From the results, it can be safely concluded that temperature shows greater correlation with the growth parameters. Precipitation also plays a vital role in vegetation dynamics. Soil parameters indicate a positive correlation with that of the growth.

Exact post-selection inference for adjusted R squared selection

Post-selection inference is developed for regression coefficients selected by the widely used model selection technique of adjusted R2. Selective inference deals with the selection aspect by conditioning inference on the model selection event. In linear models, we obtain exact post-selection inference in finite samples. Extensions to logistic regression models are discussed. A simulation study illustrates the exact type I error control, unlike classical inference. Our tests provide higher power than data splitting approaches.

A study of model selection techniques for modeling synthetic aperture sonar backscatter statistics

High resolution acoustic imaging of the seafloor, such as with synthetic aperture sonar, can reveal complex environments due to the ability to resolve scatterers of different types. The statistical distribution of the backscattered field is appropriately modeled using a mixture distribution, which consists of a sum of a K pdf for each scatterer type, weighted by the relative frequency with which they occur. In this type of modeling, the number of distributions must be selected before parameters can be fit, and as the number of components increases, so does the danger of overfitting the data. Several methods of model selection are explored here. Two methods, the Bayesian information criterion and the Akaike information criterion are based on point parameter estimates, and a penalty due to the number of model parameters. The other two, the deviance information criterion and the Watanabe-Akaike information criterion, are based on Monte-Carlo sampling of the model parameter space. These four model selection criteria are compared to each other, and to the authors manual selection of the number of components.

Machine learning for power outage prediction during hurricanes: An extensive review

The surge of machine learning (ML) applications and increasing usage of data driven approach for resilience enhancement provide great opportunities for applying ML techniques to power outage prediction (POP) during hurricanes. Considering the substantial damage that wind hazards can cause to power system network (PSN) in countries such as the USA and China, researchers have devised numerous approaches for disaster prevention to reduce their susceptibility. Since the selection of optimal ML algorithms for POP during hurricanes is a complex task, this paper helps researchers easily comprehend the numerous prediction techniques used worldwide for power outages during hurricanes. The paper provides a brief review of several categories of hurricanes and the damage it causes to the PSN more specifically to distribution side, the necessary measures to improve the PSN resilience, and finally a sequential approach for optimal ML model selection techniques that researchers and engineers can use to improve the POP during hurricanes. In this study the effectiveness of ML algorithms has been explored with the help of performance evaluation metrics which serve as the basic deciding criteria for optimal model selection. This research also highlights some of the key issues and challenges of integrating machine learning algorithms with existing PSN.

Global Model Selection via Solution Paths for Robust Support Vector Machine.

Robust support vector machine (RSVM) using ramp loss provides a better generalization performance than traditional support vector machine (SVM) using hinge loss. However, the good performance of RSVM heavily depends on the proper values of regularization parameter and ramp parameter. Traditional model selection technique with gird search has extremely high computational cost especially for fine-grained search. To address this challenging problem, in this paper, we first propose solution paths of RSVM (SPRSVM) based on the concave-convex procedure (CCCP) which can track the solutions of the non-convex RSVM with respect to regularization parameter and ramp parameter respectively. Specifically, we use incremental and decremental learning algorithms to deal with the Karush-Khun-Tucker violating samples in the process of tracking the solutions. Based on the solution paths of RSVM and the piecewise linearity of model function, we can compute the error paths of RSVM and find the values of regularization parameter and ramp parameter, respectively, which corresponds to the minimum cross validation error. We prove the finite convergence of SPRSVM and analyze the computational complexity of SPRSVM. Experimental results on a variety of benchmark datasets not only verify that our SPRSVM can globally search the regularization and ramp parameters respectively, but also show a huge reduction of computational time compared with the grid search approach.

A comparison between marginal likelihood and data augmented MCMC algorithms for Gaussian hidden Markov models

We provide a comparison between marginal likelihood and data augmented Markov chain Monte Carlo (MCMC) algorithms for Bayesian estimation of hidden Markov models. In particular, we focus on a specification of these models based on a multivariate Gaussian distribution for the response variables. We evaluate the performance of both methods on the basis of simulated samples and an empirical application, where we compare these approaches in terms of standard measures of estimation and time efficiency. The results show that the augmented algorithm is preferable, obtaining optimal convergence properties and producing larger values of the effective sample size. Instead of using variable dimensional techniques for model selection, we suggest a method based on the so-called parallel sampling for this aim.

Natural cubic splines for the analysis of Alzheimer’s trials

AbstractBackgroundMixed model repeated measures (MMRM) is the most common analysis approach in clinical trials for Alzheimer’s disease. The model treats time as a categorical variable, which can be problematic when assessments occur off‐schedule, as including off‐schedule visits can induce bias, and excluding them ignores valuable information and violates the intention to treat principle. This problem has been exacerbated by clinical trial visits which have been delayed due to the COVID19 pandemic. Since MMRM entails a mean parameter for each group and visit, the model can be inefficient for longer trials of Preclinical Alzheimer’s. As an alternative to MMRM, we propose a constrained longitudinal data analysis with natural cubic splines that treats time as continuous and uses test version effects to model the mean over time.MethodWe use model selection techniques to compare the predictive value of spline models to slope models, MMRM, and models that assume a proportional treatment effect using data from several completed trials. These trials include the ADCS trial of donepezil in MCI, trials of intranasal insulin and a fyn kinase inhibitor in mild Alzheimer’s dementia, and the A4 trial of solanezumab in Preclinical Alzheimer’s. Simulation studies assess the power and Type I error of these models for trials in Preclinical Alzheimer’s, MCI, and mild dementia.ResultThe spline model with ADAS‐Cog version effect attained the best predictive value for the trials in mild dementia. The proportional model was preferred for the trial in MCI, but the proportional effect assumption was violated, and the treatment effect estimate was in the wrong direction. Considering that the spline with two degrees of freedom and version effect provide fits that are similar to MMRM in Figure 1, we concluded that two degrees of freedom appear to be sufficient to capture enough of the general trend to assess the treatment effect. AIC results from the A4 Study will be considered and included after the study is completed in Spring 2023.ConclusionThe spline model is shown to have better predictive value and precision in real datasets and has better power and Type I error control in a variety of simulation scenarios.

The association of upper airway anatomy with cognitive test performance: the Multi-Ethnic Study of Atherosclerosis

BackgroundNumerous upper airway anatomy characteristics are risk factors for sleep apnea, which affects 26% of older Americans, and more severe sleep apnea is associated with cognitive impairment. This study explores the pathophysiology and links between upper airway anatomy, sleep, and cognition.MethodsParticipants in the Multi-Ethnic Study of Atherosclerosis underwent an upper airway MRI, polysomnography to assess sleep measures including the apnea-hypopnea index (AHI) and completed the Cognitive Abilities Screening Instrument (CASI). Two model selection techniques selected from among 67 upper airway measures those that are most strongly associated with CASI score. The associations of selected upper airway measures with AHI, AHI with CASI score, and selected upper airway anatomy measures with CASI score, both alone and after adjustment for AHI, were assessed using linear regression.ResultsSoft palate volume, maxillary divergence, and upper facial height were significantly positively associated with higher CASI score, indicating better cognition. The coefficients were small, with a 1 standard deviation (SD) increase in these variables being associated with a 0.83, 0.75, and 0.70 point higher CASI score, respectively. Additional adjustment for AHI very slightly attenuated these associations. Larger soft palate volume was significantly associated with higher AHI (15% higher AHI (95% CI 2%,28%) per SD). Higher AHI was marginally associated with higher CASI score (0.43 (95% CI 0.01,0.85) per AHI doubling).ConclusionsThree upper airway measures were weakly but significantly associated with higher global cognitive test performance. Sleep apnea did not appear to be the mechanism through which these upper airway and cognition associations were acting. Further research on the selected upper airway measures is recommended.

Enhancedb-value time-series calculation method using data-driven approach

SUMMARYThe temporal evolution analysis of b-value of the magnitude–frequency distribution (MFD) is essential for seismic risk analysis. However, ensuring the accuracy and rationality of these analyses depends on various factors, including data quality, data selection and the appropriate computation period partitioning. This study extends the data-driven b-value time-series calculation method, TbDD-BIC, by exploring different model selection techniques. To evaluate the method's effectiveness, we conducted assessments using both synthetic earthquake catalogues and actual seismic data. Our results indicate that selecting a proportion of optimal models (e.g. 5 per cent of the total number of models) using the Akaike information criterion (AIC) and computing the ensemble median yields accurate b0-values of the synthetic earthquake catalogue. This proposed method offers objective calculation rules and precise identification of abrupt b-value changes, enhancing seismicity simulation and seismic hazard analysis.

CARRoT: R-package for predictive modelling by means of regression, adjusted for multiple regularisation methods.

We present an R-package for predictive modelling, CARRoT (Cross-validation, Accuracy, Regression, Rule of Ten). CARRoT is a tool for initial exploratory analysis of the data, which performs exhaustive search for a regression model yielding the best predictive power with heuristic 'rules of thumb' and expert knowledge as regularization parameters. It uses multiple hold-outs in order to internally validate the model. The package allows to take into account multiple factors such as collinearity of the predictors, event per variable rules (EPVs) and R-squared statistics during the model selection. In addition, other constraints, such as forcing specific terms and restricting complexity of the predictive models can be used. The package allows taking pairwise and three-way interactions between variables into account as well. These candidate models are then ranked by predictive power, which is assessed via multiple hold-out procedures and can be parallelised in order to reduce the computational time. Models which exhibited the highest average predictive power over all hold-outs are returned. This is quantified as absolute and relative error in case of continuous outcomes, accuracy and AUROC values in case of categorical outcomes. In this paper we briefly present statistical framework of the package and discuss the complexity of the underlying algorithm. Moreover, using CARRoT and a number of datasets available in R we provide comparison of different model selection techniques: based on EPVs alone, on EPVs and R-squared statistics, on lasso regression, on including only statistically significant predictors and on stepwise forward selection technique.

Enhanced OCT chorio-retinal segmentation in low-data settings with semi-supervised GAN augmentation using cross-localisation

Training deep learning methods for optical coherence tomography (OCT) retinal and choroidal layer segmentation is a challenge when data is scarce. In medical image analysis, this is often the case with a lack of data sharing due to confidentiality agreements and data privacy concerns which is further exacerbated in cases of rare pathologies. Even where OCT data is readily available, performing the requisite annotations is time consuming, costly, and error-prone. Data augmentation and semi-supervised learning (SSL) are two techniques employed in deep learning to enhance training in these situations. In this study, we extend our previous work proposing an enhanced StyleGAN2-based data augmentation method for OCT images by employing SSL through a novel cross-localisation technique. The technique increases the diversity of the synthetic data by automatically incorporating styles from unlabelled data with those from labelled data. The method can be used to extend StyleGAN2 as the core idea is simple, yet highly performant. In this work, we optimise the method through a set of ablations and propose the use of a targeted task-specific model selection technique for more optimal generator selection, further boosting performance. The method is applied to OCT retinal and choroidal layer segmentation, demonstrating its effectiveness through substantial patch classification performance improvements as well as significant reductions in choroidal layer segmentation error.

Evaluation and mapping of predicted future land use changes using hybrid models in a coastal area

Future prediction modeling of land use/land cover (LULC) is crucial for coastal regions due to unique challenges and vulnerabilities associated with these areas. This research aims to evaluate the performance of three models: Logistic Regressor (LR), Artificial Neural Network (ANN), and Weight of Evidence (CA-WoE) combined with Cellular Automata (CA) to predict the future LULC change for the Sundarbans coastal area, straddling the international boundary of Bangladesh and India. Hybrid models of CA-LR, CA-ANN, and CA-WoE were applied to predict LULC changes for the Sundarbans deltaic region for 2030 and 2050, based on a host of spatial and environmental variables. The supervised machine learning algorithm of the Random Forest (RF) classification model was used to classify satellite images during 1990–2020 to create LULC maps. Image classification was based on four distinct LULC classes: (i) mangroves, (ii) waterbodies, (iii) built-up, and (iv) barren land. Respective percentage correctness (PCN) for model types of CA-LR, CA-ANN, and CA-WoE was 90.82%, 92.06%, and 92.12% with overall Cohen's kappa metrics (OKM) of 0.815, 0.841, and 0.842, respectively. Overall, CA-WoE model performance was superior to CA-ANN and CA-LR models in predicting LULC maps for 2030 and 2050. Model results suggest that future mangrove forest area in the Sundarbans will decrease, and waterbodies will increase in area by 2050. The findings of this study may guide future sustainable land use management in the Sundarbans. This study also provides model selection techniques for predicting coastal LULC changes worldwide.

Automated component analysis in DOSY NMR using information criteria

This study introduces a model selection technique based on Bayesian information criteria for estimating the number of components in a mixture during Diffusion-Ordered Spectroscopy (DOSY) Nuclear Magnetic Resonance (NMR) data analysis. As the accuracy of this technique is dependent on the efficiency of parameter estimators, we further investigate the performance of the Weighted Least Squares (WLS) and Maximum a Posteriori (MAP) estimators. The WLS method, enhanced with meticulously tuned L2-regularization, effectively detects components when the difference in self-diffusion coefficients is more than two-fold, especially when the component with the smaller coefficient has a larger weight ratio. The MAP method, strengthened by a substantial database of prior information, exhibits outstanding precision, decreasing this threshold to 1.5 times. Both estimators provide weight ratio estimates with standard deviations of approximately around 1 percentage point, although the MAP method tends to overestimate the component with a larger self-diffusion coefficient. Deviations from the expected values can exceed 10 percentage points, often due to inaccuracies in component detection. The error estimates are determined using data resampling techniques derived from a large-scale 1000-point experiment and an additional five measurements from a single-component mixture. This approach allowed us to thoroughly examine data distribution characteristics, thereby laying a robust groundwork for future refinement efforts.

Radiomics characterization of tissues in an animal brain tumor model imaged using dynamic contrast enhanced (DCE) MRI

Here, we investigate radiomics-based characterization of tumor vascular and microenvironmental properties in an orthotopic rat brain tumor model measured using dynamic-contrast-enhanced (DCE) MRI. Thirty-two immune compromised-RNU rats implanted with human U-251N cancer cells were imaged using DCE-MRI (7Tesla, Dual-Gradient-Echo). The aim was to perform pharmacokinetic analysis using a nested model (NM) selection technique to classify brain regions according to vasculature properties considered as the source of truth. A two-dimensional convolutional-based radiomics analysis was performed on the raw-DCE-MRI of the rat brains to generate dynamic radiomics maps. The raw-DCE-MRI and respective radiomics maps were used to build 28 unsupervised Kohonen self-organizing-maps (K-SOMs). A Silhouette-Coefficient (SC), k-fold Nested-Cross-Validation (k-fold-NCV), and feature engineering analyses were performed on the K-SOMs’ feature spaces to quantify the distinction power of radiomics features compared to raw-DCE-MRI for classification of different Nested Models. Results showed that eight radiomics features outperformed respective raw-DCE-MRI in prediction of the three nested models. The average percent difference in SCs between radiomics features and raw-DCE-MRI was: 29.875% ± 12.922%, p < 0.001. This work establishes an important first step toward spatiotemporal characterization of brain regions using radiomics signatures, which is fundamental toward staging of tumors and evaluation of tumor response to different treatments.

An application of copulas to OPEC’s changing influence on fossil fuel prices

This work examines how the dependence structures between energy futures asset prices differ in two periods identified before and after the 2008 global financial crisis. These two periods were characterized by a difference in the number of extraordinary meetings of OPEC countries organized to announce a change of oil production. In the period immediately following the global financial crisis, the decrease in oil prices and oil and gas demand forced OPEC countries to make frequent adjustments to the production of oil, while, since the first quarter of 2010, the recovery led to more regular meetings, with only three organized extraordinary meetings. We propose to use a copula model to study how the dependence structure among energy prices changed among the two periods. The use of copula models allows to introduce flexible and realistic models for the marginal time series; once marginal parameters are estimated, the estimates are used to fit several copula models for all asset combinations. Model selection techniques based on information criteria are implemented to choose the best models both for the univariate asset prices series and for the distribution of co-movements. The changes in the dependence structure of couple of assets are investigated through copula functionals and their uncertainty estimated through a bootstrapping method. We find the strength of dependence between asset combinations considerably differ between the two periods, showing a significant decrease for all the pairs of assets.