Model Selection Framework Research Articles

Multiset Model Selection

ABSTRACTModel selection algorithms are required to efficiently traverse the space of models. In problems with high-dimensional and possibly correlated covariates, efficient exploration of the model space becomes a challenge. To overcome this, a multiset is placed on the model space to enable efficient exploration of multiple model modes with minimal tuning. The multiset model selection (MSMS) framework is based on independent priors for the parameters and model indicators on variables. Posterior model probabilities can be easily obtained from multiset averaged posterior model probabilities in MSMS. The effectiveness of MSMS is demonstrated for linear and generalized linear models. Supplementary material for this article is available online.

Comparison of model selection technique performance in predicting the spread of newly invasive species: a case study with Batrachochytrium salamandrivorans

Species distribution models (SDMs) increasingly have been used to anticipate the spread of invasive species. However, these models are powerful conservation tools only if they are biologically relevant, and thus validation of these models is essential. Here, we evaluate four model selection frameworks for their ability to identify a best fit model of spread under low data conditions early in an invasion, specifically testing the efficacy of methods that utilize absence data in addition to presence data in evaluating models. We test this question using a simulation where we generated data with varying confidence in the accuracy of the absence data, as absences in early invasions may become presences in the future, and increasing quantity of observation data to test the models. We create these simulations based on a real-world example of a newly emergent, invasive fungal pathogen, Batrachochytrium salamandrivorans (Bsal). The simulation demonstrated that AIC and Likelihood consistently outperform both Kappa and AUC in selecting the true model as the best model when data are limited and absence data are low quality, with AIC providing the most conservative results due to penalties for overparameterization. With these results, we then used these techniques to compare five candidate models for predicting the spread of Bsal. Consistent with the simulation, the best fit model of the candidate models for Bsal was inconsistent across the four metrics. However, AIC, which performed best in the simulation study, suggested that the spread of Bsal into Western Europe was best predicted by a combination of bioclimatic suitability, salamander richness, and number of salamander imports. Our results highlight the difficulty in evaluating predictive models when data are limited and of low quality, as with a newly invasive species, but show that these challenges can be partially addressed with the appropriate model selection approach. Use of this approach is critical as SDMs of invasive species are often used to inform conservation policy and efforts before the invasion spreads, when limited data are available.

Advancing understanding and prediction in multiple stressor research through a mechanistic basis for null models.

Global environmental change is driven by multiple anthropogenic stressors. Conservation and restoration require understanding the individual and joint action of these stressors to evaluate and prioritize management measures. To date, most studies on multiple stressor effects have sought to identify potential stressor interactions, defined as deviations from null models, and related meta-analyses have focused on quantifying the relative proportion of stressor interactions across studies. These studies have provided valuable insights about the complexity of multiple stressor effects, but remain largely devoid of a theoretical framework for null model selection and prediction of effects. We suggest that multiple stressor research would benefit by (1) integrating and developing additional null models and (2) selecting null models based on their mechanistic assumptions of the stressor mode of action and organism sensitivities as well as stressor-effect relationships for individuals and populations. We present a range of null models and outline their underlying assumptions and application in multiple stressor research. Moving beyond mere description requires multiple stressor research to shift its focus from identifying statistically significant interactions to the use and development of mechanistic (null) models. Justified selection of the appropriate null model is a first step to achieve this.

Combining allele frequency and tree-based approaches improves phylogeographic inference from natural history collections.

Model selection approaches in phylogeography have allowed researchers to evaluate the support for competing demographic histories, which provides a mode of inference and a measure of uncertainty in understanding climatic and spatial influences on intraspecific diversity. Here, to rank all models in the comparison set and determine what proportion of the total support the top-ranked model garners, we conduct model selection using two analytical approaches-allele frequency-based, implemented in fastsimcoal2, and gene tree-based, implemented in phrapl. We then expand this model selection framework by including an assessment of absolute fit of the models to the data. For this, we utilize DNA isolated from existing natural history collections that span the distribution of red alder (Alnus rubra) in the Pacific Northwest of North America to generate genomic data for the evaluation of 13 demographic scenarios. The quality of DNA recovered from herbarium specimen leaf tissue was assessed for its utility and effectiveness in demographic model selection, specifically in the two approaches mentioned. We present strong support for the use of herbarium tissue in the generation of genomic DNA, albeit with the inclusion of additional quality control checks prior to library preparation and analyses with multiple approaches that incorporate various data. Analyses with allele frequency spectra and gene trees predominantly support A.rubra having experienced an ancient vicariance event with intermittent and frequent gene flow between the disjunct populations. Additionally, the data consistently fit the most frequently selected model, corroborating the model selection techniques. Finally, these results suggest that the A.rubra disjunct populations do not represent separate species.

Demographic analysis of cyanobacteria based on the mutation rates estimated from an ancient ice core.

Despite the crucial role of cyanobacteria in various ecosystems, little is known about their evolutionary histories, especially microevolutionary dynamics, because of the lack of knowledge regarding their mutation rates. Here we directly estimated cyanobacterial mutation rates based on ancient DNA analyses of ice core samples collected from Kyrgyz Republic that dates back to ~12,500 cal years before present. We successfully sequenced the 16S rRNA and 16S-23S internal transcribed spacer (ITS) region. Two cyanobacterial operational taxonomic units (OTUs) were detected from the ancient ice core samples, and these OTUs are shared with those from the modern glacier surface. The mutation rate of ITS region was estimated by comparing ancient and modern populations, and were at the magnitude of 10-7substitutions/sites/year. By using a model selection framework, we also demonstrated that the ancient sequences from the ice sample were not contaminated from modern samples. Bayesian demographic analysis based on coalescent theory revealed that cyanobacterial population sizes increased over Asia regions during the Holocene. Thus, our results enhance our understanding of the enigmatic timescale of cyanobacterial microevolution, which has the potential to elucidate the environmental responses of cyanobacteria to the drastic climatic change events of the Quaternary.

The ontogenetic allometry of body morphology and chemical composition in dairy goat wethers

We studied the ontogenetic growth of goat wethers (castrated male goats) of the Saanen and Swiss Alpine breeds based on a large range of intraspecific body mass (BM). The body parts and the chemical constituents of the empty body were described by the allometric function by using BM and the empty body mass (EBM) as the predictors for morphological traits and chemical composition, respectively. We fitted the allometric scaling function by applying the SAS NLMIXED procedure, but to evaluate assumptions regarding variances in morphological and compositional traits, we combined the scaling function with homoscedastic (MOD1), and the heteroscedastic exponential (MOD2) and power-of-the-mean (MOD3) variance functions. We also predicted the ontogenetic growth by using the traditional log-log transformation and back-transformed results into the arithmetic scale (MOD4). We obtained predictions from MOD4 in the arithmetic scale by a two-step process, and evaluated MOD1, MOD2 and MOD3 by a model selection framework, and compared MOD4 with MOD1, MOD2 and MOD3 based on goodness-of-fit measures. Based on information criteria for model selection, heterogeneous variance functions were more likely to describe 10 over 36 traits with a low level of model selection uncertainty. One trait was predicted by averaging the MOD1 and MOD2 variance functions; and nine traits were better described by averaging the MOD2 and MOD3 variance functions. The predictions for other 16 traits were averaged from MOD1, MOD2 and MOD3. However, MOD4 better described 11 traits according to the goodness-of-fit measures. Depending on the variable being analyzed, the body parts and the chemical amounts exhibited the three types of allometric behavior with respect to BM and EBM, that is, positive, negative and isometric ontogenetic growth. Reference BMs, that is, 20, 27, 35 and 45 kg, were used to compute the net protein and energy requirements based on the first derivative of the scaling function, and the results were presented in reference to the EBM and EBM0.75. Both the net protein and energy requirements scaled to EBM0.75 increased from 20 to 45 kg of BM.

Statistical Estimation of Surface Heat Control and Exchange in Endotherms

In this paper, we combine polynomial functions, Generalized Estimating Equations, and bootstrap-based model selection to test for signatures of linear or nonlinear relationships between body surface temperature and ambient temperature in endotherms. Linearity or nonlinearity is associated with the absence or presence of cutaneous vasodilation and vasoconstriction, respectively. We obtained experimental data on body surface temperature variation from a mammalian model organism as a function of ambient temperature using infrared thermal imaging. The statistical framework of model estimation and selection successfully detected linear and nonlinear relationships between body surface temperature and ambient temperature for different body regions of the model organism. These results demonstrate that our statistical approach is instrumental to assess the complexity of thermoregulation in endotherms.

Integration of artificial intelligence and production data analysis for shale heterogeneity characterization in steam-assisted gravity-drainage reservoirs

Steam-Assisted Gravity Drainage (SAGD) recovery is strongly impacted by distributions of heterogeneous shale barriers, which impede the vertical growth and lateral spread of a steam chamber and potentially reduce oil production. Conventional reservoir heterogeneities characterization workflows that entail updating static reservoir models with dynamic flow data are quite time-consuming. Furthermore, certain assumptions are often needed to approximate the complex physical processes. This study proposes a workflow integrating artificial intelligence (AI) in a model selection framework that aims to identify associated shale heterogeneities in SAGD reservoir based on extracted features from production time-series data.A series of SAGD models based on typical Athabasca oil reservoir properties and operating conditions is constructed. After constructing the base homogeneous model, the shale barriers are assigned randomly by sampling their location, lateral extent, and thickness from several probability distributions, which are inferred from field data assembled from the public domain. Sensitivity analysis is carried out to identify and analyze features in the production response that are related to shale characteristics: whenever the steam chamber encounters a shale barrier, a drop in the production is observed; this drop continues until the steam chamber has advanced past the shale barrier, and the production would rise again. Several types of input feature extraction methods are introduced in this work: piecewise linear approximation (PLA), cubic spline interpolation (CSI), and discrete wavelet transform (DWT). Next, artificial neural network (ANN) is constructed to calibrate a relationship between the retrieved production pattern parameters (inputs) and the corresponding geologic parameters describing shale heterogeneities (outputs), which include some variables capturing the location, orientation or size of a particular shale barrier encountered by the steam chamber. The final model is implemented in a novel characterization workflow to infer shale heterogeneities from production profiles. A number of realistic applications are presented to illustrate its utility.The ANN models are validated using numerous synthetic models, where the exact shale distributions are known. The trained ANN models can reliably estimate the relevant shale parameters and the associated uncertainties, while accurately predicting the corresponding production responses. It is intended to extend the proposed method to construct the ANN models directly from well logs and production data.This work presents a preliminary attempt in correlating stochastic shale parameters with observable features in production time-series data using AI techniques. The proposed method facilitates the selection of an ensemble of reservoir models that are consistent with the production history; these models can be subjected to further history-matching for a precise final match. The proposed methodology does not intend to replace traditional simulation and history-matching workflows, but it rather offers a complementary tool for extracting additional information from field data and incorporating AI-based models into practical reservoir modeling workflows.

Multiple long-term trends and trend reversals dominate environmental conditions in a man-made freshwater reservoir

Man-made reservoirs are common across the world and provide a wide range of ecological services. Environmental conditions in riverine reservoirs are affected by the changing climate, catchment-wide processes and manipulations with the water level, and water abstraction from the reservoir. Long-term trends of environmental conditions in reservoirs thus reflect a wider range of drivers in comparison to lakes, which makes the understanding of reservoir dynamics more challenging. We analysed a 32-year time series of 36 environmental variables characterising weather, land use in the catchment, reservoir hydrochemistry, hydrology and light availability in the small, canyon-shaped Římov Reservoir in the Czech Republic to detect underlying trends, trend reversals and regime shifts. To do so, we fitted linear and piecewise linear regression and a regime shift model to the time series of mean annual values of each variable and to principal components produced by Principal Component Analysis. Models were weighted and ranked using Akaike information criterion and the model selection approach. Most environmental variables exhibited temporal changes that included time-varying trends and trend reversals. For instance, dissolved organic carbon showed a linear increasing trend while nitrate concentration or conductivity exemplified trend reversal. All trend reversals and cessations of temporal trends in reservoir hydrochemistry (except total phosphorus concentrations) occurred in the late 1980s and during 1990s as a consequence of dramatic socioeconomic changes. After a series of heavy rains in the late 1990s, an administrative decision to increase the flood-retention volume of the reservoir resulted in a significant regime shift in reservoir hydraulic conditions in 1999. Our analyses also highlight the utility of the model selection framework, based on relatively simple extensions of linear regression, to describe temporal trends in reservoir characteristics. This approach can provide a solid basis for a better understanding of processes in freshwater reservoirs.

Concurrent surrogate model selection (COSMOS): optimizing model type, kernel function, and hyper-parameters

This paper presents an automated surrogate model selection framework called the Concurrent Surrogate Model Selection or COSMOS. Unlike most existing techniques, COSMOS coherently operates at three levels, namely: 1) selecting the model type (e.g., RBF or Kriging), 2) selecting the kernel function type (e.g., cubic or multiquadric kernel in RBF), and 3) determining the optimal values of the typically user-prescribed hyper-parameters (e.g., shape parameter in RBF). The quality of the models is determined and compared using measures of median and maximum error, given by the Predictive Estimation of Model Fidelity (PEMF) method. PEMF is a robust implementation of sequential k-fold cross-validation. The selection process undertakes either a cascaded approach over the three levels or a more computationally-efficient one-step approach that solves a mixed-integer nonlinear programming problem. Genetic algorithms are used to perform the optimal selection. Application of COSMOS to benchmark test functions resulted in optimal model choices that agree well with those given by analyzing the model errors on a large set of additional test points. For the four analytical benchmark problems and three practical engineering applications --- airfoil design, window heat transfer modeling, and building energy modeling --- diverse forms of models/kernels are observed to be selected as optimal choices. These observations further establish the need for automated multi-level model selection that is also guided by dependable measures of model fidelity.

Territory surveillance and prey management: Wolves keep track of space and time.

Identifying behavioral mechanisms that underlie observed movement patterns is difficult when animals employ sophisticated cognitive‐based strategies. Such strategies may arise when timing of return visits is important, for instance to allow for resource renewal or territorial patrolling. We fitted spatially explicit random‐walk models to GPS movement data of six wolves (Canis lupus; Linnaeus, 1758) from Alberta, Canada to investigate the importance of the following: (1) territorial surveillance likely related to renewal of scent marks along territorial edges, to reduce intraspecific risk among packs, and (2) delay in return to recently hunted areas, which may be related to anti‐predator responses of prey under varying prey densities. The movement models incorporated the spatiotemporal variable “time since last visit,” which acts as a wolf's memory index of its travel history and is integrated into the movement decision along with its position in relation to territory boundaries and information on local prey densities. We used a model selection framework to test hypotheses about the combined importance of these variables in wolf movement strategies. Time‐dependent movement for territory surveillance was supported by all wolf movement tracks. Wolves generally avoided territory edges, but this avoidance was reduced as time since last visit increased. Time‐dependent prey management was weak except in one wolf. This wolf selected locations with longer time since last visit and lower prey density, which led to a longer delay in revisiting high prey density sites. Our study shows that we can use spatially explicit random walks to identify behavioral strategies that merge environmental information and explicit spatiotemporal information on past movements (i.e., “when” and “where”) to make movement decisions. The approach allows us to better understand cognition‐based movement in relation to dynamic environments and resources.

Developing approaches for linear mixed modeling in landscape genetics through landscape-directed dispersal simulations.

Dispersal can impact population dynamics and geographic variation, and thus, genetic approaches that can establish which landscape factors influence population connectivity have ecological and evolutionary importance. Mixed models that account for the error structure of pairwise datasets are increasingly used to compare models relating genetic differentiation to pairwise measures of landscape resistance. A model selection framework based on information criteria metrics or explained variance may help disentangle the ecological and landscape factors influencing genetic structure, yet there are currently no consensus for the best protocols. Here, we develop landscape‐directed simulations and test a series of replicates that emulate independent empirical datasets of two species with different life history characteristics (greater sage‐grouse; eastern foxsnake). We determined that in our simulated scenarios, AIC and BIC were the best model selection indices and that marginal R 2 values were biased toward more complex models. The model coefficients for landscape variables generally reflected the underlying dispersal model with confidence intervals that did not overlap with zero across the entire model set. When we controlled for geographic distance, variables not in the underlying dispersal models (i.e., nontrue) typically overlapped zero. Our study helps establish methods for using linear mixed models to identify the features underlying patterns of dispersal across a variety of landscapes.

Unbiased risk estimates for matrix estimation in the elliptical case

This paper is concerned with additive models of the form Y = M + E , where Y is an observed n × m matrix with m < n , M is an unknown n × m matrix of interest with low rank, and E is a random noise whose distribution is elliptically symmetric. For general estimators M ̂ of M , we develop unbiased risk estimates, including in the special case where E is Gaussian with covariance matrix proportional to the identity matrix. To this end, we develop a new Stein–Haff type identity. We apply the theory to a model selection framework with estimators defined through a soft-thresholding function. We establish the robustness of our approach within a large subclass of elliptical distributions.

A decomposition model and voxel selection framework for fMRI analysis to predict neural response of visual stimuli.

This paper presents an fMRI signal analysis methodology using geometric mean curve decomposition (GMCD) and mutual information-based voxel selection framework. Previously, the fMRI signal analysis has been conducted using empirical mean curve decomposition (EMCD) model and voxel selection on raw fMRI signal. The erstwhile methodology loses frequency component, while the latter methodology suffers from signal redundancy. Both challenges are addressed by our methodology in which the frequency component is considered by decomposing the raw fMRI signal using geometric mean rather than arithmetic mean and the voxels are selected from EMCD signal using GMCD components, rather than raw fMRI signal. The proposed methodologies are adopted for predicting the neural response. Experimentations are conducted in the openly available fMRI data of six subjects, and comparisons are made with existing decomposition models and voxel selection frameworks. Subsequently, the effect of degree of selected voxels and the selection constraints are analyzed. The comparative results and the analysis demonstrate the superiority and the reliability of the proposed methodology.

Dietary diversity, feeding selectivity, and responses to fruit scarcity of two sympatric Bornean primates (Hylobates albibarbis and Presbytis rubicunda rubida).

Effectively characterizing primate diets is fundamental to understanding primate behavior, ecology and morphology. Examining temporal variation in a species’ diet, as well as comparing the responses of different species to variation in resource availability, can enhance understanding of the evolution of morphology and socioecology. In this study, we use feeding data collected over five years to describe the diets of two sympatric Southeast Asian primate species of similar body size: white-bearded gibbons (Hylobates albibarbis) and red leaf monkeys (Presbytis rubicunda rubida), in Gunung Palung National Park, West Kalimantan, Indonesia. Long-term data sets are especially important for characterizing primate diets in Southeast Asia, where the forests exhibit supra-annual mast fruiting events. We found that gibbons were mainly frugivorous, with fruit and figs comprising 70% of their 145 independent feeding observations, whereas leaf monkeys ate a substantial amount of seeds (26%), fruits and figs (26.5%) and leaves (30%, n = 219 independent feeding observations). Leaf monkeys consumed a higher number of plant genera, and this was due mostly to the non-frugivorous portion of their diet. To investigate resource selection by these primates we utilized two different approaches: the Manly Selectivity Ratio, which did not take into account temporal variation of resource availability, and a model selection framework which did incorporate temporal variation. Both species selected figs (Ficus) more than predicted based on their availability under the Manly Selectivity Ratio. Model selection allowed us to determine how these primates alter the proportion of leaves, flowers, seeds, figs and fruit in their diets in response to variation in fruit availability. When fruits were scarce, both gibbons and leaf monkeys incorporated more leaves and figs into their diets, indicating that these two food classes are fallback foods for these primates. We discuss how different measures of resource selection can provide seemingly contradictory results, and emphasize the importance of long term studies that combine independent feeding observations with rigorous assessment of temporal variation in resource availability when modelling feeding selectivity.

A Bayesian approach for envelope models

The envelope model is a new paradigm to address estimation and prediction in multivariate analysis. Using sufficient dimension reduction techniques, it has the potential to achieve substantial efficiency gains compared to standard models. This model was first introduced by [Statist. Sinica 20 (2010) 927–960] for multivariate linear regression, and has since been adapted to many other contexts. However, a Bayesian approach for analyzing envelope models has not yet been investigated in the literature. In this paper, we develop a comprehensive Bayesian framework for estimation and model selection in envelope models in the context of multivariate linear regression. Our framework has the following attractive features. First, we use the matrix Bingham distribution to construct a prior on the orthogonal basis matrix of the envelope subspace. This prior respects the manifold structure of the envelope model, and can directly incorporate prior information about the envelope subspace through the specification of hyperparamaters. This feature has potential applications in the broader Bayesian sufficient dimension reduction area. Second, sampling from the resulting posterior distribution can be achieved by using a block Gibbs sampler with standard associated conditionals. This in turn facilitates computationally efficient estimation and model selection. Third, unlike the current frequentist approach, our approach can accommodate situations where the sample size is smaller than the number of responses. Lastly, the Bayesian approach inherently offers comprehensive uncertainty characterization through the posterior distribution. We illustrate the utility of our approach on simulated and real datasets.

Abstract A11: Calibration, selection and validation of tumor growth models

Abstract As a result of the highly heterogeneous tumor microenvironment and a series of complex phenomena that occur at different scales, several modeling approaches have emerged. The tumor growth models can be categorized by the behaviors they aim to capture, the hallmarks of cancer that are being considered, and the temporal and biological scales represented. Moreover, these models can also be differentiated according to the mathematical framework that is being employed, being either discrete models, continuum models, or hybrid models. Therefore, with this large set of possible models, and with each model choice having many unknown parameters, it is important to develop an adaptive model selection and validation framework in order to choose the best model for the desired application. In this work, we employ the Occam Plausibility Algorithm (OPAL) framework [1] for model selection and validation of a large body of tumor growth models. We select several continuum tumor growth models, half of them using the phase-field modeling approach, derived from the models developed in [2], and the other half being reaction-diffusion type models derived from [3]. The OPAL framework brings together model calibration, determination of sensitivities of outputs to parameter variances, and calculation of model plausibilities for model selection. Firstly, we identify the model classes and compute the sensitivity of the quantity of interest (QoI), here the tumor volume as a function of time. The models with parameters that do not influence the QoI are eliminated. The next step consists in dividing the models into Occam categories according to their number of parameters, where the models with fewer parameters are assembled to category 1. Following the principle of Occam's Razor, one of main goals of OPAL is to select the simplest model among the several models that leads to the same prediction. The models from the first category are calibrated using Bayes' rule for a specific calibration scenario. The next step is to compute the plausibilities for the models in this category and choose the most plausible model. Finally, with the most plausible model known, we solve the problem in the validation scenario. The model is valid depending on the accuracy with which the model predictions of the QoI agree with experimental data. If the model is not valid, we move to the next category; otherwise, we select this model as being the simplest and the most plausible valid model. We demonstrate that the OPAL is able to assist in the selection of the best tumor growth model from a defined set of models. The model selected is valid for particular scenarios. In this work, we consider avascular tumor growth models to reproduce the data obtained in a murine model of glioma growth. The brain data is generated with diffusion weighted magnetic resonance imaging over a period of ten days. The methodologies and results confirm that the OPAL strategy provides a powerful framework for model selection, parameters estimation, and model validation in the presence of uncertainties.

Model selection for CO2 sequestration using surface deformation and injection data

Incorporating time-lapse observations from sources such as remote sensing and seismic is important for monitoring the migration of CO2 plumes in carbon capture and storage projects. Significant computational costs make it impractical to run coupled flow-geomechanical simulations for all plausible geologic scenarios in a prior ensemble. This study presents a model selection framework that integrates a fast approximator as a proxy to evaluate flow and geomechanical responses of geologic models. The proxy utilizes a particle tracking algorithm to mimic flow paths of injected CO2 within the geologic models. Pressure changes and rock deformation resulting from CO2 injection are estimated using a finite-element algorithm. The reliability of the proposed proxy is tested by comparing simulation and proxy results with regards to the shapes and extents of CO2 plumes, reservoir pressure, and vertical displacement at the top layer of a given reservoir. Models showing similar proxy responses are grouped into clusters by invoking multi-dimensional scaling followed by k-means clustering. A representative model of each cluster is selected, and its dynamic responses are evaluated by running flow-geomechanical simulations. The posterior ensemble consists of the models in the cluster whose representative conforms to given observation data. The proposed model selection approach is applied to history matching of a realistic channelized reservoir and a fractured reservoir inspired from In Salah, Algeria. The two case studies demonstrate that the incorporation of surface deformation data within model selection contributes to reduction in geologic uncertainty by improving the matching quality of well responses.

A Bayesian model selection analysis of equilibrium and nonequilibrium models for multiphase flow in porous media

The classical constitutive relations for multiphase flows in porous media assume instantaneous and local phase-equilibrium. Several alternative nonequilibrium/dynamic constitutive relations have been proposed in the literature including the works of Barenblatt, and Hassanizadeh and Gray. This work applies a Bayesian model selection framework in order to examine the relative efficacy of these three models to represent experimental observations. Experimental observations of multiphase displacement processes in natural porous media are often sparse and indirect, leading to considerable uncertainty in control conditions. Data from three core-scale drainage experiments are considered. Gaussian prior probability models are assumed for key multiphase flow parameters and measurements. Accurate numerical simulation approximations using the three constitutive relation models are implemented. The model selection analysis comprises a data-assimilation stage that calibrates the assumed model to the data while quantifying uncertainty. The second stage is the computation of the maximum likelihood estimate and its application to compute the Bayesian Information Criterion. It is observed that Barenblatt’s nonequilibrium model is more likely to match data from unstable displacements that involve higher viscosity ratios of the invading phase to the resident fluid. At the lowest viscosity ratio, there is no delineation between the goodness of fit obtained using the classical model and the model proposed by Hassanizadeh and Gray, and both outperform Barenblatt’s nonequilibrium model.

Generalization Error Minimization: A New Approach to Model Evaluation and Selection with an Application to Penalized Regression

We study model evaluation and model selection from the perspective of generalization ability (GA): the ability of a model to predict outcomes in new samples from the same population. We believe that GA is one way formally to address concerns about the external validity of a model. The GA of a model estimated on a sample can be measured by its empirical out-of-sample errors, called the generalization errors (GE). We derive upper bounds for the GE, which depend on sample sizes, model complexity and the distribution of the loss function. The upper bounds can be used to evaluate the GA of a model, ex ante. We propose using generalization error minimization (GEM) as a framework for model selection. Using GEM, we are able to unify a big class of penalized regression estimators, including lasso, ridge and bridge, under the same set of assumptions. We establish finite-sample and asymptotic properties (including L2-consistency) of the GEM estimator for both the n > p and the n