Inform Model Selection Research Articles

Improving reduced complexity model assessment and usability.

Reduced complexity climate models are useful tools with practical policy applications, yet evaluation of their performance and application is nascent. We call for stakeholder-driven development and assessment to address user needs, including provision of open-source code and guidance to inform model selection and application.

ON4 Heterogeneity in Survival with Immuno-Oncologic Therapy and Its Implications for Survival Extrapolations: A Case Study in Untreated Metastatic Melanoma

Short clinical trial follow-up periods at the time of submission and heterogeneity in outcomes often introduce uncertainty in the health technology assessment of immuno-oncologic (IO) therapies. To address this, some agencies prefer to use conservative methods for survival extrapolation. To evaluate the predictive accuracy of various survival extrapolation methods using data from CheckMate 067, a phase III trial of nivolumab plus ipilimumab (NIVO+IPI) vs either monotherapy in untreated metastatic melanoma. Six different extrapolation models were independently fitted to overall survival (OS) data for the three arms of CheckMate 067 using successive database locks (DBL) at: 28, 40, 52 and 60 months. NICE DSU TSD 14 informed model selection. 60-month follow-up from CheckMate 067, as well as external data for ipilimumab, were used to evaluate the predictive accuracy of the survival models. Extrapolated 20-year mean survival times (20-YR MST) were also compared with a 5% discount rate applied. For the 28-month DBL, standard parametric models, piecewise models and splines underestimated 60-month survival (range -3.8% to -11.7% across methods/arms). Mixture cure models (MCM), parametric mixture models (PMM) and response-based landmark models (RBLM) provided more accurate estimates (range -3.2% to +2.0% across methods/arms). MCM, PMM and RBLM produced closer estimates of 7-year and 10-year OS when compared to the external data for ipilimumab than other methods. 20-YR MSTs were higher for MCM, PMM and RBLM (NIVO+IPI: 6.02-7.02 years) than those for other models (NIVO+IPI: 4.56-5.62 years). Estimates of 20-YR MST for MCM, PMM and RBLM were more consistent across the successive database locks than with other methods. MCM, PMM and RBLM provided accurate and consistent estimates of OS across a range of follow-up periods. Extrapolation models that fail to account for heterogeneity in survival may result in biased assessments of the survival benefit of IO therapies for melanoma.

Investigating the implications of shifting baseline syndrome on conservation

Abstract Shifting baseline syndrome (SBS) describes a persistent downgrading of perceived ‘normal’ environmental conditions with every sequential generation, leading to under‐estimation of the true magnitude of long‐term environmental change on a global scale. The presence of SBS should be considered when local ecological knowledge and participatory techniques are involved in conservation target‐setting. However, despite increasing recognition of the phenomenon, there is little empirical evidence for SBS. Here we provide evidence of SBS, and the first empirical investigation of the impacts of SBS on public perceptions of conservation need. Large‐scale online questionnaires were used to collect public perceptions of long‐term biological change regarding 10 UK bird species, as well as demographic information and measures of knowledge and experience of the local environment (n = 330). A paired data approach compared social perceptions to a large‐scale longitudinal biological dataset. Using information theoretic and model selection techniques, we estimate the relative importance of multiple demographic, social and psychological predictors of SBS. We provide a framework for investigating evidence of SBS and its impacts on perceptions of conservation need for species in decline. Evidence of generational amnesia was found as an age‐related difference in perceptions of past ecological conditions. The perceptions of older participants had significantly higher agreement with biological data than the perceptions of younger participants. Our results therefore support the expectation that younger, less experienced people are less aware of historical ecological conditions and show greater evidence of SBS. We also present evidence of a negative impact of SBS on future conservation, as older people were more likely than younger people to perceive a greater need for conservation action for three declining species. Our research supports the need to encourage greater intergenerational communication and increase experience of local nature. Discovering evidence of SBS in public perceptions of species experienced within everyday life demonstrates SBS as a pervasive social issue with the potential to impact public perceptions of local nature. A free Plain Language Summary can be found within the Supporting Information of this article.

Equifinality in the modelling of ammonia volatilisation from a flooded rice system

Equifinality is the capability of models to produce similar model-output responses. The objective of this study is to use a numerical experimentation to investigate the equifinality of six modules for estimating ammonia (NH3) volatilisation from the floodwater in a rice system. Except for the Chowdary's, all modules can simulate the fluctuating trends of NH3 volatilisation rates. NFLOOD v.1 shows low equifinality to Jayaweera and Mikkelsen's and the regression equations in CERES-Rice, DSSAT-CSM and APSIM-Oryza in estimating the NH3 volatilisation. The equation in APSIM-Oryza has high equifinality to that in CERES-Rice. Both regression equations have low equifinality to the regression equation in DSSAT-CSM. All three regression equations show low equifinality to Jayaweera and Mikkelsen's. Findings in this study are valuable to make an informed model selection, to interpret model-output responses critically and to improve an existing model by adopting an alternative module without using experimental data explicitly.

Using graph structural information about flows to enhance short-term demand prediction in bike-sharing systems

Short-term demand prediction is important for managing transportation infrastructure, particularly in times of disruption, or around new developments. Many bike-sharing schemes face the challenges of managing service provision and bike fleet rebalancing due to the “tidal flows” of travel and use. For them, it is crucial to have precise predictions of travel demand at a fine spatiotemporal granularities. Despite recent advances in machine learning approaches (e.g. deep neural networks) and in short-term traffic demand predictions, relatively few studies have examined this issue using a feature engineering approach to inform model selection. This research extracts novel time-lagged variables describing graph structures and flow interactions from real-world bike usage datasets, including graph node Out-strength, In-strength, Out-degree, In-degree and PageRank. These are used as inputs to different machine learning algorithms to predict short-term bike demand. The results of the experiments indicate the graph-based attributes to be more important in demand prediction than more commonly used meteorological information. The results from the different machine learning approaches (XGBoost, MLP, LSTM) improve when time-lagged graph information is included. Deep neural networks were found to be better able to handle the sequences of the time-lagged graph variables than other approaches, resulting in more accurate forecasting. Thus incorporating graph-based features can improve understanding and modelling of demand patterns in urban areas, supporting bike-sharing schemes and promoting sustainable transport. The proposed approach can be extended into many existing models using spatial data and can be readily transferred to other applications for predicting dynamics in mass transit systems. A number of limitations and areas of further work are discussed.

A practical perspective on the concordance index for the evaluation and selection of prognostic time-to-event models.

Developing a prognostic model for biomedical applications typically requires mapping an individual's set of covariates to a measure of the risk that he or she may experience the event to be predicted. Many scenarios, however, especially those involving adverse pathological outcomes, are better described by explicitly accounting for the timing of these events, as well as their probability. As a result, in these cases, traditional classification or ranking metrics may be inadequate to inform model evaluation or selection. To address this limitation, it is common practice to reframe the problem in the context of survival analysis, and resort, instead, to the concordance index (C-index), which summarises how well a predicted risk score describes an observed sequence of events. A practically meaningful interpretation of the C-index, however, may present several difficulties and pitfalls. Specifically, we identify two main issues: i) the C-index remains implicitly, and subtly, dependent on time, and ii) its relationship with the number of subjects whose risk was incorrectly predicted is not straightforward. Failure to consider these two aspects may introduce undesirable and unwanted biases in the evaluation process, and even result in the selection of a suboptimal model. Hence, here, we discuss ways to obtain a meaningful interpretation in spite of these difficulties. Aiming to assist experimenters regardless of their familiarity with the C-index, we start from an introductory-level presentation of its most popular estimator, highlighting the latter's temporal dependency, and suggesting how it might be correctly used to inform model selection. We also address the nonlinearity of the C-index with respect to the number of correct risk predictions, elaborating a simplified framework that may enable an easier interpretation and quantification of C-index improvements or deteriorations.

Mutual information model selection algorithm for time series

ABSTRACT Time series model selection has been widely studied in recent years. It is of importance to select the best model among candidate models proposed for a series in terms of explaining the procedure that governs the series and providing the most accurate forecast for the future observations. In this study, it is aimed to create an algorithm for order selection in Box–Jenkins models that combines penalized natural logarithm of mutual information among the original series and predictions coming from each candidate. The penalization is achieved by subtracting the number of parameters in each candidate and empirical information the data provide.Simulation studies under various scenarios and applications on real data sets imply that our algorithm offers a promising and satisfactory alternative to its counterparts.

Endocytic Profiling of Cancer Cell Models Reveals Critical Factors Influencing LNP-Mediated mRNA Delivery and Protein Expression.

Lipid nanoparticles have great potential for delivering nucleic-acid-based therapeutics, but low efficiency limits their broad clinical translation. Differences in transfection capacity between invitro models used for nanoparticle pre-clinical testing are poorly understood. To address this, using a clinically relevant lipid nanoparticle (LNP) delivering mRNA, we highlight specific endosomal characteristics in invitro tumor models that impact protein expression. A 30-cell line LNP-mRNA transfection screen identified three cell lines having low, medium, and high transfection that correlated with protein expression when they were analyzed in tumor models. Endocyticprofiling of these cell lines identified major differences in endolysosomal morphology, localization, endocytic uptake, trafficking, recycling, and endolysosomal pH, identified using a novel pH probe. High-transfecting cells showed rapid LNP uptake and trafficking through an organized endocytic pathway to lysosomes or rapid exocytosis. Low-transfecting cells demonstrated slower endosomal LNP trafficking to lysosomes and defective endocytic organization and acidification. Our data establish that efficient LNP-mRNA transfection relies on an early and narrow endosomal escape window prior to lysosomal sequestration and/or exocytosis. Endocytic profiling should form an important pre-clinical evaluation step for nucleic acid delivery systems to inform model selection and guide delivery-system design for improved clinical translation.

Research on life cycle of bulk port production environmental management information system

Comprehensive information platform for port environmental management is necessary to be developed, due to the serious dust pollution and low efficiency of primary environmental protection equipment in the bulk port, which should be combined with life cycle of bulk port production and on the basis of the clean and efficient production needs. The aim of this study was to solve the port environmental management information platform framework, model selection and system development. It provides technical support for the establishment of port environmental management information system, realizes the automotive and global control of the port environment, and a breakthrough of the port environmental management practice.

Lattice and continuum modelling of a bioactive porous tissue scaffold.

A contemporary procedure to grow artificial tissue is to seed cells onto a porous biomaterial scaffold and culture it within a perfusion bioreactor to facilitate the transport of nutrients to growing cells. Typical models of cell growth for tissue engineering applications make use of spatially homogeneous or spatially continuous equations to model cell growth, flow of culture medium, nutrient transport and their interactions. The network structure of the physical porous scaffold is often incorporated through parameters in these models, either phenomenologically or through techniques like mathematical homogenization. We derive a model on a square grid lattice to demonstrate the importance of explicitly modelling the network structure of the porous scaffold and compare results from this model with those from a modified continuum model from the literature. We capture two-way coupling between cell growth and fluid flow by allowing cells to block pores, and by allowing the shear stress of the fluid to affect cell growth and death. We explore a range of parameters for both models and demonstrate quantitative and qualitative differences between predictions from each of these approaches, including spatial pattern formation and local oscillations in cell density present only in the lattice model. These differences suggest that for some parameter regimes, corresponding to specific cell types and scaffold geometries, the lattice model gives qualitatively different model predictions than typical continuum models. Our results inform model selection for bioactive porous tissue scaffolds, aiding in the development of successful tissue engineering experiments and eventually clinically successful technologies.

Creating a Predictive Model for Pavement Deterioration using Geographic Weighted Regression

This paper represents an effort by the New York City Department of Transportation (NYCDOT) to develop, by means of geographically weighted regression (GWR), an empirical pavement deterioration model for the five boroughs of New York City. Evaluation of probabilistic and deterministic models from existing literature, as well as statistical modeling techniques and applications, were examined to inform model selection. A model using GWR was selected as the preferred methodology. Validation tests demonstrated an adjusted R2 of .808 and a standard error (σ) representing 4.8% of the mean. Emphasis is placed on the novelty and practicality of GWR as a means for pavement deterioration prediction. Future applications of this model will aid decision makers in strategically allocating limited funds to priority projects.

Sustained Uplift at a Continental Rift Caldera

AbstractCaldera systems are often restless and experience pulses of uplift and subsidence, with a weak, but significant link to eruption. Characterizing the spatial and temporal patterns of deformation episodes provides insight into the processes responsible for unrest and the architecture of magmatic and hydrothermal systems. Here we combine interferometric synthetic aperture radar images with data from Global Positioning System and a network of seismometers at a continental rift caldera Corbetti, Ethiopia. We document inflation that started mid‐2009 and is ongoing as of 2017, with associated seismicity. We investigate the temporal evolution of the deformation source using a Hastings‐Metropolis algorithm to estimate posterior probability density functions for source model parameters and use the Akaike information criterion to inform model selection. Testing rectangular dislocation and point sources, we find a point source at a depth of 6.6 km (95% confidence: 6.3 − 6.8 km) provides the statistically justified fit. The location of this source is coincident with a conductive anomaly derived from magnetotelluric measurements. We use a joint inversion of two geodetic data sets to produce a time series, which shows a volume input of 1.0 × 107 m3/year. This is the first observation of a prolonged period of magma reservoir growth in the Main Ethiopian Rift and has implications for hazard assessment and monitoring. Corbetti is < 20 km from two major population centers and has estimated return periods of ∼500 and ∼900 years for lava flows and Plinian eruptions, respectively. Our results highlight the need for long‐term geodetic monitoring and the application of statistically robust methods to characterize deformation sources.

CULTURAL ADAPTATIONS OF EVIDENCE-BASED HOME-VISITATION MODELS IN TRIBAL COMMUNITIES.

The Tribal Maternal, Infant, and Early Childhood Home Visiting (Tribal MIECHV) Program provides federal grants to tribes, tribal consortia, tribal organizations, and urban Indian organizations to implement evidence-based home-visiting services for American Indian and Alaska Native (AI/AN) families. To date, only one evidence-based home-visiting program has been developed for use in AI/AN communities. The purpose of this article is to describe the steps that four Tribal MIECHV Programs took to assess community needs, select a home-visiting model, and culturally adapt the model for use in AI/AN communities. In these four unique Tribal MIECHV Program settings, each program employed a rigorous needs-assessment process and developed cultural modifications in accordance with community strengths and needs. Adaptations occurred in consultation with model developers, with consideration of the conceptual rationale for the program, while grounding new content in indigenous cultures. Research is needed to improve measurement of home-visiting outcomes in tribal and urban AI/AN settings, develop culturally grounded home-visiting interventions, and assess the effectiveness of home visiting in AI/AN communities.

The genesis of systems of care for transition to adulthood services: emerging models in primary and subspecialty care.

Transitioning adolescents and young adults with chronic conditions can be complex. Expert recommendations support a transition process that starts in early adolescence, provides continuous guidance and support and allows for care opportunities with patients and caregivers, with the patient alone and between paediatric and adult providers. As most of the guidance is focused on individual patients, providers and clinical programmes, much less is known about how health systems as a whole might support effective transitions of care. Many intervention studies focus on the preparation necessary to successfully transition adolescents, young adults and their families to adult care. Although randomized controlled trials of transition interventions are few and standards of care not yet established, promising models are being developed and tested. This review will describe the development and implementation of emerging models of transitional care in primary and subspecialty care, paying special attention to evaluation outcomes that can inform model selection. The emerging models described here highlight the importance of and guidance for invested clinicians and health systems to create effective methods for successful transition. Additional research using rigorous methodology is necessary.

Lifespan analysis of brain development, gene expression and behavioral phenotypes in the Ts1Cje, Ts65Dn and Dp(16)1/Yey mouse models of Down syndrome.

ABSTRACTDown syndrome (DS) results from triplication of human chromosome 21. Neuropathological hallmarks of DS include atypical central nervous system development that manifests prenatally and extends throughout life. As a result, individuals with DS exhibit cognitive and motor deficits, and have delays in achieving developmental milestones. To determine whether different mouse models of DS recapitulate the human prenatal and postnatal phenotypes, here, we directly compared brain histogenesis, gene expression and behavior over the lifespan of three cytogenetically distinct mouse models of DS: Ts1Cje, Ts65Dn and Dp(16)1/Yey. Histological data indicated that Ts65Dn mice were the most consistently affected with respect to somatic growth, neurogenesis and brain morphogenesis. Embryonic and adult gene expression results showed that Ts1Cje and Ts65Dn brains had considerably more differentially expressed (DEX) genes compared with Dp(16)1/Yey mice, despite the larger number of triplicated genes in the latter model. In addition, DEX genes showed little overlap in identity and chromosomal distribution in the three models, leading to dissimilarities in affected functional pathways. Perinatal and adult behavioral testing also highlighted differences among the models in their abilities to achieve various developmental milestones and perform hippocampal- and motor-based tasks. Interestingly, Dp(16)1/Yey mice showed no abnormalities in prenatal brain phenotypes, yet they manifested behavioral deficits starting at postnatal day 15 that continued through adulthood. In contrast, Ts1Cje mice showed mildly abnormal embryonic brain phenotypes, but only select behavioral deficits as neonates and adults. Altogether, our data showed widespread and unexpected fundamental differences in behavioral, gene expression and brain development phenotypes between these three mouse models. Our findings illustrate unique limitations of each model when studying aspects of brain development and function in DS. This work helps to inform model selection in future studies investigating how observed neurodevelopmental abnormalities arise, how they contribute to cognitive impairment, and when testing therapeutic molecules to ameliorate the intellectual disability associated with DS.This article has an associated First Person interview with the first author of the paper.

Less Is More: Cross-Validation Testing of Simplified Nonlinear Regression Model Specifications for EQ-5D-5L Health State Values.

The conventional method for modeling of the five-level EuroQol five-dimensional questionnaire (EQ-5D-5L) health state values in national valuation studies is an additive 20-parameter main-effects regression model. Statistical models with many parameters are at increased risk of overfitting-fitting to noise and measurement error, rather than the underlying relationship. To compare the 20-parameter main-effects model to simplified, nonlinear, multiplicative regression models in terms of how accurately they predict mean values of out-of-sample health states. We used data from the Spanish, Singaporean, and Chinese EQ-5D-5L valuation studies. Four models were compared: an 8-parameter model with single parameter per dimension, multiplied by cross-dimensional parameters for levels 2, 3, and 4; 9- and 11-parameter extensions with handling of differences in the wording of level 5; and the "standard" additive 20-parameter model. Fixed- and random-intercept variants of all models were tested using two cross-validation methods: leave-one-out at the level of valued health states, and of health state blocks used in EQ-5D-5L valuation studies. Mean absolute error, Lin concordance correlation coefficient, and Pearson R between observed health state means and out-of-sample predictions were compared. Predictive accuracy was generally best using random intercepts. The 8-, 9-, and 11-parameter models outperformed the 20-parameter model in predicting out-of-sample health states. Simplified nonlinear regression models look promising and should be investigated further using other EQ-5D-5L data sets. To reduce the risk of overfitting, cross-validation is recommended to inform model selection in future EQ-5D valuation studies.

Half-Normal Plots and Overdispersed Models in R: The hnp Package

Count and proportion data may present overdispersion, i.e., greater variability than expected by the Poisson and binomial models, respectively. Different extended generalized linear models that allow for overdispersion may be used to analyze this type of data, such as models that use a generalized variance function, random-effects models, zero-inflated models and compound distribution models. Assessing goodness-of-fit and verifying assumptions of these models is not an easy task and the use of half-normal plots with a simulated envelope is a possible solution for this problem. These plots are a useful indicator of goodness-of-fit that may be used with any generalized linear model and extensions. For GLIM users, functions that generated these plots were widely used, however, in the open-source software R, these functions were not yet available on the Comprehensive R Archive Network (CRAN). We describe a new package in R, hnp, that may be used to generate the half-normal plot with a simulated envelope for residuals from different types of models. The function hnp() can be used together with a range of different model fitting packages in R that extend the basic generalized linear model fitting in glm() and is written so that it is relatively easy to extend it to new model classes and different diagnostics. We illustrate its use on a range of examples, including continuous and discrete responses, and show how it can be used to inform model selection and diagnose overdispersion.

Impact of the "Linked Evidence Approach" Method on Policies to Publicly Fund Diagnostic, Staging, and Screening Medical Tests.

Background: The linked evidence approach (LEA) is used in health technology assessment (HTA) to evaluate the clinical utility of new medical tests in the absence of direct trial evidence. Objective: To determine whether use of LEA affects decisions to publicly fund medical tests. Methods: Australian HTAs that evaluated medical tests before and after LEA was mandated (in 2005) were screened for eligibility. Data were extracted and the impact of LEA and other possible clinical predictors (selected a priori) on funding decisions was modelled. Regression diagnostics were performed to estimate model fit, model specification, and to inform model selection. The unit of analysis was per clinical indication for each new test, so analyses were adjusted for clustering. Results: 83 HTAs (for 173 clinical indications) were eligible from the 259 screened. When health policy was compared before and after 2005, there was an 11% reduction in overall positive funding decisions, including a 25% decrease in “interim” (coverage with evidence development) funding decisions. The odds of obtaining interim funding reduced by 98% (odds ratio = 0.02, 95% confidence interval = 0.0005, 0.17), but there was no change in the direction of funding decisions (odds ratio = 1.36, 95% confidence interval = 0.62, 3.01). Across both time periods, when LEA was used there was a very strong likelihood that the medical test would not receive interim funding (χ2 = 12.63, df = 1, P = 0.001). For positive funding decisions, the strongest predictors were whether or not the new test would replace an existing test and whether the available evidence was limited. Conclusions: The use of LEA did not predict the direction of funding decisions. Application of the method did predict that a “coverage with evidence development” decision was unlikely. This suggests that LEA may reduce decision-maker uncertainty.

Risk based approach for design and optimization of site specific delivery of isoniazid

The research envisaged focuses on risk management approach for development and optimization of enteric coated tablet of isoniazid giving extended release in pH 6.8 phosphate buffer. Risk assessment using failure mode and effects analysis was done to depict the effects of unambiguous failure modes related to particular formulation/process variable. A 23 full factorial design was employed for optimization of core tablet to investigate effect of amount of Polyox WSR 303 (A), hardness (B) and amount of ethyl cellulose (C) on percent drug release in pH 6.8 phosphate buffer. Main effects and interaction plots were generated to study effects of variables. The selection of optimized formulation was done on overlay contour plots and desirability function. The optimized formulation exhibited percent drug release at first hour of 26.97 %, second hour of 44.20 %, fourth hour of 66.15 % and eighth hour of 97.9 % in phosphate buffer pH 6.8. Akaike information criteria and Model selection criteria revealed that the model was best described by Korsmeyer–Peppas power law. The Kopcha and Peppas–Sahlin model revealed diffusion as predominant mechanism of release which may be due to high solubility of drug and drug loading. Enteric coating optimization revealed weight gain of 10 % w/w as optimum; giving nil release of isoniazid in 0.1 N hydrochloric acid. The composite desirability for optimized formulation computed using equations and software were 0.91 and 0.90 respectively. Capability analysis on reproducibility batches revealed all indices above 1.33 signifying process was within control of producing batches as per desired specifications.

Descriptive Statistics for Modern Test Score Distributions: Skewness, Kurtosis, Discreteness, and Ceiling Effects.

Many statistical analyses benefit from the assumption that unconditional or conditional distributions are continuous and normal. More than 50 years ago in this journal, Lord and Cook chronicled departures from normality in educational tests, and Micerri similarly showed that the normality assumption is met rarely in educational and psychological practice. In this article, the authors extend these previous analyses to state-level educational test score distributions that are an increasingly common target of high-stakes analysis and interpretation. Among 504 scale-score and raw-score distributions from state testing programs from recent years, nonnormal distributions are common and are often associated with particular state programs. The authors explain how scaling procedures from item response theory lead to nonnormal distributions as well as unusual patterns of discreteness. The authors recommend that distributional descriptive statistics be calculated routinely to inform model selection for large-scale test score data, and they illustrate consequences of nonnormality using sensitivity studies that compare baseline results to those from normalized score scales.