Improvement Of Model Calibration Research Articles

A constrained maximum likelihood approach to developing well-calibrated models for predicting binary outcomes

The added value of candidate predictors for risk modeling is routinely evaluated by comparing the performance of models with or without including candidate predictors. Such comparison is most meaningful when the estimated risk by the two models are both unbiased in the target population. Very often data for candidate predictors are sourced from nonrepresentative convenience samples. Updating the base model using the study data without acknowledging the discrepancy between the underlying distribution of the study data and that in the target population can lead to biased risk estimates and therefore an unfair evaluation of candidate predictors. To address this issue assuming access to a well-calibrated base model, we propose a semiparametric method for model fitting that enforces good calibration. The central idea is to calibrate the fitted model against the base model by enforcing suitable constraints in maximizing the likelihood function. This approach enables unbiased assessment of model improvement offered by candidate predictors without requiring a representative sample from the target population, thus overcoming a significant practical challenge. We study theoretical properties for model parameter estimates, and demonstrate improvement in model calibration via extensive simulation studies. Finally, we apply the proposed method to data extracted from Penn Medicine Biobank to inform the added value of breast density for breast cancer risk assessment in the Caucasian woman population.

Geostatistical inverse modeling to characterize the transience of streambed hydraulic conductivity

Quantitative investigation of the interaction between surface water and groundwater relies on the determination of transience of streambed hydraulic conductivity (Ksb). The commonly used successive inversion approach (i.e., the inversion is repeated over successive time steps) of interpreting aquifer hydraulic response data belongs to different optimization windows consists of a sizable number of inversion sub-processes and tends to lose sharp changes in Ksb. This study treats the transience of a parameter in the time domain as the heterogeneity in 1D space domain and uses geostatistical inverse modeling to characterize such transience. The methodology is first tested through a group of ten synthetic cases with independently generated random transient patterns of Ksb. Results of these synthetic cases reveal the capability of geostatistical inverse modeling to effectively characterize the general varying pattern as well as sharp changes in Ksb. Subsequently, the methodology is applied to interpret the aquifer hydraulic response data measured at the Rhône River site, Town of Fully, Switzerland. A significant improvement in model calibration is observed when the methodology is compared to the successive inversion approach. The field investigation reveals that the transience of in-situ Ksb is closely related to stream stage variations. Particularly, multiple occurrences of sharp increases and decreases in Ksb appear when the stream stage lowers below and rises above the aquifer head. The driving mechanism for these sharp variation events is that fine-grained sediments within the streambed can be partly washed away when the aquifer water flows toward the river, and vice versa. The geostatistical approach has the potential to help investigate other physical or chemical processes in groundwater hydrology associated with transient changes in Ksb.

Breast Arterial Calcification and Cardiovascular Risk Among Postmenopausal Women

Background: Breast arterial calcification (BAC) is commonly seen in mammograms but considered an incidental finding. Prior studies have shown correlation of BAC with angiographic coronary artery disease and cardiovascular disease (CVD) outcomes. To shed light on the utility of BAC for primary prevention, we aimed to: 1) Examine the association of BAC presence and quantity with hard atherosclerotic CVD (ASCVD) and global CVD; 2) Ascertain improvement in model calibration, discrimination and reclassification of ASCVD risk; 3) Assess the joint effect of BAC presence and 10-year Pooled Cohorts Equation (PCE) risk on ASCVD. Methods: Cohort study of 5,059 women aged 60-79 years when they attended mammography screening (10/2012 - 2/2015) followed up for 6.5 years (SD=1.6 years). BAC status (presence vs. absence) and quantity (mg of calcium mass) were determined using digital mammograms. Findings: Twenty-six percent of women had BAC > 0 mg. BAC presence was associated with age, ethnicity, education level, diabetes, hypertension and parity. In adjusted Cox regression, BAC presence was associated with a 1.51 (95% CI, 1.08-2.11; p=0.02) increased hazard of ASCVD and a 1.23 (95% CI, 1.002-1.52; p=0.04) increased hazard of global CVD. There was evidence of dose-response associations and of additional risk stratification of the PCE risk by BAC status. We noted improvements in model calibration, discrimination and reclassification (22 to 28%) after adding BAC information. Interpretation: Our results add to the body of evidence that presence and quantity of BAC assessed with contemporary digital mammography has potential utility for primary CVD prevention among post-menopausal women. Funding: MINERVA is an investigator-initiated cohort study funded by the National Heart, Lung, and Blood Institute (RO1HL106-043; multiple PIs Iribarren and Molloi). Declaration of Interest: None to declare. Ethical Approval: The study was approved by the Institutional Review Boards of the participating institutions and all participants signed an informed consent.

Hydrological model optimization using multi-gauge calibration (MGC) in a mountainous region

Abstract It is a challenge for hydrological models to capture complex processes in a basin with limited data when estimating model parameters. This study aims to contribute in this field by assessing the impact of incorporating spatial dimension on the improvement of model calibration. Hence, the main objective of this study was to evaluate the impact of multi-gauge calibration in hydrological model calibration for Ikizdere basin, Black Sea Region in Turkey. In addition, we have incorporated the climate change impact assessment for the study area. Four scenarios were tested for performance assessment of calibration: (1) using downstream flow data (DC), (2) using upstream data (UC), (3) using upstream and downstream data (Multi-Gauge Calibration – MGC), and (4) using upstream and then downstream data (UCDC). The results have shown that using individual gauges for calibration (1 and 2) improve the local predictive capacity of the model. MGC calibration significantly improved the model performance for the whole basin unlike 1 and 2. However, the local gauge calibrations statistical performance, compared to MGC outputs, was better for local areas. The UCDC yields the best model performance and much improved predictive capacity. Regarding the climate change, we did not observe an agreement amongst the future climate projections for the basin towards the end of the century.

Integrating aerial geophysical data in multiple-point statistics simulations to assist groundwater flow models

The process of accounting for heterogeneity has made significant advances in statistical research, primarily in the framework of stochastic analysis and the development of multiple-point statistics (MPS). Among MPS techniques, the direct sampling (DS) method is tested to determine its ability to delineate heterogeneity from aerial magnetics data in a regional sandstone aquifer intruded by low-permeability volcanic dykes in Northern Ireland, UK. The use of two two-dimensional bivariate training images aids in creating spatial probability distributions of heterogeneities of hydrogeological interest, despite relatively ‘noisy’ magnetics data (i.e. including hydrogeologically irrelevant urban noise and regional geologic effects). These distributions are incorporated into a hierarchy system where previously published density function and upscaling methods are applied to derive regional distributions of equivalent hydraulic conductivity tensor K. Several K models, as determined by several stochastic realisations of MPS dyke locations, are computed within groundwater flow models and evaluated by comparing modelled heads with field observations. Results show a significant improvement in model calibration when compared to a simplistic homogeneous and isotropic aquifer model that does not account for the dyke occurrence evidenced by airborne magnetic data. The best model is obtained when normal and reverse polarity dykes are computed separately within MPS simulations and when a probability threshold of 0.7 is applied. The presented stochastic approach also provides improvement when compared to a previously published deterministic anisotropic model based on the unprocessed (i.e. noisy) airborne magnetics. This demonstrates the potential of coupling MPS to airborne geophysical data for regional groundwater modelling.

Spatially Variable Deoxygenation in the Danshui River: Improvement in Model Calibration

A modeling study of the Danshui River, Taiwan, reveals that in-stream BOD deoxygenation rates vary significantly along the river as a result of the highly variable strength of wastewater discharges, which directly reflect the effluent characteristics. A comprehensive field data gathering and lab analysis effort for the study site is presented. Results of the data analyses yielded spatially variable CBOD deoxygenation and nitrification rates, which were incorporated in a model of the river. The model results indicate significant improvement of model calibration, thus enhancing the predictive capability of the model for its use in water quality management. To maximize the benefits of pollution control for the Danshui River system and to achieve the water quality management goal, concurrent reductions of CBOD and ammonia loads, instead of phased reduction of CBOD followed by ammonia reduction, are strongly recommended by the model results.

Utility of genetic and non-genetic risk factors in prediction of type 2 diabetes: Whitehall II prospective cohort study

Objectives To assess the performance of a panel of common single nucleotide polymorphisms (genotypes) associated with type 2 diabetes in distinguishing incident cases of future type 2 diabetes (discrimination), and...

Generalizability of songbird habitat models in boreal mixedwood forests of Alberta

Statistical models relating habitat characteristics to species occurrences are increasingly used to evaluate the consequences of forest management activities and conservation plans over large spatial and temporal scales. In practice, such models do not always generalize to other locations, hence, they should be validated using independent data. In this paper, we assess the predictive ability of 16 songbird habitat models developed in the Calling Lake region of the boreal mixedwood forest of Alberta using both internal and external validation approaches. Internal validation relied on the same dataset used to develop the models while external validation utilised independent data collected within four validation landscapes in the same ecological region. Two aspects of predictive accuracy were evaluated: the agreement between observations and predicted values (calibration) and the models’ ability to classify locations into those in which species are present or absent (discrimination). Calibration was assessed using the Hosmer–Lemeshow (H–L) statistic and plots showing predicted versus observed probabilities of occurrence. Discrimination was assessed using receiver operating characteristic (ROC) curves and the associated area under the ROC curve. With internal validation, the calibration plots were reasonable for all species, however the H–L statistic indicated a good fit for only eight species. Model discrimination between occupied and unoccupied sites, on the other hand, was reasonable for 14 of 16 models. With external validation, both the calibration plots and the H–L statistic identified only three species with a reasonable fit. The remaining models generally over- or under-predicted probability of occurrence. External model discrimination was reasonable for 10 of 16 species. When we re-estimated the models using all available data, the calibration plots were again reasonable for all species, with the H–L statistic indicating good fit for 12 species. Similarly, model discrimination was reasonable for 14 of 16 species. Several factors may help to explain why model performance using external data was generally poorer and more variable than when using internal data. These include differences in landscape structure and disturbance history as well as differences in the frequency of occurrence of individual species. Overall, our analyses identified several species whose habitat models degraded considerably, especially as measured by calibration. By re-estimating the models, we increased the range of variation in our covariates which likely led to an improvement in model calibration. We discuss the importance of ongoing evaluation and refinement of habitat models that will be used in conservation planning or to evaluate management scenarios at large spatial and temporal scales.

Recent trends in the development of ecological models applied on aquatic ecosystems.

This paper presents an overview of the application of models on aquatic ecosystems. More than 17% of the models published in the focal journal in the field, Ecological Modelling, are aquatic ecosystem models. An increasing number of papers are dealing with the theoretical aspects of modeling – new modeling approaches and techniques, how models can be used to reveal ecosystem properties, and how models can better reflect the properties of ecosystems. This development implies that today we have more types of models. The characteristics, the advantages, and the disadvantages of these model types are presented briefly. The selection criteria for the presented model types are discussed, and the application of these types to models for aquatic ecosystems is reviewed. A recent improvement in model calibration of particular interest for aquatic ecosystems is presented, and the perspectives resulting from this new calibration procedure and from possible hybrids of the presented model types are discussed.