Split-sample Test Research Articles

Groundwater recharge predictions in contrasted climate: The effect of model complexity and calibration period on recharge rates

We systematically evaluated the effect of model complexity and calibration strategy on estimated recharge using four varyingly complex models and a unique long-term recharge data set. A differential split sample test was carried out by using six calibration periods with climatically contrasting conditions in a constrained Monte Carlo approach. All models performed better during calibration than during validation due to differences in model structures and climatic conditions. The two more complex, physically-based models predicted the observed recharge with relatively small uncertainties, even when calibration and prediction periods had different climatic conditions. In contrast, the more simplistic soil-water balance model significantly underestimated the recharge rates. The fourth, semi-mechanistic model captured the observed recharge rates, but with a larger uncertainty range than the physically-based models. Our results may have relevant implications for a broad range of applications when recharge models are used as decision-making tools.

Process-based modelling to evaluate simulated groundwater levels and frequencies in a Chalk catchment in south-western England

Abstract. Chalk aquifers are an important source of drinking water in the UK. Due to their properties, they are particularly vulnerable to groundwater-related hazards like floods and droughts. Understanding and predicting groundwater levels is therefore important for effective and safe water management. Chalk is known for its high porosity and, due to its dissolvability, exposed to karstification and strong subsurface heterogeneity. To cope with the karstic heterogeneity and limited data availability, specialised modelling approaches are required that balance model complexity and data availability. In this study, we present a novel approach to evaluate simulated groundwater level frequencies derived from a semi-distributed karst model that represents subsurface heterogeneity by distribution functions. Simulated groundwater storages are transferred into groundwater levels using evidence from different observations wells. Using a percentile approach we can assess the number of days exceeding or falling below selected groundwater level percentiles. Firstly, we evaluate the performance of the model when simulating groundwater level time series using a spilt sample test and parameter identifiability analysis. Secondly, we apply a split sample test to the simulated groundwater level percentiles to explore the performance in predicting groundwater level exceedances. We show that the model provides robust simulations of discharge and groundwater levels at three observation wells at a test site in a chalk-dominated catchment in south-western England. The second split sample test also indicates that the percentile approach is able to reliably predict groundwater level exceedances across all considered timescales up to their 75th percentile. However, when looking at the 90th percentile, it only provides acceptable predictions for long time periods and it fails when the 95th percentile of groundwater exceedance levels is considered. By modifying the historic forcings of our model according to expected future climate changes, we create simple climate scenarios and we show that the projected climate changes may lead to generally lower groundwater levels and a reduction of exceedances of high groundwater level percentiles.

A step-wise semi-distributed simulation approach to characterize a karst aquifer and to support dam construction in a data-scarce environment

Karst systems provide significant volumes of drinking water for large parts of the world population. Due to chemical weathering, karst systems are characterized by strong heterogeneity resulting in a complex flow and storage behaviour. Presently available karst modelling strategies account for the karstic heterogeneity but often a lack of data limits their applicability in data-scarce regions. In this study, a step-wise simulation approach with a semi-distributed karst model is proposed to characterize a karst aquifer at a data-scarce region in Southwest Iran and to evaluate the leakage potential related to a future dam construction project at a river that cuts through the aquifer. Observed groundwater level time series were applied to calibrate and validate the model. In order to avoid over-parameterization, the karst aquifer was split into three sections down the hydraulic gradient. At each section, groundwater level observations were used to iteratively calibrate the model from the first to the last section. A spatial split-sample test and sensitivity analysis served to evaluate the prediction performance and the identifiability of the model parameters. Finally, simple scenarios of the river infiltration into the aquifer were applied to evaluate the leakage potential of the aquifer for future dam constructions. The spatial split-sample test showed that the semi-distributed model provided reliable predictions but prediction performance and parameter identifiability decreased from the first towards the last aquifer section, most probably due to increased aquifer complexity and propagation of uncertainty from the up-gradient model section. Using sensitivity analysis, we also show that parameter sensitivities increase significantly if parameter estimation was applied simultaneously to all three aquifer subsections. Using the model to assess the leakage potential indicated that, without further technical measures, the all river flow would be able infiltrate into the aquifer and the dam would never be filled up completely.

Using high‐resolution isotope data and alternative calibration strategies for a tracer‐aided runoff model in a nested catchment

AbstractTesting hydrological models over different spatio‐temporal scales is important for both evaluating diagnostics and aiding process understanding. High‐frequency (6‐hr) stable isotope sampling of rainfall and runoff was undertaken during 3‐week periods in summer and winter within 12 months of daily sampling in a 3.2‐km2 catchment in the Scottish Highlands. This was used to calibrate and test a tracer‐aided model to assess the (a) information content of high‐resolution data, (b) effect of different calibration strategies on simulations and inferred processes, and (c) model transferability to <1‐km2 subcatchment. The 6‐hourly data were successfully incorporated without loss of model performance, improving the temporal resolution of the modelling, and making it more relevant to the time dynamics of the isotope and hydrometric response. However, this added little new information due to old‐water dominance and riparian mixing in this peatland catchment. Time variant results, from differential split sample testing, highlighted the importance of calibrating to a wide range of hydrological conditions. This also provided insights into the nonstationarity of catchment mixing processes, in relation to storage and water ages, which varied markedly depending on the calibration period. Application to the nested subcatchment produced equivalent parameterization and performance, highlighting similarity in dominant processes. The study highlighted the utility of high‐resolution data in combination with tracer‐aided models, applied at multiple spatial scales, as learning tools to enhance process understanding and evaluation of model behaviour across nonstationary conditions. This helps reveal more fully the catchment response in terms of the different mechanistic controls on both wave celerites and particle velocities.

Impact of mesoscale spatial variability of climatic inputs and parameters on the hydrological response

The spatial variability of the hydrological response is controlled by the interaction of two spatial variabilities: climatic forcing and physiographic characteristics. This paper investigates their impact on streamflow modelling throughout a large catchment. They are assessed by successively refining the spatial information of climatic inputs and model parameters, going from the lumped to the sub-catchment or mesh scale. The analysis is conducted according to a spatial split-sample test. For each resolution, the distributed model first calibrated on calibration station(s) is then evaluated in light of this(these) station(s) in addition to validation stations previously treated as ungauged. For that purpose, simulated and observed streamflows are compared focusing on four hydrological signatures: daily runoff, daily regime, flood and low flow.The above framework is applied at the daily time step over a French basin, the Loire catchment at Gien covering 35,707km2, whose streamflow data are available at the catchment outlet and 105 interior points. The results show that spatially distributed climatic forcing greatly enhances the four signatures as the resolution increases, until the right rainfall volume is provided for each simulation point. By taking advantage of the interior calibration stations to spatialise the parameters, the model’s performance progresses again but to a lesser extent. Finally, the catchment’s hydrological behaviour is mainly driven by the spatial variability of the climatic inputs (around 90% over the validation stations), while that of the parameters seems to be of second order.

Evaluating the robustness of conceptual rainfall-runoff models under climate variability in northern Tunisia

To evaluate the impact of climate change on water resources at the catchment scale, not only future projections of climate are necessary but also robust rainfall-runoff models that must be fairly reliable under changing climate conditions. The aim of this study was thus to assess the robustness of three conceptual rainfall-runoff models (GR4j, HBV and IHACRES) on five basins in northern Tunisia under long-term climate variability, in the light of available future climate scenarios for this region. The robustness of the models was evaluated using a differential split sample test based on a climate classification of the observation period that simultaneously accounted for precipitation and temperature conditions. The study catchments include the main hydrographical basins in northern Tunisia, which produce most of the surface water resources in the country. A 30-year period (1970–2000) was used to capture a wide range of hydro-climatic conditions. The calibration was based on the Kling-Gupta Efficiency (KGE) criterion, while model transferability was evaluated based on the Nash-Sutcliffe efficiency criterion and volume error. The three hydrological models were shown to behave similarly under climate variability. The models simulated the runoff pattern better when transferred to wetter and colder conditions than to drier and warmer ones. It was shown that their robustness became unacceptable when climate conditions involved a decrease of more than 25% in annual precipitation and an increase of more than +1.75°C in annual mean temperatures. The reduction in model robustness may be partly due to the climate dependence of some parameters. When compared to precipitation and temperature projections in the region, the limits of transferability obtained in this study are generally respected for short and middle term. For long term projections under the most pessimistic emission gas scenarios, the limits of transferability are generally not respected, which may hamper the use of conceptual models for hydrological projections in northern Tunisia.

Transferability of hydrological models and ensemble averaging methods between contrasting climatic periods

Understanding hydrological model predictive capabilities under contrasting climate conditions enables more robust decision making. Using Differential Split Sample Testing (DSST), we analyze the performance of six hydrological models for 37 Irish catchments under climate conditions unlike those used for model training. Additionally, we consider four ensemble averaging techniques when examining interperiod transferability. DSST is conducted using 2/3 year noncontinuous blocks of (i) the wettest/driest years on record based on precipitation totals and (ii) years with a more/less pronounced seasonal precipitation regime. Model transferability between contrasting regimes was found to vary depending on the testing scenario, catchment, and evaluation criteria considered. As expected, the ensemble average outperformed most individual ensemble members. However, averaging techniques differed considerably in the number of times they surpassed the best individual model member. Bayesian Model Averaging (BMA) and the Granger-Ramanathan Averaging (GRA) method were found to outperform the simple arithmetic mean (SAM) and Akaike Information Criteria Averaging (AICA). Here GRA performed better than the best individual model in 51%–86% of cases (according to the Nash-Sutcliffe criterion). When assessing model predictive skill under climate change conditions we recommend (i) setting up DSST to select the best available analogues of expected annual mean and seasonal climate conditions; (ii) applying multiple performance criteria; (iii) testing transferability using a diverse set of catchments; and (iv) using a multimodel ensemble in conjunction with an appropriate averaging technique. Given the computational efficiency and performance of GRA relative to BMA, the former is recommended as the preferred ensemble averaging technique for climate assessment.

Assessing the role of uncertain precipitation estimates on the robustness of hydrological model parameters under highly variable climate conditions

Abstract Study region Four headwaters in Southern Africa. Study focus The streamflow regimes in Southern Africa are amongst the most variable in the world. The corresponding differences in streamflow bias and variability allowed us to analyze the behavior and robustness of the LISFLOOD hydrological model parameters. A differential split-sample test is used for calibration using seven satellite-based rainfall estimates, in order to assess the robustness of model parameters. Robust model parameters are of high importance when they have to be transferred both in time and space. For calibration, the modified Kling-Gupta statistic was used, which allowed us to differentiate the contribution of the correlation, bias and variability between the simulated and observed streamflow. New hydrological insights Results indicate large discrepancies in terms of the linear correlation ( r ), bias ( β ) and variability ( γ ) between the observed and simulated streamflows when using different precipitation estimates as model input. The best model performance was obtained with products which ingest gauge data for bias correction. However, catchment behavior was difficult to be captured using a single parameter set and to obtain a single robust parameter set for each catchment, which indicate that transposing model parameters should be carried out with caution. Model parameters depend on the precipitation characteristics of the calibration period and should therefore only be used in target periods with similar precipitation characteristics (wet/dry).

Simulating runoff under changing climatic conditions: Revisiting an apparent deficiency of conceptual rainfall‐runoff models

AbstractHydrologic models have potential to be useful tools in planning for future climate variability. However, recent literature suggests that the current generation of conceptual rainfall runoff models tend to underestimate the sensitivity of runoff to a given change in rainfall, leading to poor performance when evaluated over multiyear droughts. This research revisited this conclusion, investigating whether the observed poor performance could be due to insufficient model calibration and evaluation techniques. We applied an approach based on Pareto optimality to explore trade‐offs between model performance in different climatic conditions. Five conceptual rainfall runoff model structures were tested in 86 catchments in Australia, for a total of 430 Pareto analyses. The Pareto results were then compared with results from a commonly used model calibration and evaluation method, the Differential Split Sample Test. We found that the latter often missed potentially promising parameter sets within a given model structure, giving a false negative impression of the capabilities of the model. This suggests that models may be more capable under changing climatic conditions than previously thought. Of the 282[347] cases of apparent model failure under the split sample test using the lower [higher] of two model performance criteria trialed, 155[120] were false negatives. We discuss potential causes of remaining model failures, including the role of data errors. Although the Pareto approach proved useful, our aim was not to suggest an alternative calibration strategy, but to critically assess existing methods of model calibration and evaluation. We recommend caution when interpreting split sample results.

Calibração hierárquica do modelo swat em uma bacia hidrográfica Catarinense

One of the main challenges of hydrological modeling is the effective calibration of model parameters. The aim of this study was to test a hierarchical calibration scheme for the Soil and Water Assessment Tool (SWAT) model by considering land use and climate changes. The study area was the Rio Negrinho watershed, located in Santa Catarina State, Brazil. Model calibration was performed with the Sequential Uncertainty Fitting (SUFI2) Algorithm through increasing complexity tests. The observed discharge data were first divided into calibration and validation periods(Split Sample Test) wherein the predicted monthly flow matched the measured values, with a Nash–Sutcliffe coefficient (NSE) ≥0.5 and Pbias<±25%. When wet and dry periods were considered separately (Differential Split-Sample Test) the NSE values of wet years (2008 a 2012) were higher than dry year values (2003 to 2007). In the SWAT validation for a watershed with similar land use (Proxy-Catchment Test), the model could not capture the flow peaks well (NSE of 0.0), however, the water balance volumes were compensated (Pbias of 0.1%). The case study demonstrates that SWAT performance varies greatly, depending on the spatial scale and calibration period. The SWAT model can be used in further studies to simulate land use and climate change scenarios however it must be calibrated and validated using spatially well distributed data and recorded series segment including dry and wet periods.

Developing Protocols for Measuring Biosolids Stability

The general goal of this project is to validate test protocols that are commonly used to assess the stability of various biosolids products, and to specify a standard for each test method. Toward this end, the project team conducted a literature review, surveyed selected practicing facilities, and conducted a sampling and analysis program. Information concerning stability-testing protocols and data for each biosolids-stabilization technology and testing method reviewed were collected and evaluated. The biosolids-stabilization technologies reviewed were aerobic and anaerobic digestion, alkaline stabilization, and composting; the testing methods reviewed were specific oxygen uptake rate, volatile solids reduction, additional volatile solids reduction, pH and changes in pH, and carbon dioxide evolution. The report recommends specific protocols for each testing method, and presents intrinsic precision data for each protocol. In addition, data is presented from facilities using each protocol, to serve as a basis for evaluating the collective degree of variability associated with each stabilization process. Collective variability results from sampling, feedstock, the stabilization process, and the intrinsic variability of each testing method. Using split sample testing, interlaboratory data (variability among different laboratories) are presented. Issues associated with variability, error sources, shortcomings, and numerical criteria values are discussed for each method. This title belongs to WERF Research Report Series ISBN: 9781843396550 (eBook)

Comparing global and local calibration schemes from a differential split-sample test perspective

This work explores the performances of the hydrologic model Hydrotel, applied to 36 catchments located in the Province of Quebec, Canada. A local calibration (each catchment taken individually) scheme and a global calibration (a single parameter set sought for all catchments) scheme are compared in a differential split-sample test perspective. Such a methodology is useful to gain insights on a model’s skills under different climatic conditions, in view of its use for climate change impact studies. The model was calibrated using both schemes on five non-continuous dry and cold years and then evaluated on five dissimilar humid and warm years. Results indicate that, as expected, local calibration leads to better performances than the global one. However, global calibration achieves satisfactory simulations while producing a better temporal robustness (i.e., model transposability to periods with different climatic conditions). Global calibration, in opposition to local calibration, thus imposes spatial consistency to the calibrated parameter values, while locally adjusted parameter sets can significantly vary from one catchment to another due to equifinality. It is hence stated that a global calibration scheme represents a good trade-off between local performance, temporal robustness, and the spatial consistency of parameter values, which is, for example, of interest in the context of ungauged catchments’ simulation, climate change impact studies, or even simply large-scale modeling.

Application of a modified conceptual rainfall–runoff model to simulation of groundwater level in an undefined watershed

Groundwater level simulation models can help ensure the proper management and use of urban and rural water supply. In this paper, we propose a groundwater level tank model (GLTM) based on a conceptual rainfall–runoff model (tank model) to simulate fluctuations in groundwater level. The variables used in the simulations consist of daily rainfall and daily groundwater level, which were recorded between April 2011 and March 2015 at two representative observation wells in Kumamoto City, Japan. We determined the best-fit model parameters by root-mean-square error through use of the Shuffled Complex Evolution-University of Arizona algorithm on a simulated data set. Calibration and validation results were evaluated by their coefficients of determination, Nash–Sutcliffe efficiency coefficients, and root-mean-square error values. The GLTM provided accurate results in both the calibration and validation of fluctuations in groundwater level. The split sample test results indicate a good reliability. These results indicate that this model can provide a simple approach to the accurate simulation of groundwater levels.

Parameter transferability under changing climate: case study with a land surface model in the Durance watershed, France

In physically-based land surface models, the parameters can all be prescribed a priori but calibration can be used to enhance the realism of the simulations in well instrumented domains. In such a case, the transferability of calibrated parameters under non-stationary conditions needs to be addressed, especially in the context of climate change. To this end, we used the Catchment Land Surface Model (CLSM) in the Upper Durance watershed located in the French Alps, which experienced a significant increase in temperature over the last century. The CLSM is forced by a 50-year meteorological dataset of good quality. Four parameters of the CLSM (one related to snow processes and three to soil properties) are calibrated against discharge observations with a multi-objective algorithm. First, the robustness of the CLSM parameterizations is tested by the Differential Split Sample Test (DSST). The simulations show good performances over a wide range of retrospective climatic conditions, except when the parameters are calibrated over a period with a large contribution of snowmelt to annual mean discharge. Then, the use of a climate change scenario reveals that the parameterizations of soil moisture processes in the CLSM are responsible for an increasing dispersion among simulations when facing dry and warm conditions. However, the differences between the simulated changes of river discharge remain very small. This work shows that calibration conveys some uncertainties, but they are moderate in the studied case, and pertain to the most conceptual parameterizations of this physically-based model.

Assessing uncertainties in climate change impacts on runoff in Western Mediterranean basins

Abstract. This paper investigates the uncertainties linked to climate change impacts on runoff in four mesoscale basins (900 to 1800 km2) in the Mediterranean region. Runoff simulations were based on a daily conceptual model including a snow module. The model was calibrated and validated according to a differential split-sample test over a 20-year period and four competing criterions aiming to represent model structural uncertainty based on the concept of Pareto optimality. Five regional climate models (RCMs) from the Med-CORDEX initiative were used to provide temperature and precipitation projections under RCP8.5 by 2050. The RCMs' inability to realistically simulate reference climate (notably precipitation) led us to apply a monthly perturbation method in order to produce a range of climate scenarios. The structural uncertainty bounds obtained from the hydrological simulations over the reference period showed that the model was able to correctly reproduce observed runoff despite contrasted hydrological conditions in (and in between) the basins. Climate projections were shown to be convergent regarding temperatures, which could increase by about +1 to 3 °C on each basin. In contrast, no clear trends in precipitation could be put in evidence, some RCMs leading to a mean annual precipitation decrease (up to 64%), and others to an increase (up to 33%). The hydrological projections resulted from the combination of the hydrological simulation bounds with the range of climate projections. Despite the propagation of those uncertainties, the 2050 hydrological scenarios agreed on a significant runoff decrease (2–77%) during spring on all basins. On the opposite, no clear trend in runoff could be observed for the other seasons.

The impact of information on public preferences for forest biodiversity preservation: a split-sample test with choice experiment method

Cet article analyse l’impact de l’information fournie aux individus sur la valeur qu’ils accordent aux attributs de la biodiversité forestière. La méthode des expériences de choix a été appliquée, sur deux sous-échantillons afin d’obtenir deux ensembles de données : le premier correspondant à des choix obtenus avec un protocole d’enquête neutre, et le second à l’issue d’un protocole délivrant aux répondants des informations spécifiques sur chaque attribut de la biodiversité. L’estimation économétrique des préférences par le modèle de choix discrets à classes latentes permet de distinguer les valeurs du consentement à payer (CAP) des individus « familiers », c’est à dire ceux qui font un usage régulier des forêts et qui sont plus sensibles aux enjeux de préservation de la biodiversité, de celles des répondants « non familiers ». L’étude conclut que donner des informations spécifiques sur chacune des composantes de la biodiversité affecte les préférences du public. Cependant, la sensibilité à la cause environnementale et la connaissance de la forêt à partir de ses usages constituent les principales variables explicatives d’un CAP plus élevé.

Reducing structural uncertainty in conceptual hydrological modelling in the semi-arid Andes

Abstract. The use of lumped, conceptual models in hydrological impact studies requires placing more emphasis on the uncertainty arising from deficiencies and/or ambiguities in the model structure. This study provides an opportunity to combine a multiple-hypothesis framework with a multi-criteria assessment scheme to reduce structural uncertainty in the conceptual modelling of a mesoscale Andean catchment (1515 km2) over a 30-year period (1982–2011). The modelling process was decomposed into six model-building decisions related to the following aspects of the system behaviour: snow accumulation and melt, runoff generation, redistribution and delay of water fluxes, and natural storage effects. Each of these decisions was provided with a set of alternative modelling options, resulting in a total of 72 competing model structures. These structures were calibrated using the concept of Pareto optimality with three criteria pertaining to streamflow simulations and one to the seasonal dynamics of snow processes. The results were analyzed in the four-dimensional (4-D) space of performance measures using a fuzzy c-means clustering technique and a differential split sample test, leading to identify 14 equally acceptable model hypotheses. A filtering approach was then applied to these best-performing structures in order to minimize the overall uncertainty envelope while maximizing the number of enclosed observations. This led to retain eight model hypotheses as a representation of the minimum structural uncertainty that could be obtained with this modelling framework. Future work to better consider model predictive uncertainty should include a proper assessment of parameter equifinality and data errors, as well as the testing of new or refined hypotheses to allow for the use of additional auxiliary observations.

Youth Training Programs and Their Impact on Career and Spell Duration of Professional Soccer Players

A unique data set of post-war English trained soccer players is used to study the impact of the youth training program they attended on their career and spell duration. Duration models in the spirit of Abbring and van den Berg are employed to estimate local treatment effects of different training programs on players — survival in the top European leagues. The results indicate that the duration patterns of players are dependent on the youth academy they attended. Certain clubs, with a well-established reputation in developing youth talent, outperform others in terms of producing and evaluating the ability of their youth players to succeed in top European leagues. The spell analysis outlines the nature of the competitive environment in which smaller clubs have a chance to keep up with the larger ones in terms of producing and holding on to homegrown talent. Finally, the results of both analyses addressed unobserved heterogeneity, allowed for nonlinearity of covariates using the cubic spline methodology, and were tested for endogeneity bias using a split sample test.

Measuring the Strength Of Color Brand-Name Links

ABSTRACT This study provides a theoretically and empirically validated approach to measuring the strength of color as a brand-identity element. The authors conducted a split sample test with 880 consumers across three categories: banking, chocolate, and hair care. Comparing four measurement approaches helped determine the effect of cuing with the brand versus color (cue direction) and prompting with response sets versus not prompting (difficulty of response). The key comparison measures were the number of people who linked the brand with color (fame) and the relative presence of competitor links (uniqueness). The color-cued, unprompted brand response measure is the recommended approach.

An adaptive workflow coupled with Random Forest algorithm to identify intact N-glycopeptides detected from mass spectrometry

Despite many attempts for algorithm development in recent years, automated identification of intact glycopeptides from LC-MS(2) spectral data is still a challenge in both sensitivity and precision. We implemented a supervised machine learning algorithm, Random Forest, in an automated workflow to identify N-glycopeptides using spectral features derived from ion trap-based LC-MS(2) data. The workflow streamlined high-confident N-glycopeptide spectral data and enabled adaptive model optimization with respect to different sampling strategies, training sample size and feature set. A critical evaluation of the features important for glycopeptide identification further facilitated effective feature selection for model improvement. Using split sample testing method from 577 high-confident N-glycopeptide spectral data, we demonstrated that an optimal true-positive rate, precision and false-positive rate of 73, 88 and 10%, respectively, can be attained for overall N-glycopeptide identification Availability and implementation: The workflow developed in this work and the application suite, Sweet-Heart, that the workflow supports for N-glycopeptide identification are available for download at http://sweet-heart.glycoproteomics.proteome.bc.sinica.edu.tw/.