Model Performance Criteria Research Articles

Şehit Ali İhsan Kalmaz Ormanı Karaçam (Pinus nigra Arnold) ağaçlandırması için tek ve çift girişli ağaç hacim tablolarının düzenlenmesi

In this study, the local single- and double-entry tree volume table was aimed to develop for Crimean pine (Pinus nigra Arnold) afforestations in Sehit Ali Ihsan Kalmaz Forest. In this study, the data used were obtained from 97 sample trees. The available data for Crimean pine tree was split into two sets: the majority (%80) was used to estimate model parameters, and the remaining data (%20) were reserved to validate the models. Sample trees were measured at diameter breast height, height and stem volume of trees in stand. Single and double-entry tree volume table construction and methods based on statistical approaches, several regression models were tested. A total of 12 tree volume models were tested for single-entry (6) and double-entry (6) tables. The performance of the models was compared and evaluated with seven model performance criteria. According to performance criteria, the best results were obtained with Dissescu-Meyer and Spurr models, respectively. As a result, regional a single and double entry volume table was created for Crimean pine. Keywords: Crimean pine, Pinus nigra, Single and double-entry tree volume table

Simulating runoff under changing climatic conditions: Revisiting an apparent deficiency of conceptual rainfall‐runoff models

AbstractHydrologic models have potential to be useful tools in planning for future climate variability. However, recent literature suggests that the current generation of conceptual rainfall runoff models tend to underestimate the sensitivity of runoff to a given change in rainfall, leading to poor performance when evaluated over multiyear droughts. This research revisited this conclusion, investigating whether the observed poor performance could be due to insufficient model calibration and evaluation techniques. We applied an approach based on Pareto optimality to explore trade‐offs between model performance in different climatic conditions. Five conceptual rainfall runoff model structures were tested in 86 catchments in Australia, for a total of 430 Pareto analyses. The Pareto results were then compared with results from a commonly used model calibration and evaluation method, the Differential Split Sample Test. We found that the latter often missed potentially promising parameter sets within a given model structure, giving a false negative impression of the capabilities of the model. This suggests that models may be more capable under changing climatic conditions than previously thought. Of the 282[347] cases of apparent model failure under the split sample test using the lower [higher] of two model performance criteria trialed, 155[120] were false negatives. We discuss potential causes of remaining model failures, including the role of data errors. Although the Pareto approach proved useful, our aim was not to suggest an alternative calibration strategy, but to critically assess existing methods of model calibration and evaluation. We recommend caution when interpreting split sample results.

Evaluation of Meso-NH and WRF/CHEM simulated gas and aerosol chemistry over Europe based on hourly observations

Gas chemistry and aerosol chemistry of 10km-resolution mesoscale models Meso-NH and WRF/CHEM were evaluated on three cases over Europe. These one-day duration cases were selected from Freney et al. (2011) and occurred on contrasted meteorological conditions and at different seasons: a cyclonic circulation with a well-marked frontal zone on winter, an anti-cyclonic situation with local storm precipitations on summer and a cold front in the northwest of Europe associated to a convergence of air masses over eastern Europe and conflicting air masses over Spain and France on autumn. To assess the performance of the two models, surface hourly databases from observation stations over Europe were used, together with airborne measurements. For both models, the meteorological fields were in good agreement with the measurements for the three days. Winds presented the largest normalised mean bias integrated over all European stations for both models. Daily gas chemistry was reproduced with normalised mean biases between −14 and 11%, a level of accuracy that is acceptable for policy support. The two models' performances were degraded during night-time quite likely due to the constant primary species emissions. The PM2.5 bulk mass concentration was overestimated by Meso-NH over Europe and slightly underestimated by WRF/CHEM. The absence of wet deposition in the models partly explains the local discrepancies with the observations. More locally, the systematic low mixing ratio of volatile organic compounds in the gas phase simulated by WRF/CHEM at three stations was correlated with the underestimation of OM (organic matter) mass in the aerosol phase. Moreover, this mass of OM was mainly composed of anthropogenic POAs (primary organic aerosols) in WRF/CHEM, suggesting a missing source for SOAs (secondary organic aerosols) mass in WRF/CHEM aerosol parameterisation. The contribution of OM was well simulated by Meso-NH, with a higher contribution for the summer case. For Meso-NH, SOA made the major contribution to the OM mass. The simulation of the mass of SO42− in particles by both models was often overestimated and correlated with an underestimation of the SO2 mixing ratio. The simulated masses of NO3− and NH4+ in particles were always higher for Meso-NH than for WRF/CHEM, which was linked to a difference in NOX mixing ratio between the models. Finally, computations of model performance criterion and model performance goals show that both models can be considered acceptable for standard modelling applications. In particular, Meso-NH model, using a gaseous chemical mechanism designed to compute the organic precursors of aerosols, shows comparable simulated amounts of SOA to observations at local sites.

Prediction of absorption and stripping factors in natural gas processing industries using feedforward artificial neural network

ABSTRACTIn dynamic simulators, mathematical models are applied in order to study the time-dependent behavior of a system, meaning the system process units and the corresponding control units. Absorption and stripping are the unit operations that are widely used in the natural gas processing industries. Many attempts have been made to define an average absorption factor method to short-cut the time consuming rigorous calculation procedures. One of the options for this complex engineering modeling problem is artificial intelligence approach. Artificial neural networks have been shown to be able to approximate any continuous nonlinear functions and have been used to build data base empirical models for nonlinear processes. In this study, feedforward neural networks (FANN) models were used to model the absorption efficiency. The mean square error (MSE), residue analysis and coefficient of determination based on the observed and prediction output is chosen as the performance criteria of model. It was found that the developed FANN models provided satisfactory model with the MSE and coefficient of determination of 0.0003 and 0.9998 for new unseen data from literature respectively.

Uncertainty and Reliability Analysis of CMIP5 Climate Projections in South Korea Using REA Method

Weighting criteria methods achieved popularity in the quantification of uncertainty in the first decade of 21st century. In this study, Reliability Ensemble Averaging (REA) method is used to provide a quantitative estimate of uncertainty range and reliability of future climate projections over Han river basin in Korea simulated by 18 GCMs under CMIP5 Project. The root mean square error (RMSE) was used to measure uncertainty range. The result indicates that historical simulations project 43% decrease in precipitation while future scenarios simulations of GCMs projects a moderate increase in mean annual temperature and precipitation. The future projections showed a reduction in uncertainty range of about 150% as compared to simple ensemble average approach. The results suggest the viability of REA method by incorporating the model performance and model convergence criteria.

Modeling a Reservoir in Arid Regions; Case of Abdelmoumen Dam, Souss Watershed Morocco

Water plays a very important role in the socio-economic development of the country and if its shortage endures, that may affect the country' seconomy, particularity the agriculture, a big user of water. For this purpose, water management aims to find the right equilibrium between resources and usage. The objective of this work is to understand and model the hydrological functioning of the Issen basin arid watershed by focusing on the realism of simulated intermediate process. SWAT modeling performance are widely illustrated and discussed. A satisfactory return of the hydrograph is possible. Therefore, we have a representative precipitation distribution of the entire basin. However, a correct return flow does not guarantee a realistic simulation of intermediate processes of the hydrological cycle. In this modeling, we used the hydrometric data from 2007 to 2015 for calibration and validation period. Obtained results are widely satisfactory with 0.74 as value of Nash coefficient. The swat model performance criteria show the ability of this model to reproduce faithfully the hydrological behavior of the Issen basin.

A 1‐D morphodynamic model of postglacial valley incision

AbstractChilliwack River is typical of many Cordilleran valley river systems that have undergone dramatic Holocene degradation of valley fills that built up over the course of Pleistocene glaciation. Downstream controls on base level, mainly blockage of valleys by glaciers, led to aggradation of significant glaciofluvial and glaciolacustrine valley fills and fan deposits, subsequently incised by fluvial action. Models of such large‐scale, long‐term degradation present a number of important challenges since the evolution of model parameters, such as the rate of bedload transport and grain size characteristics, are governed by the nature of the deposit. Sediment sampling in the Chilliwack Valley reveals a complex sequence of very coarse to fine textural modes. We present a 1‐D numerical morphodynamic model for the river‐floodplain system tailored to conditions in the valley. The model is adapted to dynamically adjust channel width to optimize sediment transporting capacity and to integrate relict valley fill material as the channel incises through valley deposits. Sensitivity to model parameters is studied using four principal criteria: profile concavity, rate of downstream grain size fining, bed surface sand content, and the timescale to equilibrium. Model results indicate that rates of abrasion and coarsening of the grain size distributions exert the strongest controls on all of the interrelated model performance criteria. While there are a number of difficulties in satisfying all model criteria simultaneously, results indicate that 1‐D models of valley bottom sedimentary systems can provide a suitable framework for integrating results from sediment budget studies and chronologies of sediment evacuation established from dating.

Combined PEST and Trial–Error approach to improve APEX calibration

The Agricultural Policy Environmental eXtender (APEX), a comprehensive hydrologic model well-suited for small watersheds, requires understanding of the input parameters for improved calibration. The “trial and error” method for calibrating the APEX model has been used very commonly in previous studies. In this study, the automatic calibration software Parameter Estimation (PEST) was combined with the conventional trial-and-error method to improve APEX calibration. The proposed Combined PEST and Trial–Error (CPTE) approach can overcome: (i) weaknesses of “Trial–Error” method in terms of tediousness and subjectivity involved in the decision to end a calibration, and (ii) drawback of PEST in that it may lead to biased simulation due to ignoring local specific condition. A case study was developed to verify the CPTE approach. The results based on APEX runoff simulation indicate that the CPTE approach greatly improved the calibration of APEX model with respect to model performance criteria. Coupling inverse modeling and trial–error manual method can be an efficient and effective alternative in calibrating the APEX model.

An Extended Model Evaluation Method under Uncertainty in Hydrologic Modeling

This paper proposes an extended model evaluation method that considers not only the model performance but also the model structure and parameter uncertainties in hydrologic modeling. A simple reservoir model (SFM) and distributed kinematic wave models (KWMSS1 and KWMSS2 using topography from 250-m, 500-m, and 1-km digital elevation models) were developed and assessed by three evaluative criteria for model performance, model structural stability, and parameter identifiability. All the models provided acceptable performance in terms of a global response, but the simpler SFM and KWMSS1 could not accurately represent the local behaviors of hydrographs. Moreover, SFM and KWMSS1 were structurally unstable; their performance was sensitive to the applied objective functions. On the other hand, the most sophisticated model, KWMSS2, performed well, satisfying both global and local behaviors. KMSS2 also showed good structural stability, reproducing hydrographs regardless of the applied objective functions; however, superior parameter identifiability was not guaranteed. A number of parameter sets could result in indistinguishable hydrographs. This result indicates that while making hydrologic models complex increases its performance accuracy and reduces its structural uncertainty, the model is likely to suffer from parameter uncertainty.

Long-term particulate matter modeling for health effect studies in California – Part 1: Model performance on temporal and spatial variations

Abstract. For the first time, a ~ decadal (9 years from 2000 to 2008) air quality model simulation with 4 km horizontal resolution over populated regions and daily time resolution has been conducted for California to provide air quality data for health effect studies. Model predictions are compared to measurements to evaluate the accuracy of the simulation with an emphasis on spatial and temporal variations that could be used in epidemiology studies. Better model performance is found at longer averaging times, suggesting that model results with averaging times ≥ 1 month should be the first to be considered in epidemiological studies. The UCD/CIT model predicts spatial and temporal variations in the concentrations of O3, PM2.5, elemental carbon (EC), organic carbon (OC), nitrate, and ammonium that meet standard modeling performance criteria when compared to monthly-averaged measurements. Predicted sulfate concentrations do not meet target performance metrics due to missing sulfur sources in the emissions. Predicted seasonal and annual variations of PM2.5, EC, OC, nitrate, and ammonium have mean fractional biases that meet the model performance criteria in 95, 100, 71, 73, and 92% of the simulated months, respectively. The base data set provides an improvement for predicted population exposure to PM concentrations in California compared to exposures estimated by central site monitors operated 1 day out of every 3 days at a few urban locations. Uncertainties in the model predictions arise from several issues. Incomplete understanding of secondary organic aerosol formation mechanisms leads to OC bias in the model results in summertime but does not affect OC predictions in winter when concentrations are typically highest. The CO and NO (species dominated by mobile emissions) results reveal temporal and spatial uncertainties associated with the mobile emissions generated by the EMFAC 2007 model. The WRF model tends to overpredict wind speed during stagnation events, leading to underpredictions of high PM concentrations, usually in winter months. The WRF model also generally underpredicts relative humidity, resulting in less particulate nitrate formation, especially during winter months. These limitations must be recognized when using data in health studies. All model results included in the current manuscript can be downloaded free of charge at http://faculty.engineering.ucdavis.edu/kleeman/ .

Empirical relations between compressive strength and microfabric properties of amphibolites using multivariate regression, fuzzy inference and neural networks: A comparative study

In this paper, microfabric properties including grain size, shape factor and quartz content are tailored to the specific evaluation of UCS of banded amphibolite rocks. However, the predicting capabilities of Artificial Neural Networks (ANNs) and Fuzzy Inference System (FIS) as well as the Multivariate Regression (MR) techniques have been evaluated and compared using the same input variables. To assess the model performances, some performance indices such as correlation coefficient (R), variance account for (VAF) and root mean square error (RMSE) were calculated and compared for the three models. The study revealed that even though the developed three models are reliable to predict the UCS, the presented ANN method displays an obvious potential for the reliable assessment of UCS according to model performance criterion. However, the outcomes of this study are quite satisfactory, which may serve microfabric characterization to be easily extended to the modeling of strength and deformation behavior of rocks in the absence of adequate budget and facility of testing UCS.

Antalya yöresi doğal kızılçam meşcereleri için genelleştirilmiş çap-boy modellerinin geliştirilmesi

Brutian pine is one of the important commercial species for forest products industry of Turkey. In this study, diameter (d)-height (h) models for natural Brutian pine stands in Antalya District were developed using the breast height diameter and some stand characteristics as regressors. In this study, a total of 16 models, the data used were obtained from 59 sample plots from natural Brutian pine stands in Antalya District, were tested. The available data for the species were split into two sets: the majority (%85) was used to estimate model parameters, and the remaining data (%15) were reserved to validate the models. The performance of the models was compared and evaluated with six model performance criteria. According to the model performance criteria, the best results were obtained with Pienaar (1991-II), Sloboda et al. (1993-I), and Sharma and Parton (2007) models, respectively. The similar results were obtained using independent dataset. Keywords: Brutian pine, Height, Diameter, Generalized models

Impacts of Stabilized Criegee Intermediates, surface uptake processes and higher aromatic secondary organic aerosol yields on predicted PM2.5 concentrations in the Mexico City Metropolitan Zone

The Community Multiscale Air Quality Model (CMAQ) with the SAPRC-99 gas phase photochemical mechanism and the AERO5 aerosol module was applied to model gases and particulate matter (PM) concentrations in the Mexico City Metropolitan Zone (MCMZ) and the surrounding regions for March 2006 using the official 2006 emission inventories, along with emissions from biogenic sources, biomass burning, windblown dust, the Tula Industrial Complex and the Popocatépetl volcano. The base case model was capable of reproducing the observed hourly concentrations of O3 and attaining CO, NO2 and NOx performance similar to previous modeling studies. Although the base case model performance of hourly PM2.5 and PM10 meets the model performance criteria, under-prediction of high PM2.5 concentrations in late morning indicates that secondary PM, such as sulfate and secondary organic aerosol (SOA), might be under-predicted. Several potential pathways to increase SOA and secondary sulfate were investigated, including Stabilized Criegee Intermediates (SCIs) from ozonolysis reactions of unsaturated hydrocarbons and their reactions with SO2, the reactive uptake processes of SO2, glyoxal and methylglyoxal on particle surface and higher SOA formation due to higher mass yields of aromatic SOA precursors. Averaging over the entire episode, the glyoxal and methylglyoxal reactive uptake and higher aromatics SOA yields contribute to ∼0.9 μg m−3 and ∼1.25 μg m−3 of SOA, respectively. Episode average SOA in the MCMZ reaches ∼3 μg m−3. The SCI pathway increases PM2.5 sulfate by 0.2–0.4 μg m−3 or approximately 10–15%. The relative amount of sulfate increase due to SCI agrees with previous studies in summer eastern US. Surface SO2 uptake significantly increases sulfate concentration in MCMZ by 1–3 μg m−3 or approximately 50–60%. The higher SOA and sulfate leads to improved PM2.5 and PM10 model performance.

Seasonal modeling of PM2.5 in California's San Joaquin Valley

California's San Joaquin Valley (SJV) is in non-attainment for the 2006 revised 24-h PM2.5 National Ambient Air Quality Standard (NAAQS or standard) established by the United States Environmental Protection Agency (U.S. EPA). As a part of the emissions control strategy development to bring the SJV into attainment of the standard, the Community Multi-scale Air Quality (CMAQ) model was used to simulate PM2.5 formation and its response to precursor emission reductions. Simulations were conducted for the first (January–March) and fourth (October–December) quarters of 2007 since the 24-h PM2.5 violations typically occur during the winter months. The CMAQ model generally met the PM2.5 model performance criteria and was suitable for the State Implementation Plan (SIP) applications. Sensitivities of the 24-h average PM2.5 concentrations to precursor emissions were investigated based on the baseline 2019 emissions forecast. Reductions in anthropogenic primary PM2.5 and oxides of nitrogen (NOx) showed the greatest impact on PM2.5 concentrations. Reducing anthropogenic ammonia and sulfur oxides emissions only slightly lowered PM2.5 concentrations, and reducing volatile organic compounds (VOC) emissions provided no benefit in reducing PM2.5 concentrations. Particularly, PM2.5 nitrate, the major PM2.5 component in the SJV showed no response to VOC reductions, which is also supported by additional modeling results from the UCD/CIT airshed model. This study demonstrated that the current strategy of controlling primary PM2.5 and NOx emissions in the SJV will continue to be effective for further reducing PM2.5 in the SJV beyond 2019. It also demonstrated how the CMAQ model can be used to design effective emissions control strategies for regulatory applications.

Applying the delta tool to support the Air Quality Directive: evaluation of the TCAM chemical transport model

This paper presents an application of the DELTA evaluation tool V3.2 to support the EU Air Quality Directive (AQD 2008). This software, designed in the frame of the FAIRMODE project (Forum for Air Quality Modelling in Europe, http://fairmode.ew.eea.europa.eu/), is currently used as support to working groups of modelers across Europe in the diagnostics and assessment of air quality model performances under the AQD (2008). The skills of the DELTA tool V3.2 are tested by looking at the results of a 1-year (2005) simulation performed using the transport chemical aerosol model (Carnevale et al. 2008) at 6 × 6-km2 resolution over the Po Valley. The modeled daily PM10 concentrations at surface level are compared to observations provided by approximately 50 stations distributed across the domain. The main statistical parameters (i.e., bias, root mean square error, correlation coefficient, standard deviation) as well as different types of diagrams (scatter plots, time series plots, Taylor and target plots) have been produced. A representation of the observation uncertainty in the target plot, used to derive model performance criteria for the main statistical indicators, is also presented and discussed.

A genetic programming approach to river flow modeling

This paper proposes the application of genetic programming (GP) to forecast monthly river flow. The river flow models were improved by the monthly rainfall and flow data from three stations for Kizilirmak River, Turkey. The coefficient of determination (R2) and root mean square error (RMSE) values were used for evaluating the accuracy of the developed models. The most appropriate GP model was determined as model having monthly flow data of Yamula and Bulakbasi stations according to the model performance criteria for testing data set. The models obtained using the GP were compared with multiple linear regression (MLR) techniques in river flow forecasting. The comparison results revealed that the suggested GP model performs quite well compared to MLR models. It was shown that the suggested GP model with R2 = 0.96 and RMSE = 8.02 m3/s for testing period could be used in planning and management of water resources.

Selection of Influential Microfabric Properties of Anisotropic Amphibolite Rocks on Its Uniaxial Compressive Strength (UCS): A Comprehensive Statistical Study

Occasionally, in complex inherent characteristics of certain rocks, especially anisotropic rocks it may be difficult to measure the uniaxial compressive strength UCS. However, the use of empirical relationships to evaluate the UCS of rock can be more practical and economical. Consequently, this study carried out to predict UCS from microfabrics properties of banded amphibolite rocks using multiple regression analysis. Based on statistical results, rock microfabric parameters, which adequately represent the UCS of a given rock type have been selected. The results show that grain size, shape factor and quartz content have high significant correlation with UCS at 95% confidence level. From multiple regression model, approximately 84% of the variance of the UCS can be estimated by the linear combination of these three parameters. However, according to model performance criteria: correlation coefficient (R = 0.919), variance account for (VAF = 97%) and root mean square error (RMSE = 4.16) the study clearly indicates that the developed model is reliable to predict the UCS. Finally, this approach can be easily extended to the modeling of rock strength in the absence of adequate geological information or abundant data.

Particle Swarm Optimization Neural Network for Flow Prediction in Vegetative Channel

AbstractFlow prediction in a vegetated channel has been extensively studied in the past few decades. A number of equations that essentially differ from each other in derivation and form have been developed. Because the process is extremely complex, getting the deterministic or analytical form of the process phenomena is too difficult. Hybrid neural network model (combining particle swarm optimization with neural network) is particularly useful in modeling processes where an adequate knowledge of the physics is limited. This hybrid model is presented here as a complementary tool to model channel flow–vegetation interactions in submerged vegetation conditions. The hybrid model is used to overcome the local minima limitations of a feed-forward neural network. The prediction capability of model has been found to be better than past empirical predictors. The model developed herein showed significantly better results in several model performance criteria compared with empirical models.

Improving real time flood forecasting using fuzzy inference system

In order to improve the real time forecasting of foods, this paper proposes a modified Takagi Sugeno (T–S) fuzzy inference system termed as threshold subtractive clustering based Takagi Sugeno (TSC-T–S) fuzzy inference system by introducing the concept of rare and frequent hydrological situations in fuzzy modeling system. The proposed modified fuzzy inference systems provide an option of analyzing and computing cluster centers and membership functions for two different hydrological situations, i.e. low to medium flows (frequent events) as well as high to very high flows (rare events) generally encountered in real time flood forecasting. The methodology has been applied for flood forecasting using the hourly rainfall and river flow data of upper Narmada basin, Central India. The available rainfall–runoff data has been classified in frequent and rare events and suitable TSC-T–S fuzzy model structures have been suggested for better forecasting of river flows. The performance of the model during calibration and validation is evaluated by performance indices such as root mean square error (RMSE), model efficiency and coefficient of correlation (R). In flood forecasting, it is very important to know the performance of flow forecasting model in predicting higher magnitude flows. The above described performance criteria do not express the prediction ability of the model precisely from higher to low flow region. Therefore, a new model performance criterion termed as peak percent threshold statistics (PPTS) is proposed to evaluate the performance of a flood forecasting model. The developed model has been tested for different lead periods using hourly rainfall and discharge data. Further, the proposed fuzzy model results have been compared with artificial neural networks (ANN), ANN models for different classes identified by Self Organizing Map (SOM) and subtractive clustering based Takagi Sugeno fuzzy model (SC-T–S fuzzy model). It has been concluded from the study that the TSC-T–S fuzzy model provide reasonably accurate forecast with sufficient lead-time.

Model quality objectives based on measurement uncertainty. Part II: NO2 and PM10

Estimating measurement uncertainty for NO2 and PM10 is a complex issue that is normally addressed by experimentalists specifically for every type of instrument and measurement. On the other hand, an estimate of the maximum expected measurement uncertainty is needed when a numerical model is to be evaluated against observations, as proposed in Thunis et al. (2012, referred to as T2012). In a companion paper (Thunis et al., 2013, referred to as T2013) a simplified formulation of the measurement uncertainty in function of the measured concentration is proposed and applied to the simpler case of Ozone. In this paper the same approach is applied for NO2 and PM10, but using different techniques for the uncertainty estimation. For NO2 the Guide to the expression of Uncertainty in Measurement JCGM (2008, referred to as GUM) approach is used and applied on each urban AirBase (1997) measurement over the year 2009. For PM10, the method of the Guide for the Demonstration of Equivalence (ECWG, 2010) is used on data obtained with two different PM samplers used in parallel either during specific monitoring campaign or as available within the AirBase database. The resulting concentration dependent measurement uncertainties are then used to update the MQO (Model Quality Objective) and MPC (Model Performance Criteria) proposed in T2012. An estimate of the measurement uncertainty for annual means is proposed as well.