Test Model Performance Research Articles

Development and Verification of Daily Gridded Climate Surfaces in the Okanagan Basin of British Columbia

Gridded estimates of daily minimum and maximum temperature and precipitation from 1960 to 2005 were prepared for the Okanagan Basin of British Columbia. The procedure utilized available daily climate data from 182 stations and employed a regression based interpolation scheme at 500 metre grid spacing. Spatial distribution of temperature took into account variations in temperature with elevation, an observed north-south temperature gradient, and proximity to several large lakes in the valley bottom. An inverse distance weighting scheme was used for the interpolation, and a constrained lapse rate approach was used to interpolate in areas that lie above the highest climate station. Temperature inversions were handled by fitting a second order polynomial and introducing a two-layer model. The precipitation routine differs from the temperature model in that the regressions were based on monthly precipitation totals, thus producing daily precipitation surfaces that incorporate an orographic component along with latitudinal trends. A "leave one out" cross-validation procedure was applied to 56 stations to test model performance. Cross-validation of the predicted surfaces indicated that, on average, the daily maximum temperature surfaces were more accurate than the daily minimum temperature surfaces. Mean Absolute Error (MAE) averaged 1.0°C for maximum temperature while for minimum temperature MAE was in the range of 1.3°C to 1.8°C depending on the season. Errors in both temperature and precipitation surfaces are largest at higher elevations where station density is low. Over all stations, monthly MAE for precipitation averaged between 10% and 18%. A MAE calculated from differences between observed and predicted annual precipitation over all stations was 27 mm with a percent error of 6.2%. The model shows a positive bias (1.4) in the daily occurrence of precipitation resulting in an over-prediction of days with drizzle. Over 80% of incorrectly predicted wet days had less than 1 mm precipitation. These error statistics compare favourably to those from similar regression-based interpolation models.

Good Research Practices for Comparative Effectiveness Research: Analytic Methods to Improve Causal Inference from Nonrandomized Studies of Treatment Effects Using Secondary Data Sources: The ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—Part III

Most contemporary epidemiologic studies require complex analytical methods to adjust for bias and confounding. New methods are constantly being developed, and older more established methods are yet appropriate. Careful application of statistical analysis techniques can improve causal inference of comparative treatment effects from nonrandomized studies using secondary databases. A Task Force was formed to offer a review of the more recent developments in statistical control of confounding. The Task Force was commissioned and a chair was selected by the ISPOR Board of Directors in October 2007. This Report, the third in this issue of the journal, addressed methods to improve causal inference of treatment effects for nonrandomized studies. The Task Force Report recommends general analytic techniques and specific best practices where consensus is reached including: use of stratification analysis before multivariable modeling, multivariable regression including model performance and diagnostic testing, propensity scoring, instrumental variable, and structural modeling techniques including marginal structural models, where appropriate for secondary data. Sensitivity analyses and discussion of extent of residual confounding are discussed. Valid findings of causal therapeutic benefits can be produced from nonrandomized studies using an array of state-of-the-art analytic techniques. Improving the quality and uniformity of these studies will improve the value to patients, physicians, and policymakers worldwide.

Challenges of simulating complex environmental systems at the landscape scale: A controversial dialogue between two cups of espresso

With the advancement of computational systems and the development of model integration concepts, complexity of environmental model systems increased. In contrast to that, theory and knowledge about > environmental systems as well as the capability for environmental systems analyses remained, to a large extent, unchanged. As a consequence, model conceptualization, data gathering, and validation, have faced new challenges that hardly can be tackled by modellers alone. In this discourse-like review, we argue that modelling with reliable simulations of human-environmental interactions necessitate linking modelling and simulation research much stronger to science fields such as landscape ecology, community ecology, eco-hydrology, etc. It thus becomes more and more important to identify the adequate degree of complexity in environmental models (which is not only a technical or methodological question), to ensure data availability, and to test model performance. Even equally important, providing problem specific answers to environmental problems using simulation tools requires addressing end-user and stakeholder requirements during early stages of problem development. In doing so, we avoid modelling and simulation as an end of its own.

Optimization and probabilistic calibration of avalanche block models

Abstract The Norwegian Geotechnical Institute (NGI) has been performing full-scale avalanche tests at Ryggfonn in western Norway for more than 30 years. During those years, measurements from about three dozen dry-snow avalanches have provided information on front velocities and runout distances. Some of those measurements were used to evaluate “optimal parameters” for a simple avalanche model and to calibrate the model, following a well-defined probabilistic method. Traditionally, parameters of those kinds of models were evaluated from runout analysis disregarding any dynamics. The set of roughly 20 observed avalanches from one single path including, estimations of the front velocities at three points in the lower third of the track provide a unique opportunity for introducing uncertainty quantification methods for evaluating the performance of similar kinds of mass block models. We present the optimization, the model calibration and results from the model performance testing.

Anticipating Welfare Impacts via Travel Demand Forecasting Models

A great disparity exists between the direction of travel demand forecasting by researchers and the travel demand models used by transportation planning organizations. Activity-based models of travel demand have become increasingly studied in the academic realm, and significant advances have been made in recent years. However, travel demand forecasting tools used in practice have lagged and rely on traditional, aggregate four- or five-step approaches. One reason behind the divergence in methods is the lack of work that directly compares performance of the two approaches. This research provides such a comparison, with an emphasis on calculations of traveler welfare. A traditional, aggregate model and an activity-based microsimulation model of travel demand were developed in parallel by using the same data for Austin, Texas. The models were applied for a base scenario and for several policy scenarios to test model performance and sensitivity to inputs. The spatial distribution of traveler welfare implied by these scenarios illuminates a variety of key differences in the models’ performance and suggests that the activity-based model enjoys a greater sensitivity to inputs. Additional outputs demonstrate the level of segmentation that can be attained in model outputs using microsimulation methods. The comparative analysis of these two competing approaches to travel demand forecasting also offers some insight into the practical benefits of an activity-based approach.

Lagrangian validation of numerical drifter trajectories using drifting buoys: Application to the Agulhas system

The skill of numerical Lagrangian drifter trajectories in three numerical models is assessed by comparing these numerically obtained paths to the trajectories of drifting buoys in the real ocean. The skill assessment is performed using the two-sample Kolmogorov–Smirnov statistical test. To demonstrate the assessment procedure, it is applied to three different models of the Agulhas region. The test can either be performed using crossing positions of one-dimensional sections in order to test model performance in specific locations, or using the total two-dimensional data set of trajectories. The test yields four quantities: a binary decision of model skill, a confidence level which can be used as a measure of goodness-of-fit of the model, a test statistic which can be used to determine the sensitivity of the confidence level, and cumulative distribution functions that aid in the qualitative analysis. The ordering of models by their confidence levels is the same as the ordering based on the qualitative analysis, which suggests that the method is suited for model validation. Only one of the three models, a 1/10° two-way nested regional ocean model, might have skill in the Agulhas region. The other two models, a 1/2° global model and a 1/8° assimilative model, might have skill only on some sections in the region.

Improved model identification for non-linear systems using a random subsampling and multifold modelling (RSMM) approach

In non-linear system identification, the available observed data are conventionally partitioned into two parts: the training data that are used for model identification and the test data that are used for model performance testing. This sort of ‘hold-out’ or ‘split-sample’ data partitioning method is convenient and the associated model identification procedure is in general easy to implement. The resultant model obtained from such a once-partitioned single training dataset, however, may occasionally lack robustness and generalisation to represent future unseen data, because the performance of the identified model may be highly dependent on how the data partition is made. To overcome the drawback of the hold-out data partitioning method, this study presents a new random subsampling and multifold modelling (RSMM) approach to produce less biased or preferably unbiased models. The basic idea and the associated procedure are as follows. First, generate K training datasets (and also K validation datasets), using a K-fold random subsampling method. Secondly, detect significant model terms and identify a common model structure that fits all the K datasets using a new proposed common model selection approach, called the multiple orthogonal search algorithm. Finally, estimate and refine the model parameters for the identified common-structured model using a multifold parameter estimation method. The proposed method can produce robust models with better generalisation performance.

Non-asymptotic tests of model performance

This paper describes a non-asymptotic approach to the problem of selection bias in economic forecasting. By using non-asymptotic measure concentration results, it is possible to deal with settings in which the class of potential models is large with respect to the number of data points. The bounds on p values obtained by these methods are necessarily conservative, but they provide a useful benchmark for model selection in settings where asymptotics may not apply.

Simulation of phosphorus losses from lysimeters

AbstractBecause of water resources eutrophication and the need for water protection strategies, the estimation of diffuse phosphorus (P) leaching losses from agricultural soils has become an important issue. The objective of this study was to numerically depict P leaching and transformation processes to clarify the dominant factors controlling the P dynamics in soils and to investigate interactions between P and carbon (C) and nitrogen (N) cycles. We expanded the existing WASMOD modeling package with a P subroutine and tested model performance on an experimental data set from a 12 y lysimeter study.

Predictive Model of Habitat Suitability for the Marbled Murrelet in Western Washington

Abstract: The marbled murrelet (Brachyramphus marmoratus) is a small Pacific seabird with a breeding range that extends from the Aleutian Islands to central California. Throughout most of its breeding range, it uses mature and old‐growth coniferous forests as nesting habitat. Although most murrelets seem to nest within 60 km of the coast, occupied nesting habitat has been identified as far as 84 km from the ocean in Washington state. Due to the extensive inland distances within which birds are known to breed, the area requiring surveys to identify breeding sites can be enormous. Therefore, the standard 2‐year survey protocol can be expensive and time‐consuming for forest management agencies and companies to administer. We developed a logistic regression model to determine whether a suite of forest structural characteristics could be used to reliably predict occupancy of a forest patch by marbled murrelets. We tested the performance of the final model using cross‐validation procedures and a sample of independent sites. We used 50 sites surveyed for marbled murrelets to estimate the model, and 48 independent sites were available to test model performance. All 50 sites were on private forestland owned by Rayonier located in the western lowlands of Olympic Peninsula within the Sitka spruce and western hemlock transition zones. We sampled forest habitat at each site, and we collected information on 15 explanatory variables. The best‐fitting logistic regression model contained variables that measured number of canopy layers (P<0.001, approx. F test) and mistletoe (Arceuthobium sp.) abundance (P = 0.031, approx. F test). The model misclassified 2 of 33 (94% correct) unoccupied sites as occupied using a classification cut‐off (c) of c=0.25. In the other direction, under cross‐validation the final model misclassified 2 of 17 (88% correct) occupied sites as unoccupied. On a test of the model against an independent sample, using a classification cut‐off value of c = 0.25, the final model correctly classified 36 of 48 sites (75% correct). The final model misclassified 3 of 31 occupied sites as unoccupied (90% correct). Use of predictive models could greatly reduce the amount of forest that requires surveys by screening out those sites with little probability of use and by focusing remaining effort on higher probability sites, resulting in a higher likelihood of identifying occupied sites and thereby more efficiently conserving marbled murrelet nesting habitat.

Separating spatial and temporal sources of variation for model testing in precision agriculture

The application of crop simulation models in precision agriculture research appears to require only the specification of some input parameters and then running the model for each desired location in a field. Reports in the extensive literature on modeling have described independent tests for different cultivars, soil types and weather, and these have been presumed to validate the model performance in general. However, most of these tests have evaluated model performance for simulating mean yields for multiple plots in yield trials or in other large-area studies. Precision agriculture requires models to simulate not only the mean, but also the spatial variation in yield. No consensus has emerged about how to test model performance rigorously, or what level of performance is sufficient. In addition, many measures of goodness of fit between the observed and simulated data (i.e., model performance) depend on the range of variation in the observed data. If, for example, inter-annual and spatial sources of variation are combined in a test, poor performance in one might be masked by good performance in the other. Our objectives are to: (1) examine research aims that are common in precision agriculture, (2) discuss expectations of model performance, and (3) compare several traditional and some alternative measures of model performance. These measures of performance are explained with examples that illustrate their limitations and strengths. The risk of relying on a test that combines spatial and temporal data was shown with data where the overall fit was good (r 2 > 0.8), but the fit within any year was zero. Information gained using these methods can both guide and help to build confidence in future modeling efforts in precision agriculture.

Converting the UK calculation of road traffic noise (CORTN) to a model capable of calculating LAeq,1h for the Tehran’s roads

This paper describes the methodology through which the UK calculation of road traffic noise (CORTN) has been converted to the algorithms that are able to calculate hourly A-weighted equivalent sound pressure level ( L Aeq,1h) for the Tehran’s roads. The methodology adopts two different approaches to model calibration and performance test through the holdout validation method on the basis of the database including 52 samples taken from 52 sampling stations located alongside 5 roads of Tehran at distances less than 4 m from the nearside carriageway edge. As to the CORTN manual the distances less than 4 m are considered to be equal to 4 m. In the first approach the model is calibrated through carrying out nonlinear regression parameter estimation using 50% of samples to replace the basic noise level parameters with the new ones that are presumably able to satisfy the objective of the study with an acceptable fitness of the model. In the second approach the model calibration is carried out on the basis of 30 measurements taken from 2 roads. In the next step the other subsets of samples are introduced into the calibrated equations to conduct the performance test. Non parametric goodness of fit tests, i.e. two related samples Wilcoxon and two independent samples Kolmogorov–Smirnov, respectively conducted for the calibration and the performance test steps; indicate satisfactory results for both approaches.

Identifying Sites for Elk Restoration in Arkansas

Abstract: We used spatial data to identify potential areas for elk (Cervus elaphus) restoration in Arkansas. To assess habitat, we used locations of 239 elk groups collected from helicopter surveys in the Buffalo National River area of northwestern Arkansas, USA, from 1992 to 2002. We calculated the Mahalanobis distance (D2) statistic based on the relationship between those elk‐group locations and a suite of 9 landscape variables to evaluate winter habitat in Arkansas. We tested model performance in the Buffalo National River area by comparing the D2 values of pixels representing areas with and without elk pellets along 19 fixed‐width transects surveyed in March 2002. Pixels with elk scat had lower D2 values than pixels in which we found no pellets (logistic regression: Wald χ2 = 24.37, P < 0.001), indicating that habitat characteristics were similar to those selected by the aerially surveyed elk. Our D2 model indicated that the best elk habitat primarily occurred in northern and western Arkansas and was associated with areas of high landscape heterogeneity, heavy forest cover, gently sloping ridge tops and valleys, low human population density, and low road densities. To assess the potential for elk‐human conflicts in Arkansas, we used the analytical hierarchy process to rank the importance of 8 criteria based on expert opinion from biologists involved in elk management. The biologists ranked availability of forage on public lands as having the strongest influence on the potential for elk‐human conflict (33%), followed by human population growth rate (22%) and the amount of private land in row crops (18%). We then applied those rankings in a weighted linear summation to map the relative potential for elk‐human conflict. Finally, we used white‐tailed deer (Odocoileus virginianus) densities to identify areas where success of elk restoration may be hampered due to meningeal worm (Parelaphostrongylus tenuis) transmission. By combining results of the 3 spatial data layers (i.e., habitat model, elk‐human conflict model, deer density), our model indicated that restoration sites located in west‐central and north‐central Arkansas were most favorable for reintroduction.

A validated risk model for early neutropenic events in older cancer patients receiving systemic chemotherapy

9036 Background: A prospective, nationwide study was undertaken to develop and validate a risk model for early neutropenic events (NE) in older cancer patients undergoing chemotherapy. Methods: 1,386 patients =65 years of age with lung, breast, colorectal, ovarian cancer or lymphoma were prospectively registered at 117 randomly selected sites. Data on up to 4 cycles were collected upon initiation of chemotherapy. A logistic regression model for cycle 1 NE consisting of febrile neutropenia (FN; fever/infection and absolute neutrophil count nadir <1x109/L) or severe neutropenia (SN; neutrophils <.5x109/L) was derived on 1,378 patients with available data. Validation was performed using a split sample random selection process. Results: No significant differences in distribution of NE or predictive factors were observed between derivation dataset (n=922) and validation dataset (n=464). Major independent baseline clinical risk factors for cycle 1 NE in the derivation model (DM) included: anthracycline based regimens (p<.001), non-chemotherapy immune-modulatory agents (p=.003), elevated bilirubin (p=.016), reduced glomerular filtration rate (p<.001), cancer type (p=.02), planned relative dose intensity =85% (p=.027), and regimens containing cyclophosphamide (p<.001), etoposide (p=.002) or ifosfamide (p=.032). Reduced risk of cycle 1 NE was associated with myeloid growth factor (MGF) prophylaxis (p<.001). DM R2 was 0.478 and c-statistic 0.88 [95% CI 0.86–0.91; p<.001]. At median predicted risk of cycle 1 NE of 7%, model test performance (MTP) showed: sensitivity 90%; specificity 59%; and predictive value positive and negative of 32% and 97%, respectively. Cycle 1–4 FN risk in the DM was 16.6% and 3.3% among high and low risk patients, respectively. The validation model (VM) R2 was 0.508 and c-statistic 0.89 [95% CI: 0.86–0.93; p<.001]. MTP in the VM demonstrated: sensitivity 90%; specificity 65%; predictive value positive and negative of 36% and 97%, respectively. Cycle 1–4 FN risk in the VM was 16.8% and 1.6% in high and low risk patients, respectively. Conclusions: This validated risk model demonstrated good discrimination between older cancer patients at decreased risk for NE, and those at increased risk who may benefit from targeted prophylaxis with MGF. No significant financial relationships to disclose.

Simulating SOC changes in 11 land use change chronosequences from the Brazilian Amazon with RothC and Century models

Land use and land cover changes in the Brazilian Amazon have major implications for regional and global carbon (C) cycling. Cattle pasture represents the largest single use (about 70%) of this once-forested land in most of the region. The main objective of this study was to evaluate the accuracy of the RothC and Century models at estimating soil organic C (SOC) changes under forest-to-pasture conditions in the Brazilian Amazon. We used data from 11 site-specific ‘forest to pasture’ chronosequences with the Century Ecosystem Model (Century 4.0) and the Rothamsted C Model (RothC 26.3). The models predicted that forest clearance and conversion to well managed pasture would cause an initial decline in soil C stocks (0–20 cm depth), followed in the majority of cases by a slow rise to levels exceeding those under native forest. One exception to this pattern was a chronosequence in Suia-Missu, which is under degraded pasture. In three other chronosequences the recovery of soil C under pasture appeared to be only to about the same level as under the previous forest. Statistical tests were applied to determine levels of agreement between simulated SOC stocks and observed stocks for all the sites within the 11 chronosequences. The models also provided reasonable estimates (coefficient of correlation = 0.8) of the microbial biomass C in the 0–10 cm soil layer for three chronosequences, when compared with available measured data. The Century model adequately predicted the magnitude and the overall trend in δ 13C for the six chronosequences where measured δ 13C data were available. This study gave independent tests of model performance, as no adjustments were made to the models to generate outputs. Our results suggest that modelling techniques can be successfully used for monitoring soil C stocks and changes, allowing both the identification of current patterns in the soil and the projection of future conditions. Results were used and discussed not only to evaluate soil C dynamics but also to indicate soil C sequestration opportunities for the Brazilian Amazon region. Moreover, modelling studies in these ‘forest to pasture’ systems have important applications, for example, the calculation of CO 2 emissions from land use change in national greenhouse gas inventories.

EVALUATING REGIONAL PREDICTIVE CAPACITY OF A PROCESS-BASED MERCURY EXPOSURE MODEL, REGIONAL-MERCURY CYCLING MODEL, APPLIED TO 91 VERMONT AND NEW HAMPSHIRE LAKES AND PONDS, USA

Regulatory agencies must develop fish consumption advisories for many lakes and rivers with limited resources. Process-based mathematical models are potentially valuable tools for developing regional fish advisories. The regional mercury cycling model (R-MCM) specifically was designed to model a series of lakes for a given region with site-specific data and parameterization for each application. In this paper, we explore the feasibility of R-MCM application to develop regional fish advisories from existing data by testing model performance across 91 Vermont ([VT], USA) and New Hampshire ([NH], USA) lakes. We use a progressive method of parameter refinement ranging from simple defaults specified by the model to site-specific parameterization to evaluate potential improvements in model prediction. Model applications and parameter refinement tiers are based on Regional Environmental Monitoring Assessment Program (REMAP) data. Results show that R-MCM generally underpredicts water column methylmercury and total mercury concentrations and overpredicts sediment methylmercury concentrations. Default level input parameterization produced the largest amount of random scatter in model forecasted values. Using site-specific values for the default level characteristics reduced this variability but did not improve overall model performance. By separating the observed and predicted data by lake characteristics, we identify some overall trends in bias and fit, but are unable to identify systematic biases in model performance by lake type. This analysis suggests that process-based models like R-MCM cannot be used for a priori predictive applications at the regional scale at this time. Further, this work reinforces the need for additional research on the transport and transformation of mercury to elucidate parameterization useable in a modeling framework to help refine predictive capabilities of process-based models.

TMPT – conclusions and consequences for the industry from the industry

After a short introduction, the first part of this publication shows the main topics of vehicle dynamics from the computational simulation point of view: vehicle handling, comfort and durability. It is discussed, which effects of the physical tire behavior are included in different tire models for simulation. Additionally limitations of these models are given briefly. In the second part, the tire parameterization process is shown, starting from the tire measurement to the release of the tire data set. One essential point of this process is the validation and plausibility check of the virtual tire behavior. Finally, the general requirements of tire models are discussed, based on the results of the Tire Model Performance Test. The acknowledgment of the virtual tire characteristics is a fundamental part of correctly interpreting simulation results for decisions in the development process.

Developments in vehicle dynamics and the tire model performance test

Aspects of the vehicle dynamics like handling, ride, comfort, and durability as well as of controlled subsystems are discussed. The increasing importance of simulation to support the development and application process of vehicles with control systems is shown. The role of the tire and the corresponding tire models is highlighted and requirements for the modeling and parameterization of tire models are derived.

CDTire: a tire model for comfort and durability applications

This paper discusses the tire model performance test (TMPT) participation of the LMS International comfort and durability tire model (LMS CDTire). CDTire is a family of three models based on a macroscopic physical description of tires, which is a compromise between scope of applicability and calculation time. A short overview of the CDTire model suite, including parameter identification (PI) software and road surface models, is complemented with application examples of full vehicle simulations utilizing different sub-models and different road surface models. The PI procedure for the TMPT tire, a non-commercial 205/55 R 16 passenger car tire, is reviewed and the results of the TMPT validation and capability tests are discussed.

TMPT: multi-body package SIMPACK

Within the framework of the tire model performance test (TMPT), SIMPACK, among other packages, was used as simulation environment to carry out the capability and validation tests with different tire models. This article gives an overview on the multi-body software SIMPACK with focus on automotive applications such as handling, ride, comfort and durability. The description of the basic structure of the motion equations and the solver modes will be followed by a classification of the range of applications of SIMPACK–automotive for vehicle dynamics design and assessment and the corresponding classification of tire models within SIMPACK. The classification will be done along occurring oscillation amplitudes ranging from very small to large part displacements and along occurring frequencies ranging from quasi-static movements up to the acoustic frequency. The basic structure of the interfaces between tire model, road description and multi-body model will be revealed, and the different call modes of tire models in multi-body codes will be summarized. The TMPT test rig set-up in SIMPACK will be explained describing the kinematic and force element set-up and covering issues such as initial conditions and test-case-specific changes thereof.