Implications For Performance Assessment Research Articles

Inconsistent strategies to spin up models in CMIP5: implications for ocean biogeochemical model performance assessment

Abstract. During the fifth phase of the Coupled Model Intercomparison Project (CMIP5) substantial efforts were made to systematically assess the skill of Earth system models. One goal was to check how realistically representative marine biogeochemical tracer distributions could be reproduced by models. In routine assessments model historical hindcasts were compared with available modern biogeochemical observations. However, these assessments considered neither how close modeled biogeochemical reservoirs were to equilibrium nor the sensitivity of model performance to initial conditions or to the spin-up protocols. Here, we explore how the large diversity in spin-up protocols used for marine biogeochemistry in CMIP5 Earth system models (ESMs) contributes to model-to-model differences in the simulated fields. We take advantage of a 500-year spin-up simulation of IPSL-CM5A-LR to quantify the influence of the spin-up protocol on model ability to reproduce relevant data fields. Amplification of biases in selected biogeochemical fields (O2, NO3, Alk-DIC) is assessed as a function of spin-up duration. We demonstrate that a relationship between spin-up duration and assessment metrics emerges from our model results and holds when confronted with a larger ensemble of CMIP5 models. This shows that drift has implications for performance assessment in addition to possibly aliasing estimates of climate change impact. Our study suggests that differences in spin-up protocols could explain a substantial part of model disparities, constituting a source of model-to-model uncertainty. This requires more attention in future model intercomparison exercises in order to provide quantitatively more correct ESM results on marine biogeochemistry and carbon cycle feedbacks.

Can Choice of the Sample Population Affect Perceived Performance: Implications for Performance Assessment

There is accelerating interest in measuring and reporting the quality of care delivered by health care providers and organizations, but methods for defining the patient panels for which they are held accountable are not well defined. To examine the potential impact of using alternative algorithms to define accountable patient populations for performance assessment. We used administrative data regarding Community Health Center (CHC) visits in simulations of performance assessment for breast, cervical, and colorectal cancer screening. Fifteen CHC sites in the northeastern US. We used three different algorithms to define patient populations eligible for measurement of cancer screening rates and simulated center-level performance rates based on these alternative population definitions. Focusing on breast cancer screening, the percentage of women aged 51-75 eligible for this measure across CHCs, if using the most stringent algorithm (requiring a visit in the assessment year plus at least one visit in the 2 years prior), ranged from 28% to 60%. Analogous ranges for cervical and colorectal cancer screening were 18-59% and 26-62%, respectively. Simulated performance data from the centers demonstrate that variations in eligible patient populations across health centers could lead to the appearance of large differences in health center performance or differences in expected rankings of CHCs when no such differences exist. For instance, when holding performance among similar populations constant, but varying the proportion of populations seen across different health centers, simulated health center adherence to screening guidelines varied by over 15% even though actual adherence for similar populations did not differ. Quality measurement systems, such as those being used in pay-for-performance and public reporting programs, must consider the definitions used to identify sample populations and how such populations might differ across providers, clinical practice groups, and provider systems.

Uncertainties in geochemical models of natural systems: Implications for performance assessments

AbstractLicensing a geologic repository for nuclear waste will be based primarily upon the results of a performance assessment (PA), which evaluates compliance of the repository with radiation exposure limits by modeling the engineered and natural barrier systems over extended periods (104 to 106 years). The properly completed PA must include an analysis of the sources of uncertainty, as well as the range of input parameter values, in order to determine whether the analysis is “robust”, “realistic” or “conservative”. This paper examines the sources of uncertainty that may be encountered in source-term and near-field models in a performance assessment of a repository for spent nuclear fuel by examining the uncertainties in modeling the geochemical behavior of the Bangombé natural reactor in Gabon.

Patterns of Writing Skills Derived From Cluster Analysis of Direct-Writing Assessments

Current interest in performance assessment in education raises many measurement questions, one of which is concern for the validity and usefulness of holistic ratings versus multiple-dimension ratings of student work. A study of more than 10,000 students in Grades 3 and 8 was conducted to explore the advantages of analytical scoring in writing assessment. Trained readers scored student essays on a 6-trait analytical scoring system. Cluster analyses were employed to identify subgroups of students who showed similar patterns of trait scores. Eleven patterns of student scores, reflecting different patterns of strengths and weaknesses in writing, were replicated across five modes of writing (e.g., descriptive and persuasive). Limitations of the study and implications for performance assessment are discussed.

Measurement and Meaningful Management

It is commonly assumed that management is responsible for the performance of a system. Since measurement is closely identified with management intentions and practices, as the latter change, the former requires adaptation. Shifts in the reality of the workplace will have implications for performance assessment and implementation. One of the major implications of ongoing changes in the workplace is that traditional measurement systems are becoming more complex. It no longer suffices for a measurement system to be valid; it must also be legitimate and functional. Three conditions-validity, legitimacy, and functionality-have to be considered simultaneously for a performance measurement system to be fit for use. Each dimension shifts with changes in the environment. Performance measurement systems that do not take these shifts into account will eventually get out of sync with management ends. It is therefore necessary to continuously adapt measures and measurement practices to the changing requirements of systems and their environment.

Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment

Recent developments in the philosophy of validity, highlighting the importance of investigating the consequences of assessment use, provide theoretical support for the move toward performance assessment. The problem for validity researchers is finding the appropriate set of criteria and standards to simultaneously support the validity of an assessment-based interpretation and the validity of its impact on the educational system. My intent is to provide an integrative and critical review of the guidance available for conducting validity inquiry in the context of performance assessment. In the first section is a summary of the emerging consensus among measurement scholars—not reflected in the 1985 Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME])—about the centrality of construct validity to the evaluation of any assessment-based interpretation and about the importance of expanding the concept of validity to include explicit consideration of the consequences of assessment use. The description of this emerging consensus suggests general epistemological principles for validity inquiry. In the second section is a description and synthesis of various categories of questions, evidence, or criteria that have been used to describe or guide validity inquiry, either for assessment in general or for performance assessment in particular. These analytic schemes, like the traditional construct-content-criterion categories, highlight specific issues that their authors consider important for validity researchers to address. Each balances technical concerns about such issues as reliability, generalizability, and comparability with concerns about the consequences of assessment. In the final section is an overview of concerns, expressed largely by interpretive researchers, about validity criteria that privilege standardized forms of assessment, whether performance-based or multiple-choice. These arguments suggest the importance of further expanding the conception of validity inquiry to treat as problematic the epistemological principles used to warrant validity conclusions.

Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment

Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment