Setting Performance Standards Research Articles

University of Virginia School of Medicine.

University of Virginia School of Medicine.

Meanings of Merit

What does discussion of higher education issues tell us about fundamental societal questions? Merit appears to underpin the major issues in higher education at the moment. Yet an inspection of these issues suggests that this is really about larger tensions that we may wish to avoid, such as the role of experts in a democracy, specifically their degree of professional autonomy in setting content and performance standards. A new unifying purpose for higher education—articulate judgement—may allow us to address these societal issues more clearly and the role of higher education more publicly and directly. Fundamentally, we must decide what role we want for the academic professions and how we place articulate judgement in their endeavor. Do we have the moral confidence in our enterprise to assert a capacity and a responsibility for independent, informed, deliberative, and humane judgment and for the capacity to elicit the same in our students?

Conspiring fruitfully with professionals: new management roles for professional organisations

Professionalism still is on the way up. However, the working methods of managers and professionals do not develop at the same pace. Professionals often seek out their workplace within an organisation but then proceed to act as soloists, which makes fragmentation, mediocrity and non‐commitment the rule rather than the exception. The manager’s reflex dictates that he/she tackles problems with control and command, resulting in all sorts of conflicts. It is argued that this dysfunctional habit can be corrected by introducing a clear division of roles: professionals manage the primary process, managers the secondary processes. Peace then will be restored but fragmentation, mediocrity and non‐commitment are still evident. We have found heuristically that these core problems only can disappear when professionals and managers tackle them in a concerted action by developing a collective ambition, investing in mutual learning and setting performance standards. Although professionals’ loner genes hold them back in concerted action on these matters, they can overcome this resistance when managers are willing and able to orchestrate a dialogue on these matters. This is a tall order, but the core problems can thus be transformed into inspiration, growth and dedication.

Setting Performance Standards for an Anesthesia Department

Setting Performance Standards for an Anesthesia Department

Performance standards and monitoring requirements of surface coal mine reclamation success in mountainous jurisdictions of western North America: a review

The regulatory performance standards and monitoring requirements that are used to evaluate reclamation success in nine western North America jurisdictions were reviewed. Ecosystem-based reclamation monitoring, a crucial part of regulatory enforcement and adaptive environmental management, was lacking in most jurisdictions. Reclamation practices and regulatory control in each jurisdiction would be improved greatly by effective long-term monitoring. A set of performance standards (including productivity, land use capability and functional dynamics) is proposed which better integrates with a phased bond release process. As well, a long-term hierarchical (population, community, ecosystem, landscape) monitoring programme is proposed.

Setting Performance Standards for an Anesthesia Department

The Stanford University Department of Anesthesia established performance standards by identifying aspects of their service that were related to an important “customer’s” perception of quality. A “quality grid” targeted service attributes that surgeons scored high for importance and low for performance. Control charts and flow charts helped establish reasonable performance levels for “timely first case starts” and “turnaround time.” Control charts indicated that a reasonable performance standard for timely first case starts was “less than 20% of first case delays will be related to anesthesia activities.” For turnaround time, the standard was set at “less than 10% of all turnaround times will be greater than 15 minutes, because of anesthesia-related activities.” After instituting performance standards, the performance for first case start times improved from a 36% defective rate to a 9% defective rate. Anesthesia-related delays in turnaround times stabilized at a 16% defective rate. Using appropriate service standards can improve performance.

Setting Performance Standards for Professional Licensure and Certification

Credentialing programs were surveyed to ascertain the procedures that they use to set performance standards on multiple-choice and open-ended assessments. For multiple-choice assessments, these programs mostly employ variations on the Angoff (1971) standard-setting method. Procedures used with open-ended questions showed more divergence; some agencies use a question by question approach, whereas others utilize methods that consider the assessment results more holistically. Implications of these standard-setting practices from credentialing agencies to the National Assessment of Educational Progress (NAEP), including the consequences of the assessment on the individual candidate, the matrix sampling construction of NAEP assessments, the multiple cutpoints of the NAEP assessment program, and the types of validity evidence that are typically gathered to support the validity of the performance standard, are discussed. Generalizations of these standard-setting methods from the field of professional licensure and certification should be made with caution.

The Role of Automation and Labor in Determining Customer Satisfaction in a Telephone Repair Service Process

ABSTRACTTelephone repair process and performance relationships are modeled using a structural equation approach. The basic structure of the causal model is a function of process flow. The data analysis and modeling approach advocated here can be applied to any manufacturing or service process that has a recurrent process flow and good internal and external performance data. Our primary objectives are to define quantitatively (1) how the performance of repair persons dispatched to the customers' premises is related to automatic testing and repair, and (2) how internal process performance is linked to external customer satisfaction. A secondary objective of this article is to explain briefly some of the practical problems of modeling process‐level performance relationships. Results indicate that the structural equation models are statistically significant and have a meaningful interpretation. Examples of “what if” scenarios using these structural equation models are presented to help managers predict customer satisfaction levels given certain process performance improvements, set standards of performance, and better understand the interdependencies between labor and automation resources in this telephone repair process.

Perspectives on U.S. Army Physical Requirements: Problems and Current Approaches to Solutions

Of the various types of occupational standards (e.g., cognitive, formal education, manual dexterity) two of the more important considerations for military operations are physical demands and anthropometric characteristics. Fifty-seven percent of Army entry level jobs have a physical demands rating of heavy or very heavy (i.e., occasional lifting of 100 or more pounds, & frequent or constant lifting of 50 pounds). Also, the issue of fit is important given that current assignment opportunities for women have greatly increased, and given that typically the design envelope on a given dimension for most equipment has been the 5th to 95th percentile male. Inadequate matching of personnel capabilities and task assignments results in increased costs in the form of supplemental training, inadequate performance, and work-related injuries. Summary of the Four Presentations In order to address these issues, the mismatches in physical demands and anthropometric design parameters must first be clearly defined. The first two presentations in this symposium address these issues. The initial presentation demonstrates an occupation specific data base management system which can be used to assist research efforts, identify jobs most in need of redesign, and set training and performance standards. The second presentation describes a research effort that identifies anthropometric dimensions of military clothing, equipment, and workstations and the level of compatibility (height-reach characteristics) of these fielded systems with female soldier anthropometrics. Once the potential problems are clearly defined, remedial methods must be explored. Task/equipment redesign, personnel selection, and training are methods typically suggested for addressing ergonomic issues and enhancing job performance. Two of these key methods are addressed in this symposium. The third presentation discusses physical training issues, including general versus task-specific training regimens and their potential influence in improving military performance, increasing soldier safety, and decreasing attrition. The final presentation describes a procedure for identifying, evaluating, and redesigning physically demanding tasks and equipment. This solution is the recommended procedure for ensuring soldier safety by designing a task to eliminate the hazard itself. Project Goals This symposium follows an ergonomic systems approach of analyzing the problem, designing and implementing appropriate interventions and establishing methods for evaluating the effectiveness of those interventions (e.g., analyzing cost-benefit tradeoffs of potential implementations). By closely matching job demands with human capabilities and limitations, the ability of all soldiers to perform their duties is enhanced. The goals of these projects and similar military research efforts represent a unified approach to enhance personnel utilization, sustain performance, and ensure the health and safety of our service members.

Legal Defensibility of Standards: Issues and Policy Perspectives

What legal issues may arise in setting content or performance standards? What precedents are relevant? What practices are defensible?

What Do International Assessments Imply for World-Class Standards?

The focus of this article is on the implications of international assessments in setting world-class performance standards. Illustrations of possible uses of results of recent international assessments as benchmarks against which to compare the achievement of U.S. students are presented. A distinction is made between defining a cutting score on an assessment and setting performance standards that indicate whether students have met established performance standards. Several characteristics of international assessments that affect their usefulness as the source of achievement benchmarks (e.g., the content coverage and weighting of the assessment tasks, the definition of the population to be assessed, and the quality of the samples obtained) are discussed.

When Audience Presence Is Enjoyable: The Influence of Audience Awareness of Prior Success on Performance and Task Interest

Subjects received positive feedback on a practice anagram task and then performed a similar task in front of either an audience that was aware of their prior success or an audience that was unaware of their prior success. Two control conditions were included in which subjects performed alone after receiving positive feedback or performed alone without feedback. Results indicated subjects' task interest and performance increased under conditions in which they performed in front of unaware audiences. In addition, subjects' task interest and performance levels decreased in the presence of audiences that were aware of their prior success. These results point to an important role of audience awareness in setting performance standards and expectations. These findings are interpreted from a resource-investment analysis of task value and performance. Implications for surveillance in public settings and in the workplace (e.g., automated computer monitoring) are discussed.

Setting Performance Standards Through Two-Stage Judgmental Policy Capturing

A performance-standard-setting procedure termed judgmental policy capturing (JPC), and its application, are described. The JPC procedure was examined and evaluated as a possible means of establishing performance standards for classroom teachers who seek certification from the National Board for Professional Teaching Standards. The study demonstrated the feasibility of using the method for setting performance standards on assessments composed of complex, multiply scored performance-based exercises. The results of the study suggested that (a) the JPC procedures and associated instructions were clear to panelists, (b) panelists could provide judgments in response to as many as 200 five-dimensional profiles in a reasonably short time without undue fatigue, and (c) most panelists could achieve high levels of consistency in their responses to profiles, as reflected in coefficients of multiple determination between 0.55 and 0.83.

Issues and problems of performance appraisal systems for university general staff: a study in a U.S. university

Both benefits and problems appear to be associated with performance appraisal systems, but the research literature includes relatively few workplace‐based studies which explore the reasons for the difficulties surrounding it. This lack of work‐place information is particularly true of performance appraisal for university general staff. This paper aims to fill this gap, at least partly, through an investigation of the effectiveness of a performance appraisal system for general staff in an American university. Staff in the University of California Office of the President, the central administration of a very large multi‐campus American university, were surveyed for their views on five major areas that are thought to be influenced positively by an effective performance appraisal system. These areas are summarised under the headings: communication and feedback, work objectives, job duties, communication with supervisors, and effectiveness of communication, training and rewards. The survey also sought information about staff views on performance appraisal in general, and also about what special issues performance appraisal presents for supervisors. Results of the study supported many of the hypotheses of the research literature about how to install an effective appraisal system, including the need for continuing training, support from the top of the organisation, clear objectives for the system, and an appropriate match between the type of appraisal system and its uses. There were few differences in perceptions about these matters between staff and their supervisors. The type of work done in higher educational institutions, however, as well as aspects of those organisations’ cultures, gave rise to special problems for their performance appraisal systems, so that the system's benefits appeared limited in the eyes of many of its users. The problems included difficulties in coming to grips with the organisation's goals, and hence in setting performance standards, and enforcing the requirement that everyone in the organisation be appraised. The attempt to link the performance appraisal system with merit pay increases also appeared problematic given the vulnerable nature of higher education funding, the variability of the task environment and a collegial culture which extends to general staff as well as academics.

Training in Government--Setting Performance Standards

Training in Government--Setting Performance Standards

Managing performance

Discusses cultural change within a leading division of a finance house, which had managed change through turbulent times by emphasizing teamwork. Outlines procedures for managerial change, with particular reference to managing difficult people and poor performers. Explores techniques of setting performance standards.

Six years of comprehensive, clinical, performance-based assessment using standardized patients at the Southern Illinois University School of Medicine

By the end of 1990-91, the Southern Illinois University School of Medicine had had six years of experience with comprehensive, performance-based examinations of senior medical students' levels of clinical competence; this report assesses the psychometric aspects of the six examinations given during that period. The examinations were aimed at determining the students' readiness for postgraduate training. Compared with other clinical performance-based assessments that use standardized patients (SPs), these examinations had two important and unique features: (1) the examinations assessed a comprehensive range of clinical skills and reasoning; and (2) they approximated the challenges of real clinical practice wherein a practitioner's skills need to be orchestrated and prioritized in order to meet the challenges of the case encountered. Each year, the performance-based assessment given was an intensive clinical examination requiring each student to work up 13 to 18 SP problems over a three-day period. To administer an examination to an entire class of students took three weeks. Because all students after the first year of administration (1986) were required to pass these examinations, the fairness of test design and scoring and the setting of performance standards for the examinations became important issues for the faculty. The results, accumulated over six years and based on a total of 6,804 student-patient encounters involving 405 students, indicate that this kind of clinical performance-based examination can discriminate a wide range of students' clinical performances. The results provide evidence for the examinations' test security, content validity, construct validity, and reliability.

Assessment of constituency opinion about NBME examination standards

The National Board of Medical Examiners (NBME) reviewed its procedures for setting pass-fail standards in conjunction with the introduction of its comprehensive Part I and Part II examinations in 1991. A component of the review was the 1990 survey of 12 NBME constituencies to assess opinions about pass-fail rates and standard setting. The survey responses, generally consistent across groups, revealed (1) support for a more even distribution of fail rates across the three examinations, (2) support for a fixed standard, with pass-fail rates that change depending on the performance of a cohort of examinees, and (3) a favorable reaction to setting performance standards based on a review of the examination content. These responses may serve as a resource and a frame of reference for determining the standards for the comprehensive Part I and Part II examinations.

A Validation of Ebel's Method for Performance Standard Setting through its Application with Comparison Approaches to a Selected Criterion-Referenced Test

This paper seeks to describe the research and development into performance standard setting processes used by the U.S. Navy's Strategic Weapon System Training Program for developing Criterion-Referenced System Achievement Tests (C-R SAT) for selected Naval Enlisted Classifications (NECs) or job titles in the Strategic Weapon System (SWS). The design of the C-R SAT involves identification of essential skills and knowledge for each NEC. This process entails planned workshops involving selected senior fleet and supervisory personnel drawn from the NEC. Following this test development process each C-R SAT is then subjected to a performance standard setting process. As all performance standard setting methodologies are subjective, the decision was made to validate the Ebel process and its outputs. Two alternative performance standard setting methodologies were applied to data generated from the two versions (watch stander and supervisor) of a recently administered TRIDENT Technician C-R SAT.

Model Standards impact on local health department performance in California.

An evaluation of a state-local negotiation process to implement Model Standards in 18 California local health departments was conducted during 1985-87. Model Standards specific to California were developed by state and local public health officials for seven programs. Nine counties participated in state-local negotiations to set performance standards in these seven programs. During the two-year effort, greater use of Model Standards and more improvement in local program performance occurred in local health departments that participated in the state-local negotiation process. The use of Model Standards appeared to contribute to establishing program priorities, emphasizing the measurement of outcomes, improving data management systems, and evaluating the current performance of programs. While this state-local negotiation process is suggested as the preferred approach to encouraging the use of Model Standards in local health departments, it must be recognized that this process requires time and commitment of resources from both the state and local levels.