Experienced Assessors Research Articles

Construct Validity of a Tool for Assessing Competency in Polypectomy

IntroductionThere is a void in the formal assessment of technical skills in polypectomy which is now considered an integral part of colonoscopy. It has been shown to reduce the incidence...

Measuring caloric response: Comparison of different analysis techniques

Electronystagmography (ENG) testing has been supplanted by newer techniques of measuring eye movement with infrared cameras (VNG). Most techniques of quantifying caloric induced nystagmus measure the slow phase velocity in some manner. Although our analysis is carried out by very experienced assessors, some systems have computer algorithms that have been "taught" to locate and quantify maximum responses. We wondered what differences in measurement might show up when measuring calorics using different techniques and systems, the relevance of this being that if there was a change in slow phase velocity between ENG and VNG testing when measuring caloric response, then normative data would have to be changed. There are also some subjective but important aspects of ENG interpretation which comment on the nature of the response (e.g. responses which might be "sporadic" or "scant"). Our experiment compared caloric responses in 100 patients analyzed four different ways. Each caloric was analyzed by our old ENG system, our new VNG system, an inexperienced assessor and the computer algorithm, and data was compared. All four systems made similar measurements but our inexperienced assessor failed to recognize responses as sporadic or scant, and we feel this is a limitation to be kept in mind in the rural setting, as it is an important aspect of assessment in complex patients. Assessment of complex VNGs should be left to an experienced assessor.

Contaminated land risk assessment: variability in site assessment and decision making in the UK

Judgement forms an integral part of a risk-based approach to the assessment of land affected by contamination. Legislation and guidance suggest that the assessor should use a rational step-wise process to identify pollutant linkages in order to assess risk from land contamination. The present study aims to investigate the decision-making processes that are used by experienced contaminated land assessors. This study required 29 participants with a minimum of five years’ relevant experience to rate the level of risk from land contamination on 27 hypothetical housing development sites. Each site was designed with specific information (variables) used as indicators of the potential for unacceptable risk. Linear regression analysis was used to identify the significance of each of the variables in determining the level of risk assessed by participants. The first of the key findings was that considerable disagreement was observed between participants, and this was correlated to cases with contradictory information. This may have also been related to the participant’s perception of the available risk scale. The linear regression analysis showed that the most influential variables were chemical-test data and the presence of human-exposure pathways. These findings would suggest that experienced assessors focus on a few key aspects of the information available to assess risk from land contamination. However, analysis of the qualitative data collected in the study supported a more holistic decision-making process, in line with use of pollutant linkages described in guidance. The results suggest that when presented with limited data for development sites, assessors may rely on a few variables to rate the risk, but that a coherent picture of the interaction of all of the variables is required for a more confident assessment. The findings of the study presented here can be used to inform training and future guidance in this sector.

Interobserver Reliability of Classification Systems to Rate the Quality of Femoral Neck Fracture Reduction

We compared the interrater reliability of five classification systems and surgeons' global impressions of the quality of femoral neck fracture reduction. Six orthopedic trauma surgeons, six orthopedic nontrauma surgeons, and six orthopedic residents from three sites each rated 50 radiographs of postoperative femoral neck fractures fixated with cannulated screws or a sliding hip screw, using their overall impression, the Garden Index, Lowell's criteria, Lindquist and Tornkvist's criteria, the Western Infirmary Glasgow (WIG) angle, and standards established by a working group of orthopedic trauma surgeons. Reliability estimates for the Garden Index, Lowell's criteria, and the working group standards all fell within the range of moderate agreement [intraclass correlation coefficient (ICC) range 0.41-0.48], with no instrument achieving higher reliability than reviewers' overall impressions (ICC 0.49, 95% CI 0.39-0.61). Reviewers reached only fair agreement using Lindquist and Tornkvist's criteria and the WIG angle (kappa = 0.27 and 0.39, respectively). Trauma surgeons consistently achieved higher agreement than did nontrauma surgeons and trainees. Future studies using quality of reduction as an outcome measure or exploring the prognostic importance of reduction quality should measure this with trauma surgeons' overall impression, rather than with less experienced assessors or using one of the alternative instruments.

Reliability and validity of motion analysis in children treated for congenital clubfoot according to the Clubfoot Assessment Protocol (CAP) using inexperienced assessors.

BackgroundThe Clubfoot Assessment Protocol (CAP) was developed for follow-up of children treated for clubfoot. The objective of this study was to analyze reliability and validity of the six items used in the domain CAPMotion Quality using inexperienced assessors.FindingsFour raters (two paediatric orthopaedic surgeons, two senior physiotherapists) used the CAP scores to analyze, on two different occasions, 11 videotapes containing standardized recordings of motion activity according to the domain CAPMotion Quality These results were compared to a criterion (two raters, well experienced CAP assessors) for validity and for checking for learning effect.Weighted kappa statistics, exact percentage observer agreement (Po), percentage observer agreement including one level difference (Po-1) and amount of scoring scales defined how reliability was to be interpreted. Inter- and intra rater differences were calculated using median and inter quartile ranges (IQR) on item level and mean and limits of agreement on domain level.Inter-rater reliability varied between fair and moderate (kappa) and had a mean agreement of 48/88% (Po/Po-1). Intra -rater reliability varied between moderate to good with a mean agreement of 63/96%. The intra- and inter-rater differences in the present study were generally small both on item (0.00) and domain level (-1.10). There was exact agreement of 51% and Po-1 of 91% of the six items with the criterion. No learning effect was found.ConclusionThe CAPMotion quality can be used by inexperienced assessors with sufficient reliability in daily clinical practice and showed acceptable accuracy compared to the criterion.

Comprehensive quality assessment of healthy school interventions

Objectives. The number of healthy school interventions of unknown quality overwhelms schools. Quality is a construct that is differently interpreted by teachers and health promoters. The schoolBeat checklist for quality assessment of healthy school interventions incorporates the quality perceptions of both professional groups. To support quality improvements – and thus effectiveness – in school health promotion, this study evaluates the schoolBeat checklist. Methods. Twenty-nine healthy school interventions were assessed in the Netherlands, each by two health promoters and two teachers—individually and at a consensus meeting. Generalizability coefficients were calculated for the nine specific quality criteria. Results. The mean consensus score differs from the mean average individual score for two out of nine criteria. To obtain a threshold Generalizability coefficient of 0.70, the number of assessors required per criterion ranges from 1.6 to 10.8, with an average of 4.7. Conclusion. Quality assessment procedures of healthy school interventions using the schoolBeat checklist require about four experienced assessors from each professional domain to facilitate reliable quality scores based on individual assessment only. Publicly available quality scores enable the inclusion of high quality interventions in school policies in order to increase the impact of school health.

A survey of assessor beliefs and practices related to faking

PurposeTo gather information on assessor beliefs and behaviors in relation to assessee faking issues on a personality inventory in the individual assessment process.Design/methodology/approachA survey approach was used in this research. Totally 77 experienced assessors who conducted individual assessments for an international consulting firm responded to the survey. Analyses of mean item rankings were used to answer several research questions.FindingsMajor results of the study were: assessors believe faking is a problem; assessors believe they can detect faking; and assessors believe they can effectively eliminate all of the effects of faking when evaluating the candidates.Practical implicationsThe first implication from this research is that assessors believe that they can detect and deal with faking despite a paucity of evidence to support it. The second implication is that organizations may be reluctant to continue to develop effective methods of identifying and dealing with faking if their assessors mistakenly believe they are already successfully doing so.Originality/valueThis study is the first to survey experience assessors regarding their beliefs and perceptions of faking issues in the individual assessment process and is designed to garner immediate practical insights and ideas for future testable hypotheses.

Precision And Accuracy Of Pest And Pathogen Damage Assessment In Young Eucalypt Plantations

Fungal pathogens, browsing mammals, birds, insects, nutrient deficiencies, drought, frost and waterlogging are all damaging agents to plantation species. The subsequent loss in leaf tissue or reduced photosynthetic potential can reduce growth and potentially lead to tree death. The Crown Damage Index (CDI) was developed in Australia to quantify damage in young eucalypt plantations. The accuracy and precision of assessing damage at a tree level were determined to ensure the reliability, objectivity and repeatability of the CDI method. Nine assessors, with varying levels of experience, estimated damage on three plots of fifty trees each, to obtain an understanding of the subjectivity of assessing damage caused by insects (e.g. Chrysophtharta spp.) and fungal pathogens (e.g. Mycosphaerella spp.) on Eucalyptus globulus. Damage levels were measured by destructive sampling to enable direct comparisons between estimates and damage levels to be made. The most experienced assessors provided the most repeatable estimates and were generally the most accurate. The incidence of foliar necrosis was the least subjective measure while defoliation was the most subjective and the least accurate of the indices measured. All assessors, regardless of experience, were able to predict the Crown Damage Index (a combined index of all damage classes) to within 12% of measured damage levels.

Controlled continuous flow delivery system for investigating taste-aroma interactions.

A multichannel flavor delivery system, Dynataste, was developed. Controlled amounts of isoamyl acetate (100 ppm) and sucrose (0-3%) solution was administered to experienced and naïve assessors who used time intensity techniques to record perceived 'fruit' flavor intensity. In-nose volatile delivery was monitored using atmospheric pressure chemical ionization-mass spectrometry. Results indicated that sucrose is a key driver of fruit flavor intensity but that the magnitude of the effect varies between individuals. The combined temporal analysis of chemical stimuli in vivo and sensory data indicate evidence of interactions at a perceptual level. Comparison of experienced and naïve assessors revealed cross-modal interactions in each group, although a subgroup of experienced assessors was unaffected by changes in sucrose concentration. This raises the question of the selected use of experienced panels in cross modal investigations.

Assessment Center Procedures: Cognitive Load During the Observation Phase

This study explores the traditional procedure of observing assessment center exercises while taking notes vs. an alternative procedure where assessors merely observe and postpone note‐taking until immediately after the exercise. The first procedure is considered to be cognitively demanding due to the requirement of simultaneous note‐taking and observing. Also, dual task processing (concurrent observing and note‐taking) is considered to be especially demanding for assessors without rating experience. The procedures are evaluated using a 2 × 2 design (with note‐taking/without note‐taking × experienced/inexperienced). Some 121 experienced and inexperienced assessors rated videotaped candidates, observing either with or without taking notes. Results showed that experienced assessors yield significantly higher differential accuracy than inexperienced assessors. We did not find an effect of observation procedure on accuracy, interrater reliability or halo. Implications for future research are described.

A Teaching Method for Multicultural Assessment: Psychological Report Contents and Cultural Competence

An earlier training model emphasized systematic feedback on accuracy of concepts contained in student assessment reports prepared with common data sets from standard tests. This feedback resulted in consistent increments in concept agreements between reports prepared by students and more experienced assessors. This model was applied in courses with multicultural students by contrasting standard and multicultural assessment and using data sets from multicultural assessees. These students experienced greater difficulty in demonstrating consistent increments of agreement in report concepts due to uneven knowledge of specific cultures, inapplicability of available norms, and interpretation issues. The effectiveness of combining standard and multicultural training should be compared with conventional standard training followed by culturally focused specialized assessment courses.

Alternative Perspectives on Playroom Practice Autres Perspectives sur les Pratiques en Salle de Jeux Perspectivas Alternativas sobre la Pra´ctica en el Cuarto de Juegos

This paper reports on research designed to explore alternative perspectives on playroom practice. Two perspectives on the role of practitioners were articulated. One was an outside, context-free perspective constructed in consultation with a group of experienced assessors of pre-school provision and presented as a Framework for Good Practice. Observations of acknowledged good practice (using the Framework as the observation tool) and interviews with practitioners supported its ready applicability to the range of pre-school settings typical in Scotland. The second perspective was an 'insider' view encapsulating the way in which practitioners think about their work in the playroom, their professional craft knowledge. The characteristics of each perspective and the insights that can be derived from each are discussed, along with the value of considering first the 'insider' perspective when staff are engaged in continuing professional development. Cet article fait état de recherches conçues dans le but d'explorer de nouvelles perspectives sur les pratiques en salle de jeux. Deux perspectives sur le rôle des praticiens ont été retenues. L'une consiste en une perspective externe, totalement dénuée de contextuatisation, élaborée en consultation avec un groupe d'inspecteurs d'établissements préscolaires expérimentés et présentée en tant que Structure de bonne pratique. Des observations de bonne pratique reconnue (se référant à cette Structure comme outil d'observation) ainsi que des entretiens auprès de praticiens sont venus étayer son application pratique dans la gamme d'établissement pré-scolaires typiques en Écosse. La seconde perspective représente une vue e interne f qui retrace la façon dont les praticiens réfléchissent àleur travail en salle de jeux, leurs connaissances professionnelles sur le terrain. Les caractéristiques de ces deux perspectives et les aperçus qui sont offerts sur chacune d'entre elles font l'objet de discussions. Est également débatue la validité d'introduire en premier la perspective e interne f dans le cadre d'un programme de formation continue éventuellement suivi par le personnel. Este artículo informa sobre una investigación diseñada para explorar las perspectivas alternativas sobre la práctica de los juegos en el aula. Se articulan dos perspectivas sobre el papel de los docentes. Una era una perspectiva externa, libre de contexto construida tras consultar a un grupo de asesores con experiencia en la disposición preescolar y presentada como un Marco para la Buena Práctica . Las observaciones de una reconocida buena práctica (usando el Marco como una herramienta de observación) y las entrevistas con los docentes apoyaron su pronta aplicación en el ámbito de los escenarios preescolares típicos en Escocia. La segunda perspectiva es una visión interna que engloba la manera en la que los docentes reflexionan sobre su trabajo en el aula de juego, el conocimiento profesional de su oficio. Se habla de las características de cada perspectiva y de las percepciones que pueden extraerse de cada una, junto con el valor que tiene considerar primero la perspectiva interna cuando los docentes están comprometidos con un continuo desarrollo profesional.

Rating the process attribute utilizing AHP in SPICE‐based process assessments

AbstractIn SPICE (Software Process Improvement and Capability dEtermination) assessments, the rating of a process attribute is determined by the aggregated achievements of associated MPs (management practices). Previous rating scale usability studies revealed that experienced assessors had difficulty distinguishing the boundaries between adjacent ratings: Fully and Largely, Largely and Partially, and Partially and Not achieved. To cope with these boundary problems, this study presents an AHP (analytic hierarchy process) approach as an objective aggregation method for the achievements of MPs. This research analyzed assessors' responses to our questions. Results indicated that the AHP approach was a systematic method and helpful in determining assessors' consensus in boundary problems. Furthermore, respondents answered that they would again use the AHP approach for tackling boundary problems for future assessments. In addition, they felt that the AHP approach could provide the assessors with confidence in the results of an assessment. While assessors responded that the AHP approach was easy to understand, some assessors had difficulty in creating an AHP comparison matrix in detail. Assessors with AHP experience expressed their support through such statements as ‘reduce the assessment effort’ and ‘provide confidence in the assessment results’. These results suggest the AHP approach could be employed as a method in solving the boundary problem in SPICE‐based process assessments. Copyright © 2001 John Wiley & Sons, Ltd.

Investigating Social Processes in Group Musical Composition

In recent years, the benefits of social interaction have been recognised in learning and working environments, the impact of social support on individual confidence and creativity being widely reported. In many domains including music, ability, personality, gender and friendship relationships have been identified as influencing the success or failure of group interactions. Despite the assumption in educational practice that group work can improve individuals' performance, however, social facilitation literature indicates the presence of others can sometimes be a hindrance (Hunt & Hillery, 1973). The main aim of the current paper is to investigate how various groupings of individuals (here, children) influenced creative musical tasks. In particular, friendship as opposed to random, single and both male and female sex groups were contrasted. It was hypothesised that quality of musical outcome would be influenced to some degree by the standard of social interaction and thus impact on individual sense of self within the task context. Fifty-nine children aged 11 participated in the study. All sessions were video-recorded. A researcher and class teachers observed the children's musical interactions using a check-list and a detailed structured questionnaire to guide observations whilst the children worked in groups of six participants in the following configurations: Randomly assigned, friendship based single and both sex, non-friendship based single and both sex. In each context, the children completed a compositional task, which was then assessed by independent, experienced assessors. The children were all individually questioned post-task about their experiences. The results suggested that social grouping did not influence the standard of the creative product, but it did affect the children's social interaction and thus their personal sense of achievement and enjoyment on task.

Derivation of Risk Management Criteria for Chemicals of Unknown Toxic Potency at Contaminated Sites

Environmental investigations of former industrial sites often detect the presence of chemicals for which no soil criteria exist and for which regulatory agencies have not derived estimates of toxic potency. This poses a considerable problem for making informed risk management decisions involving sites where such chemicals are present. As a result, a methodology has been developed for making risk-based decisions for chemicals of unknown toxic potency in soil at contaminated sites. The method is based on principles and procedures used by the US Food and Drug Administration (USFDA), the US Environmental Protection Agency (USEPA) and the Canadian Council of Ministers of the Environment (CCME). After analyzing the data on hundreds of carcinogenic and non-carcinogenic substances, the USFDA and other leading researchers have concluded that, if no toxicological data is available on a chemical, exposures less than 1.5 µg/person/day (i.e., 0.02 µg/kg body weight/day) are unlikely to result in appreciable health risks even if the substance was later found to be a carcinogen. To develop maximum soil concentrations that will be protective of human health (i.e., Risk Management Criteria or RMC), the above exposure limit of 0.02 µg/kg body weight/day has been assumed to be protective of risks from exposure to chemicals lacking toxicological data. Using a stochastic risk assessment model for estimating exposures to chemicals from contaminated sites, our analyses indicate that a soil concentration of 2 µg/g would be protective of human health for land uses that include residential, commercial, and industrial development provided no major indirect pathways exist at the site. If indirect pathways exists (e.g., vapor infiltration of soil gases, uptake of chemicals into garden produce, etc.), alternate RMC could be developed, that include such indirect pathways, using the methodology provided in this paper. Used by experienced risk assessors, the approach is a scientifically defensible screening method that will preclude many chemicals from unnecessary evaluation, while allowing risk assessors to focus efforts on chemicals of greater concern and make informed risk management decisions.

A tool for assessing perineal trauma.

The initial aim of this study was to develop a reliable visual tool to assist in the assessment of the severity of oedema and bruising in perineal trauma, using a categorical scale of 'none', 'mild', 'moderate' and 'severe'. A standardised set of photographs was selected by 10 clinically experienced midwives to represent these categories. The tool was tested in a clinical trial involving 77 women, recording and monitoring changes in the level of oedema and bruising during the first 48 hours following suturing of an episiotomy. The results demonstrated a statistically significant change for both oedema and bruising. However, the less experienced assessors reported some uncertainties in assessing a small proportion of women. A combined method, using this tool with a categorical scoring scale, was used to minimise difficulties in the assessment. Two pairs of midwives, one pair very experienced in assessing perineal trauma and the other less experienced, evaluated the combined method. The results showed a high level of agreement. The standard set of photographs and categorical scoring scale together resulted in a reliable and sensitive assessment tool to evaluate severity of perineal trauma.

Are replicate evaluations of triangle tests during a session good practice?

Results from four experiments in which replicated triangle tests were performed are reported. Two experiments were carried out with non experienced assessors and the two other ones with experienced assessors. Results from the non experienced assessors showed a slight performance increase from the beginning to the end of the session whereas results from the experienced assessors did not. This is explained by an attentional process allowing non experienced assessors to gradually focus their attention on the difference between the two tested products. In some specific occasions, replicate triangle tests might be a good methodological practice when the purpose of the test is to study the effect of some modifications on the product. Replicates increase the detection ability of the panel and the power of the test when the number of assessors is limited.

Ankylosing spondylitis antirheumatic drug trials: Effects of a standardized instructional viddeotape on the reliability of observer-dependent outcome measures

A study was designed to assess the effects of a standardized instruction videotape on reducing interobserver variability for several commonly used observer-dependent outcome measures. During a single day, six rheumatologists independently examined six patients with ankylosing spondylitis (AS) in a predetermined order using a Latin square design, before and after viewing a standardized videotape demonstrating 14 examination techniques. Reliability coefficients were calculated based on the variance components of the analysis of variance (ANOVA) table. Prestandardization reliability coefficients were <0.80 for three measures. Following standardization 12 reliability coefficients exceeded 0.80. For the majority of measures prestandardization reliability coefficients were high and no further improvement in reliability could be demonstrated. For one measure of cervical extension, but not another, an important and beneficial effect in reliability was noted. It was not possible to achieve adequate reliability in the performance of the chest excursion measurement. It is usually assumed that serial measurement in clinical trials should be performed by the same assessor because of concern regarding interobserver variability. However, the high levels of prestandardization interobserver reliability observed in this study indicate that for these variables serial measurements in a clinical trial could be made by different assessors, assuming they were equally skilled. This observation has important implications for outcome measurement in AS clinical trials. Although high levels of prestandardization reliability precluded the demonstration of any significant effect, we speculate that the videotape might be effective in training less experienced assessors. Nevertheless, an alternative approach to standardization may be required for the chest excursion measurement. Reductions in observer variability have the potential to diminish sample size requirements for AS antirheumatic drug studies. The use of a videotape to achieve this goal offers cost and convenience advantages over one-on-one training procedures, and this method should be further assessed in a group of less experienced assessors.

The Frankfort horizontal as a basis for cephalometric analysis

A random sample of 79 British 12-year-old children was studied from tracings registered in the natural head position (NHP). The angle between the Frankfort horizontal (FH) and the horizontal at right angles to the plumb line was measured. Two experienced assessors checked every tracing; for those children perceived to show unnatural head position this was adjusted to what they considered to be the natural head orientation (NHO). The NHO was defined as the head orientation of the subject perceived by the clinician, based on general experience, as the NHP in a standing, relaxed body and head posture, when the subject is looking at a distant point at eye level. The FHK horizontal angle was then measured, now related to the corrected head position. The standard deviation for the latter angle was smaller than that of the uncorrected, but still too large for the FH to be considered reliable as a basis for clinical cephalometric analysis. No statistically significant difference in variability as found between FH and the sella-nasion line. The extracranial horizontal plane related to NHO was recommended as the least variable of the references studied. (AM J ORTHOD DENTOFAC ORTHOP 1995;107:537-40.)

Assessment of corticosteroid-induced skin blanching: evaluation of the Minolta Chromameter CR200

The major criticism of the human bioassay currently employed for corticosteroid activity is that it requires the use of experienced assessors subjectively to determine and rank the degree of skin pallor (blanching) induced in the skin by the topical application of a formulated product. Recently a number of studies have suggested that the use of tri-stimulus colorimetry (using the Minolta Chromameter CR 200) provides an instrumental and hence objective means of assessing such topical blanching activity. The aim of the present study was to evaluate further the possibility of employing the Chromameter routinely in the bioassay of topical corticosteroids. In one study, employing ten volunteers, the effects of pressure of the measuring device on the skin and the influence of site of application were determined, prior to the application of any active agent. Blanching profiles were determined for clobetasol propionate (0.05%) and betamethasone 0.1% (as valerate) creams under occluded and unoccluded conditions (in a further two studies, each comprising 10 volunteers) using both visual assessment and instrumental measurement. It was established that: (1) application of pressure to the skin induces a measurable change in skin colour and must be avoided to prevent error in instrumental readings, (2) there is a variation in natural skin colour from elbow to wrist, (3) there is a marked but reproducible diurnal variation in skin colour, (4) providing the diurnal variation in skin colour is subtracted from the readings obtained as a result of corticosteroid-induced blanching, then a good correlation exists between the visual and Chromameter techniques both for very potent and potent corticosteroids. On the basis of these results, it is recommended that the use of the Chromameter in skin-blanching assays still requires careful development and more validation before its use can be recommended in a bioassay intended to replace the traditional visual assessment method.