Assessor Accuracy Research Articles

Scoring Running Records: Complexities and affordances

Running Records are thought to be an excellent formative assessment tool because they generate results that educators can use to make their teaching more responsive. Despite the technical nature of scoring Running Records and the kinds of important decisions that are attached to their analysis, few studies have investigated assessor accuracy. We measured precision across 114 teachers who were given a pre-coded Running Record to analyze by comparing their quantification and interpretation of the record against a scoring key. We also used Rasch measurement to examine which items teachers found easy and hard to score accurately. Analyses revealed wide variation in teachers’ accuracy, particularly in interpretation, that, according to item difficulty analysis, were caused by a few specific mistakes in scoring. Implications to improve training so that individuals administer Running Records more reliably are shared.

Craniocervical Posture Assessed With Photogrammetry and the Accuracy of Palpation Methods for Locating the Seventh Cervical Spinous Process: A Cross-sectional Study

ObjectiveThe purpose of this study was to compare craniocervical posture assessed by photogrammetry using 2 distinct palpation methods for locating the spinous process of the seventh cervical vertebra (C7SP). MethodsThis cross-sectional study was conducted in 2 phases. In phase I (n = 42), the assessor's accuracy in locating the C7SP using the flexion-extension and the modified thorax-rib static methods was compared to radiography. In phase II (n = 68), the craniocervical posture was analyzed with photogrammetry after palpation using the 2 methods. Neck pain intensity and disability were also determined. ResultsThe accuracy in locating the C7SP was higher using the modified thorax-rib static method (67%, 95% confidence interval [CI], 55-79) compared to the flexion-extension method (38%, 95% CI, 26-50, P = .016). Lower values of the craniocervical angle were obtained with the flexion-extension method than the modified thorax-rib static method (mean difference = −1.1°, 95% CI, −1.6 to −0.6, P < .001). However, both palpation methods resulted in similar classifications of participants as with or without forward head posture (P = .096). Weak correlations were observed between the craniocervical angle and neck pain intensity (ρ = −0.088 and −0.099, respectively) and disability (ρ = −0.231 and −0.249, respectively). ConclusionCraniocervical angles obtained using palpation methods with different accuracies were different, although the magnitude of the difference was insufficient to lead to different classifications of a forward head posture in adults with mild neck pain and disability. Craniocervical posture was weakly correlated with neck-pain intensity and disability.

Diagrammatic scale for assessing foliar symptoms of alternaria brown spot in citrus

Alternaria brown spot (ABS) is the main fungal disease of mandarins. Cause damage to fruits, branches and leaves. For determinations of disease severity in fruit, there is a specific diagrammatic scale, which does not occur for foliar lesions. In order to standardize leaf reviews of ABS in citrus, a diagrammatic scale was developed from the collection of symptomatic Dancy tangerine and Murcott tangor leaves, with ten levels, from zero to 97% of the area damaged by the pathogen. They were tested by five evaluators who analyzed 100 pictures of symptomatic leaves with and without the use of the proposed scale. To validate the scale, precision and accuracy of assessors in rating the images were compared. Results showed that the use of the scale improved the accuracy of the evaluators, with increased accuracy. This shows that the leaf scale can be used to determine the severity of alternaria brown spot in citrus.

Achieving assessor accuracy on the International Standards for Neurological Classification of Spinal Cord Injury.

A retrospective audit of assessor accuracy using the International Standards for Neurological Classification of Spinal Cord Injury (ISNCSCI) in three multicentre randomised controlled trials (SCIPA: Spinal Cord Injury and Physical Activity) spanning 2010-2014 with standards revised in 2011. To investigate assessor accuracy of neurological classification after spinal cord injury. Australia and New Zealand. ISNCSCI examinations were undertaken by trained clinicians prior to randomisation. Data were recorded manually and ISNCSCI worksheets circulated to panels, consensus reached and worksheets corrected. An audit team used a 2014 computerised ISNCSCI algorithm to check manual worksheets. A second audit team assessed whether the 2014 computerised algorithm accurately reflected pre- and post-2011 ISNCSCI standards. Of the 208 ISNCSCI worksheets, 24 were excluded. Of the remaining 184 worksheets, 47 (25.5%) were consistent with the 2014 computerised algorithm and 137 (74.5%) contained one or more errors. Errors were in motor (30.1%) or sensory (12.4%) levels, zone of partial preservation (24.0%), motor/sensory scoring (21.5%), ASIA Impairment Scale (AIS, 8.3%) and complete/incomplete classification (0.8%). Other difficulties included classification when anal contraction/sensation was omitted, incorrect neurological levels and violation of the 'motor follows sensory rule in non-testable myotomes' (7.4%). Panel errors comprised corrections that were incorrect or missed or incorrect changes to correct worksheets. Given inaccuracies in the manual ISNCSCI worksheets in this long-term clinical trial setting, continued training and a computerised algorithm are essential to ensure accurate scoring, scaling and classification of the ISNCSCI and confidence in clinical trials.

‘All the better for not seeing you’: Effects of communicative context on the speech of an individual with acquired communication difficulties

Evidence shows that speakers adjust their speech depending on the demands of the listener. However, it is unclear whether people with acquired communication disorders can and do make similar adaptations. This study investigated the impact of different conversational settings on the intelligibility of a speaker with acquired communication difficulties. Twenty-eight assessors listened to recordings of the speaker reading aloud 40 words and 32 sentences to a listener who was either face-to-face or unseen. The speaker's ability to convey information was measured by the accuracy of assessors’ orthographic transcriptions of the words and sentences. Assessors’ scores were significantly higher in the unseen condition for the single word task particularly if they had heard the face-to-face condition first. Scores for the sentence task were significantly higher in the second presentation regardless of the condition. The results from this study suggest that therapy conducted in situations where the client is not able to see their conversation partner may encourage them to perform at a higher level and increase the clarity of their speech.Learning outcomes: Readers will be able to describe: (1) the range of conversational adjustments made by speakers without communication difficulties; (2) differences between these tasks in offering contextual information to the listener; and (3) the potential for using challenging communicative situations to improve the performance of adults with communication disorders.

Risk-Benefit Analysis of Sampling Methods for Fine-Needle Aspiration Cytology: A Mathematical Modeling Approach

The effectiveness of fine-needle aspiration (FNA) increases with the number of needle passes, but needle passes are also associated with increased risk of adverse events. The trade-off between needle passes and adequacy has not been well characterized. Clinical studies are limited because of their inherent variability and limited sample size. We developed mathematical models to compare the performance of a variety of sampling protocols under a wide range of conditions. Specifically, we compared the performance of sampling methods using a fixed number of needle passes with sampling methods using a rapid onsite evaluation (ROSE) with a variable number of needle passes. Variable sampling with ROSE generally required fewer needle passes than fixed sample size policies to achieve a desired adequacy rate. Variable sampling policies using ROSE achieve greater per-case adequacy with fewer needle passes than sampling policies using a fixed number of passes if assessor accuracy is high.

Classification of Diapause Status by Color Phenotype inLygus hesperus

Recent studies on adult diapause in the western tarnished plant bug, Lygus hesperus Knight (Heteroptera: Miridae), have highlighted the need to identify a reliable external marker for the internal changes that differentiate a normal animal from one that is overwintering. To test the efficacy of a color based discrimination system, L. hesperus of both genders were reared from eggs through day 10 of adulthood under a 10 hour photophase at a constant temperature. They were separated into three color groups (yellow, pale green, dark green), then dissected for diapause categorization based on internal development. Most yellow individuals were in diapause, dark green individuals were not, and pale green ones were mixed. A group of 25 assessors, naïve with regard to L. hesperus development, were then asked to use a simplified color criteria (yellow = diapause, green = non-diapause) to estimate the status of a mixture of diapausing and non-diapausing adults of both genders aged two to seven days post-eclosion. After dissection to verify diapause status, assessor accuracy was found to be ineffective for assessing adults of both sexes younger than four days because color differences, which increased with age, were subtle or non-existent at this stage. For four to seven-day-old bugs, 84% of females and 67% of males were correctly categorized, on average. Incorrect assessments in all but the youngest males over-identified diapause, but for females there was no trend in miscategorizations. Overall, the results indicate that diapause status can be adequately discriminated by color assessment, and with greatest accuracy when sampling older females. However, factors other than photoperiod appear to also influence coloration.

Observed consultation: confidence and accuracy of assessors

Judgments made by the assessors observing consultations are widely used in the assessment of medical students. The aim of this research was to study judgment accuracy and confidence and the relationship between these. Assessors watched recordings of consultations, scoring the students on: a checklist of items; attributes of consultation; a passmark scale and lastly their confidence in this last judgment. Then they were interviewed using stimulated recall to explain their rationale for scoring and confidence. Twenty-three staff assessors watched two consultations. Assessor confidence and accuracy were least for those student performances perceived near the passmark standard. The difference between confidence and accuracy, over-confidence, was greatest at this level, although at the extremes under-confidence was found. In the interviews the assessors were aware of a variety of factors that they perceived affected their scoring and confidence. As in other contexts confidence and accuracy vary and over-confidence increases the more difficult the judgments. However, this study also demonstrated under-confidence for less difficult judgments. The perception that more information would help, may improve confidence but not necessarily accuracy, so increasing over-confidence.

Patterns of Performance and Rating Frequency: Influence on the Assessment of Performance

A laboratory simulation was conducted in order to examine the influence of assessees' pattern of performance (ascending, consistently average, descending) on: (a) assessor ratings of assessee performance, (b) assessor attributionsfor assessee performance and, (c) rating accuracy. The influence of rating frequency (i.e., rating at the end of each exercise versus rating after all exercises have been concluded) on assessor accuracy was also examined. As predicted, assessees who exhibit different patterns of performance receive different ratings even though their overall level of performance is similar. More specifically, the ascending performer was rated higher in terms of leadership, decision-making, and persuasiveness, than either the consistently average or descending performer. It was alsofound that descending and ascending performance was attributed more to motivation causes than consistently average performance, and that descending performance was attributed more to ability causes (i.e., a lack of ability) than either the ascending or consistently average performer. With respect to rating accuracy, it was found that descending and ascending performance was rated more accurately than consistently average performance and that subjects who made ratings at the conclusion of each exercise did not make more accurate overall assessment ratings than those subjects who made overall assessment ratings only. Instead, it was found that the accuracy of overall assessment ratings was greatest when ratings collected after each exercise were averaged. The implications of these results for assessment center procedures is discussed.

Number of assessment center dimensions as a determinant of assessor accuracy.

Number of assessment center dimensions as a determinant of assessor accuracy.

Number of assessment center dimensions as a determinant of assessor accuracy.

Number of assessment center dimensions as a determinant of assessor accuracy.

SOME TESTS OF BINOMIAL PROBABILITY ASSESSMENTS

AbstractA replication of the Alpert‐Raiffa probability assessment experiment is used to motivate the development of several tests for evaluating assessors. The tests permit an evaluation of an assessor's accuracy in estimating target quantities and in estimating his own uncertainty as well.Assumption of normal subjective distributions permits the determination of subjective variances. Data from trial assessments are employed to test models which assert that assessed quantities are equal to subjectively assessed error variances. Several tests of the latter type are compared using data from individual, as well as pooled, assessors.