Non-expert Raters Research Articles

Assessing students’ communication skills: validation of a global rating

Communication skills training is an accepted part of undergraduate medical programs nowadays. In addition to learning experiences its importance should be emphasised by performance-based assessment. As detailed checklists have been shown to be not well suited for the assessment of communication skills for different reasons, this study aimed to validate a global rating scale. A Canadian instrument was translated to German and adapted to assess students' communication skills during an end-of-semester-OSCE. Subjects were second and third year medical students at the reformed track of the Charité-Universitaetsmedizin Berlin. Different groups of raters were trained to assess students' communication skills using the global rating scale. Validity testing included concurrent validity and construct validity: Judgements of different groups of raters were compared to expert ratings as a defined gold standard. Furthermore, the amount of agreement between scores obtained with this global rating scale and a different instrument for assessing communication skills was determined. Results show that communication skills can be validly assessed by trained non-expert raters as well as standardised patients using this instrument.

Do clinical clerks provide candidates with adequate formative assessment during Objective Structured Clinical Examinations?

Various research studies have examined the question of whether expert or non-expert raters, faculty or students, evaluators or standardized patients, give more reliable and valid summative assessments of performance on Objective Structured Clinical Examinations (OSCEs). Less studied has been the question of whether or not non-faculty raters can provide formative feedback that allows students to take advantage of the educational opportunity that OSCEs provide. This question is becoming increasingly important, however, as the strain on faculty resources increases. A questionnaire was developed to assess the quality of feedback that medical examiners provide during OSCEs. It was pilot tested for reliability using video recordings of OSCE performances. The questionnaires were then used to evaluate the feedback given during an actual OSCE in which clinical clerks, residents, and faculty were used as examiners on two randomly selected test stations. The inter-rater reliability of the 19-item feedback questionnaire was 0.69 during the pilot test. The internal consistency was found to be 0.90 during pilot testing and 0.95 in the real OSCE. Using this form, the feedback ratings assigned to clinical clerks were significantly greater than those assigned to faculty evaluators. Furthermore, performance on the same OSCE stations eight months later was not impaired by having been evaluated by student examiners. While evidence of mark inflation within the clinical clerk examiners should be addressed with examiner training, the current results suggest that clerks are capable of giving adequate formative feedback to more junior colleagues.

Evaluating home environments of persons with dementia: interrater reliability and validity of the Home Environmental Assessment Protocol (HEAP)

Purpose : This article reports on an interrater reliability and preliminary validity study of an investigator-developed tool, the Home Environmental Assessment Protocol (HEAP) for use in homes of persons with dementia. Method : The HEAP consists of 192 items that are summed into separate indices representing the number of hazards, adaptations, and level of clutter and comfort in eight areas of the home. Interrater reliability was examined among four raters, two environmental experts and two non-experts, who observed 22 dementia households. Results : The Kappa statistic was used to evaluate agreement level for each measured item and found that agreement ranged from slight to almost perfect. Intraclass correlations (ICCs), were used to evaluate agreement level for indices. The hazard index in each room ranged from fair (0.36) to moderate (0.66) for all raters. For the adaptation, clutter and comfort indices in each room, ICCs ranged from 0.51 to 0.90 for all raters. Agreement level between expert and non-expert raters differed minimally for all indices. Adaptations to dining rooms (r = -0.080, p = 0.001), kitchens (r = -0.52, p = 0.02) and bedrooms (r = -0.76, p = 0.001) were associated with patient deficits such that more adaptations were made in homes of dependent persons. Low Mini-Mental Status Examination scores were associated with fewer hazards, more adaptations, and less clutter. Conclusion : Findings show that both experts and non-expert raters use the HEAP consistently. Also, measured attributes are related to cognitive and functional status in the expected direction.

Validation of a Typology of Religious Experience and its Relationship to the Psychotic Experience

A typology of religious experience was validated and an empirical determination of the similarities between the religious and psychotic experiences was made. Expert and non-expert raters (n=16) were asked to differentiate religious experiences (n=12), fabricated religious experiences (n=12), and psychotic experiences (n=12) using the typology of religious experience. A split plot, repeated measures analysis of variance yielded three significant findings: (a) Expert raters performed significantly better than non-expert raters at identifying the three types of experiences. (b) Non-expert raters with the typology performed as well as expert raters and significantly better than non-expert raters without the typology. (c) Psychotic experiences were more easily identified than were religious or fabricated experiences.