Multiple Ratings Research Articles

Sleep deprivation response in seasonal affective disorder during a 40-h constant routine

Background: There are no controlled studies investigating the response of patients with seasonal affective disorder (SAD) to a total sleep deprivation (SD). Methods: The clinical response to SD of patients with SAD in winter was investigated under the stringently controlled conditions of a 40-h constant routine protocol. Results: 52% of the SAD patients ( N=11 women) improved, using a mean of a multiple ratings. This is in the range of response found for non-seasonal major depression. In contrast, controls ( N=8 women) showed less improvement of mood (29%). Conclusion: SAD patients respond to SD as do non-seasonal major depressives. The best discrimination of response was obtained in an observer rating (Clinical Global Impression: global severity improvement), and the morning values of two different self ratings (v. Zerssen depression scale, 100 mm VAS with the criterion of ≥10 mm improvement). Limitation: A more reliable estimate of the SD response rate in SAD patients would require a larger group. Clinical Relevance: SAD patients do not differ from other subgroups of major depression in their response to SD, and therefore this is an additional treatment option to light therapy.

Detecting Gender Bias in Peer Marking of Students’ Group Process Work

Concerns relating to the reliability of teacher and student peer assessments are discussed, and some correlational analyses comparing student and teacher marks described. The benefits of the use of multiple ratings are elaborated. The distinction between gender differences and gender bias is drawn, and some studies which have reported gender bias are reviewed. The issue of ‘blind marking’ is addressed. A technique for detecting gender bias in cases where student raters have awarded marks to same and opposite sex peers is described, and illustrated by data from two case studies. Effect sizes were found to be very small, indicating an absence of gender bias in these two cases. Results are discussed in relation to task and other contextual variables. The authors conclude that the technique described can contribute to the good practice necessary to ensure the success of peer assessment in terms of pedagogical benefits and reliable and fair marking outcomes.

A new method for assessing interexaminer agreement when multiple ratings are made on a single subject: applications to the assessment of neuropsychiatric symptomatology

A new method is introduced for assessing levels of interexaminer agreement when multiple ratings are made on a single subject, with an application in psychiatric research. It is designed to provide an overall level of interexaminer agreement and separate indices of agreement for each examiner. These indices are based on biostatistical and clinical criteria to determine whether the ratings of any given examiner are appreciably higher or lower than the group average, or a consensus diagnosis. A number of examples, from ongoing psychiatric research, are provided to illustrate conditions favoring the application of the new methodology. Finally, the necessary software for performing the analyses is available to clinical investigators with interest in this area of assessment. © 1997 Elsevier Science Ireland Ltd.

A computer program for assessing interexaminer agreement when multiple ratings are made on a single subject

This report describes a computer program for applying a new statistical method for determining levels of agreement, or reliability, when multiple examiners evaluate a single subject. The statistics that are performed include the following: an overall level of agreement, expressed as a percentage, that takes into account all possible levels of partial agreement; the same statistical approach for deriving a separate level of agreement of every examiner with every other examiner; and tests of the extent to which a giver examiner's rating (say a symptom score of three on a five-category ordinal rating scale) deviates from the group or overall average rating. These deviation scores are interpreted as standard Z statistics. Finally, both statistical and clinical criteria are provided to evaluate levels of interexaminer agreement. © 1997 Elsevier Science Ireland Ltd.

Assessment and evaluation of contractor data against client goals using PERT approach

A methodology for assessing and evaluating contractor data for the purpose of prequalification and bid evaluation is presented. The PERT approach is used to develop a linear model for the assessment of contractor data. The model incorporates a multiple ratings permitting the uncertainty in contractor data to be evaluated. An empirical study investigating the importance of different contractor criteria is described. A lexicographical ordering with aspiration levels and risk analysis with sensitivity methods are used to evaluate and select or rank-order contractors against the main client goals of time, cost, and quality. A literature review is reported regarding client goals and current evaluation strategies. The assumptions, advantages and disadvantages of this work as well as an example are also presented.

The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational I settings

The increasing demand for performance assessment of speaking skills in second languages has led to logistic complications, for example, the delivery of tests in overseas locations. One solution to the problem has been to train native speaker interlocutors to carry out a series of oral interactions with the candidate, with assessment from audiorecordings of the test session postponed and conducted cen trally by a small team of trained raters. This technique is currently used in two large-scale occupationally related ESP tests administered internationally on behalf of the Australian government. But these procedures raise questions about the effect of such facets of the assessment situation as interlocutor variables and the quality of the audiotape recording. Recent developments in multifaceted Rasch measure ment have significantly broadened the possibilities for investigation of these issues. The research presented in this article investigates potential problems associated with the above approach to the offshore testing of speaking skills. Data from audiotape-based assessments of 70 offshore candidates from two administrations of the Occupational English Test, an advanced-level ESP test for health pro fessionals, are considered. In addition to multiple ratings of candidate perform ance, each recording is rated for perceptions of the competence of the interlocutor, the rapport established between the candidate and the interlocutor, and the audi bility of the interaction. These aspects of the assessment situation are treated as facets in a series of multifaceted Rasch analyses of the data. The results of the analysis reveal the effects of interlocutor variability and audio tape quality on ratings. The article concludes with an evaluation of the overall feasibility of the procedure, and implications for test administration are considered. The study is also a further demonstration of the application of multifaceted Rasch measurement in performance assessment settings.

The validity of personality, service orientation, and reading comprehension measures as predictors of Flight Attendant training performance

The present study examined the predictive power of Big Five personality dimensions, a situational measure of service orientation and a reading comprehension measure as predictors of Flight Attendant training criteria. The criteria consisted of multiple ratings made by trainers of trainees across a number of training dimensions. At least partial data was collected on 424 Flight Attendant trainees. Results indicated significant correlations between all three predictors and a number of the training criteria.

Time For a Review of Admission to Medical School?

Appropriate selection of medical students is a fundamental prerequisite if medical schools are to produce competent and caring doctors. The selection criteria for entry to the medical degree course at the University of Newcastle, New South Wales, are unique in Australia. The purpose of this study was to identify admission criteria that may predict performance in the first postgraduate (intern) year. Performance ratings were obtained from the clinical supervisors of two graduating classes of Newcastle University medical students during their five terms in internship (first postgraduate year). At least one rating was obtained for 93% of interns. A subset analysis of interns with multiple ratings (57%) showed that combining previous study in both humanities and science before medical school entry was predictive of higher intern performance ratings. These interns were rated more favourably than those who had studied science alone. Moreover, students who had earlier studied both humanities and science were twice as likely to complete their medical degree as those who had studied science alone. Age, gender, admission interview results, written psychometric test scores, academic marks, and whether previous tertiary study had been undertaken prior to medical school entry were not predictive of intern performance ratings. Subject spread, including a background in humanities, is important for effective medical practice, at least in the immediate postgraduate period. Perhaps it is time to evaluate the admission criteria by which medical students are selected.

Interobserver variability on the histopathologic diagnosis of cutaneous melanoma and other pigmented skin lesions.

To assess the interobserver agreement on the diagnosis and classification of cutaneous melanoma. A set of 140 slides of cutaneous melanoma, including a small subset of benign pigmented skin lesions, were circulated to four experienced histopathologists. The kappa statistic for multiple ratings per subject was calculated using the method described by Fleiss. The kappa value on the diagnosis of cutaneous melanoma versus benign lesions was 0.61. There was some discordance on the diagnosis in 37 of 140 cases (26%). For the histopathologic classification of cutaneous melanoma, the highest kappa values were attained for Breslow thickness (kappa = 0.76) and presence of ulceration (kappa = 0.87). The agreement was generally poor for other histologic features, such as level of dermal invasion (kappa = 0.38), presence of regression (kappa = 0.27), and lymphocytic infiltration (kappa = 0.27). Our study suggests considerable disagreement among pathologists on the diagnosis of melanoma versus other pigmented lesions. Tumor thickness and presence of ulceration are the most reproducible histologic features of cutaneous melanoma.

Reasoning and the weighting of attributes in attitude judgments.

Two experiments examined processes by which analyzing reasons may influence attitude judgments. Participants made multiple liking judgments on sets of stimuli that varied along 6 a priori dimensions. In Study 1, the stimulus set consisted of 64 cartoon faces with 6 binary-valued attributes ( e.g., a straight vs. a crooked nose). In Study 2, the stimuli were 60 digitized photographs from a college yearbook that varied along 6 dimensions uncovered through multidimensional scaling. In each experiment, half of the participants were instructed to think about the reasons why they liked each face before making their liking rating. Participants' multiple liking ratings were then regressed on the dimension values to determine how they weighted each dimension in their liking judgments. The results support a process whereby reasoning leads to increased variability and inconsistency in the weighting of stimulus information. Wilson's model of the disruptive effects of reasoning on attitude judgments (e.g., T. D. Wilson, D. S. Dunn, D. Kraft, & D. J. Lisle, 1989) is discussed.

The Importance of Task in Evaluating Positive Marital Interactions

This article illustrates the importance of task context for evaluating the reliability and validity of observer assessments of warm/supportive marital interactions. Multiple observer ratings of 424 families were obtained within and across two observational task situations using the Iowa Family Interaction Rating Scales (Melby et al., 1990). The psychometric properties of observer ratings and family member reports were assessed simultaneously through the use of structural equation modeling. In general, the findings support the reliability of the global assessments of warm/supportive marital behaviors. As hypothesized, a marital discussion task elicited significantly higher levels of spousal warmth than the problem-solving task. Validity, however, varied across the two interactional contexts. These inconsistencies underscore the importance of task context, both for eliciting warm/support behaviors and for evaluating the validity of observational measures of such behaviors. Although it seems intuitively appealing that warmth, support, and demonstrations of positive affect in marital interaction should promote couple satisfaction and harmony, in general the evidence for such an association is inconsistent and far less robust than support for the detrimental influence of hostile or negative marital interactions (Markman & Notarius, 1987). There is a general trend for observer ratings of marital warmth and support, when compared with measures of other observed behaviors (e.g., hostile interactions), to show lower reliability and validity (Hooley & Hahlweg, 1989; Markman, 1991). In this study, we test the hypothesis that the relatively weak association found in earlier studies between positive marital interaction and marital quality, as well as the lower reliability and validity obtained for measures of positive marital interaction, derive in large part from difficulties in adequately measuring these dimensions of spousal behavior. Numerous family researchers have noted that it is not uncommon to find cross-method correspondence among measures from various sources (e.g., among observer-, self-, and spouse-report measures) (Furman, Jones, Buhrmester, & Adler, 1989; Jacob, Tennenbaum, & Krahn, 1987; Kenny, 1991; Moskowitz, 1990). Although a variety of factors may influence degree of correspondence among such measures of spousal behaviors (Howe & Reiss, 1993; Markman, 1990), in this article we specifically examine the effects of the contexts or trigger situations used for the obtained ratings. Given the phenomenal growth in the use of observational methodology (Bennion, 1993) and the increasing recognition of the importance of using observational measures for assessing family processes (Baucom & Sayers, 1989; Hetherington, 1994), assessing the adequacy of various report sources is of particular significance. It is essential, however, that evaluations regarding insider (e.g., self or spouse) and outsider (e.g., trained observer) assessments of behaviors such as warmth and support in marriage be made using procedures that allow direct comparisons among measures obtained by different methods or from different sources. Reliability and validity of such measures should be considered simultaneously in order to ensure their adequacy (Zeller & Carmines, 1980). THE PRESENT STUDY The purpose of this study is to evaluate the reliability and validity of global observer ratings of positive, warm, and supportive marital interactions using the Iowa Family Interaction Rating Scales (Melby et al., 1990). Prior analyses of observational ratings of hostility and coercion obtained using this coding system have demonstrated (a) the reliability and validity of such observational ratings obtained from a marital interaction task (Melby, Conger, Ge, & Warner, 1995) and (b) the ability of either a family problem-solving task or a marital discussion to yield acceptable observational assessments of such behaviors (Melby, Conger, Ge, & Warner, 1994). …

A meta-analysis of interrater and internal consistency reliability of selection interviews.

A meta-analysis of 111 interrater reliability coefficients and 49 coefficient alphas from selection interviews was conducted. Moderators of interrater reliability included study design, interviewer training, and 3 dimensions of interview structure ( standardization of questions, of response evaluation, and of combining multiple ratings). Interactions showed that standardizing questions had a stronger moderating effect on reliability when coefficients were from separate ( rather than panel) interviews, and multiple ratings were useful when combined mechanically (there was no evidence of usefulness when combined subjectively). Average correlations (derived from alphas) between ratings were moderated by standardization of questions and number of ratings made. Upper limits of validity were estimated to be.67 for highly structured interviews and.34 for unstructured interviews.

Task Involvement and Satisficing with Motivational Scales

Three measures related to motivation to perform a rating task accurately were obtained from 49 students, who also performed goal-attribute ratings. Less motivated students surpassed more motivated ones on several indices of rating bias. Previous evidence for differential relationships among goal attributes was clearly replicated only with the ratings of more motivated subjects. These findings suggest that many respondents may strive for “satisfactory” rather than optimal goal ratings (“satisficing”), that this tendency is a strong potential source of error with such data, but that individual differences related to satisficing may be used to examine such effects and partially bracket them out. It is suggested that measures of subjects' motivation or other related personality characteristics be routinely gathered when multiple ratings from subjects are required.

Caffeine and time of day effects on a force discrimination task in humans

The effects of caffeine (0 mg/kg, 1 mg/kg, 3 mg/kg) and time of day (TOD) on human performance were studied using a multiple forceband discrimination task (MFDT) and subjective ratings. Selfrated measures of energy level were affected by TOD and caffeine, while mood was affected by TOD. Energy level decreased throughout the day and was offset by caffeine which increased energy level independent of TOD. Self-reported anxiety was not affected by TOD, or caffeine. Mood was affected by TOD in a complex cubic trend with late morning and late evening peaks 12 h apart. MFDT performance was affected by TOD, caffeine dosage, and their interaction. Trend analyses showed varying patterns of TOD effects across peak force variability, response latency, response duration, and correct responding. Results support and extend previous findings of TOD influences on the MFDT and support the utility of multicomponent proprioceptive tasks for examining drug effects on performance.

Test of a Latent Structure of Performance Ratings Derived from Wherry’s (1952) Theory of Rating

A model of the latent structure of performance ratings based on Wherry's (19.52) theory of rating was developed which sought to explain relationships between multiple ratings, each of multiple performance dimensions, obtained from multiple rating sources. The model was tested using data obtained from first-and second-level supervisors who rated the performance of approximately 700 entry level clerical and professional/technical employees of 96 insurance, financial, and service organizations. Results only partially confirmed the a priori model, supporting specifications of dimensional true performance and overall rater bias effects, but failing to support specifications of areal (dimensional) bias effects on ratings.

Test of a latent structure of performance ratings derived from Wherry's (1952) theory of rating

A model of the latent structure of performance ratings based on Wherry's (1952) theory of rating was developed which sought to explain relationships between multiple ratings, each of multiple performance dimensions, obtained from multiple rating sources. The model was tested using data obtained from first- and second-level supervisors who rated the performance of approximately 700 entry level clerical and professional/technical employees of 96 insurance, financial, and service organizations. Results only partially confirmed the a priori model, supporting specifications of dimensional true performance and overall rater bias effects, but failing to support specifications of areal (dimensional) bias effects on ratings.

A Latent Trait Finite Mixture Model for the Analysis of Rating Agreement

This article presents a latent distribution model for the analysis of agreement on dichotomous or ordered category ratings. The model includes parameters that characterize bias, category definitions, and measurement error for each rater or test. Parameter estimates can be used to evaluate rater performance and to improve classification or measurement with use of multiple ratings. A simple maximum likelihood estimation procedure is described. Two examples illustrate the approach. Although considered in the context of analyzing rater agreement, the model provides a general approach for mixture analysis using two or more ordered-caregory measures.

Higher-Order Kappa-Type Statistics for a Dichotomous Attribute in Multiple Ratings

We introduce a sequence of parameters called canonical moments to characterize the predictive value of the pattern of consensus among (kappa-1) ratings to an additional rating. The usual kappa coefficient of agreement among ratings (two or more) is just the second canonical moment. By using the higher canonical moments, we can evaluate the consensus of multiple binary ratings.

Perceptions or reality: Is multi‐perspective measurement a means or an end?

AbstractThis article takes an epistemological perspective on “perceptions or reality” when examining multi‐rater assessment measures as means versus ends and contrasts the scientist's and practitioner's perspectives. Multiple ratings—when used to map differences as well as similarities in perceptions about common reference points—can be powerful tools for individual or organizational development applications. The application of measurement tools for development purposes is quite different from the psychometric focus of measurement, however. In the latter, measurement becomes an end in itself; in the former, it is more a means to an end. © 1993 by John Wiley & Sons, Inc.

A Neural Network That Learns to Interpret Myocardial Planar Thallium Scintigrams

The planar thallium-201 (201Tl) myocardial perfusion scintigram is a widely used diagnostic technique for detecting and estimating the risk of coronary artery disease. Interpretation is currently based on visual scoring of myocardial defects combined with image quantitation and is known to have a significant subjective component. Neural networks learned to interpret thallium scintigrams as determined by both individual and multiple (consensus) expert ratings. Four different types of networks were explored: single-layer, two-layer backpropagation (BP), BP with weight smoothing, and two-layer radial basis function (RBF). The RBF network was found to yield the best performance (94.8% generalization by region) and compares favorably with human experts. We conclude that this network is a valuable clinical tool that can be used as a reference "diagnostic support system" to help reduce inter- and intraobserver variability. This system is now being further developed to include other variables that are expected to improve the final clinical diagnosis.