Satisfactory Inter-rater Agreement Research Articles

Exploring the Dynamic Relationship between Pushing Behavior and Crowd Dynamics

Crowds, subjects of considerable complexity, have been extensively studied both as homogeneous entities and as collective sums of individual movements in various studies. However, crowd models, being grounded in physics, are limited in terms of incorporating psychological perspectives on individual behavior. Building upon the premise that crowd behavior is heterogeneous and dynamic, particularly in bottleneck scenarios, this study aims to explore the nuances of forward motion. Adopting the category system proposed by Lügering et al. (2022) (consisting the following categories: strong pushing, mild pushing, just walking, falling behind), this paper investigates the circumstances and locations where pushing or non-pushing behaviors arise, intensify, or cease within crowds approaching bottlenecks. The study utilized 14 video materials obtained from previous laboratory pedestrian experiments to examine the spatial characteristics of forward motion and pushing behavior in relation to corridor widths and varied motivational instructions. Two trained raters independently annotated these videos, achieving satisfactory inter-rater agreement (KALPHA = .65) , and a joint dataset was then created for each video. These videos consisted both high (7 videos) and low (7 videos) motivation scenarios. The importance of corridor width was also considered: four videos featured a 5.6m width, another four featured a 4.5m width, and the remaining videos displayed widths of 3.4m, 2.3m, and 1.2m twice. Our findings suggest a tendency for increased pushing behavior or an increase in the categories as individuals approach the bottleneck, regardless of the width of the corridor or the motivational instruction. Furthermore, non-pushing behaviors were predominantly observed in the areas farther away from the bottleneck. A noticeable trend was observed in high motivation scenarios, which generally exhibited more instances of pushing behavior. The effect of corridor width indicated that, in certain cases, pedestrians who push in wider corridors experience faster access to the bottleneck. However, this effect is less significant in narrower widths.

Harmonization of sensorimotor deficit assessment in a registered multicentre pre-clinical randomized controlled trial using two models of ischemic stroke

Multicentre preclinical randomized controlled trials (pRCTs) are a valuable tool to improve experimental stroke research, but are challenging and therefore underused. A common challenge regards the standardization of procedures across centres. We here present the harmonization phase for the quantification of sensorimotor deficits by composite neuroscore, which was the primary outcome of two multicentre pRCTs assessing remote ischemic conditioning in rodent models of ischemic stroke. Ischemic stroke was induced by middle cerebral artery occlusion for 30, 45 or 60 min in mice and 50, 75 or 100 min in rats, allowing sufficient variability. Eleven animals per species were video recorded during neurobehavioural tasks and evaluated with neuroscore by eight independent raters, remotely and blindly. We aimed at reaching an intraclass correlation coefficient (ICC) ≥0.60 as satisfactory interrater agreement. After a first remote training we obtained ICC = 0.50 for mice and ICC = 0.49 for rats. Errors were identified in animal handling and test execution. After a second remote training, we reached the target interrater agreement for mice (ICC = 0.64) and rats (ICC = 0.69). In conclusion, a multi-step, online harmonization phase proved to be feasible, easy to implement and highly effective to align each centre’s behavioral evaluations before project’s interventional phase.

Interrater reliability for the detection of cortical lesions on phase-sensitive inversion recovery magnetic resonance imaging in patients with multiple sclerosis.

To assess the reliability of phase-sensitive inversion recovery (PSIR) magnetic resonance imaging (MRI) and its accuracy for determining the topography of demyelinating cortical lesions in patients with multiple sclerosis (MS). This was a cross-sectional study conducted at a tertiary referral center for MS and other demyelinating disorders. We assessed the agreement among three raters for the detection and topographic classification of cortical lesions on fluid-attenuated inversion recovery (FLAIR) and PSIR sequences in patients with MS. We recruited 71 patients with MS. The PSIR sequences detected 50% more lesions than did the FLAIR sequences. For detecting cortical lesions, the level of interrater agreement was satisfactory, with a mean free-response kappa (κFR) coefficient of 0.60, whereas the mean κFR for the topographic reclassification of the lesions was 0.57. On PSIR sequences, the raters reclassified 366 lesions (20% of the lesions detected on FLAIR sequences), with excellent interrater agreement. There was a significant correlation between the total number of lesions detected on PSIR sequences and the Expanded Disability Status Scale score (ρ = 0.35; p < 0.001). It seems that PSIR sequences perform better than do FLAIR sequences, with clinically satisfactory interrater agreement, for the detection and topographic classification of cortical lesions. In our sample of patients with MS, the PSIR MRI findings were significantly associated with the disability status, which could influence decisions regarding the treatment of such patients.

Interactions with Standardized Patients to Evaluate Students’ Psychotherapy-Competencies

Abstract. The use of standardized patients (SPs) in the training of prospective practitioners is a well-established didactic tool in medical schools. Only recently have simulations of patients in psychotherapy been introduced into the training of psychologists. By integrating psychotherapy training into university-level master’s programs, German law now requires licensing exams for psychotherapists (i. e., Approbationsprüfung) to include an assessment of therapeutic competencies in simulated interactions with SPs. Yet, it has not been examined whether these simulations are useful for a reliable assessment of competencies in psychotherapy trainees. Also, we need to develop standardized instruments to evaluate competencies in entry-level psychotherapists. As part of a university course, we trained master’s-level students from three cohorts in clinical interviewing techniques (course title: Klinisch-psychologische Gesprächsführung). We analyzed videotaped 20-min sequences of N = 104 students while they interviewed one of N = 38 trained SPs. The students’ task was to interview the SP, conduct a brief case history, and use the interviewing skills they had learned in class. Two independent raters evaluated their psychotherapeutic competencies with an adapted version of the German Cognitive Therapy Scale (CTS). Raters evaluated students’ performance on two subscales and the total score with satisfactory interrater agreement (intraclass correlations). In general, students performed well in the interviews: They structured the sessions sufficiently, and their global psychotherapeutic competencies were satisfactory. However, the psychotherapeutic competencies of master’s students fell short of the benchmark derived from experienced psychotherapists. This pilot study provides first evidence that simulated interviews with SPs may be a reliable tool in the assessment of practical competencies in psychotherapy trainees at an early stage of their training. Moreover, we found that the CTS, which has demonstrated validity to quantify competencies of psychotherapists, is applicable and reliable in this training context as well. In sum, this suggests that simulated interviews with SPs may be useful for evaluating psychotherapeutic competencies of psychotherapy trainees.

Pilot Validation Study on the Tavistock Relationships Observational Scale for the Assessment of the Quality of Dyadic Relationships in Couples in Which One Partner Has Dementia.

Many people with dementia are cared for by their spouses. Poor quality of the dyadic relationship is a known risk factor for carer burden and mistreatment of the person with dementia. This study aimed to design and cross validate the Tavistock Relationships Observational Scale (TROS) for assessment of the quality of the relationship in such couples. The study was conducted within the context of the therapeutic intervention with n = 11 couples. Couples' video-recorded interactions were assessed independently by three psychologists using the 13-item TROS. Cronbach alphas of the scale computed for the three raters were in the range 0.76-0.88 (on average 0.83), thus demonstrating high internal consistency of the scale. Fleiss Kappa coefficient for the three rates computed for 22 assessed videos ranged between 0.45 (moderate agreement) and 0.85 (almost perfect agreement) with 50% of coefficients being above 0.50, suggesting the good performance of the TROS in terms of yielding a satisfactory inter-rater agreement. The TROS also showed good external validity when using as validation criteria caregiving partner's burden (measured with the Zarid Burden Interview: the scales Social burden [Rho = -0.65, p = 0.021] and Role strain [Rho = -0.68, p = 0.014]), depression of the person with dementia (measured with the Cornell Scale for Depression in Dementia: Rho = -0.78, p = 0.020), and relationship satisfaction of the caregiving partner (measured with the dyadic adjustment scale: Rho = 0.52, p = 0.038). The TROS also showed a good capacity to detect the change in scores from time 1 to time 2; moreover, power analysis showed that the TROS can detect the change in scores even in very small samples (n = 5 couples). The TROS may be a useful tool for a standardized assessment of the quality of dyadic relationships in couples with dementia in research, nursing, or clinical settings to assist clinical psychologist, nurses, or social workers in a preliminary evaluation of the situation of couples with dementia, to choose a suitable care or intervention, to assess the effectiveness of interventions focusing on improvement of dyadic relationships, or to assess the risk of spousal abuse or neglect of a person with dementia.

Inter-rater reliability of four validated diabetic foot ulcer classification systems

BackgroundThe need to classify, score and describe diabetic foot ulcerations is needed for accurate clinical documentation. An agreed wound classification can improve communication between health care professionals, and improve ease of discussion regarding potential outcomes. This study aimed to determine the inter-rater reliability between four validated wound classification systems. Research design and methodA prospective non-experimental comparative study was conducted. Forty patients with diabetes presenting with a new or long-standing ulceration were recruited. Each ulcer was graded/scored by three clinicians using four widely used classification systems; namely the University of Texas, SINBAD, Meggitt-Wagner and the PEDIS scoring systems. ResultsAll classifications had a satisfactory inter-rater agreement when evaluated by three raters of various clinical experience, however, the strength of the agreement varied between classifications. The Meggitt-Wagner system had an almost perfect agreement, the SINBAD and UT systems had a strong inter-rater agreement whilst the PEDIS had a moderate inter-rater agreement. ConclusionResults of study serve as assurance and provides the evidence that these systems may be reliably used by different clinicians, even those with varied experience in wound care.

Development and validation of the specific instrument for assistance complexity of puerperal and newborns: Fantinelli Scale

Abstract Objectives: to develop and validate a care complexity assessment system adapted for mothers and newborns, based on two pre-existing scales. Methods: this is a cross-sectional study of psychometric validation of instruments, applied in a roomming-in, in 2016. The instrument has 13 indicators, submitted to the evaluation of a panel of judges for validation. For the psychometric analysis of the instrument, five criteria were adopted: a) internal consistency through Cronbach's alpha (αC); (b) intraclass correlation coefficient (ICC); c) correlation between categorical items, mediating Kendall correlation; d) ceiling and floor effect; e) Kaiser-Meyer-Olkin test. Results: after evaluating the judges on the categories, the agreement was satisfactory, being equal to or greater than 80%. The mean total score of the 13 items was 22.5 ± 4.2, with an intermediate level score of 52.9%. For psychometric validation purposes, the instrument presented αC scores of 0.73 points, between minimum items of 0.70 and maximum of 0.74 points. In the correlation of αC by items and the total score, the instrument showed high relationships (r2: 0.84). In the ICC assessment of 0.73 points (CI95%= 0.71-0.75; p&lt;0.001). Conclusion: the questionnaire presented psychometric indicators of acceptable content validity, with statistical evidence of reliability, obtained by satisfactory inter-rateragreement.

Developing a task-based approach to assessment in an Asian context

The article addresses the need to devise assessment methods appropriate for task-based instruction in an Asian context. An initial attempt was made to use an existing (non-Chinese) system to establish the difficulty level of test tasks, and rating systems appropriate to them. This attempt, although promising, indicated a need for modifications to be viable in this new context. Consequently, a series of studies were undertaken, each building on its predecessor, which developed systems for assessing task difficulty and for rating the performances which resulted with test-tasks. A multi-category task-analysis scheme was devised, trialled and modified, and specific ratings scales were developed for each category. This scheme was used to assess task difficulty itself; and task performance, at a detailed subscale level, with these ratings being accomplished by ELT professionals and novices. The results indicate agreement, between task generators and task raters, on a three-step task difficulty scale. They also demonstrate satisfactory inter-rater agreement on performance itself, including agreement between ELT professionals and novices. The implications of this are discussed for the development of curriculum-linked task-based assessment, in general, and the usefulness of the scheme for class teachers focussed on developing within-class achievement tests.

Validation of a simple screening tool for early diagnosis of advanced Parkinson\u2019s disease in daily practice: the CDEPA questionnaire

Early clinical diagnosis of advanced Parkinson’s disease (APD) may be difficult. This study aimed to validate a simple screening tool, the CDEPA questionnaire (“Cuestionario De Enfermedad de Parkinson Avanzada” [Questionnaire for Advanced Parkinson’s Disease]), for the identification of APD in daily practice. The study included 173 consecutively selected patients with PD (40% were women, mean age was 68.4 ± 10.5 years), stratified according to the Hoehn and Yahr (HY) scale. The CDEPA questionnaire defined APD as the presence of severe disability requiring help for activities of daily living (ADL), motor fluctuations with limitation or inability to perform ADL, severe dysphagia, recurrent falls, or dementia. The diagnostic performance of the questionnaire was assessed against the gold standard criterion based on clinical judgment. PD was categorized as advanced in 65 (38%) patients when using the gold standard and in 109 (63%) patients when the CDEPA questionnaire was used. The CDEPA questionnaire and the gold standard agreed moderately (kappa statistic of 0.48, P < 0.001). The CDEPA classified APD with a sensitivity of 97%; specificity of 57%; total accuracy of 72.3%; and area under the curve (for a binary classifier) of 77.2%. Significant differences were found between the groups created by the CDEPA in several usual PD evaluations (HY Scale, SCOPA Motor Scale, Non-motor Symptoms Scale for PD, Clinical Impression of Severity Index for PD, Clinical Global Impression–Severity Scale, and Patient Global Impression–Severity Scale). CDEPA showed satisfactory inter-rater agreement (kappa = 0.88) and test–retest concordance (kappa 0.83). In conclusion, the CDEPA questionnaire is a valid, reliable, and useful instrument for easily screening APD.

Patient Classification System for alcohol and other drugs: construction and validation.

To construct and validate a nursing care dependency classification system for alcohol and other drugs. A psychometric study. We used the agreement percentage and intraclass correlation coefficient for inter-rater agreement. The Receiver Operating Characteristic was used to determine values and accuracy of the cut-off points. The steps were literature review, content validation and clinical validation. Fifteen (15) judges, 6 nurses and 65 users of psychiatric emergency and hospitalization units participated. There were 11 indicators divided into three subcategories of dependency, and a sum between 11 and 33. The agreement regarding the judges' evaluation on the categories was satisfactory, being equal to or higher than 80%. Agreement was satisfactory among the nurses in most of the indicators (K>0.400), and the Intraclass Correlation Coefficient was equal to 0.723. The Classification System of Patients for Alcohol and other Drugs presents statistical evidence of reliability, obtained by a satisfactory inter-rater agreement. Its applicability is widely believed to assist in the management of nursing care and team sizing.

Validity of Criteria-Based Content Analysis (CBCA) at trial in free-narrative interviews

Objective The reliability of child witness testimony in sexual abuse cases is often controversial, and few tools are available. Criteria-Based Content Analysis (CBCA) is a widely used instrument for evaluating psychological credibility in cases of suspected child sexual abuse. Only few studies have evaluated CBCA scores in children suspected of being sexually abused. We designed this study to investigate the reliability of CBCA in discriminating allegations of child sexual abuse during court hearings, by comparing CBCA results with the court's final, unappealable sentence. We then investigated whether CBCA scores correlated with age, and whether some criteria were better than others in distinguishing cases of confirmed and unconfirmed abuse. Methods From a pool of 487 child sexual abuse cases, confirmed and unconfirmed cases were selected using various criteria including child IQ ≥ 70, agreement between the final trial outcome and the opinion of 3 experts, presence of at least 1 independent validating informative component in cases of confirmed abuse, and absence of suggestive questions during the child's testimonies. This screening yielded a study sample of 60 confirmed and 49 unconfirmed cases. The 14 item version of CBCA was applied to child witness testimony by 2 expert raters. Results Of the 14 criteria tested, 12 achieved satisfactory inter-rater agreement (Maxwell's Random Error). Analyses of covariance, with case group (confirmed vs. unconfirmed) and gender as independent variables and age as a covariate, showed no main effect of gender. Analyses of the interaction showed that the simple effects of abuse were significant in both sex. Nine CBCA criteria were satisfied more often among confirmed than unconfirmed cases; seven criteria increased with age. Conclusion CBCA scores distinguish between confirmed and unconfirmed cases. The criteria that distinguish best between the 2 groups are Quantity of Details, Interactions, and Subjective Experience. CBCA scores correlate positively with age, and independently from abuse; all the criteria test except 2 (Unusual Details and Misunderstood Details) increase with age. The agreement rate could be increased by merging criteria Unusual and Superfluous details that achieve a low inter-rater agreement when investigated separately. Practice implication Given its ability to distinguish between confirmed and unconfirmed cases of suspected child abuse, the CBCA could be a useful tool for expert opinion. Because our strict selection criteria make it difficult to generalize our results, further studies should investigate whether the CBCA is equally useful in the cases we excluded from our study (for example mental retardation).

Intra- and interrater agreement of pressure pain thresholds in children with orthopedic disorders

Pressure algometry is widely used to obtain measures of mechanical pain sensitivity in adults, both in experimental and clinical pain conditions. Only very few studies describe the use of pressure algometry in children. The purpose of this study was to produce reference values of pressure pain thresholds and to determine the intra- and interrater agreement, in order to introduce pressure algometry as a standardized method for the assessment of pain thresholds in children with orthopedic disorders. Pressure pain thresholds were determined at the leg and at the thenar in 50 children aged 4-12years. All were scheduled for physical examination at the outpatient Orthopedic Clinic at Aarhus University Hospital, Denmark, because of various orthopedic disorders such as clubfoot and Baker's cyst. Pressure pain thresholds were determined by two different raters: 20 children were examined twice by rater 1 (intrarater agreement), and another 20 children were examined by both rater 1 and rater 2 (interrater agreement). An additional ten children were examined once by rater 1 to increase the number of children used to determine the reference values. The pressure pain threshold was 183.1kPa (mean, SD: 90.7) at the leg and 179.1kPa (mean, SD: 97.4kPa) at the thenar (n=50). Pressure pain thresholds were similar in boys (n=27) and girls (n=23). Also, pressure pain thresholds were not affected by age. Bland-Altman plots showed excellent intrarater agreement and satisfactory interrater agreement. Pressure algometry has excellent intrarater agreement and satisfactory interrater agreement. Pressure algometry was well-tolerated even by very young children. The method deserves more widespread use both in clinical and experimental settings.

Medida Interpessoal de Psicopatia (IM-P): estudo preliminar no contexto brasileiro

The direct observation of interpersonal behaviors is an important resource in the description and diagnosis of the psychopathic personality. The Interpersonal Measure of Psychopathy (IM-P) is a psychometric instrument comprised of 21 items, designed to be applied in association with other instruments that also evaluate psychopaths. It focuses specifically on interpersonal and non-verbal behaviors that become evident during the interaction between the interviewer and individuals with psychopathic characteristics. To describe preliminary results obtained in the assessment of interpersonal behaviors of psychopathic individuals using the IM-P, including instrument translation/adaptation and inter-rater reliability results, in a sample of Brazilian adolescents. This was a cross-sectional, descriptive, correlation study conducted with a sample of 20 institutionalized male adolescents from the metropolitan area of Porto Alegre, state of Rio Grande do Sul, southern Brazil. Following instrument translation, interviewer training, and a pilot study conducted using a semi-structured interview, the IM-P was scored by three independent raters. The statistical results obtained using Kendall's coefficient of concordance revealed a high and satisfactory inter-rater agreement for IM-P total scores (W = 0.84; p < 0.001). Some items showed conflicting results and were therefore discussed and evaluated separately. The psychometric properties of the Brazilian Portuguese version of the IM-P, as assessed in this preliminary study, showed consistent inter-rater concordance, suggesting instrument reliability. Some study limitations are discussed, as are some relevant aspects that should be taken into consideration in future research involving the IM-P.

Development of the Hong Kong Version of the Functional Test for the Hemiplegic Upper Extremity (FTHUE-HK)

To develop the Hong Kong version of the Functional Test for the Hemiplegic Upper Extremity (FTHUE-HK) for patients with stroke in a Hong Kong Chinese community. Content validity of the prototype was established by a panel of occupational therapists who are experts in the field of neurological rehabilitation. New items relevant to the local culture were constructed. All items were reviewed with regard to internal consistency, and false positive and negative rates after implementation in a preliminary sample of 40 stroke patients. Fifty-seven stroke patients were recruited from eight local hospitals to form a convenience sample in which the inter-rater and test-retest reliabilities, and concurrent validity of the final version of the FTHUE-HK could be established. There was high sensitivity and specificity for test items within each functional level (i.e. recovery stage). Satisfactory inter-rater agreement on both testing procedure and functional levels was documented. Significant correlations between the functional levels and the upper extremity and hand subscores of the Fugl-Meyer Assessment (FMA) ( r = 0.88 and r = 0.88, respectively), and the self-care subscore of the Functional Independence Measure (FIM) ( r = 0.46) were found. Regression analysis indicated that 86% of the variance of the functional levels could be explained by hand and upper extremity subscores of the FMA. However, there was no significant relationship with age, gender, side of hemiparesis, and duration post-stroke. The findings substantiate the psychometric properties of the instrument, and the application of this unique hemiplegic upper extremity functional test is highly indicated in the context of the local culture.

Development of the Comprehensive Psychotherapeutic Interventions Rating Scale (CPIRS)

Three studies on the reliability and validity of the Comprehensive Psychotherapeutic Interventions Rating Scale (CPIRS) for measuring the presence and distribution of interventions in psychotherapy are reported. The CPIRS consists of 76 interventions using 6 types of therapy: experiential, psychodynamic, directive behavioral, cognitive, group dynamic, and systemic. An 18-hr training period yielded a satisfactory interrater agreement. The 1st and 2nd studies showed some negative findings regarding construct validity when comparing video fragments of individual client-centered, psychodynamic, cognitive, and behavior therapies. In the 3rd study, experiential, psychodynamic, directive behavioral, and group dynamic interventions discriminated among real-life sessions of individual (client centered, psychodynamic, and behavioral), group (psychodynamic and behavioral), and couple therapies. The sessions were representative of specific psychotherapies. However, interventions from other orientations were also present.

Rating the severity of delirium: The delirium assessment scale

AbstractExplicit operational definitions of the DSM‐3 criteria for delirium have recently been published. In this study, the reliability of the Delirium Assessment Scale (DAS), which uses the operationalized criteria to determine the severity of delirium symptoms, was examined. Three independent assessors examined four groups of patients —patients with delirium, patients with delirium and dementia, patients with dementia alone and patients without cognitive impairment. Sensitivity (83–88%) and specificity (79–88%) of the operationalized definitions for the diagnosis of delirium were high. Satisfactory interrater agreement was obtained for all individual symptoms of the DAS (kappa = 0.66–0.99). These results suggest that the DAS may be suitable for monitoring the severity of delirium; prospective studies are required to confirm this hypothesis.

The ethics objective structured clinical examination.

To develop objective structured clinical examination (OSCE) stations to assess the ability of physicians to address selected clinical-ethical situations, and to evaluate inter-rater agreement in these stations. Two ten-minute OSCE stations were developed using video-taped encounters between attending physicians and standardized patients. One scenario involved a daughter requesting a do-not-resuscitate (DNR) order for her competent mother without the mother's knowledge; the other involved a competent elderly woman requesting not to be re-intubated if her congestive heart failure worsened. The scenarios were evaluated using foreign medical graduates taking an OSCE. Each candidate was scored on his or her interaction with a standardized patient in the two OSCE stations by two independent observers. Eight attending physicians from the Division of General Internal Medicine at the Toronto Hospital were used to develop the OSCE stations, and 69 foreign medical graduates taking the University of Toronto Pre-Internship Program OSCE were used to evaluate the stations. The inter-rater reliability coefficients for the DNR and intubation scenarios were 0.79 (95% CI 0.69-0.87) and 0.75 (95% CI 0.62-0.84), respectively. For the DNR station, the scores of the two examiners, on a scale of 0 to 10, agreed exactly for 34 candidates (50%), within one mark for 59 candidates (87%), and within two marks for 65 candidates (96%). For the intubation station, the scores of the two examiners agreed exactly for 27 candidates (40%), within one mark for 56 candidates (84%), and within two marks for 63 candidates (94%). The authors produced ethics OSCE stations with face and content validity and satisfactory inter-rater agreement. Ethics OSCE stations may be suitable for evaluating the ability of medical students and residents to address selected clinical-ethical situations.

The Consortium to Establish a Registry for Alzheimer's Disease (CERAD). Part III. Reliability of a standardized MRI evaluation of Alzheimer's disease.

The Consortium to Establish a Registry for Alzheimer's Disease (CERAD) has developed procedures for standardized imaging and reporting of magnetic resonance (MR) findings in Alzheimer's disease (AD) for use by neuroradiologists in multiple medical centers using a variety of MR equipment and field strengths. After initial pretesting, we revised the protocol, expanded the summary rating scale to seven points, and added more illustrations. Fourteen participating neuroradiologists evaluated 28 MR scans of elderly patients, giving us the basis for judging interrater agreement. We obtained acceptable intraclass correlations (greater than 0.79) for rating the size of the lateral and third ventricles and the temporal horn. Less satisfactory intraclass correlations occurred when rating other areas, including (1) global atrophy of the brain (0.70); (2) dilatation of the sulci of the temporal lobe (0.66); (3) frequency, location, and severity of white matter lesions (0.77); (4) sylvian fissure enlargement (0.70); and (5) cerebral sulcal dilatation (0.64). We also saw considerable variation in the reporting of cortical and lacunar infarcts. Despite careful design of the rating methodology and readings by experienced neuroradiologists, we did not find satisfactory interrater agreement for interpreting MR findings in elderly subjects. These findings may explain the difficulties encountered in applying similar subjective rating techniques that meet with success at one institution to multicenter studies. More objective and reproducible procedures are needed for interpretation of neuroimaging findings of AD in multicenter studies.