Strong Inter-rater Reliability Research Articles

Abstract 2284: Reliability and Validity of the MGHS Dysphagia Screening Tool for Stroke Patients

Background Dysphagia occurs in approximately 55% of acute stroke patients and is associated with aspiration pneumonia. Clinical practice guidelines recommend that all patients who are admitted with the diagnosis of stroke or suspected stroke should be screened for dysphagia utilizing a validated screening tool prior to receiving any food, fluid, or medication by mouth. Purpose The purpose of the study is to determine the reliability and validity of the Marquette General Health Systems Dysphagia Screen for identifying patients at risk for aspiration. Method This non-randomized study utilized a collaborative tracking tool that compared results of the Nursing Dysphagia Screen (NDS) to the Speech Language Pathologist Swallow Evaluation (SLPSE). Two Registered Nurses rated each patient utilizing the NDS and then the patient was rated on the SLPSE. The NDS includes10 items: and the Speech Language Pathologist portion of the tool consisted of 8 items. Results One hundred subjects ranged in age from 20-94 (M=70.31, SD=14.6). There were more males (n=58) than females (n=42). The majority were admitted to the medical unit (64%), followed by cardiac (14%), ICU (12%) and CCU (10%). Length of stay ranged from 1-15 days (M=3.73, SD=2.8). NIH scores ranged from 0-20 (M=6.8, SD=7.4). Kappa was calculated to determine inter-rater reliability. For 63 pairs of ratings between two staff nurses, Kappa=.81 indicates strong reliability. Scores on the Nursing Dysphagia Screen were compared to 86 Speech Language Pathologist Swallow Evaluations. The Phi Coefficient was not significant indicating that the nurses’ and speech therapists ratings’ were similar. In addition, when there were differences it was false positives by nurses. Conclusion The MGHS NDS is a quick screening tool used by nursing in an effort to prevent aspiration. Prior to implementing the use of the tool, it was important to establish the reliability and validity of the tool. Results indicate that there is strong inter-rater reliability. Validity was established given that there were no significant difference in ratings between the nurses and speech therapists. It appears that the MGHS NDS is an effective early screening tool for dysphagia.

Effectiveness of Nutritional Supplements for Reducing Symptoms in Autism-Spectrum Disorder: A Case Report

This report presents a case of a 9-year-old boy with autism that responded positively to nutritional supplements. The supplements were dimethylglycine and a combination of a large dose of vitamin B6 (pyridoxal HCl) and magnesium. The Autism Treatment Evaluation Checklist (ATEC) was used for outcome assessment and administered by 2 of his family members both before the nutritional supplements and 5 months into the supplements, thereby referred to as pretest and post-test, respectively. Two (2) assessors independently performed evaluations. The ATEC evaluates four areas: communication, sociability, sensory/cognitive awareness, and behavior. The lower the scores are, the less severe the symptoms are. The ATEC evaluations by 2 independent assessors showed that the changes in total ATEC were from 63 at pretest to 33 at post-test, and from 64 at pretest to 30 at post-test, respectively. These changes represented reductions of 47.6% and 53.1%. A strong inter-rater reliability was demonstrated, with an intraclass correlation coefficient of 0.988. The school teachers also noticed improvements in various areas consistent with the ATEC evaluations. Although the reported findings cannot be generalized, this case report provides useful preliminary evidence to an accumulating body of literature supporting the theory and efficacies of nutritional supplements in autism-spectrum disorders.

Initial Description of a Quantitative, Cross-Species (Chimpanzee–Human) Social Responsiveness Measure

Comparative studies of social responsiveness, an ability that is impaired in autism spectrum disorders, can inform our understanding of both autism and the cognitive architecture of social behavior. Because there is no existing quantitative measure of social responsiveness in chimpanzees, we generated a quantitative, cross-species (human-chimpanzee) social responsiveness measure. We translated the Social Responsiveness Scale (SRS), an instrument that quantifies human social responsiveness, into an analogous instrument for chimpanzees. We then retranslated this "Chimpanzee SRS" into a human "Cross-Species SRS" (XSRS). We evaluated three groups of chimpanzees (n = 29) with the Chimpanzee SRS and typical and human children with autism spectrum disorder (ASD; n = 20) with the XSRS. The Chimpanzee SRS demonstrated strong interrater reliability at the three sites (ranges for individual ICCs: 0.534 to 0.866; mean ICCs: 0.851 to 0.970). As has been observed in human beings, exploratory principal components analysis of Chimpanzee SRS scores supports a single factor underlying chimpanzee social responsiveness. Human subjects' XSRS scores were fully concordant with their SRS scores (r = 0.976, p = .001) and distinguished appropriately between typical and ASD subjects. One chimpanzee known for inappropriate social behavior displayed a significantly higher score than all other chimpanzees at its site, demonstrating the scale's ability to detect impaired social responsiveness in chimpanzees. Our initial cross-species social responsiveness scale proved reliable and discriminated differences in social responsiveness across (in a relative sense) and within (in a more objectively quantifiable manner) human beings and chimpanzees.

Personality disorders in women with severe premenstrual syndrome

Premenstrual syndrome (PMS) and its more severe form, premenstrual dysphoric disorder, affect up to 18% of women. Both are commonly associated with other mood-related disorders such as major depression, and cause significant life impairment, but their relationship with personality disorders is less clear. After completing the Structured Clinical Interview for DSM-IV-TR disorders, 33 women with severe PMS and 26 asymptomatic women, counterbalanced for menstrual cycle phase, were administered the Structured Interview for DSM-IV Personality Disorders, a diagnostic interview with low transparency, strong inter-rater reliability, and good diagnostic clarity. Women with severe PMS had a higher prevalence of personality disorders (p = 0.003) than asymptomatic women (27% versus 0%), and were more likely to have odd-eccentric, dramatic-erratic, and anxious-fearful personality disorder traits (p < 0.05). Obsessive-compulsive personality disorder (OCPD) was the most common character pathology in the PMS group (n = 6, 18%). OCPD, although not necessarily associated with greater severity of premenstrual symptoms, was related to poorer life functioning in women with PMS. The comorbidity of a personality disorder and severe PMS places an additive burden on general life functioning and may have implications for psychiatric treatment or medication given to those with severe premenstrual symptoms.

Development and Validation of a Parent Report Measure for Assessing Social-Emotional Competencies of Children and Adolescents

Based on the premises that strength-based assessment of children and adolescents is an important emerging area, and that additional tools for this purpose are needed, this study details development and validation efforts on a new strength-based assessment: the Social-Emotional Assets and Resilience Scale, parent form (SEARS-P). Following careful development of a comprehensive research prototype assessment, a large and diverse nationwide sample of more than 2,000 ratings of school-age children and adolescents were obtained from their parents and other caregivers. Factor analytic procedures revealed a robust and replicable underlying factor structure, including Self-Regulation/Responsibility, Social Competence, and Empathy factors. The factor scores and total score of the SEARS-P were shown to have strong internal consistency reliability, as well as strong interrater reliability between mother-father pairs who rated the same child. Convergent construct validity of the SEARS-P was established through findings of significant correlations with two established strength-based rating scales for use by parents. Construct validity of the SEARS-P was further bolstered through findings of significant gender differences in scores (with females rated as having higher levels of competency than males), as well as significant differences in scores based on educational disability status. Limitations and future research needs are discussed, as are implications of this study for research and practice with children and families.

Pain assessment in the intensive care unit: Development and psychometric testing of the nonverbal pain assessment tool

Few standardized tools measure pain in nonverbal patients in the intensive care unit (ICU). Evidence exists that patient behaviors provide important information about pain in those unable to report their pain. The Nonverbal Pain Assessment Tool (NPAT), a behavioral pain assessment tool, was developed for the nonverbal adult patient in the ICU. Content and criterion validity and internal and interrater reliability were evaluated. The validity of NPAT was assessed against the patients' self-reports of pain. Interrater reliability was tested by 5 teams of 2 nurses assessing pain in nonverbal adult patients in 4 different ICUs. The tool underwent 2 revisions and involved 270 patients. During the final evaluation, strong internal reliability (chronbach's α, .82; 95% confidence interval), moderately strong interrater reliability (concordance coefficient, .72; 95% confidence interval; and weighted κ statistic, .35), and moderately strong validity (concordance coefficient, .66; 95% confidence interval) were achieved. The NPAT was found to possess moderately strong validity and strong internal and interrater reliability. It is easy to use, and provides a standard approach to assessing pain in the nonverbal adult patient.

Entorhinal cortex volume in older adults: Reliability and validity considerations for three published measurement protocols

Measuring the entorhinal cortex (ERC) is challenging due to lateral border discrimination from the perirhinal cortex. From a sample of 39 nondemented older adults who completed volumetric image scans and verbal memory indices, we examined reliability and validity concerns for three ERC protocols with different lateral boundary guidelines (i.e., Goncharova, Dickerson, Stoub, & deToledo-Morrell, 2001; Honeycutt et al., 1998; Insausti et al., 1998). We used three novice raters to assess inter-rater reliability on a subset of scans (216 total ERCs), with the entire dataset measured by one rater with strong intra-rater reliability on each technique (234 total ERCs). We found moderate to strong inter-rater reliability for two techniques with consistent ERC lateral boundary endpoints (Goncharova, Honeycutt), with negligible to moderate reliability for the technique requiring consideration of collateral sulcal depth (Insausti). Left ERC and story memory associations were moderate and positive for two techniques designed to exclude the perirhinal cortex (Insausti, Goncharova), with the Insausti technique continuing to explain 10% of memory score variance after additionally controlling for depression symptom severity. Right ERC-story memory associations were nonexistent after excluding an outlier. Researchers are encouraged to consider challenges of rater training for ERC techniques and how lateral boundary endpoints may impact structure-function associations.

A scoring system for elective triage of referrals: Spine Severity Score

Background context The Spine Severity Score (SSS) is a 15-point scoring system devised for the purpose of triaging elective surgical spine referrals. From the referral letter and the accompanying radiology report, a total score is calculated based on clinical, pathological, and radiological criteria; a maximum score of 5 can be obtained within each category. A higher total score represents a referral that should be seen more urgently. Purpose To report interrater and intrarater reliability for the SSS and compare the scoring system against the traditional system for triage, that is, the surgeon's clinical experience. Study design A prospective cross-comparison design was used to evaluate the reliability and convergent validity of the SSS using spine case referrals. Methods Four spine surgeons (experts) and three administrative assistants (nonexperts) at the University of Calgary scored 25 referrals. A second iteration of scoring was performed with a minimum time interval elapsed of 6 weeks. Raters were instructed to choose the most concerning (the one with the highest associated score) descriptor in each category that was thought relevant to the individual referral. No further instructions were given on how to interpret the referral letter or the radiology report. The surgeons also scored the referrals using their own preexisting four-point scoring systems. The results were analyzed with independent and dependent t tests, Pearson product moment correlation coefficient, and generalizability and decision analysis. Results An independent-measures t test (p>.05) revealed no statistical differences between experts and nonexperts (ie, interrater reliability) for both Iterations 1 and 2 on total scores of the SSS and a moderately strong relationship between their ratings across iterations (r=0.79, p<.001). Similarly, a paired-samples t test (p>.05) indicated a nonsignificant mean difference between Total SSS ratings at Time 1 and Time 2 (ie, intrarater reliability) and a high degree of agreement (r=0.96, p<.001) between the two iterations. These results were confirmed with correlational analyses. Pearson product moment correlation coefficients between the gold standard and the mean score were calculated from expert ratings on Total SSS at Time 1 (r=0.71, p<.001) and at Time 2 (r=0.69, p<.001). Conclusions The SSS is a reliable scoring system for triage of elective spine referrals, even among nonexperts. We have been able to demonstrate strong interrater and intrarater reliability for the SSS and moderately strong correlation with the traditional triage system.

Evaluation of Pathologists (Histopathology) and Radiologists (Cone Beam Computed Tomography) Differentiating Radicular Cysts from Granulomas

Introduction This study investigated the differentiation of radicular cysts from granulomas. Cone beam computed tomography (CBCT) imaging was compared with the existing standard, biopsy and histopathology. Methods Forty-five patients scheduled for an apicoectomy received a CBCT scan of the involved arch. Two oral and maxillofacial radiologists, working independently and using the same criteria, categorized the CBCT images as one of the following: cyst, likely cyst, likely granuloma, granuloma, or other. After apicoectomies, two oral pathologists, working independently and using the same criteria, diagnosed the surgical specimens as one of the following: radicular cyst, granuloma, or other. We examined the following: (1) interrater agreement between pathologists as to the biopsy diagnosis, (2) interrater agreement between radiologists as to the CBCT diagnosis, and (3) accuracy of radiologists' diagnostic assessments using histopathology as the standard. Results Findings showed strong interrater reliability between pathologists (κ = 0.79, z = 5.46, p < 0.0001) and weak interrater reliability between radiologists (κ = 0.14, p = not significant). Accuracy (true-positives plus true-negatives) for the two radiologists was 51% and 63%. Conclusion Under the conditions of this study, based on the inconsistency of the radiologists' reports as evidenced by statistical analyses, it was concluded that CBCT imaging is not a reliable diagnostic method for differentiating radicular cysts from granulomas. Surgical biopsy and histopathological evaluation remain the standard procedure for differentiating radicular cysts from granulomas.

Reliability of the School Food Checklist for in‐school audits and photograph analysis of children’s packed lunches

Assessment of children's diets is problematic, typically relying on error-prone parent or child recall or reporting, or resource intensive direct observation. The School Food Checklist (SFC) is an objective instrument comprising of 20 food and beverage categories designed to measure the foods contained in children's packed lunches. The present study aimed to assess intra-rater and inter-rater reliability of each of the food and beverage categories of the SFC for both in-school audits and photograph analysis of children's school lunches. Participants comprised 176 children aged 5-8 years from five primary schools in Northern Metropolitan Melbourne. The SFC was used to measure the foods contained in children's packed lunches in the school setting and using photographs. Photograph analysis was conducted by the auditors 2-3 months after completion of in-school audits. Both intra-rater [intra-class correlation coefficient (ICC) = 0.78-1] and inter-rater (ICC = 0.50-0.95) reliability analysis indicated strong agreement for in-school auditing. With the exception of the food category titled 'leftovers', there was strong intra-rater reliability for auditors' live audits and their analysis of photographs [ICC = 0.57-0.98 (Auditor 1); ICC = 0.72-0.90 (Auditor 2)], and strong inter-rater reliability for photograph analysis (ICC = 0.68-0.92). The SFC is a reliable method of measuring the foods and beverages contained in children's packed lunches when used in the school setting or for photograph analysis. This finding has broad implications, particularly for the use of photograph analysis, because this approach offers a convenient and cost effective method of measuring what food and beverages children bring to school in home packed lunches.

Development and Evaluation of an Observational Tool for Assessing Surgical Flow Disruptions and Their Impact on Surgical Performance

Many researchers have previously explored the correlation between surgical flow disruptions and adverse events in cardiac surgery; however, there is no reliable tool to prospectively categorize surgical flow disruptions and the conditions that predispose a surgical team to adverse events. Two independent raters of different medical and human factors expertise observed 12 cardiovascular operations and iteratively designed a surgical flow disruption tool (SFDT) to characterize surgical flow disruptions and the latent factors that contribute to adverse events. Categories to characterize surgical flow disruptions were created based on human factors models of human error. After the design period, both raters observed ten surgical cases using the tool to assess validity and inter-rater reliability. Rating agreement (weighted kappa) for each category across the ten surgeries was moderate to very high, resulting in strong inter-rater reliability for each category on the surgical flow disruption tool. Use of the SFDT was simple and clear for observers of diverse backgrounds, including human factors experts and medical personnel. This research depicts the development and utility of a tool to analyze surgical flow disruptions in the cardiovascular operating room with satisfactory inter-rater reliability. This tool is an important first step in systematically categorizing and measuring surgical flow disruptions and their impact on patient safety in the operating room.

Developing a Tool for Reliably Identifying Distractions and Interruptions during Surgery

Background: Many researchers have previously explored the correlation between surgical flow disruptions (distractions and interruptions) and error in cardiac surgery, however there is no reliable tool to prospectively categorize surgical flow disruptions and the conditions that predispose a surgical team to error. Methods and Results: Two independent raters of different medical and human factors expertise observed 12 cardiovascular operations and iteratively designed a tool to characterize surgical flow disruptions and the latent factors that contribute to error. Categories to characterize surgical flow disruptions were based on human factors models of human error. Following the design period, both raters observed 10 surgical cases using the tool, to assess validity and inter-rater reliability. Rating agreement (weighted kappa) for each category across the 10 surgeries was moderate to very high, resulting in strong inter-rater reliability for each category on the surgical flow disruption tool (SFDT). Conclusions: This research depicts the development and utility of a tool to analyze surgical flow disruptions in the cardiovascular operating room with satisfactory inter-rater reliability. This tool is an important first step in systematically categorizing and measuring surgical flow disruptions and their impact on patient safety in the operating room.

Interrater reliability of the mind map assessment rubric in a cohort of medical students

BackgroundLearning strategies are thinking tools that students can use to actively acquire information. Examples of learning strategies include mnemonics, charts, and maps. One strategy that may help students master the tsunami of information presented in medical school is the mind map learning strategy. Currently, there is no valid and reliable rubric to grade mind maps and this may contribute to their underutilization in medicine. Because concept maps and mind maps engage learners similarly at a metacognitive level, a valid and reliable concept map assessment scoring system was adapted to form the mind map assessment rubric (MMAR). The MMAR can assess mind map depth based upon concept-links, cross-links, hierarchies, examples, pictures, and colors. The purpose of this study was to examine interrater reliability of the MMAR.MethodsThis exploratory study was conducted at a US medical school as part of a larger investigation on learning strategies. Sixty-six (N = 66) first-year medical students were given a 394-word text passage followed by a 30-minute presentation on mind mapping. After the presentation, subjects were again given the text passage and instructed to create mind maps based upon the passage. The mind maps were collected and independently scored using the MMAR by 3 examiners. Interrater reliability was measured using the intraclass correlation coefficient (ICC) statistic. Statistics were calculated using SPSS version 12.0 (Chicago, IL).ResultsAnalysis of the mind maps revealed the following: concept-links ICC = .05 (95% CI, -.42 to .38), cross-links ICC = .58 (95% CI, .37 to .73), hierarchies ICC = .23 (95% CI, -.15 to .50), examples ICC = .53 (95% CI, .29 to .69), pictures ICC = .86 (95% CI, .79 to .91), colors ICC = .73 (95% CI, .59 to .82), and total score ICC = .86 (95% CI, .79 to .91).ConclusionThe high ICC value for total mind map score indicates strong MMAR interrater reliability. Pictures and colors demonstrated moderate to strong interrater reliability. We conclude that the MMAR may be a valid and reliable tool to assess mind maps in medicine. However, further research on the validity and reliability of the MMAR is necessary.

A psychometric investigation of speech, language and cognitive communicative rating scales for adults with acquired neurogenic communication disorders in the acute hospital setting

Speech-language pathologists working in hospitals have limited opportunities to identify patients with newly acquired communication related impairments and to support patients with communication related impairments to communicate their healthcare needs. The International Classification of Functioning, Disability and Health conceptualizes these different roles as within the Body Functions and Structures component and within the Activities and Participation component respectively. The Inpatient Functional Communication Interview (IFCI) is a measure of how well a patient is able to communicate their healthcare needs in hospital. This study investigated whether a speech-language pathologist could conduct the IFCI and have sufficient information to rate a patient's level of communication related impairments, on newly developed speech, language and cognitive communicative rating scales (named the OHW scales) in a reliable and valid way. This research indicated that the OHW scales had strong and significant concurrent criterion validity and significant interrater reliability. However the most important aspect of interrater reliability for the OHW scales was interrater agreement. Interrater agreement was moderately high for the OHW speech and cognitive communicative scales but low for the OHW language scale. Interrater agreement on the OHW language scale requires further investigation.

The utility of the Generalized Anxiety Disorder Severity Scale (GADSS) with older adults in primary care

The Generalized Anxiety Disorder Severity Scale (GADSS) is an interview rating scale designed specifically for assessing symptom severity of generalized anxiety disorder (GAD), which has demonstrated positive psychometric data in a sample of adult primary care patients with GAD and panic disorder. However, the psychometric properties of the GADSS have not been evaluated for older adults. This study evaluated the psychometric properties of the GADSS, administered via telephone, with a sample of older primary care patients (n=223) referred for treatment of worry and/or anxiety. The GADSS demonstrated adequate internal consistency, strong inter-rater reliability, adequate convergent validity, poor diagnostic accuracy, and mixed discriminant validity. Results provide mixed preliminary support for use of the GADSS with older adults.

Quality and Use of Trauma Histories Obtained From Psychiatric Outpatients: A Ten-Year Follow-Up

In 1996 Eilenberg and colleagues reviewed 180 charts and reported that trauma histories were poorly documented at a general psychiatric clinic. This study is a ten-year replication of that work. A total of 107 randomly selected charts from the same clinic were reviewed for assessment of trauma history. The quality of the assessments was rated. Fifty-six percent of charts contained a thorough description of trauma severity, compared with 15% in the initial study. Documentation of posttraumatic symptoms improved significantly in all categories but remained low. Documentation of diagnostic formulations and treatment plans for patients with a trauma history was thorough in a small proportion of charts (14%), as was found in the initial study (9%). Although some improvements were found in recording trauma histories, eliciting such a history did not lead to thorough documentation of posttraumatic symptoms or to inclusion of the history in the diagnostic formulation or treatment plan.

Assessing fidelity in individual and family therapy for adolescent substance abuse

This study introduces an observational measure of fidelity in evidence-based practices for adolescent substance abuse treatment. The Therapist Behavior Rating Scale—Competence (TBRS-C) measures adherence and competence in individual cognitive–behavioral therapy and multidimensional family therapy for adolescent substance abuse. The TBRS-C assesses fidelity to the core therapeutic goals of each approach and also contains global ratings of therapist competence. Study participants were 136 clinically referred adolescents and their families observed in 437 treatment sessions. The TBRS-C demonstrated strong interrater reliability for goal-specific ratings of treatment adherence, and modest reliability for goal-specific and global ratings of therapist competence, evidence of construct validity, and discriminant validity with an observational measure of therapeutic alliance. The utility of the TBRS-C for evaluating treatment fidelity in field settings is discussed.

Microsurgical lab testing is a reliable method for assessing ophthalmology residents’ surgical skills

Background:Formal assessment of clinical competencies is necessary to ensure that all residents are acquiring important skills and, in the United States, will soon become a requirement for residency programme accreditation...

Self-Injurious Thoughts and Behaviors Interview: Development, reliability, and validity in an adolescent sample.

The authors developed the Self-Injurious Thoughts and Behaviors Interview (SITBI) and evaluated its psychometric properties. The SITBI is a structured interview that assesses the presence, frequency, and characteristics of a wide range of self-injurious thoughts and behaviors, including suicidal ideation, suicide plans, suicide gestures, suicide attempts, and nonsuicidal self-injury (NSSI). This initial study, based on the administration of the SITBI to 94 adolescents and young adults, suggested that the SITBI has strong interrater reliability (average kappa = .99, r = 1.0) and test-retest reliability (average kappa = .70, intraclass correlation coefficient = .44) over a 6-month period. Moreover, concurrent validity was demonstrated via strong correspondence between the SITBI and other measures of suicidal ideation (average kappa = .54), suicide attempt (kappa = .65), and NSSI (average kappa = .87). The authors concluded that the SITBI uniformly and comprehensively assesses a wide range of self-injury-related constructs and provides a new instrument that can be administered with relative ease in both research and clinical settings.

Compliance with guidelines for ADHD: A pilot study of an evaluation tool

The investigators developed a 1-page chart abstraction tool to evaluate compliance with practice guidelines for attention deficit hyperactivity disorder. The tool had strong inter-rater reliability, with kappa of 0.81. Pilot testing for 57 children with attention deficit hyperactivity disorder showed only 12% documented full compliance with assessment guidelines and 44% with treatment guidelines.