Inter-rater Percent Agreement Research Articles

Adaptation of a Simulation Model and Checklist to Assess Pediatric Emergency Care Performance by Prehospital Teams.

Simulation tools to assess prehospital team performance and identify patient safety events are lacking. We adapted a simulation model and checklist tool of individual paramedic performance to assess prehospital team performance and tested interrater reliability. We used a modified Delphi process to adapt 3 simulation cases (cardiopulmonary arrest, seizure, asthma) and checklist to add remote physician direction, target infants, and evaluate teams of 2 paramedics and 1 physician. Team performance was assessed with a checklist of steps scored as complete/incomplete by raters using direct observation or video review. The composite performance score was the percentage of completed steps. Interrater percent agreement was compared with the original tool. The tool was modified, and raters trained in iterative rounds until composite performance scoring agreement was 0.80 or greater (scale <0.20 = poor; 0.21-0.39 = fair, 0.40-0.59 = moderate; 0.60-0.79 = good; 0.80-1.00 = very good). We achieved very good interrater agreement for scoring composite performance in 2 rounds using 6 prehospital teams and 4 raters. The original 175 step tool was modified to 171 steps. Interrater percent agreement for the final modified tool approximated the original tool for the composite checklist (0.80 vs. 0.85), cardiopulmonary arrest (0.82 vs. 0.86), and asthma cases (0.80 vs. 0.77) but was lower for the seizure case (0.76 vs. 0.91). Most checklist items (137/171, 80%) had good-very good agreement. Among 34 items with fair-moderate agreement, 15 (44%) related to patient assessment, 9 (26%) equipment use, 6 (18%) medication delivery, and 4 (12%) cardiopulmonary resuscitation quality. The modified checklist has very good agreement for assessing composite prehospital team performance and can be used to test effects of patient safety interventions.

Design and development of the home healthcare worker observation tool.

The rapidly growing number of home healthcare workers (HHCW) are exposed to unique occupational hazards within each patient home. This article describes the development of an observation tool to document occupational hazards HHCWs encounter. Tool development followed three steps: determining content domain, content validity, and inter-rater agreement. Expert feedback guided the revision of content domain to 636 items. Scale level content validity index (S-CVI) was 0.90. Inter-rater agreement tests resulted in percent agreement and accuracy mean of 89.5% and frequency variables resulted in standard deviations from 0 to 8.62. The observation tool encompasses the diverse range of occupational hazards HHCWs encounter; inter-rater percent agreement and overall accuracy scores were acceptable. Future pilot testing of this tool among broader raters and populations is recommended to characterize its usability, internal consistency, and reliability to assess HHCW occupational hazards.

Personalized warning signals for depressive relapse: A qualitative study

IntroductionAn important aspect of depression relapse prevention programs is identifying personalized warning signals (PWS). These PWS are typically defined as depressive symptoms. Yet, no study has investigated to what extend PWS fit within the diagnostic classification framework, and how this compares to a more transdiagnostic, integrative approach towards depression.ObjectivesTo examine how well PWS reflect depressive symptoms, describe the remaining PWS, and examine how well PWS can be assigned to domains of an existing transdiagnostic and integrative framework, the positive health concept.Methods162 PWS of 66 individuals with a history of depression were labeled as one or more symptoms of depression or to a residual category. The same process was repeated for labeling the domains of the positive health model. Labeling was done by three independent reviewers (inter-rater percent agreement: symptoms: 0.83 &amp; positive health domains: 0.73). Disagreements were resolved by discussion.ResultsThe three most commonly reported depressive symptoms were insomnia/hypersomnia, anhedonia and fatigue/loss of energy. However, sixty-five percent of the PWS were not depressive symptoms, but other symptoms (e.g. irritability, rumination) or aspects of functioning (e.g. withdrawing, managing time). The positive health domains captured all the PWS. However, 44% of PWS were labeled as multiple positive health domains, whereas labeling as symptoms of depression resulted in almost no such overlap.ConclusionsA more transdiagnostic and integrative approach seems necessary to capture PWS. Depending on one’s purpose, one may consider expanding the definition with other symptoms and aspects of functioning, or using the positive health concept.

Interrater Reliability of Four Neurological Scales for Patients Presenting to the Emergency Department

BackgroundAltered mental status (AMS) comprises a group of clinical symptoms rather than a specific diagnosis. A variety of neurological scales have been used to monitor these patients, including the Glasgow coma scale (GCS), AVPU (alert, responds to verbal stimuli, responds to painful stimuli, unresponsive) scale, and ACDU (alert, confused, drowsy, unresponsive) scale. The simplified motor scale (SMS) has been found to be a potential replacement for the GCS. In this study, we compare the interrater reliability of the GCS (individual components), AVPU, ACDU, and SMS to patients presenting to the ED with AMS.Materials and methodsThis was a prospective observational study conducted in the emergency department of an urban tertiary-care hospital in New Delhi, India. Patients with AMS (traumatic and nontraumatic) presenting to the ED were assessed by two emergency physicians within 10 minutes of each other using the four neurological scales. Percentage agreement and kappa coefficient score were used to determine interrater reliability for SMS, individual components of GCS, AVPU, and ACDU.ResultsThe SMS had the best inter-rater percent agreement with 83.91% (95% CI 77.7–88.6%), followed by ACDU which was 76.44% (95% CI 69.0–81.6%), AVPU 75.29% (95% CI 67.8–80.6%), GCS Eye 74.14% (95% CI 66.6–79.6%), GCS Verbal 67.82% (95% CI 60.6–74.3%), and GCS Motor was 64.94% (95% CI 57.6–71.6%).The kappa coefficient for SMS was 0.75 (95% CI 0.67–0.83), followed by GCS Eye which was 0.63 (95% CI 0.54–0.72), AVPU 0.62 (95% CI 0.52–0.72), ACDU 0.60 (95% CI 0.49–0.71), and GCS Verbal 0.58 (0.49–0.66) and GCS Motor was 0.53 (95% CI 0.44–0.63).ConclusionIt can be said that SMS has the best interrater reliability in the ED and can be recommended for clinical use in the emergency departments for patients presenting with AMS.How to cite this articleHaldar M, Verma A, Jaiswal S, Sheikh WR. Interrater Reliability of Four Neurological Scales for Patients Presenting to the Emergency Department. Indian J Crit Care Med 2020;24(12):1198–1200.

Inter-rater reliability in diagnosis and treatment of type one laryngeal cleft: A blinded observational study

IntroductionType one laryngeal cleft (T1LC) has been implicated as a major contributor to aspiration in non-syndromic pediatric dysphagia. Despite an increasing incidence, there remains controversy in diagnosis and treatment algorithms. ObjectivesThe primary objective of this study was to evaluate the inter-rater reliability (IRR) for the diagnosis and treatment of T1LC. MethodsA retrospective analysis was conducted to identify children evaluated for a T1LC from 2016 to 2017 at a single tertiary care center. The microlaryngoscopy video recordings depicting palpation of the interarytenoid region with a right-angle probe were reviewed. These recordings were shown to blinded pediatric otolaryngologists and each surgeon's determination of the presence or absence of a T1LC as well as recommended treatment (observation, injection laryngoplasty, or endoscopic cleft repair) was recorded and compared against the other blinded surgeons. Fleiss's kappa was calculated to evaluate IRR in both diagnosis and treatment. ResultsEight pediatric otolaryngologists were included in the study with a mean post-training experience of 15 years (range 1–35 years). The inter-rater percent agreement in diagnosis of our patient population was 28.6% (range 3.7–71%) with a kappa value of 0.31 (p < 0.0001). In regard to management, the inter-rater percent agreement in treatment was 11.4% (range 0–35%) with a kappa value of 0.14 (p = 0.01). ConclusionThis study highlights the challenges and variation that exists among surgeons in diagnosing and managing potential T1LC. Further standardizing the endoscopic examination and treatment algorithm may reduce diagnostic and treatment discordance.

A Standardized Electrode Nomenclature for Stereoelectroencephalography Applications.

Stereoelectroencephalography (SEEG) is widely performed on individuals with medically refractory epilepsy for whom invasive seizure localization is desired. Despite increasing adoption in many centers across the world, no standardized electrode naming convention exists, generating confusion among both clinical and research teams. We have developed a novel nomenclature, named the Standardized Electrode Nomenclature for SEEG Applications system. Concise, unique, informative, and unambiguous labels provide information about entry point, deep targets, and relationships between electrodes. Inter-rater agreement was evaluated by comparing original electrode names from 10 randomly sampled cases (including 136 electrodes) with those prospectively assigned by four additional blinded raters. The Standardized Electrode Nomenclature for SEEG Application system was prospectively implemented in 40 consecutive patients undergoing SEEG monitoring at our institution, creating unique electrode names in all cases, and facilitating implantation design, SEEG recording and mapping interpretation, and treatment planning among neurosurgeons, neurologists, and neurophysiologists. The inter-rater percent agreement for electrode names among two neurosurgeons, two epilepsy neurologists, and one neurosurgical fellow was 97.5%. This standardized naming convention, Standardized Electrode Nomenclature for SEEG Application, provides a simple, concise, reproducible, and informative method for specifying the target(s) and relative position of each SEEG electrode in each patient, allowing for successful sharing of information in both the clinical and research settings. General adoption of this nomenclature could pave the way for improved communication and collaboration between institutions.

Validation Of Visual Inspection For Determining Accelerometer Wear Days

Visual inspection of accelerometer output may provide an alternative to needing participant-reported wear logs for ascertaining wear days when monitors are sent and returned by mail, for participants not completing logs. However, there are no data on the reliability and validity of visual inspection compared to wear logs. PURPOSE: To compare estimates of wear days and physical activity using visual inspection, with participant-reported wear logs. METHODS: Participants (N = 197, mean age = 71.0 years) were from the Women’s Health Study. In 2011-2014, women were mailed an accelerometer (ActiGraph GT3X+) to be worn for 7 days and then returned in the mail. They also indicated on a log the days they wore the monitor. Accelerometer days were classified as “worn” or “not worn” based on participant-reported wear logs. For each participant, we created a series of line charts of accelerometer counts over the course of the day (1 chart per day of accelerometer data). Two raters independently inspected these graphs to identify each day as ‘worn’ or ‘not worn’. We calculated the sensitivity and specificity for each rater, compared against log determined wear days, as well as the percent agreement between raters. We tested for differences in median wear time and physical activity estimates between visual inspection and wear logs using Wilcoxon Signed-Rank tests. RESULTS: The median (interquartile range) number of valid wear days for both visual inspection and log was 7 (7, 7). For rater 1, the sensitivity was 99.7% (95% confidence interval (CI): 99.2%, 99.9%); specificity, 97.2% (95.2%, 98.6%). For rater 2, the sensitivity was 99.7% (99.2%, 99.9%); specificity, 97.0% (95% CI: 94.9%, 98.4%). The inter-rater percent agreement was 99.5%. There was no significant difference in the median number of minutes of moderate-to-vigorous physical activity per day between the two methods (visual inspection: 20.7 (8.9, 37.4); wear logs: 20.0 (8.9, 37.4); P = 0.95). CONCLUSIONS: Visual inspection of accelerometer data is a valid and reliable alternative to participant-reported wear logs to determine accelerometer wear days, for those not providing logs, in direct mail study designs.

Translation, cross-cultural adaptation to Brazilian- Portuguese and reliability analysis of the instrument Rapid Entire Body Assessment-REBA.

Background: Observational instruments, such as the Rapid Entire Body Assessment, quickly assess biomechanical risks present in the workplace. However, in order to use these instruments, it is necessary to conduct the translational/cross-cultural adaptation of the instrument and test its measurement properties. Objectives: To perform the translation and the cross-cultural adaptation to Brazilian-Portuguese and test the reliability of the REBA instrument. Method:The procedures of translation and cross-cultural adaptation to Brazilian-Portuguese were conducted following proposed guidelines that involved translation, synthesis of translations, back translation, committee review and testing of the pre-final version. In addition, reliability and the intra- and inter-rater percent agreement were obtained with the Linear Weighted Kappa Coefficient that was associated with the 95% Confidence Interval and the cross tabulation 2×2. Results: The procedures for translation and adaptation were adequate and the necessary adjustments were conducted on the instrument. The intra- and inter-rater reliability showed values of 0.104 to 0.504, respectively, ranging from very poor to moderate. The percentage agreement values ranged from 5.66% to 69.81%. The percentage agreement was closer to 100% at the item 'upper arm' (69.81%) for the Intra-rater 1 and at the items 'legs' and 'upper arm' for the Intra-rater 2 (62.26%). Conclusions: The processes of translation and cross-cultural adaptation were conducted on the REBA instrument and the Brazilian version of the instrument was obtained. However, despite the reliability of the tests used to correct the translated and adapted version, the reliability values are unacceptable according to the guidelines standard, indicating that the reliability must be re-evaluated. Therefore, caution in the interpretation of the biomechanical risks measured by this instrument should be taken.

System for Observing Staff Promotion of Activity and Nutrition (SOSPAN)

Frontline-staff are critical to achieving policies related to child physical activity and nutrition (PAaN) in out-of-school-time programs (OSTP). Recent policies call upon staff to demonstrate behaviors related to PAaN. Currently, no instrument exists to measure these behaviors. This study fills the gap between policy mandates and staff behaviors by describing the development of the System for Observing Staff Promotion of Activity and Nutrition (SOSPAN) in OSTP. SOSPAN items were aligned with existing OSTP policies. Reliability and validity data of SOSPAN were collected across 8 OSTP: 4 summer day camps and 4 afterschool programs. Validity of SOSPAN staff behaviors/management of PA was established using the percent of children active measured concurrently via direct observation. A total of 6437 scans were performed. Interrater percent agreement ranged from 74%-99% across PAaN behaviors. Children's activity was associated with staff facilitative behaviors/management, such as playing with the children and providing 2 or more activities for children to choose, while prohibitive behaviors/management, such as waiting in line were related to increased sedentary behavior. Staff nutrition behaviors were observed in less than 0.6% of scans. SOSPAN is a reliable and valid tool to assess staff behaviors/management of PAaN in OSTPs.

The Healthy Afterschool Activity and Nutrition Documentation Instrument

Policies call on afterschool programs to improve the physical activity and nutrition habits of youth attending. No tool exists to assess the extent to which the afterschool program environment meets physical activity and nutrition policies. To describe the development of the Healthy Afterschool Activity and Nutrition Documentation (HAAND) instrument, which consists of two subscales: Healthy Afterschool Program Index for Physical Activity (HAPI-PA) and the HAPI-Nutrition (HAPI-N). Thirty-nine afterschool programs took part in the HAAND evaluation during fall/spring 2010-2011. Inter-rater reliability data were collected at 20 afterschool programs during a single site visit via direct observation, personal interview, and written document review. Validity of the HAPI-PA was established by comparing HAPI-PA scores to pedometer steps collected in a subsample of 934 children attending 25 of the afterschool programs. Validity of the HAPI-N scores was compared against the mean number of times/week that fruits and vegetables (FV) and whole grains were served in the program. Data were analyzed in June/July 2011. Inter-rater percent agreement was 85%-100% across all items. Increased pedometer steps were associated with the presence of a written policy related to physical activity, amount/quality of staff training, use of a physical activity curriculum, and offering activities that appeal to both genders. Higher servings of FV and whole grains per week were associated with the presence of a written policy regarding the nutritional quality of snacks. The HAAND instrument is a reliable and valid measurement tool that can be used to assess the physical activity and nutritional environment of afterschool programs.

Evaluation of Clinical Assessment Methods for Scapular Dyskinesis

The purposes of this study were to (1) assess the inter-rater reliability and validity of 2 clinical assessment methods of categorizing scapular dyskinesis and (2) quantify the frequency of asymmetry of bilateral scapular motion in injured and uninjured shoulders by use of 3-dimensional (3D) kinematic analysis. We evaluated 56 subjects, 35 with shoulder injury and 21 with no symptoms. Two blinded evaluators categorized the scapular motion of all subjects to determine inter-rater reliability using 2 observational methods ("yes/no" and "4 type") to evaluate scapular dyskinesis. Subjects were also instrumented with electromagnetic receivers to assess bilateral 3D scapular kinematics to determine the presence of dyskinesis and establish criterion validity of the 2 methods. The inter-rater percent agreement and the degree of this agreement as measured by kappa statistic showed that the yes/no method produced a higher inter-rater percent agreement (79%, kappa = 0.40) than the 4-type method (61%, kappa = 0.44). The yes/no method had a higher sensitivity (76%) and positive predictive value (74%) when compared with the 3D criterion. A chi(2) analysis found significantly more multiple-plane asymmetries in symptomatic subjects (54%) in flexion compared with asymptomatic subjects (14%) (P = .002). The yes/no method allows multiple-plane asymmetries to be considered in clinical assessment and therefore renders this a good screening tool for the presence of scapular dyskinesis. Kinematic analysis shows that asymmetries are common in symptomatic and asymptomatic populations. Testing in flexion showed a higher frequency of multiple-plane scapular asymmetries in the symptomatic group. Identification of scapular dyskinesis is a key component of the shoulder examination. The clinician's ability to establish the presence or absence of scapular dyskinesis by observation is enhanced using a simple yes/no method especially when testing subjects in shoulder forward flexion. Although scapular asymmetries appear to be a prevalent finding, dyskinesis in the presence of shoulder symptoms should be considered a potential factor contributing to the dysfunction in the presence of shoulder symptoms should be considered a potential factor contributing to the dysfunction.

Interrater reliability of identifying indicators of posterior ligamentous complex disruption when plain films are indeterminate in thoracolumbar injuries

BackgroundThe Spine Trauma Study Group (STSG) has proposed a novel thoracolumbar injury classification system and score (TLICS) in an attempt to define traumatic spinal injuries and direct appropriate management schemes objectively. The TLICS assigns specific point values based on three variables to generate a final severity score that guides potential treatment options. Within this algorithm, significant emphasis has been placed on posterior ligamentous complex (PLC) integrity. The purpose of this study was to determine the interrater reliability of indicators surgeons use when assessing PLC disruption on imaging studies, including computed tomography (CT) and magnetic resonance imaging (MRI). MethodsOrthopedic surgeons and neurosurgeons retrospectively reviewed a series of thoracolumbar injury case studies. Thirteen case studies, including images, were distributed to STSG members for individual, independent evaluation of the following three criteria: (1) diastasis of the facet joints on CT; (2) posterior edema-like signal in the region of PLC components on sagittal T2-weighted fat saturation (FAT SAT) MRI; and (3) disrupted PLC components on sagittal T1-weighted MRI. Interrater agreement on the presence or absence of each of the three criteria in each of the 13 cases was assessed. ResultsAbsolute interrater percent agreement on diastasis of the facet joints on CT and posterior edema-like signal in the region of PLC components on sagittal T2-weighted FAT SAT MRI was similar (agreement 70.5%). Interrater agreement on disrupted PLC components on sagittal T1-weighted MRI was 48.9%. Facet joint diastasis on CT was the most reliable indicator of PLC disruption as assessed by both Cohen’s kappa (κ = 0.395) and intraclass correlation coefficient (ICC 0.430). ConclusionsThe interrater reliability of assessing diastasis of the facet joints on CT had fair to moderate agreement. The reliability of assessing the posterior edema-like signal in the region of PLC components was lower but also fair, whereas the reliability of identifying disrupted PLC components was poor.