Interrater Agreement Statistics Research Articles

Testing of the Facial Expression Coding System (FACES) for middle school–aged special education students and development of a training protocol

Emotion regulation is a key developmental skillset, but many existing measures rely on self-report or laboratory-based measurement approaches. This study aimed to develop a training and implementation protocol for the widely used Facial Expression Coding System (FACES) to be used in real-world settings with pre-recorded video data. A revised coding system with supplemental guidelines and training procedures was developed to use FACES with video data recorded in special education classrooms. This system resulted in adequate interrater reliability as well as reduced training time for coders. Specific training methods included close study of code definitions, coding of practice video, quantitative analysis of observation data to generate interrater agreement and kappa statistics, review of comparison charts to identify discrepancies between coder and training observations using the Noldus Observer XT software, and post-observation discussions. The revised FACES protocol and new training method presented here offer a more robust, efficient, and versatile tool that can be applied to systematic behavior observations conducted of students in real-world classroom settings.

The impact of implant-related characteristics on dental implant blooming: An in vitro study.

To assess, in vitro, variables potentially influencing implant blooming using a human-like imaging phantom and 3D-printed mandibles. Sixty implants were inserted in 3D-printed mandibles in 26 different configurations in order to examine the impact of implant diameter, presence of a cover screw, implant design/material, implant position, and the presence of additional implants on implant blooming using two cone-beam computed tomography (CBCT) devices (Accuitomo [ACC] and NewTom [NWT]). Two observers measured the amount of implant blooming in both buccolingual and mesiodistal directions. Inter-rater agreement and descriptive statistics, grouped by implant characteristic and CBCT device, were calculated. Both CBCT devices increased implant diameter (a mean increase of 9.2% and 11.8% for titanium, 20.3% and 24.4% for zirconium, for ACC and NWT, respectively). An increase in implant diameter did not increase the amount of blooming, whereas placing a cover screw did (from 8.0% to 10.9% for ACC, and from 10.0% to 15.6% for NWT). Moreover, implant design, anatomical region, and the presence of another implant also affected the extent of the blooming. Dental implants show a clear diameter increase on CBCT, with the effect being more pronounced for zirconium than for titanium implants. Similar effects are likely to occur in the clinical setting, potentially masking nonosseointegration, reducing the dimensions of peri-implant defects, and/or causing underestimation of the buccal bone thickness.

Improving Prostate Cancer Detection With MRI: A Multi-Reader, Multi-Case Study Using Computer-Aided Detection (CAD)

To evaluate whether addition of a computer-aided diagnostic (CAD) generated MRI series improves detection of clinically significant prostate cancer. Nine radiologists retrospectively interpreted 150 prostate MRI examinations without and then with an additional random forest-based CAD model-generated MRI series. Characteristics of biopsy negative versus positive (Gleason ≥ 7 adenocarcinoma) groups were compared using the Wilcoxon test for continuous and Pearson's chi-squared test for categorical variables. The diagnostic performance of readers was compared without versus with CAD using MRMC methods to estimate the area under the receiver operator characteristic curve (AUC). Inter-reader agreement was assessed using weighted inter-rater agreement statistics. Analyses were repeated in peripheral and transition zone subgroups. Among 150 men with median age 67 ± 7.4 years, those with clinically significant prostate cancer were older (68 ± 7.6 years vs. 66 ± 7.0 years; p < .02), had smaller prostate volume (43.9 mL vs. 60.6 mL; p < .001), and no difference in prostate specific antigen (PSA) levels (7.8 ng/mL vs. 6.9 ng/mL; p=.08), but higher PSA density (0.17 ng/mL/cc vs. 0.10 ng/mL/cc; p < .001). Inter-rater agreement (IRA) for PI-RADS scores was moderate without CAD and significantly improved to substantial with CAD (IRA=0.47 vs. 0.65; p < .001). CAD also significantly improved average reader AUC (AUC=0.72, vs. AUC=0.67; p=.02). Addition of a random forest method-based, CAD-generated MRI image series improved inter-reader agreement and diagnostic performance for detection of clinically significant prostate cancer, particularly in the transition zone.

An Empirical Comparative Assessment of Inter-Rater Agreement of Binary Outcomes and Multiple Raters

Background: Many methods under the umbrella of inter-rater agreement (IRA) have been proposed to evaluate how well two or more medical experts agree on a set of outcomes. The objective of this work was to assess key IRA statistics in the context of multiple raters with binary outcomes. Methods: We simulated the responses of several raters (2–5) with 20, 50, 300, and 500 observations. For each combination of raters and observations, we estimated the expected value and variance of four commonly used inter-rater agreement statistics (Fleiss’ Kappa, Light’s Kappa, Conger’s Kappa, and Gwet’s AC1). Results: In the case of equal outcome prevalence (symmetric), the estimated expected values of all four statistics were equal. In the asymmetric case, only the estimated expected values of the three Kappa statistics were equal. In the symmetric case, Fleiss’ Kappa yielded a higher estimated variance than the other three statistics. In the asymmetric case, Gwet’s AC1 yielded a lower estimated variance than the three Kappa statistics for each scenario. Conclusion: Since the population-level prevalence of a set of outcomes may not be known a priori, Gwet’s AC1 statistic should be favored over the three Kappa statistics. For meaningful direct comparisons between IRA measures, transformations between statistics should be conducted.

GRADING JOURNALS IN ECONOMICS: THE ABCS OF THE ABDC

AbstractMany institutions and governments grade academic journals for the evaluation of research. In this paper, we implement a multi‐bibliometric methodology for the evaluation of such a list of journal grades. We examine the grades assigned by the Australian Business Deans Council (ABDC) for over 750 journals in the fields of economics and statistics.Firstly, we generate up to 48 bibliometric‐based grades for each journal based on the grade distribution implied by the ABDC. Secondly, we categorize the bibliometrics employing a cluster analysis of an interrater agreement statistic. Thirdly, we present a visualization of the consistency of the grading by journal. Finally, we list those journals where the majority of the matched bibliometrics indicate a higher or lower grade than their ABDC grade.

Liver Needle Biopsies are Poor Predictors of Histologic Tumor Grade for Midgut Neuroendocrine Tumors

Background: Our group has previously shown than neuroendocrine tumors (NETs) are heterogeneous neoplasms having histologic and functional differences between their primary tumor, lymph node, and hepatic metastases. Due to the heterogeneity of these malignancies, we hypothesized that there would be discordance between the histologic grade of surgical specimens and that predicted by preoperative biopsies. Methods: Twenty consecutive patients diagnosed with NETs of the ileum and hepatic metastasis were included. Ki-67 proliferative index and WHO 2010 histologic grade were recorded for preoperative hepatic needle biopsy and subsequent tissue-matched surgical specimens. Concordance between sample values was determined. Results: Ten males and 10 females were included in this analysis. Five and 15 patients had fine-needle aspirate (FNA) and core needle biopsies, respectively. Preoperative biopsies predicted the histologic grade of subsequent tissue-matched surgical specimens in only 65% of samples (13/20). Of the 7 values that changed grade (7/20, 35%), 4 went from intermediate (G2) to low (G1) grade [1 FNA and 3 core biopsies] and 3 went from low (G1) to intermediate (G2) grade [1 FNA and 2 core biopsies]. The corresponding interrater agreement statistic (K) was 0.251±0.230 (95% CI: -0.199-0.702), with 0.21&lt;K&lt;0.40 indicating fair strength of agreement. Conclusion: Preoperative fine-needle aspirates and core needle biopsies of hepatic metastasis have a 35% error rate in predicting the histologic grade from subsequent tissue-matched surgical NET specimens. Clinicians should be cognizant of this error rate when making decisions on systemic treatment and consider repeat needle biopsy or open biopsy if the actual clinical course does not match predicted behaviour.

Frequency of positive ANCA test in a population with clinical symptoms suggestive of autoimmune disease and the interference of ANA in its interpretation

BackgroundAntibodies against neutrophil cytoplasm (ANCA) are associated with vasculitis. There are different methods to determine their presence. The interference of antinuclear antibodies (ANA) in the differentiation between P-ANCA and C-ANCA patterns has been described. ObjectiveTo determine the frequency of ANCA in a population with manifestations of autoimmune disease, and evaluate the interference of ANA in its interpretation. Materials and methodsRetrospective, descriptive nonexperimental cross-sectional study, including 3330 data. The presumptive diagnosis was autoimmune disease and a test for ANCA was requested. The ANCA and ANA determinations were made by indirect immunofluorescence, L-ANCA® and CytoBead® ANCA. Anti-proteinase 3 and anti-myeloperoxidase were detected by ELISA and CytoBead® ANCA. ResultsANCAs were positive in 10.21% and 12.64% of those positive for ANCA were positive for ANA. The inter-rater agreement statistic (Kappa) for anti-PR3 between CytoBead ANCA and ELISA was 100% (K=1.00; P<.05) and the agreement between anti-myeloperoxidase by ELISA and CytoBead® ANCA was high (K=0.94; P<.05). 30% of those with ANCAs had a diagnosis of a type of vasculitis; 20% of them had an autoimmune disease. ConclusionsThe results suggest an overestimated request for ANCAs as a diagnostic aid in primary care which was not addressed. For an adequate evaluation of ANCAs, the indirect immunofluorescence technique should be implemented for the control and confirmation with the determination of specific antigens for anti-proteinase 3 and anti-myeloperoxidase in any of the confirmatory assays. The high concordance shown by ANCA CytoBeads makes us consider the use of this alternative for the determination of ANCAs and the confirmation. Given the interference of ANAs, the ANA test by IFI in the presence of positive P-ANCA results is recommended in order to minimise “false positives”.

The Papanicolaou Society of Cytopathology guidelines for respiratory cytology: Reproducibility of categories among observers.

Introduction:The Papanicolaou Society of Cytopathology (PSC) has developed a set of guidelines for reporting respiratory cytology. While the malignancy risk for each category is known, the interobserver reproducibility of these diagnostic categories has not been well described.Methods:Fifty-five cytologic specimens obtained by fine needle aspiration from the pulmonary nodules were independently reviewed by four board-certified cytopathologists and assigned to the diagnostic categories described by the PSC guidelines for respiratory specimens. Statistical analysis for diagnostic accuracy was performed for absolute agreement and chance-corrected agreement (kappa). Differences in frequency of distribution of diagnoses between raters were assessed using the Kruskal–Wallis test.Results:No significant differences in distribution of scores by raters were observed. On average, the absolute agreement was 49.5% and the chance-corrected agreement (kappa) was 20%. 34.5% of interrater comparisons were in full agreement, and total lack of agreement between the four categories was found in 3% of cases. Combining the “suspicious for malignancy” category with the “malignant” category did not significantly alter interrater agreement statistics.Conclusions:The PSC categories showed only fair reproducibility among four cytopathologists. Agreement between raters was at best fair and did not improve significantly when the categories “suspicious for malignancy” and “malignant” were combined. The most common source of disagreement appeared to be between the categories “suspicious” and “malignant.”

Identifying and prioritizing the benefits of integrating BIM and sustainability practices in construction projects: A Delphi survey of international experts

The recent initiatives of the construction industry to embed sustainable strategies in its processes can be enhanced when clear and practical benefits of such integration are available to project stakeholders to support their decision-making. Hence, this study purports to evaluate the perceived benefits of integrating BIM initiatives and sustainability practices in construction projects. Delphi survey technique was used to solicit the perceptions of expert panel on the 36 identified benefits. Statistical tools were employed to analyze the derived data, and the consensus reached by the expert panel was validated using the interrater agreement statistics. The three most important benefits included the ability to enhance overall project quality and efficiency and improve the ability to simulate building performances and energy usage and facilitate better design products and multi-design alternatives. Comparative analyses among the expert groups lend credence to the strong consensus reached by the expert panel. Meanwhile, the study recommended strategies to enable the construction industry to key-in to these benefits as well as identifying prevalent research gaps in practice. The studyös findings will enhance the drive for the realization of the sustainable smart city as well as equip various stakeholders of the possibilities in its full adoption and implementation.

One-year experience of a regional service model of teleconsultation for planning and treatment of complex thoracoabdominal aortic disease

ObjectiveThe objective of this study was to report the methodology and 1-year experience of a regional service model of teleconsultation for planning and treatment of complex thoracoabdominal aortic disease (TAAD). MethodsComplex TAADs without a feasible conventional surgical repair were prospectively evaluated by vascular surgeons of the same public health service (National Health System) located in a huge area of 22,994 km2 with 3.7 million inhabitants and 11 tertiary hospitals. Surgeons evaluated computed tomography scans and clinical details that were placed on a web platform (Google Drive; Google, Mountain View, Calif) and shared by all surgeons. Patients gave informed consent for the teleconsultation. The surgeon who submits a case discusses in detail his or her case and proposes a possible therapeutic strategy. The other surgeons suggest other solutions and options in terms of grafts, techniques, or access to be used. Computed tomography angiography, angiography, and clinical outcomes of cases are then presented at the following telemeetings, and a final agreement of the operative strategy is evaluated. Teleconsultation is performed using a web conference service (WebConference.com; Avaya Inc, Basking Ridge, NJ) every month. An inter-rater agreement statistic was calculated, and the κ value was interpreted according to Altman's criteria for computed tomography angiography measurements. ResultsThe rate of participation was constant (mean number of surgeons, 11; range, 9-15). Twenty-four complex TAAD cases were discussed for planning and operation during the study period. The interobserver reliability recorded was moderate (κ = 0.41-0.60) to good (κ = 0.61-0.80) for measurements of proximal and distal sealing and very good (κ = 0.81-1) for detection of any target vessel angulation >60 degrees, significant calcification (circumferential), and thrombus presence (>50%). The concordance for planning and therapeutic strategy among all participants was complete in 16 cases. In one case, the consultation was decisive for creating an innovative therapeutic strategy; in the remaining seven cases, the strategy proposed by the patient's surgeon was changed completely after the discussion. Technical success was the same (100%) if concordance in planning was present initially or not. Overall 6-month mortality was 4%, 0% for those patients with initial concordance in planning vs 12% for those without initial concordance (P = .33). Surgery was always performed in a tertiary hospital by local surgeons, and in two cases (8%) external surgeons joined the local surgical team. ConclusionsSuch a regional service of teleconsultation may be of value in standardizing the treatment and derived costs of complex TAADs in a huge region under the same health provider. The shared decision-making strategy may be of medical-legal value as well.

Grading Journals in Economics: The ABCs of the ABDC

Many institutions and governments grade academic journals for the evaluation of research. In this paper, we implement a multi‐bibliometric methodology for the evaluation of such a list of journal grades. We examine the grades assigned by the Australian Business Deans Council (ABDC) for over 750 journals in the fields of economics and statistics. Firstly, we generate up to 48 bibliometric‐based grades for each journal based on the grade distribution implied by the ABDC. Secondly, we categorize the bibliometrics employing a cluster analysis of an interrater agreement statistic. Thirdly, we present a visualization of the consistency of the grading by journal. Finally, we list those journals where the majority of the matched bibliometrics indicate a higher or lower grade than their ABDC grade.

Validity of a new comprehensive pterygia grading scale for use in clinical research and clinical trial.

To determine the reliability and agreement of a new comprehensive pterygium grading scale for use in clinical research and clinical trials. Thirty eyes with pterygia were enrolled in this study. Primary gaze position and lateral gaze position images were taken of each eye with a modified single-lens reflex camera system. Our grading scale includes five parameters: two hyperemia parameters of pterygia on two different gaze position images and three size parameters, quantifying length, width, and area of the cornea encroachment of pterygium, using ImageJ software. All images were graded on the five parameters by two masked, certified reading center graders. Two graders independently graded all the images to determine inter-grader reliability. One grader regraded the images after 3days to determine intra-grader reliability. Intraclass correlation coefficient (ICC) and inter-rater agreement statistic (κ) calculations were performed. The intra-grader reliability for hyperemia grading was high on both primary and lateral gazing positions (κ value is 0.93 and 0.96). The inter-grader reliability for hyperemia grading was also good (κ value is 0.85 and 0.87). The mean value of width, length, and area of the cornea encroachment of pterygium was 4.31±2.04mm, 2.08±1.43mm, and 7.84±7.62mm2, respectively. The intra-grader agreement on width, length, and area were excellent, with ICCs of 0.98 (95% CI 0.96-0.99), 0.99 (95% CI 0.98-1.0), and 0.97 (95% CI 0.94-0.99), respectively. The inter-grader agreement on width, length, and area were also excellent, with ICCs of 0.96 (95% CI 0.90-0.98), 0.99 (95% CI 0.98-0.99), and 0.99 (95% CI 0.97-0.99), respectively. There was excellent intra- and inter-observer reproducibility with the new comprehensive grading scale. This scale could lead to the development of standardized grading assessments and quantification of pterygia that would be valid in clinical research and clinical trials.

An Overview of Interrater Agreement on Likert Scales for Researchers and Practitioners.

Applications of interrater agreement (IRA) statistics for Likert scales are plentiful in research and practice. IRA may be implicated in job analysis, performance appraisal, panel interviews, and any other approach to gathering systematic observations. Any rating system involving subject-matter experts can also benefit from IRA as a measure of consensus. Further, IRA is fundamental to aggregation in multilevel research, which is becoming increasingly common in order to address nesting. Although, several technical descriptions of a few specific IRA statistics exist, this paper aims to provide a tractable orientation to common IRA indices to support application. The introductory overview is written with the intent of facilitating contrasts among IRA statistics by critically reviewing equations, interpretations, strengths, and weaknesses. Statistics considered include rwg, , r′wg, rwg(p), average deviation (AD), awg, standard deviation (Swg), and the coefficient of variation (CVwg). Equations support quick calculation and contrasting of different agreement indices. The article also includes a “quick reference” table and three figures in order to help readers identify how IRA statistics differ and how interpretations of IRA will depend strongly on the statistic employed. A brief consideration of recommended practices involving statistical and practical cutoff standards is presented, and conclusions are offered in light of the current literature.

Social participation following right hemisphere stroke: influence of a cognitive-communication disorder

ABSTRACTBackground: The ability to return to social activities and roles is an important focus of rehabilitation for people affected by stroke. Rehabilitation professionals currently have limited evidence on the impact of a cognitive-communication disorder (CCD) following right hemispheric (RH) stroke on social participation, and how to support return to social activities and roles.Aims: This study describes (1) social participation change as reported by people with RH stroke; (2) compares social participation change across two groups based on the presence or absence of CCD following RH stroke; and (3) compares self and proxy reports of changes in participation.Methods & Procedures: A telephone survey was conducted with randomly selected people with first onset, unilateral RH stroke, and with matched significant others (proxies). The Sydney Psychosocial Reintegration Scale (SPRS-2) was utilised to explore change associated with stroke across Occupational Activities (OA), Interpersonal Relationships (IR) and Independent Living Skills (LS). Survey responses were compared utilising independent t-tests across two groups based on the presence or absence of CCD. Participant-proxy agreement was established using inter-rater correlation coefficient.Outcomes & Results: Change in social participation was reported for thirty-six participants with an average age of 65.95 years (SD = 13.09), of whom 58.3% were male and 38.9% employed at the time of the stroke. Time post-stroke to completion of the SPRS-2 ranged from six to 33 months. Most participants (94.4%, n = 34) reported change in at least one of the SPRS-2 domains. The presence of CCD had an impact on social participation as measured on the SPRS-2, which was significantly different to what was reported by participants without a communication impairment post-RH stroke (p = 0.02.) Inter-rater agreement statistic (Kappa) indicated a fair participant-proxy dyad agreement in the group with CCD, and a moderate agreement in the group without CCD across the SPRS-2 domains.Conclusions: Changes in social participation following RH stroke occur across Occupational Activities, Interpersonal Relationships and Independent Living Skills for the majority of people and occur with greater frequency and degree where a CCD is present. This exploratory study highlights the importance of rehabilitation goals that address social participation to reduce potential social isolation in people with CCD post-RH stroke. Further identification of risk factors for social participation restriction in this population is required to better inform rehabilitation timing and focus.

Interrater Agreement for Consensus Definitions of Delayed Ischemic Events After Aneurysmal Subarachnoid Hemorrhage.

Thirty percent of patients with subarachnoid hemorrhage experience delayed cerebral ischemia or delayed ischemic neurologic decline (DIND). Variability in the definitions of delayed ischemia makes outcome studies difficult to compare. A recent consensus statement advocates standardized definitions for delayed ischemia in clinical trials of subarachnoid hemorrhage. We sought to evaluate the interrater agreement of these definitions. Based on consensus definitions, we assessed for: (1) delayed cerebral infarction, defined as radiographic cerebral infarction; (2) DIND type 1 (DIND1), defined as focal neurologic decline; and (3) DIND2, defined as a global decline in arousal. Five neurologists retrospectively reviewed electronic records of 58 patients with subarachnoid hemorrhage. Three reviewers had access to and reviewed neuroradiology imaging. We assessed interrater agreement using the Gwet kappa statistic. Interrater agreement statistics were excellent (95.83%) for overall agreement on the presence or absence of any delayed ischemic event (DIND1, DIND2, or delayed cerebral infarction). Agreement was "moderate" for specifically identifying DIND1 (56.58%) and DIND2 (48.66%) events. We observed greater agreement for DIND1 when there was a significant focal motor decline of at least 1 point in the motor score. There was fair agreement (39.20%) for identifying delayed cerebral infarction; CT imaging was the predominant modality. Consensus definitions for delayed cerebral ischemia yielded near-perfect overall agreement and can thus be applied in future large-scale studies. However, a strict process of adjudication, explicit thresholds for determining focal neurologic decline, and MRI techniques that better discriminate edema from infarction seem critical for reproducibility of determination of specific outcome phenotypes, and will be important for successful clinical trials.

Interrater agreement for Critical Care EEG Terminology.

The interpretation of critical care electroencephalography (EEG) studies is challenging because of the presence of many periodic and rhythmic patterns of uncertain clinical significance. Defining the clinical significance of these patterns requires standardized terminology with high interrater agreement (IRA). We sought to evaluate IRA for the final, published American Clinical Neurophysiology Society (ACNS)-approved version of the critical care EEG terminology (2012 version). Our evaluation included terms not assessed previously and incorporated raters with a broad range of EEG reading experience. After reviewing a set of training slides, 49 readers independently completed a Web-based test consisting of 11 identical questions for each of 37 EEG samples (407 questions). Questions assessed whether a pattern was an electrographic seizure; pattern location (main term 1), pattern type (main term 2); and presence and classification of eight other key features ("plus" modifiers, sharpness, absolute and relative amplitude, frequency, number of phases, fluctuation/evolution, and the presence of "triphasic" morphology). IRA statistics (κ values) were almost perfect (90-100%) for seizures, main terms 1 and 2, the +S modifier (superimposed spikes/sharp waves or sharply contoured rhythmic delta activity), sharpness, absolute amplitude, frequency, and number of phases. Agreement was substantial for the +F (superimposed fast activity) and +R (superimposed rhythmic delta activity) modifiers (66% and 67%, respectively), moderate for triphasic morphology (58%), and fair for evolution (21%). IRA for most terms in the ACNS critical care EEG terminology is high. These terms are suitable for multicenter research on the clinical significance of critical care EEG patterns. A PowerPoint slide summarizing this article is available for download in the Supporting Information section http://dx.doi.org/10.1111/epi.12653/supinfo.

Organizational readiness for implementing change: a psychometric assessment of a new measure

BackgroundOrganizational readiness for change in healthcare settings is an important factor in successful implementation of new policies, programs, and practices. However, research on the topic is hindered by the absence of a brief, reliable, and valid measure. Until such a measure is developed, we cannot advance scientific knowledge about readiness or provide evidence-based guidance to organizational leaders about how to increase readiness. This article presents results of a psychometric assessment of a new measure called Organizational Readiness for Implementing Change (ORIC), which we developed based on Weiner’s theory of organizational readiness for change.MethodsWe conducted four studies to assess the psychometric properties of ORIC. In study one, we assessed the content adequacy of the new measure using quantitative methods. In study two, we examined the measure’s factor structure and reliability in a laboratory simulation. In study three, we assessed the reliability and validity of an organization-level measure of readiness based on aggregated individual-level data from study two. In study four, we conducted a small field study utilizing the same analytic methods as in study three.ResultsContent adequacy assessment indicated that the items developed to measure change commitment and change efficacy reflected the theoretical content of these two facets of organizational readiness and distinguished the facets from hypothesized determinants of readiness. Exploratory and confirmatory factor analysis in the lab and field studies revealed two correlated factors, as expected, with good model fit and high item loadings. Reliability analysis in the lab and field studies showed high inter-item consistency for the resulting individual-level scales for change commitment and change efficacy. Inter-rater reliability and inter-rater agreement statistics supported the aggregation of individual level readiness perceptions to the organizational level of analysis.ConclusionsThis article provides evidence in support of the ORIC measure. We believe this measure will enable testing of theories about determinants and consequences of organizational readiness and, ultimately, assist healthcare leaders to reduce the number of health organization change efforts that do not achieve desired benefits. Although ORIC shows promise, further assessment is needed to test for convergent, discriminant, and predictive validity.

Interrater agreement statistics with skewed data: evaluation of alternatives to Cohen's kappa.

In this study, we aimed to evaluate interrater agreement statistics (IRAS) for use in research on low base rate clinical diagnoses or observed behaviors. Establishing and reporting sufficient interrater agreement is essential in such studies. Yet the most commonly applied agreement statistic, Cohen's κ, has a well known sensitivity to base rates that results in a substantial penalization of interrater agreement when behaviors or diagnoses are very uncommon, a prevalent and frustrating concern in such studies. We performed Monte Carlo simulations to evaluate the performance of 5 of κ's alternatives (Van Eerdewegh's V, Yule's Y, Holley and Guilford's G, Scott's π, and Gwet's AC₁), alongside κ itself. The simulations investigated the robustness of these IRAS to conditions that are common in clinical research, with varying levels of behavior or diagnosis base rate, rater bias, observed interrater agreement, and sample size. When the base rate was 0.5, each IRAS provided similar estimates, particularly with unbiased raters. G was the least sensitive of the IRAS to base rates. The results encourage the use of the G statistic for its consistent performance across the simulation conditions. We recommend separately reporting the rates of agreement on the presence and absence of a behavior or diagnosis alongside G as an index of chance corrected overall agreement.

Supplemental Material for Interrater Agreement Statistics With Skewed Data: Evaluation of Alternatives to Cohen’s Kappa

Supplemental Material for Interrater Agreement Statistics With Skewed Data: Evaluation of Alternatives to Cohen’s Kappa

Statistical significance criteria for the rWG and average deviation interrater agreement indices.

Despite the widespread use of interrater agreement statistics for multilevel modeling and other types of research, the existing guidelines for inferring the statistical significance of interrater agreement are quite limited. They are largely relevant only under conditions that numerous researchers have argued rarely exist. Here we address this problem by generating guidelines for inferring statistical significance under a number of conditions via a computer simulation. As a set, these guidelines cover many of the conditions researchers commonly face. We discuss how researchers can use the guidelines presented to more reasonably infer the statistical significance of interrater agreement relative to using the limited guidelines available in the extant literature.