Interrater Kappa Coefficients Research Articles

Intrarater and Inter-rater Reliability of Tibial Plateau Fracture Classifications: Systematic Review and Meta-Analysis.

The interobserver and intraobserver reliability of various tibial plateau fracture (TPF) classifications has been examined in recent literature using radiography, computed tomography, and magnetic resonance imaging. The question remains as to which classification system provides the highest reliability. In this systematic review, we are going to evaluate the overall interobserver and intraobserver reliability of various TPF classifications in different imaging modalities. We conducted a systematic review following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. In February 2023, predefined terms were used for database search (Embase, PubMed, Scopus, Cochrane, and Web of Science). Meta-analysis of intrarater and inter-rater kappa coefficients was performed for each of the classifications in each modality. Thirty-four studies were included in this review. Schatzker's classification was more frequently used than others. It had a better intrarater kappa coefficient than the Hohl and Moore and Arbeitsgemeinschaft für Osteosynthesefragen/Orthopedic Trauma Association (AO/OTA) classifications in radiography (κ = 0.72, 95% confidence interval [CI] = 0.67-0.76, p < 0.01). The Schatzker and AO/OTA classifications had similar inter-rater reliability in the radiography modality (κ = 0.53, 95% CI = 0.51-0.54, p < 0.01; κ = 0.53, 95% CI = 0.5-0.55, p < 0.01; respectively). In 3-dimensional computed tomography, the Luo classification system showed the highest intrarater (κ = 0.85, 95% CI = 0.35-0.66) and inter-rater (κ = 0.77, 95% CI = 0.73-0.81) kappa coefficients. Three-column classification proposed by Luo et al. was able to reach the highest degree and was the only classification with near-excellent inter-rater reliability.

Catégorisation de la densité mammaire et dépistage organisé du cancer du sein

The objective of this study was to evaluate the intra- and inter-rater agreement of radiologists regarding the evaluation of breast density. Breast density assessments of 120 cases were performed by four radiologists in the city of Ouagadougou according to the fifth edition of the American College of Radiology BI-RADS atlas. Cohen's weighted kappa coefficients and Fleiss kappa coefficients were used to estimate agreement between observers and with a panel of three experts radiologists. A new evaluation of the 120 cases was performed by all raters one month after the initial evaluation. Inter-rater kappa coefficients ranged from 0.55 to 0.74. The Fleiss kappa coefficient was 0.58, 0.43, 0.41, and 0.43 for categories A, B, C, and D respectively. In terms of classification into "sparse breasts" and "dense breasts", the kappa coefficients ranged from 0.47 to 0.67. Taking the results of the expert panel as a reference, the proportion of false positives in the diagnosis "dense breasts" ranged from 18.6% to 26.8%. Intraobserver agreement was good. Our study showed moderate to good intra- and inter-raters agreements. Upgrading and harmonisation of practices will be used to empower radiologists to participate in organised breast cancer screening in Burkina Faso.

No Difference in Two-Dimensional Kinematic Assessment of a 45-Degree Sidestep Cut Compared to Qualitative Assessment.

The Expanded Cutting Alignment Scoring Tool (E-CAST) is a two-dimensional qualitative scoring system that has demonstrated moderate inter-rater and good intra-rater reliability for the assessment of trunk and lower extremity alignment during a 45-degree sidestep cut. The primary purpose of this study was to examine the reliability of the quantitative version of the E-CAST among physical therapists and to compare the reliability of the quantitative E-CAST to the original qualitative E-CAST. The hypothesis was that the quantitative version of the E-CAST would demonstrate greater inter-rater and intra-rater reliability compared to the qualitative E-CAST. Observational cohort, repeated measures reliability study. Twenty-five healthy female athletes (age 13.8±1.4 years) performed three sidestep cuts with two-dimensional video capturing frontal and sagittal views. Two physical therapist raters independently scored a single trial using both views on two separate occasions. Based on the E-CAST criteria, select kinematic measurements were extracted using a motion analysis phone application. Intraclass correlation coefficients and 95% confident intervals were calculated for the total score, and kappa coefficients were calculated per kinematic variable. Correlations were converted to z-scores and compared to the six original criteria for significance (α<0.05). Cumulative intra- and inter-rater reliability were both good (ICC=0.821, 95% CI 0.687-0.898 and ICC=0.752, 95% CI 0.565-0.859). Cumulative intra-rater kappa coefficients ranged from moderate to almost perfect, and cumulative inter-rater kappa coefficients ranged from slight to good. No significant differences were observed between the quantitative and qualitative criteria for either inter- or intra-rater reliability (Zobs(intra)= -0.38, p=0.352 and Zobs(inter)= -0.30, p=0.382). The quantitative E-CAST is a reliable tool to assess trunk and lower extremity alignment during a 45-degree sidestep cut. No significant differences were observed in reliability of the quantitative versus qualitative assessment. 3b.

Inter- and Intrarater Reliability and Agreement Among Danish Head and Neck Pathologists Assessing Extranodal Extension in Lymph Node Metastases from Oropharyngeal Squamous Cell Carcinomas.

Extranodal extension (ENE) in lymph node metastases is one of the most important prognostic factors in head and neck squamous cell carcinomas. Studies have shown inconsistency among pathologists in the assessment of ENE. The aims of this study were: (1) to determine the interrater and intrarater reliability and agreement in the assessment of ENE among Danish pathologists and (2) to test if a standardized assessment method may increase interrater agreement. Four Danish head and neck pathologists assessed ENE presence or absence in 120 histological slides from lymph nodes with oropharyngeal squamous cell carcinoma metastases (first round). Subsequently, guidelines were introduced to the pathologists and a new assessment was performed (second round). Finally, two of the pathologists assessed the slides to determine intrarater reliability and agreement (third round). Interrater kappa coefficients varied between 0.57 and 0.67 in the first round and between 0.59 and 0.72 in the second round. The intrarater agreement between round 2 and 3 was 0.88 for pathologist 1 and 0.92 for pathologist 2 with resulting kappa coefficients of 0.76 (95% CI 0.64-0.88) and 0.84 (95% CI 0.74-0.94), respectively. We found a moderate level of reliability and agreement among pathologists for ENE in lymph node metastases from oropharyngeal squamous cell carcinomas. The intrarater reliability and agreement was generally higher than interrater measures. Interrater agreement was slightly improved by standardized assessment.

The Reliability of Two Dimensional Kinematic Assessment of a 45-Degree Side Step Cut

Background:Up to 70% of all anterior cruciate ligament (ACL) injuries occur via a non-contact mechanism involving deceleration and direction change on a planted foot. Clinical screening tools analyzing movement patterns during a cutting task are reliable and valid. The Expanded Cutting Alignment Scoring Tool (E-CAST) is a reliable tool to qualitatively assess trunk and lower extremity (LE) alignment during a 45-degree side step cut (SSC). Previous works found two-dimensional (2D) kinematic analysis more reliable than qualitative assessments when evaluating jumping and squatting movements. However, there is limited evidence on the reliability of 2D assessments to evaluate SSC tasks.Hypothesis/Purpose:To determine if using 2D kinematic analysis with the E-CAST improves the tool’s reliability for evaluating trunk and LE alignment during a 45-degree SSC.Methods:25 healthy females (age 13.8 ± 1.4 years, mass 52.4 ± 9.3 kg, height 161.7 ± 6.0 cm) who regularly participated in cutting or pivoting sports were included. A repeated measures study design was used. Each subject performed three SSC trials with one trial randomly selected for analysis. Trials were recorded in the frontal and sagittal planes. Two physical therapist raters independently viewed and scored each video on two separate occasions. Videos orders were randomized and a two-week wash out period occurred between rounds. The E-CAST evaluates six movement variables (Figure 1). The assessment criteria for each variable was adapted to utilize the Dartfish motion analysis application on a smart phone (Figure 2). If a movement variable falls outside of the motion analysis definition, the subject scores a point, therefore a higher score represents poorer technique. Intraclass correlation coefficients (ICC) and 95% confident intervals (95% CI) were calculated for the E-CAST total score, and a kappa coefficient was calculated for each variable. Correlations were converted to z scores and compared for significance.Results:The cumulative intra-rater reliability was good (ICC= 0.821, 95% CI 0.687 – 0.898) and the cumulative inter-rater reliability was moderate (ICC= 0.752, 95% CI 0.565-0.859). Intra-rater kappa coefficients ranged from moderate to almost perfect for all variables (k= 0.505-0.875) and inter-rater kappa coefficients ranged from slight to good for all variables (k=0.336-0.733).Conclusion:The use of 2D kinematic analysis resulted in higher intra-and inter-rater reliability compared to the traditional qualitative E-CAST (ICC: 0.77, 95% CI 0.59-0.96 and ICC: 0.71, 95% CI 0.50-0.91), however, the difference was not significant (Zobs = -0.46 and Zobs = -0.30).

Reliability of the Expanded Cutting Alignment Scoring Tool (E-CAST) to Assess Trunk and Limb Alignment During a 45-Degree Side-Step Cut.

BackgroundCurrent clinical screening tools assessing risky movements during cutting maneuvers do not adequately address sagittal plane foot and ankle evaluations. The Cutting Alignment Scoring Tool (CAST) is reliable in evaluating frontal plane trunk and lower extremity alignment during a 45-degree side-step cut. The Expanded Cutting Alignment Scoring Tool (E-CAST) includes two new sagittal plane variables, knee flexion and ankle plantarflexion angle.Hypothesis/PurposeTo assess the inter-and intra-rater reliability of the E-CAST to evaluate trunk and lower extremity alignment during a 45-degree side-step cut.Study DesignRepeated MeasuresMethodsParticipants included 25 healthy females (13.8 ± 1.4 years) regularly participating in cutting or pivoting sports. Participants were recorded performing a side-step cut in frontal and sagittal planes. One trial was randomly selected for analysis. Two physical therapists independently scored each video using the E-CAST on two separate occasions, with randomization and a two-week wash-out between rounds. Observed movement variables were awarded a score of “1”, with higher scores representing poorer technique. Intraclass correlation coefficients (ICC) and 95% confident intervals (95% CI) were calculated for the total score, and a kappa coefficient (k) was calculated for each variable.ResultsThe cumulative intra-rater reliability was good (ICC=0.78, 95% CI 0.59-0.96) and the cumulative inter-rater reliability was moderate (ICC=0.71, 95% CI 0.50-0.91). Intra-rater kappa coefficients ranged from moderate to excellent for all variables (k= 0.50-0.84) and inter-rater kappa coefficients ranged from slight to excellent for all variables (k=0.20-0.90).ConclusionThe addition of two sagittal plane variables resulted in lower inter-rater ICC compared to the CAST (ICC= 0.81, 95% CI 0.64-0.91). The E-CAST is a reliable tool to evaluate trunk and LE alignment during a 45-degree side-step cut, with good intra-rater and moderate inter-rater reliability.Level of EvidenceLevel 2, Diagnosis

The Norwich Osteoarthritis of the Ankle MRI Score (NOAMS): a reliability study

To define and test the inter- and intra-rater reliability of a grading system for staging osteoarthritis (OA) of the ankle with magnetic resonance imaging (MRI) (Norwich Osteoarthritis of the Ankle MRI Score, NOAMS). The MRI features to be included in the score were defined by a multidisciplinary expert panel through a Delphi process. An anonymised randomised dataset of 50 MRI studies was created from patients with concurrent plain radiographs to include 10 ankles of each of the Kellgren-Lawrence grades 0 to 4. Two experienced musculoskeletal radiologists and two trainees scored each ankle MRI twice independently and blinded to the plain radiographs. The inter-rater kappa coefficient of agreement for cartilage disease was 0.88 (95% confidence interval [CI]: 0.85, 0.91) for experienced raters and 0.71 (95% CI: 0.67, 0.76) for trainees. Inter-rater agreement for subchondral bone marrow oedema and cysts varied from 0.73 to 0.82 for experienced raters and from 0.63 to 0.75 for trainees with lowest 95% CI of 0.48 and 0.63. When bone marrow lesions were combined into a total joint score the level of agreement increased to between 0.88 and 0.97 with lowest 95% CI of 0.86. Combining cartilage zone scores did not increase the reliability coefficients. An expert panel considered that cartilage degradation and subchondral bone marrow lesions were the most important features for staging the severity of ankle OA on MRI. Experienced observers can grade the severity of ankle OA on MRI with a clinically useful high degree of reproducibility.

Venous tortuosity as a novel biomarker of rupture risk in arteriovenous malformations: ARI score

BackgroundRisk of rupture in arteriovenous malformations (AVMs) varies considerably among series. Hemodynamic factors, especially within the venous side of the circuit, seem to be responsible but are not yet well...

Comprehensive classification system for multirod constructs across three-column osteotomies: a reliability study.

In this study, the authors' goal was to determine the intra- and interobserver reliability of a new classification system that allows the description of all possible constructs used across three-column osteotomies (3COs) in terms of rod configuration and density. Thirty-five patients with multirod constructs (MRCs) across a 3CO were classified by two spinal surgery fellows according to the new system, and then were reclassified 2 weeks later. Constructs were classified as follows: the number of rods across the osteotomy site followed by a letter corresponding to the type of rod configuration: "M" is for a main rod configuration, defined as a single rod spanning the osteotomy. "L" is for linked rod configurations, defined as 2 rods directly connected to each other at the osteotomy site. "S" is for satellite rod configurations, which were defined as a short rod independent of the main rod with anchors above and below the 3CO. "A" is for accessory rods, defined as an additional rod across the 3CO attached to main rods but not attached to any anchors across the osteotomy site. "I" is for intercalary rod configurations, defined as a rod connecting 2 separate constructs across the 3CO, without the intercalary rod itself attached to any anchors across the osteotomy site. The intra- and interobserver reliability of this classification system was determined. A sample estimation for validation assuming two readers and 35 subjects results in a two-sided 95% confidence interval with a width of 0.19 and a kappa value of 0.8 (SD 0.3). The Fleiss kappa coefficient (κ) was used to calculate the degree of agreement between interrater and intraobserver reliability. The interrater kappa coefficient was 0.3, and the intrarater kappa coefficient was 0.63 (good reliability). This scenario represents a high degree of agreement despite a low kappa coefficient. Correct observations by both observers were 34 of 35 and 33 of 35 at both time points. Misclassification was related to difficulty in determining connectors versus anchors. MRCs across 3COs have variable rod configurations. Currently, no classification system or agreement on nomenclature exists to define the configuration of rods across 3COs. The authors present a new, comprehensive MRC classification system with good inter- and intraobserver reliability and a high degree of agreement that allows for a standardized description of MRCs across 3COs.

Reliability of CSF turbulence and choroid plexus visualization on fast-sequence MRI in pediatric hydrocephalus.

OBJECTIVE Endoscopic third ventriculostomy with choroid plexus cauterization for the treatment of neonatal and infant hydrocephalus has gained popularity in the past decade. Identifying treatment failure is critically important. Results of a pilot study of 2 novel imaging markers seen on fast-sequence T2-weighted axial MRI showed potential clinical utility. However, the reliability of multiple raters detecting these markers must be established before a multicenter validation study can be performed. METHODS Two sets of de-identified single-shot T2-weighted turbo spin-echo axial images were prepared from scans of patients before and after they underwent endoscopic third ventriculostomy with choroid plexus cauterization between March 2013 and January 2016. The first set showed the lateral and third ventricles for visualization of turbulent CSF dynamics, and the second set showed the lateral ventricular atria for choroid plexus glomus detection. Three raters (Group 1) received written instructions before evaluating each image set once and then again 1 week later. Another 8 raters (Group 2) evaluated both image sets after oral instruction and group training on a pretest image set. Fleiss' kappa coefficients with 95% CIs were calculated for intrarater and interrater reliability in Group 1 and interrater reliability in Group 2. RESULTS Intrarater reliability kappa coefficients for Group 1 were ≥ 0.74 for turbulence and ≥ 0.80 for choroid plexus; their interrater kappa coefficients at the initial assessment were 0.50 (95% CI 0.37-0.62) and 0.56 (95% CI 0.43-0.69), respectively. The Group 2 interrater kappa scores were 0.82 (95% CI 0.78-0.86) for turbulence and 0.62 (95% CI 0.58-0.66) for choroid plexus. CONCLUSIONS With minimal training, intrarater reliability on visualization of turbulence and the choroid plexus was substantial, but interrater reliability was only moderate. After modestly increasing training, interrater reliability improved to near perfect and to substantial reliability for visualization of turbulence and choroid plexus, respectively. Given adequately trained observers, a multicenter investigation into the validity and potential clinical utility of the imaging markers seems feasible.

Comparative performance of three TSE rapid tests for surveillance in healthy sheep affected by scrapie

Rapid tests specific for sheep and goats became part of European Union-wide active scrapie surveillance in 2006. Performance of three approved TSE rapid tests for the detection of sheep infected with scrapie in field cases in the pre-clinical stage of the disease was compared.The medulla oblongata of 969 asymptomatic sheep of various genotype and breed aged over 18 months from 23 Italian flocks affected with scrapie, were tested by the Bio-Rad TeSeE Sheep/Goat (A), the IDEXX HerdChek BSE-Scrapie Antigen Test Kit, EIA (B) and the Prionics®-Check Western Small Ruminant (C) rapid tests.Of 136 positive samples of classical scrapie, as confirmed by Western blot assay, 132 were positive with test A (Se 97.06%, CI 95% 92.64–99.19); 135 with test B (Se 99.26%, 95% CI 95.97–99.98) and 128 with test C (Se 94.12%, 95% CI 88.74–97.43).Tests A and B showed the best performance on analytical sensitivity. All three systems demonstrated good reproducibility: being the intrarater and interrater kappa coefficients always over 0.83.The one available atypical scrapie sample was positive with tests A and B, negative with test C.Considering the discrepant results in the detection of low PrPsc concentrations and of the atypical case, differences can be expected in the efficacy of an active surveillance system, depending on the test adopted.

Schedule for affective disorders and schizophrenia for school-age children (K-SADS-PL) for the assessment of preschool children – A preliminary psychometric study

Objective To assess the psychometrics of the schedule for affective disorders and schizophrenia for school-age children present and lifetime version (K-SADS-PL) in diagnosing DSM-IV psychiatric disorders and subsyndromal symptomatology in preschool children. Method Parents were interviewed about their children using the K-SADS-PL, and they completed the early childhood inventory-4 (ECI-4) and child behavior checklist for ages 1½–5 years (CBCL). Discriminant, divergent, and convergent validity of the K-SADS-PL were evaluated in 204 offspring ages 2–5 years old of parents from an ongoing study. Inter-rater reliability as well as predictive validity of intake diagnoses at second assessment approximately two years after intake were evaluated. Fourteen children were also assessed by the preschool age psychiatric assessment (PAPA). Results Children who were diagnosed with oppositional defiant disorder, attention deficit hyperactivity disorder, anxiety, mood, or elimination disorders had significantly higher scores on the ECI-4 than children without these disorders. Significant correlations were found for all convergent CBCL scales. Divergent validity was acceptable for emotional disorders. Inter-rater kappa coefficients for all diagnoses were good. Above noted results were similar for children with at least one positive K-SADS-PL key screen symptom. A significantly higher percentage of children with an intake diagnosis had a diagnosis approximately two years after intake compared to those without an intake disorder. Overall, there was consistency between the PAPA and the K-SADS-PL. Conclusions Pending further testing, the K-SADS-PL may prove useful for the assessment of psychopathology in preschoolers.

Failures in a Proton Pump Inhibitor Therapeutic Substitution Program: Lessons Learned

The pathogenesis of patient dissatisfaction following involuntary therapeutic substitutions involving proton pump inhibitors (PPIs) is poorly understood. The aim of this study was to describe the patient population experiencing therapeutic failure and investigate whether failure was related to individual differences in response to the different PPIs. Treatment failures in a lansoprazole-rabeprazole therapeutic substitution program were compared to switch successes. A subgroup was randomized in a double-blind, double-dummy, crossover study to four 2-week periods of lansoprazole-rabeprazole-lansoprazole-rabeprazole or vice versa. Measures included overall rating of gastrointestinal reflux disease (GERD) symptoms for the past week as well as the frequency and distress scales of the GERD Symptom Assessment Scale. One hundred fifteen nonresponders were compared with 54 successful responders. Nonresponders consisted primarily of patients with GERD (74%, vs. 44% of responders; P = 0.005) who had undergone upper gastrointestinal endoscopy (50%, vs. 31% of responders; P = 0.02). Twelve patients completed the randomized treatment study. The interrater kappa coefficient for responder status was estimated to be 0.80 for lansoprazole and 0.21 for rabeprazole. The majority of PPI nonresponders had a clinical diagnosis of GERD and were receiving >/=40 mg of rabeprazole daily. This pilot study provides new insights into the design of subsequent studies of nonresponders in PPI therapeutic substitution programs.

1160 Retrospective assessment of performance status (PS)–accuracy and relation to survival in patients (PTS) with urinary transitional cell cancer (TCC)

The aim of the study was to evaluate the reliability and validity of retrospective assessment of PS in pts with TCC. The clinical records of 149 pts with primary and 53 pts with recurrent TCC were scored twice by the two authors according to the WHO PS scale and related to survival. The results were analyzed in a Cox multivariate model together with other potential prognostic factors for survival. The median survival was 20.4 months; 13 pts were alive at follow up. The intra- and interrater Kappa coefficients of the PS scores were 63–72% and 49–68%. The four assessments of PS were all significantly related to survival. Multivariate analyses showed that sex, ASAT, and each of the four PS scores was an independent prognostic factor. Retrospective scoring of PS is a reproducible and reliable method. Optimal retrospective studies of the effect of multiple prognostic factors should include an assessment of the functional status of the pts.

The Research Diagnostic Criteria: Historical Background, Development, Validity, and Reliability*

This article reviews the theoretical papers and research studies concerning the Research Diagnostic Criteria (RDC), a research tool used to group psychiatric sugjects into homogenous diagnostic groups. The RDC contains specified inclusion and exclusion criteria for 25 diagnostic categories. The contributions made by the Feighner and Spitzer groups in the field of diagnostic reliability and validity are briefly outlined. The development of the RDC and studies regarding its reliability and validity are presented and discussed. A review of the reliability studies, which basically employed an interrater design, indicated that the RDC is a relatively reliable diagnostic tool, obtaining interrater kappa coefficients which are consistently higher than those obtained, for similar categories, in studies using other diagnostic systems such as the DSM-II. Very few studies, however, directly investigated the validity of the RDC. Its assumed validity is dependent on the findings from studies which used similar diagnostic criteria. It was suggested that more studies would be required to establish the validity of the RDC. Several limitations of the RDC were pointed out. Included among these were: its use of an exclusion method to determine diagnostic criteria, the difficulty and great amount of time required for its use, and the fact that it has so far been used only with adult inpatients. Despite its limitations, it was concluded that the RDC remains one of the best tools available to a researcher who wishes to group psychiatric patients into homogenous groups, and that the clinician could advantageously use it in his clinical practice.