Quadratic Weighted Kappa Statistics Research Articles

Predicting the pathological invasiveness of early lung adenocarcinoma prior to surgery using Deauville criteria: reliability and validity

PurposeThis retrospective study aimed to investigate the validity and reliability of FDG-PET/CT visual assessment using Deauville criteria to predict pathological invasiveness of early lung adenocarcinoma prior to surgery.Materials and methodsBetween April 2020 and January 2022, 51 patients who underwent surgery for pathological stage 0/I lung adenocarcinoma were enrolled. The pulmonary lesions were divided into two groups according to pathological invasiveness: less invasive (including adenocarcinoma in situ and minimally invasive adenocarcinoma and invasive adenocarcinoma. We compared CT size (total and solid size), SUVmax, and Deauville score between the two groups. Furthermore, we investigated inter-rater and intra-rater agreements regarding the Deauville score. Receiver operating characteristic (ROC) curve analysis was performed to identify the diagnostic performance of each method.ResultsBased on pathologic diagnoses, 51 lesions in the 51 patients were divided into 6 less invasive and 45 invasive adenocarcinoma lesions. According to quadratic-weighted Kappa statistics, inter-rater (k = 0.93) and intra-rater (k = 0.97) agreements among all five components of the Deauville score indicated high agreement. There was a statistically significant difference in CT solid size, SUVmax, and Deauville score between the two groups. There were no significant differences between CT solid size and FDG-PET/CT assessments (AUC = 0.93 for Deauville score and SUVmax, AUC = 0.84 for CT solid size).ConclusionFDG-PET/CT visual assessment using the Deauville score could assist in deciding upon minimally invasive surgery for early lung adenocarcinoma.

LO81: Interrater agreement and time it takes to assign a Canadian Triage and Acuity Scale score pre and post implementation of eCTAS

Introduction: In addition to its clinical utility, the Canadian Triage and Acuity Scale (CTAS) has become an administrative metric used by governments to estimate patient care requirements, emergency department (ED) funding and workload models. The electronic Canadian Triage and Acuity Scale (eCTAS) initiative aims to improve patient safety and quality of care by establishing an electronic triage decision support tool that standardizes that application of national triage guidelines across Ontario. The objective of this study was to evaluate triage times and score agreement in ED settings where eCTAS has been implemented. Methods: This was a prospective, observational study conducted in 7 hospital EDs, selected to represent a mix of triage processes (electronic vs. manual), documentation practices (electronic vs. paper), hospital types (rural, community and teaching) and patient volumes (annual ED census ranged from 38,000 to 136,000). An expert CTAS auditor observed on-duty triage nurses in the ED and assigned independent CTAS in real time. Research assistants not involved in the triage process independently recorded triage time. Interrater agreement was estimated using unweighted and quadratic-weighted kappa statistics with 95% confidence intervals (CIs). Results: 1491 (752 pre-eCTAS, 739 post-implementation) individual patient CTAS assessments were audited over 42 (21 pre-eCTAS, 21 post-implementation) seven-hour triage shifts. Exact modal agreement was achieved for 567 (75.4%) patients pre-eCTAS, compared to 685 (92.7%) patients triaged with eCTAS. Using the auditor's CTAS score as the reference standard, eCTAS significantly reduced the number of patients over-triaged (12.0% vs. 5.1%; Δ 6.9, 95% CI: 4.0, 9.7) and under-triaged (12.6% vs. 2.2%; Δ 10.4, 95% CI: 7.9, 13.2). Interrater agreement was higher with eCTAS (unweighted kappa 0.89 vs 0.63; quadratic-weighted kappa 0.91 vs. 0.71). Research assistants captured triage time for 3808 patients pre-eCTAS and 3489 post implementation of eCTAS. Median triage time was 312 seconds pre-eCTAS and 347 seconds with eCTAS (Δ 35 seconds, 95% CI: 29, 40 seconds). Conclusion: A standardized, electronic approach to performing CTAS assessments improves both clinical decision making and administrative data accuracy without substantially increasing triage time.

LO70: Interrater agreement and time it takes to assign a Canadian Triage and Acuity Scale score pre and post implementation of eCTAS

Introduction: In addition to its clinical utility, the Canadian Triage and Acuity Scale (CTAS) has become an administrative metric used by governments to estimate patient care requirements, ED funding and workload models. The Electronic Canadian Triage and Acuity Scale (eCTAS) initiative aims to improve patient safety and quality of care by establishing an electronic triage decision support tool that standardizes the application of national triage guidelines (CTAS) across Ontario. The objective of this study was to evaluate the implementation of eCTAS in a variety of ED settings. Methods: This was a prospective, observational study conducted in 7 hospital EDs, selected to represent a mix of triage processes (electronic vs. manual), documentation practices (electronic vs. paper), hospital types (rural, community and teaching) and patient volumes (annual ED census ranged from 38,000 to 136,000). An expert CTAS auditor observed on-duty triage nurses in the ED and assigned independent CTAS in real time. Research assistants not involved in the triage process independently recorded the triage time. Interrater agreement was estimated using unweighted and quadratic-weighted kappa statistics with 95% confidence intervals (CIs). Results: 1200 (738 pre-eCTAS, 462 post-implementation) individual patient CTAS assessments were audited over 33 (21 pre-eCTAS, 11 post-implementation) seven-hour triage shifts. Exact modal agreement was achieved for 554 (75.0%) patients pre-eCTAS, compared to 429 (93.0%) patients triaged with eCTAS. Using the auditors CTAS score as the reference standard, eCTAS significantly reduced the number of patients over-triaged (12.1% vs. 3.2%; 8.9, 95% CI: 5.7, 11.7) and under-triaged (12.9% vs. 3.9%; 9.0, 95% CI: 5.9, 12.0). Interrater agreement was higher with eCTAS (unweighted kappa 0.90 vs 0.63; quadratic-weighted kappa 0.79 vs. 0.94). Research assistants captured triage time for 4403 patients pre-eCTAS and 1849 post implementation of eCTAS. Median triage time was 304 seconds pre-eCTAS and 329 seconds with eCTAS ( 25 seconds, 95% CI: 18, 32 seconds). Conclusion: A standardized, electronic approach to performing CTAS assessments improves both clinical decision making and administrative data accuracy without substantially increasing triage time.

LO75: Interrater agreement and time it takes to assign a Canadian Triage and Acuity Scale score in 7 emergency departments

Introduction: The Canadian Triage and Acuity Scale (CTAS) is the standard used in all Canadian (and many international) emergency departments (EDs) for establishing the priority by which patients should be assessed. In addition to its clinical utility, CTAS has become an important administrative metric used by governments to estimate patient care requirements, ED funding and workload models. Despite its importance, the process by which CTAS scores are derived is highly variable. Emphasis on ED wait times has also drawn attention to the length of time the triage process takes. The primary objective of this study was to determine the interrater agreement of CTAS in current clinical practice. The secondary objective was to determine the time it takes to triage in a variety of ED settings. Methods: This was a prospective, observational study conducted in 7 hospital EDs, selected to represent a mix of triage processes (electronic vs. manual), documentation practices (electronic vs. paper), hospital types (rural, community and teaching) and patient volumes (annual ED census ranged from 38,000 to 136,000). An expert CTAS auditor observed on-duty triage nurses in the ED and assigned independent CTAS in real time. Research assistants not involved in the triage process independently recorded the triage time. Interrater agreement was estimated using unweighted and quadratic-weighted kappa statistics with 95% confidence intervals (CIs). Results: 738 consecutive patient CTAS assessments were audited over 21 seven-hour triage shifts. Exact modal agreement was achieved for 554 (75.0%) patients. Using the auditor’s CTAS score as the reference standard, on-duty triage nurses over-triaged 89 (12.1%) and under-triaged 95 (12.9%) patients. Interrater agreement was “good” with an unweighted kappa of 0.63 (95% CI: 0.58, 0.67) and quadratic-weighted kappa of 0.79 (95% CI: 0.67, 0.90). Research assistants captured triage time for 3808 patients over 69 shifts at 7 different EDs. Median (IQR) triage time was 5.2 (3.8, 7.3) minutes and ranged from 3.9 (3.1, 4.8) minutes to 7.5 (5.8, 10.8) minutes. Conclusion: Variability in the accuracy, and length of time taken to perform CTAS assessments suggest that a standardized approach to performing CTAS assessments would improve both clinical decision making, and administrative data accuracy.

Reliability of Canadian Emergency Department Triage and Acuity Scale (CTAS) in Saudi Arabia.

BackgroundThe Canadian Emergency Department Triage and Acuity Scale (CTAS) is an integral part of the Canadian emergency medicine triaging system. There is growing interest and implementation of CTAS worldwide. However, little is known about its reliability outside Canada. The aim of this study was to determine the reliability agreement of CTAS in a tertiary care emergency center in Saudi Arabia.MethodsTen triage nurses (five senior and five junior nurses) utilized CTAS guidelines to independently assign a triage level for 160 real case-based scenarios. Quadratic weighted kappa statistics were used to measure raters’ agreements.ResultsRaters provided 1600 triage category assignments to case scenarios for analysis. Intra-rater agreement was similar for both senior and junior nurses; for senior nurses (SN1) kappa 0.871 95 % CI (0.840–0.897), and for junior nurses (SN2) kappa 0.871 95 % CI (0.839–0.898). Inter-rater agreement for the SN1 versus SN2 nurses had statistically meaningful agreement across different triage levels (weighted kappa = 0.770) 95 % CI (0.742–0.797).ConclusionsCTAS has good reliability among emergency department (ED) triage nurses in King Abdulaziz Medical City (KAMC), Saudi Arabia. The findings suggest that CTAS might be a reliable instrument when applied in countries outside Canada.

Agreement and association between different indicators of body image and body mass index in adolescents.

The aim of the study was to examine the correlation among different indicators of body image; between each one of these and nutritional status; and the association of these indicators with the Body Mass Index (BMI) of adolescents. A random sample of 152 students from public and private schools in the city of Rio de Janeiro, Brazil, was studied. On four occasions, two silhouette scales and two questions regarding the opinion of the student about his/her body and weight were applied and weight and height were measured. The BMI was examined both as a continuous and as a categorical variable. The agreement between the variables was analyzed using the quadratic weighted Kappa statistics. The association between body image variables and BMI was examined by the comparison among median, mean, standard deviation and 95% confidence interval of BMI for each category of the body image variables. In general, the correlation among the body image variables ranged from reasonable to good; between these and the variable nutritional status, correlation ranged from regular to reasonable. Best results were observed among boys and students from private schools. All body image variables showed good discriminatory power for BMI, when it was analyzed as a continuous variable, even when controlling for potential confounders. The question about body seems to be better than that about weight to compose the questionnaire of a surveillance system for risk and protective factors for adolescent health.

Patient-reported outcomes are associated with patient-oncologist agreement of performance status in a multi-ethnic Asian population.

This study aims to compare the performance status assessed by breast cancer patients and their oncologist and investigate the factors associated with the patient-oncologist disagreement in a multi-ethnic Asian population. This is a cross-sectional study of 270 female breast cancer patients in Singapore. Patients' performance status was rated by the patients and their treating oncologists. Patients were also asked to complete two instruments regarding their health status and health-related quality of life. The proportions of patients with assessment of performance status the same as and different from their oncologist were calculated. The level of agreement was quantified by Kappa statistics together with the confidence interval (CI). Multinomial logistic regression analyses were performed to examine potential factors associated with the patient-oncologist disagreement. Approximately half (136/270) of the patients agreed with the oncologist in assessing performance status. Among the disagreeing half, 48 (17.8 %) patients rated better and 86 (31.9 %) poorer than the oncologists. The unweighted and quadratic-weighted Kappa statistics were, respectively, 0.21 (95 % CI = 0.13 to 0.29) and 0.34 (95 % CI = 0.26 to 0.42). Multivariable multinomial logistic regressions showed that outpatient, language used, and evidence of disease were associated with patients' better assessment than the oncologists, while age, physical well-being, and pain/discomfort were associated with patients' poorer assessment. The patient-oncologist agreement was weak to moderate. When discrepant, patients tended to rate themselves worse than the oncologists assessed. Poorer assessment rated by patients was associated with their physical well-being. Patients who self-rated poorer may require further evaluation of any unobserved symptoms.

Confiabilidade teste-reteste do Índice de Capacidade para o Trabalho (ICT) em trabalhadores de enfermagem

This paper assesses the test retest reliability of the Work Ability Index (WAI) in nursing workers. A self-administered questionnaire was applied twice to a group of 80 workers (nurses and nursing aides/assistants) at a public hospital in Rio de Janeiro, Brazil within an interval from seven to fifteen days. The reliability was estimated using quadratic weighted kappa statistics, interclass correlation coefficient (ICC) and the Bland and Altman plot. Eighty-one percent of participants were women aged between from 22 to 67 years (mean =39.1; SD=10.8 years); 36.3% had completed higher education. The global score of the WAI presented ICC= 0.79 (IC95% 0.67 to 0.86) and weighted kappa=0.69 (CI95% 0.50 to 0.80) for categorical WAI (classified as low, moderate, good and excellent). The quadratic weighted kappa of the WAI items ranged from 0.39 to 0.82 and the Bland and Altman plot did not show a systematic pattern. The agreement between the test and retest measures shows an acceptable degree of reliability, suggesting the adequacy of the assessment process among nursing workers.

Time‐resolved MR venography of the pulmonary veins precatheter‐based ablation for atrial fibrillation

To evaluate time-resolved magnetic resonance angiography (TR-MRA) of the pulmonary venous circulation using the time-resolved angiography with interleaved stochastic trajectories (TWIST) method and compare it with the more commonly used conventional contrast-enhanced magnetic resonance angiography (CE-MRA) approach in atrial fibrillation patients referred for preablation pulmonary vein mapping. This study was approved by the Institutional Review Board. Twenty-six patients (15 males; age 59.6 ± 12.7 years) referred for preablation pulmonary vein mapping underwent both conventional CE-MRA and TR-MRA with TWIST. Imaging was performed on a 1.5 T (Magnetom Avanto, Siemens Healthcare, Erlangen, Germany) MRI scanner. Source partition and maximum intensity projection (MIP) images were evaluated retrospectively. For quantitative analysis, pulmonary vein ostium orthogonal dimensions were measured using double oblique multiplanar reformatting. The results were analyzed using paired t-tests, Lin's concordance correlation coefficient, and Bland-Altman plots. For qualitative analysis, both source partition images and MIP images were assessed by two observers (A.P. and M.G.). The presence of common ostiums or accessory veins was recorded and analyzed using unweighted Cohen's kappa. Pulmonary vein conspicuity was scored on a scale of 1-4 (1 = poor, 2 = fair, 3 = good, 4 = excellent) and analyzed using paired t-tests, intraclass correlation coefficients, and quadratic weighted kappa statistics. Orthogonal venous diameters were comparable for both TR-MRA and conventional CE-MRA (1.34 ± 0.37 vs. 1.38 cm ± 0.36, respectively). Results of paired t-tests, Lin's concordance correlation coefficient, and Bland-Altman analysis revealed relatively close comparison between methods. The magnitude of the mean difference for any of the statistical comparisons did not exceed 0.10 cm. The visualization of variant pulmonary vein anatomy was very similar for both techniques. Agreement between techniques was determined to be "good" to "very good" (κ = 0.78-0.85). Conspicuity scores for each pulmonary vein were also very close. Paired t-tests, intraclass correlation coefficients, and quadratic weighted kappa statistics all revealed strong agreement between methods. TR-MRA using TWIST produces comparable anatomic images and pulmonary venous dimensions to the more widely used CE-MRA technique. Additionally, the TWIST technique improves arterio-venous separation, does not need exact bolus timing, requires less gadolinium, and gives additional information on vein perfusion.

Intra-rater reliability of the Modified Modified Ashworth Scale (MMAS) in the assessment of upper-limb muscle spasticity

The aim of this study was to investigate the intra-rater reliability of the Modified Modified Ashworth Scale (MMAS) in the upper limb of patients with hemiparesis and to determine the effect of pain and contracture presence on the reliability of the MMAS. For this test-retest study 30 patients with hemiparesis were included. One physiotherapist using the MMAS, randomly rated the spasticity of shoulder adductors, elbow flexors, and wrist flexors in the affected upper limb of each patient with hemiparesis twice with at least a 1 week interval between testing sessions. The presence of pain and contracture during passive stretch was recorded. The magnitude of the contracture was measured by a goniometer. The quadratic weighted kappa statistics was very good for the upper limb spasticity (κw= 0.84). Intra-rater reliability was good for shoulder adductors (κw=0.75), and very good for elbow flexors and wrist flexors (κw 0.86 and 0.90, respectively). There were no differences between the weighted kappa values for muscle groups (p>0.05). The intra-rater reliability was also good in the presence of pain or contracture. The MMAS had very good intra-rater reliability in the assessment of upper limb spasticity in patients with hemiparesis. The presence of pain during shoulder abduction or contracture of the shoulder adductors had no influence on the reliability of the MMAS (κ w=0.75, 0.77, respectively).

Optimization of myocardial nulling in pediatric cardiac MRI

Current protocols to determine optimal nulling time in late enhancement imaging using adult techniques may not apply to children. To determine the optimal nulling time in anesthetised children, with the hypothesis that this occurs earlier than in adults. Sedated cardiac MRI was performed in 12 children (median age: 12months, range: 1-60months). After gadolinium administration, scout images at 2, 3, 4 and 10min and phase sensitive inversion recovery (PSIR) images from 5 to 10min were obtained. Signal-to-noise ratio (SNR) and inversion time (TI) were determined. Quality of nulling was assessed according to a grading score by three observers. Data was analysed using linear regression, Kruskal-Wallis and quadratic-weighted kappa statistics. One child with a cardiomyopathy had late enhancement. Good agreement in nulling occurred for scout images at 2 (κ = 0.69) and 3 (κ = 0.66) min and moderate agreement at 4min (κ = 0.57). Agreement of PSIR images was moderate at 7min (κ = 0.44) and poor-fair at other times. There were significant correlations between TI and scout time (r = 0.61, P < 0.0001), and SNR and kappa (r = 0.22, P = 0.017). Scout images at 2-4min can be used to determine the TI with little variability. Image quality for PSIR images was highest at 7min and SNR optimal at 7-9min. TI increases with time and should be adjusted frequently during imaging. Thus, nulling times in children differ from nulling times in adults when using standard adult techniques.

Timing and variability of late myocardial enhancement imaging in anesthetised pediatric patients

Methods Sedated cardiac MRI was performed on a 1.5 T Siemens Sonata. After IV injection of 0.2 mmol/kg of gadolinium (Magnovist), scout images were obtained at 2, 3, 4 and 10 minutes, and turboFLASH PSIR were obtained using standard views at 5-9 minutes. All images were assessed according to a grading score: 0 = none; 1 = reverse; 2 = poor; 3 = partial; and 4 = good nulling. The inversion time (TI) was determined from the best nulled scout image. Images were analysed by 3 independent observers blinded to the clinical information. The mean and standard deviation of the grading score was analysed using the KruskalWallis analysis and interobserver variability was determined by quadratic weighted kappa statistics.

Validity and reliability of an FFQ for use with adolescents in Ho Chi Minh City, Vietnam

The present study evaluates the reliability and validity of an FFQ designed for use with adolescents in urban Vietnam. A cohort study was conducted between December 2003 and June 2004. The FFQ was administered three times over a 6-month period (FFQ 1-3) and nutrient intakes were compared to those obtained from four 24 h recalls collected over the same period (24 h recalls 1-4) using crude, energy-adjusted and de-attenuated correlation coefficients. The level of agreement between the two measurements was also evaluated with Bland-Altman analysis. The percentage of nutrient intakes classified within one quintile, as well as quadratic-weighted kappa statistics, were calculated. Ho Chi Minh City, Vietnam. A total of 180 students were recruited in three junior high schools. Coefficients ranged from 0.22 for retinol to 0.78 for fibre for short-term reliability, and from 0.30 for retinol to 0.81 for zinc for long-term reliability. Coefficients for nutrient intakes between the mean of the three FFQ and mean of four 24 h recalls were mostly around 0.40, but higher for energy-adjusted nutrients. After allowing for within-person variation, the mean coefficient was 0.52 for macronutrients and 0.46 for micronutrients. There were a relatively high proportion of nutrient intakes classified within one quintile and a small number grossly misclassified. Kappa values shows 'fair' to 'good' agreement for all food/nutrient categories, while the Bland-Altman plots indicated that the FFQ is accurate in assessing nutrient intake at a group level. This newly developed FFQ is a valid tool for measuring nutrient intake in adolescents in urban Vietnam.

Dental arch relationships after two-flap palatoplasty in Taiwanese patients with unilateral cleft lip and palate

The authors evaluated the dental arch relationships of patients with unilateral cleft lip and palate treated with two-flap palatoplasty at the Chang Gung Craniofacial Center, Taipei, Taiwan, and compared them with the 6 centers from the Eurocleft study. Intraoral dental photographs of 60 consecutively treated Taiwanese patients with complete unilateral cleft lip and palate were used. The dental arch relationships were rated with the Goslon yardstick and compared with the Goslon outcome of the 6 center Eurocleft study. The strength of agreement of rating was assessed with quadratic-weighted kappa statistics. Inter- and intra-examiner agreements were high. 75% of patients were classified into poor or very poor categories, and the mean Goslon score was 3.92. These results showed no significant difference with Eurocleft center D, which had relatively poor treatment outcome. These data suggest that intraoral dental photographs provide a reliable method for rating dental arch relationships. Dental arch relationships in patients with unilateral cleft lip and palate at Chang Gung Craniofacial Center were unfavorable. This seemed to be attributable to the surgical procedures and racial differences in craniofacial morphology.

Reproducibility of the Wisconsin cataract grading system in the Blue Mountains Eye Study

Problem To measure the inter- and intra-grader reliability of the Wisconsin cataract grading system for grading of nuclear, cortical, and posterior subcapsular cataract (PSC) in the Blue Mountains Eye Study.Methods Random samples of lens photographs from study subjects were regraded by each grader according to the Wisconsin cataract grading system. The results were compared using quadratic weighted kappa statistics to derive measures of inter- and intra-grader reliability for the grading of nuclear (5 grades), PSC (3 grades), and cortical cataract (3 grades).Results Inter-grader reliability (quadratic weighted kappa) for grading of nuclear cataract varied between 0.82 and 0.79; kappa was 0.78 for cortical cataract and 0.57 for PSC. Intra-grader reliability was slightly higher than inter-grader reliability for all three cataract types.Conclusion In the Blue Mountains Eye Study, the Wisconsin cataract grading system provided good reproducibility, comparable to that reported from the Beaver Dam Eye Study, endorsing its usefulness in cataract epidemiology.