Fair Kappa Value Research Articles

Significant inter-observer variation in the diagnosis of extrapancreatic necrosis and type of pancreatic collections in acute pancreatitis – An international multicenter evaluation of the revised Atlanta classification

BackgroundFor consistent reporting and better comparison of data in research the revised Atlanta classification (RAC) proposes new computed tomography (CT) criteria to describe the morphology of acute pancreatitis (AP). The aim of this study was to analyse the interobserver agreement among radiologists in evaluating CT morphology by using the new RAC criteria in patients with AP. MethodsPatients with a first episode of AP who obtained a CT were identified and consecutively enrolled at six European centres backwards from January 2013 to January 2012. A local radiologist at each center and a central expert radiologist scored the CTs separately using the RAC criteria. Center dependent and independent interobserver agreement was determined using Kappa statistics. ResultsIn total, 285 patients with 388 CTs were included. For most CT criteria, interobserver agreement was moderate to substantial. In four categories, the center independent kappa values were fair: extrapancreatic necrosis (EXPN) (0.326), type of pancreatitis (0.370), characteristics of collections (0.408), and appropriate term of collections (0.356). The fair kappa values relate to discrepancies in the identification of extrapancreatic necrotic material. The local radiologists diagnosed EXPN (33% versus 59%, P < 0.0001) and non-homogeneous collections (35% versus 66%, P < 0.0001) significantly less frequent than the central expert. Cases read by the central expert showed superior correlation with clinical outcome. ConclusionDiagnosis of EXPN and recognition of non-homogeneous collections show only fair agreement potentially resulting in inconsistent reporting of morphologic findings.

Reproducibility of subgingival bacterial samples from patients with peri-implant mucositis

The aim of the present study was to investigate the reproducibility of bacterial enumeration from subsequent subgingival samples collected from patients with peri-implant mucositis. Duplicate microbial samples from 222 unique implant sites in 45 adult subjects were collected with paper points and analyzed using the checkerboard DNA-DNA hybridization technique. Whole genomic probes of 74 preselected bacterial species were used. Based on the bacterial scores, Cohen's kappa coefficient was used to calculate the inter-annotator agreement for categorical data. The percentage agreement was considered as "good" when the two samples showed the same score or differed by 1 to the power of 10. Moderate to fair kappa values were displayed for all bacterial species in the test panel (range 0.21-0.58). There were no significant differences between Gram-positive and Gram-negative species. The percentage of good agreement between the first and second samples averaged 74.7 % (n = 74; range 56-83 %), while the proportion of poor agreement ranged from 1 to 19 % for the various strains. While an acceptable clinical agreement was obtained in most cases, diverging bacterial scores may appear in subgingival samples collected at the same time point from patients with peri-implant mucositis. The broad bulky base of implant crowns may present an obstacle for the collection of reproducible subgingival samples with paper points.

Reproducibility of 6- and 4-category faces pain scale to the assessment of temporomandibular joint pain and muscle in school-age children

The aim of this study was to compare the levels of reproducibility between two Faces Pain Scales (FPS) of 6 and 4 categories to the assessment of pain intensity on children with and without Temporomandibular joint and muscle pain. A total of 29 children were recruited: 13 symptomatic (9.79 ± 1.36 years old) and 16 asymptomatic (8.69 ± 0.87 years old). One previously-trained examiner applied manual palpation to evaluate orofacial structures, and FPS to assess pain intensity. All children were initial evaluated using 6-category FPS and after 3 days using 4-category FPS. The children were assessed after a seven day time interval from initial and second sessions. Weighted Kappa coefficient was used to verify reproducibility levels. Similar levels of reproducibility (moderate and fair kappa values) have been verified with the application of the 6-category FPS on symptomatic and asymptomatic children. Similar results were verified using 4-category FPS on symptomatic and asymptomatic children (poor and fair kappa values). Higher levels of reproducibility were verified with the application of the 6-category FPS in both groups and considering symptomatic and asymptomatic as single group. In this way, the 6-category FPS should be preferred over the 4-category FPS to assess pain intensity on children's orofacial structures.

Content validity and internal consistency of the Dutch translation of the Safety Attitudes Questionnaire: An observational study

BackgroundPatient safety is fundamental to healthcare quality. Attention has recently focused on the patient safety culture of an organisation and its impact on patient outcomes. A strong safety climate appears to be an essential condition for safe patient care in the hospital. A number of instruments are used to measure this patient safety climate or culture. The Safety Attitudes Questionnaire is a validated, widely used instrument to investigate multiple dimensions of safety climate at the clinical level in a variety of inpatient and outpatient settings. ObjectivesThe purpose of this study is to explore the face- and content validity and the internal consistency of the Safety Attitudes Questionnaire in a large Belgian academic medical center. MethodThe translation into Dutch was done by three researchers. A panel of fifteen Dutch speaking experts evaluated the translation and its content validity. Content validity was quantified by the content validity index (CVI) and a modified kappa index. Face validity was evaluated by two nurses and two physicians who assessed the Dutch version of the SAQ. A cross-sectional design was used to test internal consistency of the SAQ items by calculating Cronbach's alpha and corrected item-total correlations. ResultsTwenty-three of the 33 SAQ items showed excellent and seven items showed good content validity. One item had a fair kappa value (item 20) and two items had a low content validity index (items 15 and 16). The average CVI of the total scale was 0.83 and ranged from 0.55 to 0.97 for the six subscales. The face-validity was good with no fundamental remarks given. The SAQ's overall Cronbach's alpha was 0.9 and changed minimally when removing items. The item-total correlations ranged from 0.10 to 0.63, no single items were strongly correlated with the sum of the other items. ConclusionWe conclude that in this study the Dutch version of the Safety Attitudes Questionnaire showed acceptable to good psychometric properties. In line with previous evidence, this instrument seems to be an acceptable to adequate tool to evaluate the safety climate.

Reliability of Prognostic Factors in Breast Carcinoma Determined by Core Needle Biopsy

The aim of this study was to evaluate the reliability of information obtained by core needle biopsy (CNB). We studied 111 women (112 lesions) with breast cancer who underwent CNB and subsequent surgical excision. Six factors (histological type, nuclear grade, histological grade, estrogen receptor (ER) status, progesterone receptor (PR) status, and human epidermal growth factor receptor-2 (HER2) status) were evaluated in a blinded fashion at CNB and at surgical excision. The histological type at CNB correlated exactly with that of the excisional specimen in 83% (87/105) of the cases. Of the 45 in situ lesions at CNB, 16 (36%) were found to have invasive carcinoma at surgical excision. The difference between the specimens from CNB and those from surgery in terms of the absolute concordance rate and kappa statistic value were 61% with a fair kappa value (0.26) in the nuclear grade, 75% with a moderate kappa value (0.55) in the histological grade, 95% with an almost perfect kappa value (0.84) in ER, 88% with a substantial kappa value (0.70) in PR and 88% with a substantial kappa value (0.65) in HER2. Regarding the evaluation of nuclear and histological grades, a trend toward greater accuracy was observed when thicker specimens were used. CNB provided reliable information on the histological type of invasive carcinoma. It also evaluated ER, PR and HER2 (only in cases where the score was 3+) accurately in spite of the limited quantity of the specimen obtained with the thin (16-gauge) needle.

Inter-tester reliability of passive intervertebral and active movements of the cervical spine

Measurements of active range of motion (AROM) and passive intervertebral movements (PIM) of the cervical spine are frequently used for patients with neck pain. However, there is a paucity of studies that investigate the psychometric properties of these measurements. Objectives of this study were to: (1) determine the inter-tester reliability of PIM, AROM, and the effects of AROM on symptom provocation; (2) establish the minimal detectable change (MDC) in cervical AROM; and (3) determine the association between AROM and disability. Thirty subjects (age 41+/- 12) with neck pain participated in this study. Two masked examiners performed the measurements during the same testing session. PIM was assessed manually and recorded as hypomobile or normal. AROM was measured in degrees with a gravity goniometer. The effect of AROM on patient's symptoms was recorded as no change, decreased, increased, centralization, or peripheralization. Measures of AROM had moderate to substantial reliability (.78 - .91) and resulted in a MDC adequate for clinical use (from 9 degrees to 16 degrees). The effect of AROM on symptom provocation resulted in Kappa values that ranged from slight to substantial (.25 - .87). Measures of PIM resulted in substantial and moderate reliability of assessing occipital-atlas mobility, tenderness of the transverse processes of atlas, and symptom provocation during PIM testing of the lower cervical segments. Fair Kappa values were observed during judgment of mobility in the C2 segment and symptom reproduction during PIM of C2 and C5. The additional PIM had Kappa values that ranged from none to slight. Low prevalence of positive findings likely resulted in an artificial deflation of the Kappa statistic during some PIM measures. Measures of AROM in saggital and transverse planes were associated with disability scores (r = .43 and .40, respectively). Findings are relevant to the planning of future studies to establish the criterion validity of these tests to guide the selection of interventions and establish prognosis in patients with neck pain.

A prospective assessment of diagnostic efficacy of blind protective bronchial brushings compared to bronchoscope-assisted lavage, bronchoscope-directed brushings, and blind endotracheal aspirates in ventilator-associated pneumonia.

The purpose of this study is to compare techniques for the diagnosis of suspected ventilator-associated pneumonia in the trauma patient. Per the literature, bronchoscope protected brushings and bronchoalveolar lavage were set as the standards for comparison because of their high specificity and sensitivity. We hypothesized that blind protected brushings were equivalent to bronchoscope-directed techniques and that endotracheal aspirates (ETA) were not. With informed consent, 90 trauma patients with two or more of the following were accepted into the study: 48 hours or more on the ventilator, new or increasing infiltrate on chest radiograph, excess or purulent secretions, suspected aspiration, temperature of 38.5 degrees C or above, white blood cell count greater than or equal to 12,000/mm3, and respiratory distress. Four samplings were performed on each patient using bronchoscope-assisted and nonbronchoscopic techniques. Each patient had cultures obtained by and significances quantified as follows: ETA, moderate/many/abundant; bronchoscope-directed protected brushings (BDPB), 103 colony-forming units [CFU]/mL; blind protected brushing via endotracheal tube (BPB), 103 CFU/mL; and bronchoscopic bronchoalveolar lavage (BAL), 104 CFU/mL. Quantitative cultures were obtained and compared for the following pathogens: gram-positive cocci, gram-positive rods, gram-negative cocci, gram-negative rods, anaerobic bacteria, and yeast. An assessment of agreement for cultured pathogens between the sampling modalities was completed using kappa (kappa) analysis, and significance was set at p < or = 0.05. With patients used as their own controls, Gram's stain and pathogens cultured from the various sampling techniques were compared for agreement by kappa analysis. BDPB and BAL were set as the "gold standards" for comparison against each other and against the BPB and ETA. Kappa analysis was used to measure the strength of agreement for these findings; individual values from the comparisons of Gram's stain were then averaged for descriptive purposes of the data. Most kappa values were associated with a statistically significant value of p < 0.05. The greatest strength of agreement was found to be moderate comparing Gram's stain results of BPB and BDPB (kappa = 0.467), ETA and BAL (kappa = 0.535), and BPB and BAL (kappa = 0.547). Fair kappa values were shown in comparing Gram's stain results of ETA and BDPB (kappa = 0.382) and BAL and BDPB (kappa = 0.390). A quantitative analysis of bacteriologic cultures obtained by four standard sampling techniques has demonstrated with statistical significance that no difference exists between modality of sampling in reliability or in obtaining clinically significant pathogens. In reviewing the literature, this study is the first assessment of agreement for cultured pathogens between the four different sampling modalities and the largest to assess the efficacy of the blind protected brush technique.