Prevalence And Bias Adjusted Kappa Research Articles

Comparison of serological assays for detecting antibodies in ducks exposed to H5 subtype avian influenza virus

BackgroundChicken red blood cells (RBCs) are commonly used in hemagglutination inhibition (HI) tests to measure hemagglutinating antibodies against influenza viruses. The use of horse RBCs in the HI test can reportedly increase its sensitivity when testing human sera for avian influenza antibodies. This study aims to compare the proportion of positives detected and the agreement between two HI tests using either chicken or horse red blood cells for antibody detection in sera of ducks experimentally infected or naturally exposed to Indonesian H5 subtype avian influenza virus. In addition, comparison with a virus neutralisation (VN) test was conducted with the experimental sera.ResultsIn the experimental study, the proportion of HI antibody-positive ducks increased slightly, from 0.57 when using chicken RBCs to 0.60 when using horse RBCs. The HI tests indicated almost perfect agreement (kappa = 0.86) when results were dichotomised (titre ≥ 4 log2), and substantial agreement (weighted kappa = 0.80) for log titres. Overall agreements between the two HI tests were greater than between either of the HI tests and the VN test. The use of horse RBCs also identified a higher proportion of antibody positives in field duck sera (0.08, compared to chicken RBCs 0.02), with also almost perfect agreements for dichotomized results (Prevalence and bias adjusted Kappa (PABAK) = 0.88) and for log titres (weighted PABAK = 0.93), respectively. Factors that might explain observed differences in the proportion of antibody-positive ducks and in the agreements between HI tests are discussed.ConclusionIn conclusion, we identified a good agreement between HI tests. However, when horse RBCs were used, a higher proportion of sera was positive (titre ≥ 4 log2) than using chicken RBCs, especially during the early response against H5N1 virus. The HRBC-HI might be more responsive in identifying early H5N1 HPAI serological response and could be a recommended assay for avian influenza sero-surveillance in both wild and domestic birds.

Interrater agreement of nasal endoscopy in patients with a prior history of endoscopic sinus surgery

Nasal endoscopy is an important part of the clinical evaluation of patients with chronic rhinosinusitis. However, the objectivity and interrater agreement of the procedure related findings have not been well studied, especially in patients who have previously had sinus surgery. Patients with a history of endoscopic sinus surgery for chronic rhinosinusitis were prospectively enrolled from a tertiary rhinology practice. Fourteen endoscopic nasal examinations were recorded using digital video capture software. Each patient also underwent computed tomography (CT) and completed the Sinonasal Outcome Test (SNOT-22). Blinded review of inflammatory and anatomic findings for each video was independently performed by 5 academic rhinologists at separate institutions. Comparisons were performed using the unweighted Fleiss' kappa statistic (K(f) ) and the prevalence- and bias-adjusted kappa (PABAK). There were no significant correlations between age, Lund-Mackay score, or SNOT-22 score. Interrater agreement was variable across the characteristics studied. Mean PABAK was excellent for the assessment of polyps (K(f) = 0.886); moderate for the assessments of middle turbinate (MT) integrity (K(f) = 0.543), MT position (K(f) = 0.443), maxillary sinus patency (K(f) = 0.593), and ethmoid sinus patency (K(f) = 0.429); fair for discharge (K(f) = 0.314), synechiae (K(f) = 0.257), and middle meatus patency (K(f) = 0.229); and poor for MT mucosal changes (K(f) = 0.148) and uncinate process (K(f) = 0.126). This study was notable for variability in the interrater agreement among the inflammatory and anatomic attributes that were examined. Further standardization of nasal endoscopy with regard to interpretation may improve the reliability of this procedure in clinical practice.

Pulmonary embolism: Are we there yet?

Clinical prediction rules (such as Wells model) are a reliable assessment tool for diagnostic work-up of suspected pulmonary embolism (PE). When used as part of a clinical algorithm and in combination with a D-Dimer, the model can safely exclude PE in low-risk groups and indicate when further investigations are unnecessary. The purpose of this study was to investigate the level of adherence to local diagnostic imaging guidelines for suspected PE and to ascertain the impact of interventions. Retrospective search of all patients referred from the Emergency Department (ED) of Royal Perth Hospital for computed tomography pulmonary angiography (CTPA) or V/Q scan between 11 September 2005 to 10 March 2006 (pre-intervention) and 1 January 2008 to 31 March 2008 (post-intervention) was conducted. The guidelines on 'Diagnostic Imaging Pathways' were considered as gold standard. Interventions included orienting ED doctors to guidelines and modified request forms for mandatory completion of Wells score. A prevalence- and bias-adjusted kappa (PABAK) score analysed the level of agreement between documentation on notes (R-score) and stamp (S-score). Thirty-five per cent (n = 187) and 22% (n = 109) deviated from the pathway pre-intervention and post-intervention, respectively (13% absolute reduction; P = 0.017). Stamp compliance was only 55% despite mandatory filling requirement. PABAK for 'PE as most likely diagnosis' was 0.25 for V/Q group and - 0.26 for CTPA. In addition, 44/60 (73%) had an intermediate or high S-score, yet only 11 of those 44 had a matched intermediate to high R-Score. Interventions reduced inappropriate practice but did not eliminate it completely. Compliance issues may be managed in the future via the introduction of electronic request linked to decision support.

Reliability of change in lumbar MRI findings over time in patients with and without disc prosthesis—comparing two different image evaluation methods

To assess the reliability of change in lumbar magnetic resonance imaging (MRI) findings evaluated retrospectively by direct comparison of images and by non-comparison. Pre-treatment and 2-year follow-up MRI was performed in 126 patients randomized to disc prosthesis surgery or non-surgical treatment. Two experienced radiologists independently evaluated progress and regress for Modic changes, disc findings, and facet arthropathy (FA) at L3/L4, L4/L5, and L5/S1, both by non-comparison and by comparison of initial and follow-up images. FA was evaluated at all levels, and other findings at non-operated levels. We calculated prevalence- and bias-adjusted kappa (PABAK) values for interobserver agreement. The impact of an adjacent prosthesis (which causes artefacts) and image evaluation method on PABAK was assessed using generalized estimating equations. Image comparison indicated good interobserver agreement on progress and regress (PABAK 0.63-1.00) for Modic changes, posterior high-intensity zone, disc height, and disc contour at L3-S1 and for nucleus pulposus signal and FA at L3/L4; and moderate interobserver agreement (PABAK 0.46-0.59) on decreasing nucleus signal and increasing FA at L4-S1. Image comparison indicated lower (but fair) interobserver agreement (PABAK 0.29) only for increasing FA at L5/S1 in patients with prosthesis in L4/L5 and/or L5/S1. An adjacent prosthesis had no overall impact on PABAK values (p ≥ 0.22). Comparison yielded higher PABAK values than non-comparison (p < 0.001). Regarding changes in lumbar MRI findings over time, comparison of images can provide moderate or good interobserver agreement, and better agreement than non-comparison. An adjacent prosthesis may not reduce agreement on change for most findings.

Interexaminer reliability of orthopaedic special tests used in the assessment of shoulder pain

Orthopaedic special tests (OST) are commonly used in the assessment of the painful shoulder to assist to rule-in or rule-out specific pathology. A small number of tests with high levels of diagnostic accuracy have been identified but interexaminer reliability data is variable or lacking. The aim of this study was to determine the interexaminer reliability of a group of OST with demonstrated diagnostic accuracy at primary care level. Forty consecutive subjects with shoulder pain were recruited. Six tests were performed by two examiners (physiotherapists) on the same day. Tests included the active compression test, Hawkins–Kennedy test, drop-arm test, crank test, Kim test and belly-press test. ‘Fair’ reliability (kappa 0.36–0.38) was observed for the active compression test (labral pathology), Hawkins–Kennedy test and crank test. Prevalence of positive agreements was low for the active compression test (acromioclavicular joint), drop-arm test, Kim test and belly-press test. Prevalence and bias adjusted kappa (PABAK) values indicated ‘substantial’ reliability (0.65–0.78) for these tests. The active compression test (acromioclavicular joint), belly-press tests (observation and weakness), Kim test and drop-arm test demonstrate acceptable levels of interexaminer reliability in a group of patients with sub-acute and chronic shoulder conditions.

Agreement Between Pharmacists for Problem Identification: An Initial Quality Measurement of Cognitive Services

Recent changes in national reimbursement policies expand the ability of pharmacists to seek reimbursement for cognitive services. The quality of pharmacist-provided cognitive services has, until now, remained unassessed. Pharmacists should demonstrate the quality and value of their work to ensure the continued and expanded acceptance of reimbursement for their services. A preliminary step in assessing quality is to compare agreement between pharmacists for basic problem identification. To quantify agreement between pharmacist reviewers for problem identification among Utah Medicaid recipients. Five pharmacists retrospectively reviewed drug regimens, patient characteristics, diagnosis codes, and procedures for 80 Medicaid patients in September 2008 and identified drug-related problems (DRPs) in 15 predetermined categories. Data for each patient were reviewed twice, and each combination of 2 pharmacists reviewed the same 8 patients' information. We calculated a reliability coefficient to compare the number of DRPs identified and used prevalence and bias adjusted kappa (PABAK) to determine interrater reliability for the presence of a specific DRP. Of the 15 DRPs categorized by pharmacist reviewers, 1 (untreated indications) had a PABAK coefficient of 0.20, indicating a relatively low level of agreement between reviewers. All other DRP categories had good to excellent agreement, with PABAK coefficients ranging between 0.43 and 0.98. Pharmacist reviewers exhibited less variability in DRP identification or categorization than had been expected for most categories. This work supports the conclusion that pharmacists in our center provide a basic and necessary level of quality for problem assessment. Future work is needed to document the impact of this quality on patient outcomes.

Usability of quality measures for online health information: Can commonly used technical quality criteria be reliably assessed?

Many criteria have been developed to rate the quality of online health information. To effectively evaluate quality, consumers must use quality criteria that can be reliably assessed. However, few instruments have been validated for inter-rater agreement. Therefore, we assessed the degree to which two raters could reliably assess 22 popularly cited quality criteria on a sample of 42 complementary and alternative medicine Web sites. We determined the degree of inter-rater agreement by calculating the percentage agreement, Cohen's kappa, and prevalence- and bias-adjusted kappa (PABAK). Our un-calibrated analysis showed poor inter-rater agreement on eight of the 22 quality criteria. Therefore, we created operational definitions for each of the criteria, decreased the number of assessment choices and defined where to look for the information. As a result 18 of the 22 quality criteria were reliably assessed (inter-rater agreement > or = 0.6). We conclude that even with precise definitions, some commonly used quality criteria cannot be reliably assessed. However, inter-rater agreement can be improved with precise operational definitions.