Overall Percentage Of Agreement Research Articles

Clinical performance of an automated reader in interpreting malaria rapid diagnostic tests in Tanzania

BackgroundParasitological confirmation of malaria is now recommended in all febrile patients by the World Health Organization (WHO) to reduce inappropriate use of anti-malarial drugs. Widespread implementation of rapid diagnostic tests (RDTs) is regarded as an effective strategy to achieve this goal. However, the quality of diagnosis provided by RDTs in remote rural dispensaries and health centres is not ideal. Feasible RDT quality control programmes in these settings are challenging. Collection of information regarding diagnostic events is also very deficient in low-resource countries.MethodsA prospective cohort of consecutive patients aged more than one year from both genders, seeking routine care for febrile episodes at dispensaries located in the Bagamoyo district of Tanzania, were enrolled into the study after signing an informed consent form. Blood samples were taken for thick blood smear (TBS) microscopic examination and malaria RDT (SD Bioline Malaria Antigen Pf/Pan™ (SD RDT)). RDT results were interpreted by both visual interpretation and Deki Reader™ device. Results of visual interpretation were used for case management purposes. Microscopy was considered the “gold standard test” to assess the sensitivity and specificity of the Deki Reader interpretation and to compare it to visual interpretation.ResultsIn total, 1,346 febrile subjects were included in the final analysis. The SD RDT, when used in conjunction with the Deki Reader and upon visual interpretation, had sensitivities of 95.3% (95% CI, 90.6-97.7) and 94.7% (95% CI, 89.8–97.3) respectively, and specificities of 94.6% (95% CI, 93.5–96.1) and 95.6% (95% CI, 94.2–96.6), respectively to gold standard. There was a high percentage of overall agreement between the two methods of interpretation.ConclusionThe sensitivity and specificity of the Deki Reader in interpretation of SD RDTs were comparable to previous reports and showed high agreement to visual interpretation (>98%). The results of the study reflect the situation in real practice and show good performance characteristics of Deki Reader on interpreting malaria RDTs in the hands of local laboratory technicians. They also suggest that a system like this could provide great benefits to the health care system. Further studies to look at ease of use by community health workers, and cost benefit of the system are warranted.

Primary Cervical Cancer Screening and Triage Using an mRNA Human Papillomavirus Assay and Visual Inspection

Mexican Cervical Cancer Screening Study II (MECCS II) seeks to develop a highly sensitive and highly specific screening program able to be adapted to all socioeconomic levels in Mexico. The objectives of MECCS II are (1) to compare the sensitivity and specificity for cervical intraepithelial neoplasia (CIN) 3 or cancer of self-collected vaginal specimens tested for high-risk types of the human papillomavirus (HR-HPV) by APTIMA with those tested for HR-HPV by Hybrid Capture 2 (HC2); and (2) determine the efficacy of cryotherapy in the treatment of HR-HPV-positive and acetic acid-aided visual inspection (VIA)-positive and -negative women after VIA triage. The study was conducted in rural Mexico. Women aged 30 to 50 years, nonpregnant, with no history of hysterectomy or pelvic irradiation and varied histories of screening, participated. A direct endocervical sample was tested for cytology, HC2, and APTIMA assay (AHPV). Subjects positive on any test were recalled for triage VIA, biopsies, and immediate cryotherapy. Tests were compared using McNemar test. Two thousand forty-nine patients have complete results. Mean age of the patients was 39.2 years; 7.7% presented with ≥atypical squamous cells of uncertain significance (ASCUS), 1.8% ≥low-grade squamous intraepithelial neoplasia, and 0.5% ≥high-grade squamous intraepithelial neoplasia. Two percent of patients had ≥CIN2, and 0.78% had ≥CIN3 (including 2 with invasive disease). The sensitivity of ThinPrep (>ASCUS), HC2, and AHPV for >CIN3 for direct endocervical collection was 87.5%, 100%, and 100%, respectively. The specificity of ThinPrep (>ASCUS), HC2, and AHPV for >CIN3 was 94.1%, 92.2%, and 93.5%, respectively. Specificities of HC2 and AHPV differed significantly. The overall percentage of agreement among HPV assays (HC2 vs APTIMA) is 97%. Four hundred sixty-nine women returned for VIA. Two hundred ninety-one women were treated with cryotherapy. The specificity of the APTIMA assay along with high sensitivity is an advantage for primary screening. Follow-up evaluation will be important to determine the true impact of potential undertreatment in the screening algorithm. Self-sampling applications are explored.

Three-dimensional turbo spin-echo magnetic resonance imaging (MRI) and semiquantitative assessment of knee osteoarthritis: comparison with two-dimensional routine MRI

The aim of this study was to evaluate three-dimensional (3D) turbo spin-echo (TSE) magnetic resonance imaging (MRI) for semiquantitative assessment of knee OA. Twenty subjects fulfilling the American College of Rheumatology clinical criteria of knee OA underwent both two-dimensional (2D) and 3D MRIs on the same day. The 2D MRI protocol included triplanar fat-suppressed (FS) intermediate-weighted (Iw) TSE. For the 3D TSE technique, a sagittal FS Iw sequence was acquired and triplanar reformations were constructed. 2D and 3D MRIs were read separately by two radiologists using the Whole-Organ Magnetic Resonance Imaging Score (WORMS) system. Agreement was determined using weighted kappa statistics and percentage of overall agreement. The diagnostic performance of WORMS readings using 3D TSE MRI to detect the presence or absence of features was assessed using readings from 2D TSE images as a reference. Agreement for the scored features ranged between 0.62 (osteophytes (OS)) and 0.94 (meniscal extrusion). The sensitivity of WORMS readings using the 3D TSE technique ranged between 80% (periarticular cysts) and 100% (several features), the specificity ranged between 62.3% (OS) and 100% (several features), and accuracy ranged between 77.2% (OS) and 99.3% (subchondral cysts). Semiquantitative assessment of knee OA can be reliably performed using 3D TSE MRI, showing substantial to almost perfect agreement and high accuracy when compared to routine 2D TSE MRI. 3D TSE MRI also takes less time, which is important for large OA studies.

Antenatal Ultrasonographic Anteroposterior Renal Pelvis Diameter Measurement: Is It a Reliable Way of Defining Fetal Hydronephrosis?

Purpose. It was to quantify the intraobserver and interobserver variability of the sonographic measurements of renal pelvis and classify hydronephrosis severity. Methods. Two ultrasonographers evaluated 17 fetuses from 23 to 39 weeks of gestation. Renal pelvis APD were taken in 50 renal units. For intraobserver error, one of them performed three sequential measurements. The mean and standard deviation from the absolute and percentage differences between measurements were calculated. Bland-Altman plots were used to visually assess the relationship between the precision of repeated measurements. Hydronephrosis was classified as mild (5.0 to 9.9 mm), moderate (10.0 to 14.9 mm), or severe (≥15.0 mm). Interrater agreement were obtained using the Kappa index. Results. Absolute intraobserver variation in APD measurements was 5.2 ± 3.5%. Interobserver variation of ultrasonographers was 9.3 ± 9.7%. Neither intraobserver or interobserver error increased with increasing APD size. The overall percentage of agreement with the antenatal hydronephrosis diagnosis was 64%. Cohen's Kappa to hydronephrosis severity was 0.51 (95% CI, 0.33 to 0.69). Conclusion. Inter and intraobserver APD measurement errors were low in these group, but the agreement to hydronephrosis diagnosis and classification was fair. We suggest that standard and serial APD measurement can better define and evaluate fetal hydronephrosis.

Labial salivary gland biopsies in Sjögren's syndrome: still the gold standard?

The accuracy and diagnostic benefits of the labial salivary gland (LSG) biopsy for Sjögren's syndrome (SS) have received mixed reviews. This study was conducted to assess (1) the inter-rater agreement among 5 pathologists, and (2) the relationship between biopsy findings and clinical disease parameters. Three oral pathologists (OP) and two surgical pathologists (SP) provided independent diagnoses, focus scores, and plasma cell characterizations for 37 LSG biopsies. Inter-rater reliability was assessed using percentage of overall agreement and intraclass correlation coefficients. Relationships between diagnoses and clinical parameters were assessed by nonparametric correlations. Overall agreement among the pathologists was poor, although the intra-specialty agreement was good. The ratings of OP were most highly correlated with serological measures, while those of SP were correlated with salivary flow rate and disease damage. Since the LSG biopsy can be the determining factor in SS diagnoses, these demonstrated inconsistencies merit further consideration.

Direct Comparison of the BD Phoenix System with the MicroScan WalkAway System for Identification and Antimicrobial Susceptibility Testing of Enterobacteriaceae and Nonfermentative Gram-Negative Organisms

The Phoenix automated microbiology system (BD Diagnostics, Sparks, MD) is designed for the rapid identification (ID) and antimicrobial susceptibility testing (AST) of clinically significant human bacterial pathogens. We evaluated the performance of the Phoenix instrument in comparison with that of the MicroScan WalkAway system (Dade Behring, West Sacramento, CA) in the ID and AST of gram-negative clinical strains and challenge isolates of Enterobacteriaceae (n = 150) and nonfermentative gram-negative bacilli (NFGNB; 45 clinical isolates and 8 challenge isolates). ID discrepancies were resolved with the API 20E and API 20NE conventional biochemical ID systems (bioMerieux, Durham, NC). The standard disk diffusion method was used to resolve discordant AST results. The overall percentages of agreement between the Phoenix ID results and the MicroScan results at the genus and species levels for clinical isolates of Enterobacteriaceae were 98.7 and 97.7%, respectively; following resolution with conventional biochemical testing, the accuracy of the Phoenix system was determined to be 100%. For NFGNB, the levels of agreement were 100 and 97.7%, respectively. Both systems incorrectly identified the majority of the uncommon nonfermentative nonpseudomonal challenge isolates recovered from cystic fibrosis patients; these isolates are not included in the databases of the respective systems. For AST of Enterobacteriaceae, the rate of complete agreement between the Phoenix results and the MicroScan results was 97%; the rates of very major, major, and minor errors were 0.3, 0.2, and 2.7%, respectively. For NFGNB, the rate of complete agreement between the Phoenix results and the MicroScan results was 89.1%; the rates of very major, major, and minor errors were 0, 0.5, and 7.7%, respectively. Following the confirmatory testing of nine clinical isolates initially screened by the MicroScan system as possible extended-spectrum-beta-lactamase (ESBL)-producing organisms (seven Klebsiella pneumoniae isolates and two Escherichia coli isolates), complete agreement was achieved for eight isolates (one ESBL positive and seven negative); one false positive was obtained with the Phoenix instrument. The MicroScan system correctly detected the 10 ESBL challenge isolates, versus the 6 detected by the Phoenix system. Overall, there was good correlation between the Phoenix instrument and the MicroScan system for the ID and AST of Enterobacteriaceae and common NFGNB. The Phoenix system is a reliable method for the ID and AST of the majority of clinical strains encountered in the clinical microbiology laboratory. Until additional performance data are available, results for all Klebsiella pneumoniae or Klebsiella oxytoca and E. coli isolates screened and confirmed as ESBL producers by any automated system should be confirmed by alternate methods prior to the release of final results.

An interrater reliability study of the Braden scale in two nursing homes

Background Adequate risk assessment is essential in pressure ulcer prevention. Assessment scales were designed to support practitioners in identifying persons at pressure ulcer risk. The Braden scale is one of the most extensively studied risk assessment instruments, although the majority of studies focused on validity rather than reliability. Objectives The first aim was to measure the interrater reliability of the Braden scale and its individual items. The second aim was to study different statistical approaches regarding interrater reliability estimation. Design and methods An interrater reliability study was conducted in two German nursing homes. Residents ( n = 152) from 8 units were assessed twice. The raters were trained nurses with a work experience ranging from 0.5 to 30 years. Data were analysed using an overall percentage of agreement, weighted and unweighted kappa and the intraclass correlation coefficient. Results Differences between nurses rating the overall Braden score ranged from 0 up to 9 points. Interrater reliability expressed by the intraclass correlation coefficient ranged from 0.73 (95% CI 0.26–0.91) to 0.95 (95% CI 0.87–0.98). Calculated intraclass correlation coefficients for individual items ranged from 0.06 (95% CI −0.31 to 0.48) to 0.97 (95% CI 0.93–0.99) with the lowest values being measured for the items “sensory perception” and “nutrition”. There was no association between work experience and the level of interrater reliability. With two exceptions, simple kappa-values were always lower than weighted kappa-values and intraclass correlation coefficients. Conclusions Although the calculated interrater reliability coefficients for the total Braden score were high in some cases, several clinically relevant differences occurred between the nurses. Due to interrater reliability being very low for the items “sensory perception” and “nutrition”, it is doubtful if their assessment contributes to any valid results. The calculation of weighted kappa or intraclass correlation coefficients is the most appropriate interrater reliability estimates.

Interpreting interrater reliability coefficients of the Braden scale: A discussion paper

There are many studies investigating psychometric properties of the Braden scale, a scale that predicts the risk for pressure ulcers. The main focus of these studies is validity as opposed to reliability. In order to estimate the degree of interrater reliability a literature review revealed that numerous statistical approaches and coefficients were used (Pearson's product-moment correlation, Cohen's kappa, overall percentage of agreement, intraclass correlation). These coefficients were calculated for the individual items and the overall Braden score and were used inconsistently. The advantages and limitations of every coefficient are discussed and it is concluded that most of them are inappropriate measures. Therefore, estimating the degree of the Braden scale interrater reliability is limited to a certain extent. It is shown that the intraclass correlation coefficient is an appropriate statistical approach for calculating the interrater reliability of the Braden scale. It is recommended to present intraclass correlation coefficients in combination with the overall percentage of agreement.

Gene copy number of epidermal growth factor receptor (EGFR) by fluorescent in situ hybridization (FISH) in head and neck squamous cell carcinomas (HNSCC) is closely correlated with EGFR protein levels assessed by automated quantitative protein analysis (AQUA)

6023 Background: EGFR overexpression correlates with recurrence and with treatment resistance in HNSCC. The mechanisms of EGFR protein overexpression are poorly understood. Nonetheless, previous investigators have not demonstrated a correlation between EGFR gene copy number and protein content, using conventional immunohistochemistry (IHC). The aim of this study was to evaluate the relationship of EGFR gene copy number and protein expression utilizing fluorescence in situ hybridization (FISH) and AQUA, a novel, immunohistochemical method of automated quantitative in situ proteomic analysis which permits subcellular localization. Methods: A tissue microarray composed of 137 HNSCC treated with (chemo)radiation was constructed and analyzed for EGFR copy number by FISH (Vysis/Abbot) and EGFR protein expression (DAKO antibody) using AQUA analysis of EGFR staining scored on a scale of 0–255 and by conventional IHC. Agreement was assessed using kappa. Results: Sixteen (15%) of one-hundred six tumors with FISH results demonstrated EGFR high polysomy and/or gene amplification (FISH+). AQUA demonstrated a range of 3.6–102.2; protein levels assessed by AQUA in the FISH amplified cases were significantly higher (p =0.008) than in the FISH non- amplified ones. Using the EGFR 75th percentile as a cut-off, AQUA and FISH showed significant agreement (percentage of overall agreement 82%, kappa=0.458, p=0.003). To the contrary there was no concordance between FISH and conventional IHC results in this series. Conclusions: The discrepancy between EGFR gene amplification rate and protein expression by IHC reported previously may be due to the limitations and nonquantitative nature of conventional IHC. EGFR protein content correlates with gene copy number if protein content is quantitated and automatically analyzed, as with AQUA. No significant financial relationships to disclose.

Reliability and Validity of the Evaluation to Sign Consent Measure

The purpose of this report is to evaluate the reliability and validity of the five-item Evaluation to Sign Consent (ESC), a measure that can guide determination of an older adult's capacity to consent for research. Information was obtained from 346 nursing home residents from six facilities who were being enrolled into a randomized controlled trial testing a restorative care intervention. In addition to the ESC, the resident's cognitive status and demographic information was obtained. The average age of the participants was 86.1 +/- 7.3 years; most of the participants were female (84%) and Caucasian (95%). The mean Mini-Mental State Exam score was 18.0 +/- 7.4. A total of 218 residents (63%) did not pass the ESC. According to a Rasch analysis and the inter-rater reliability (r =.81), there was some evidence of reliability and validity with this measure. Logistic regression showed that Items 1 (describing two risks to participation in the study) and 2 (knowing what is associated with participation) had the greatest overall percentage of agreement with the full ESC, and the Mini-Mental State Exam was the only resident-tested variable to predict the results of the ESC. This study provides useful information about the ESC. It indicates a reason and a method to move beyond cognitive testing that can more appropriately evaluate the capacity to consent to participate in research.

Effects of item and rater characteristics on checklist recording: what should we look for?

Examinations based on using standardised patients (SPs) commonly use checklist recordings to evaluate students' clinical performance. This paper examines whether and to what extent item and rater characteristics affect the reliability of history checklist recording in an SP-based assessment. Checklist items were reviewed for the presence or absence of 5 item characteristics and a 2-point versus 3-point scoring scale. Agreement between checklist recordings obtained from SPs and clinician-examiners (CEs) were compared by item characteristics, scoring scale and CEs' level of involvement in the assessment. Based on 3179 pairs of recordings, the overall percentage of agreement between SPs and CEs was 83% (kappa = 0.64). Agreement was significantly higher for items scored on a 2-point than on a 3-point scale, and when the CE was also the author and the trainer of the station. After controlling for other factors, item characteristics were only marginally associated with level of interrater agreement. This study suggests that attention should be paid to specific aspects of checklist development and checklist recording training when an SP or CE is used as recorder.

CALIBRATING INPUFF–2 MODEL BY RESIDENT–PANELISTS FOR LONG–DISTANCE ODOR DISPERSION FROM ANIMAL PRODUCTION SITES

An air dispersion model, INPUFF–2, was calibrated for long–distance odor estimation. Odor events were monitored from June to November 1999 by 19 trained resident–panelists living on a 4.8
4.8 km (3
3 miles) grid of farmland containing 20 livestock/poultry farms. The panelists used a numerical 0 to 3 scale to assess the odor intensity. To convert odor intensity to concentration, the correlation between odor intensity (on a 0 to 5 n–butanol water solution scale) and concentration (in OU/m 3 ) for swine and cattle odors were developed. Odor emissions from the farms were measured and weather conditions were monitored over the same period. A total of 296 odor events were detected among which 170 odor events were simulated by the model. One hundred twenty three of these 170 odor events, i.e. 72.3%, were classified as faint odor (intensity 1). The comparison between the modeled and the measured odor intensity indicated that the model successfully estimated odor intensity 1 (faint odor) traveling up to 3.2 km under stable atmospheric conditions (P > 0.05). However, the model underestimated moderate to strong or very strong odors and odors that occurred during neutral or unstable weather as compared with the field measured data (P < 0.05). The overall percentage of agreement was 81.8%. The results indicate that the model could serve as an effective tool for agriculture odor dispersion estimation from animal production sources.

Karnofsky and ECOG performance status scoring in lung cancer: A prospective, longitudinal study of 536 patients from a single institution

The Karnofsky's index of performance status (KPS) and the Eastern Cooperative Oncology Group Performance Status Scale (ECOG PS) are widely used methods of assessing the functional status of cancer patients. In this study, we compare their predictive validity, and suggest a table of transformation between scales. 536 consecutive lung cancer patients were assigned both KPS and ECOG PS scores before, during and after treatment (in all, 1656 assignments). Patients were accurately staged at diagnosis, and carefully re-evaluated at each follow-up visit. Multiple clinical, laboratory and instrumental data were recorded along with performance status assessments. Survival times were measured from the pathological diagnosis. KPS and ECOG PS assignments were strongly related to each other (Spearman R = − 0.869). Correlation between scales persisted unchanged in pretreatment and post-treatment assessments, advanced and limited diseases, response or non-response to treatment, and different assessors (R indices ranging from −0.825 to −0.901). A three-point conversion table showed the highest rate of success with an overall percentage of agreement exceeding 84% (grade 1: KPS = 100, 90, 80 and ECOG PS = 0, 1; grade 2: KPS = 70, 60 and ECOG PS = 2; grade 3: KPS < 60 and ECOG PS = 3, 4). Both univariate and multivariate analyses of survival documented the predictive validity of the two scales. However, KPS showed less ability than ECOG PS to discriminate patients with different prognosis. Because of the better predictive ability shown in this study, ECOG PS should be preferred to KPS. A general consensus on the scale to use could avoid problems of conversion, which is not always easy and free of errors.

Are cystometric and cystoscopic examinations of any value for disabled incontinent elderly?

According to assessments made by the personnel and patients' own reports 42% of males and 54% of females aged 65 years or over and attending the day hospital in the town of Tampere were found to be incontinent. Incontinence was related to the frequency of urinating in the daytime. All incontinent patients of the day hospital were referred to the urologist at the University Hospital. Only about one third of these patients participated in the cystometric and cystoscopic examinations. The main reasons for non-attendance were unwillingness of the patient, referral of the patient to another place of care and resistant urinary infection. Urinary retention and small bladder capacity were common findings by cystometry and cystoscopy. A surgical management was recommended to seven per cent and conservative management to 93% of the patients. The concordance rates between the classification of the incontinence by anamnestic questions and the classification made by the urologist were not very high: the overall percentage of agreement was 40%. It is concluded, that not all incontinent elderly patients should be sent for cystometric and cystoscopic examination. Residual urine should be measured, if urinary retention is thought to be associated with incontinence. Furthermore, the classification of incontinence cannot be made only by using anamnestic questions.

Determinants of Professional-Parental Agreement for the Developmental Status of Young Handicapped Children

Several factors have resulted in an increased interest on the part of professionals in collecting and using data from parents during assessment procedures conducted with young handicapped children. However, the results of recent studies which have examined differences between the developmental status of the child as described by parents and as evaluated by professionals have been contradictory. Some have concluded that parents provide significantly higher estimates of the child's development, while others have found no such differences. This study was designed to investigate relationships between parent- and child-related demographic variables and percentages of agreement calculated between both mothers and fathers and a diagnostician. Forty mothers and 32 fathers of handicapped children enrolled in early intervention programs were interviewed in order to complete the Developmental Profile. An overall percentage of agreement was calculated for each child by comparing parental data with that obtained by a diagnostician who scored the Developmental Profile by direct testing and observation. Significant relationships were found between some of the demographic variables and percentages of agreement. Results are discussed in terms of their implications for assessment procedures.

Effects of Training on Behavioral Observations by Nurses

The effect of training institutional personnel to produce observational records oj patient behavior is described. Fifteen nurses were randomly assigned to three teams for training in the observation and recording of patient behavior as a part of a workshop at the University of Kansas Medical Center. At different stages in their training, each team dictated observational records from a videotape of patient activities in six typical behavioral settings within the hospital. Analysis of the coded records indicated high percentages of agreement among observers in the proportions of patient self-involvement, self-initiation of activities, and time spent alone, regardless of the kind of training received. Significant variance differences occurred among teams, as predicted, but were relatively small compared to the overall percentages of agreement.