Inter-rater Agreement Analysis Research Articles

Do epilepsy specialist nurses use a similar history-taking process as consultant neurologists in the differential diagnosis of patients presenting with a first seizure?

The development of specialist nursing practice has blurred the boundaries between medicine and nursing. This mainly qualitative study compares the structure of epilepsy specialist nurse (ESN) and consultant neurologist (CN) clinical interviews at first seizure presentation and opinion on diagnosis. Twenty patients with a suspected first seizure were randomly allocated for clinical review with an ESN and then a CN, or vice versa. Clinical interviews were unstructured and audio-recorded. The ESN and CN reached an independent diagnosis for each patient. Audiotapes were transcribed verbatim. Emergent themes were identified, catalogued and grouped into major thematic areas. Annotated audio recordings, medical notes and dictated clinic letters were used to validate findings. Statistical analysis of inter-rater agreement of diagnosis was evaluated using Kappa. The clinical interviews of CN and ESN were similar in structure. Differences demonstrated CNs concentrated on the prodrome to events and expressed less diagnostic uncertainty. ESNs concentrated on post-ictal recovery and used more investigations. Complete disagreement on diagnosis occurred in 5 (25%) patients. Kappa score = 0.510, demonstrating a moderate level of inter rater agreement on diagnosis between the CN and ESN.

A Twenty-First Century Assessment of Values Across the Global Workforce

This article provides current Schwartz Values Survey (SVS) data from samples of business managers and professionals across 50 societies that are culturally and socioeconomically diverse. We report the society scores for SVS values dimensions for both individual- and societal-level analyses. At the individual-level, we report on the ten circumplex values sub-dimensions and two sets of values dimensions (collectivism and individualism; openness to change, conservation, self-enhancement, and self-transcendence). At the societal-level, we report on the values dimensions of embeddedness, hierarchy, mastery, affective autonomy, intellectual autonomy, egalitarianism, and harmony. For each society, we report the Cronbach’s α statistics for each values dimension scale to assess their internal consistency (reliability) as well as report interrater agreement (IRA) analyses to assess the acceptability of using aggregated individual level values scores to represent country values. We also examined whether societal development level is related to systematic variation in the measurement and importance of values. Thus, the contributions of our evaluation of the SVS values dimensions are two-fold. First, we identify the SVS dimensions that have cross-culturally internally reliable structures and within-society agreement for business professionals. Second, we report the society cultural values scores developed from the twenty-first century data that can be used as macro-level predictors in multilevel and single-level international business research.

SPECT/CT stabilizes the interpretation of somatostatin receptor scintigraphy findings: a retrospective analysis of inter-rater agreement

Correlating the anatomical information from CT with the functional information from SPECT improves diagnostic accuracy of somatostatin-receptor-scintigraphy (SRS) in patients with neuroendocrine tumors (NET). The aim of the present study was to investigate the impact of dual modality SPECT/CT on the inter-rater agreement in SRS. Twenty-five unselected patients with suspected or histologically proven NET in whom whole body planar imaging and low-dose SPECT/CT had been performed after injection of 200 MBq In-111-octreotide were included retrospectively. Images were interpreted independently by 2 nuclear medicine physicians, an experienced one and an inexperienced one. Both readers first re-evaluated the planar whole-body images alone, then added the SPECT images, and finally the CT-images. Lesions with pathologically increased tracer uptake were categorized according to the following 3-point score: equivocal, probably pathologic, and definitely pathologic. Cohen's linear-weighted kappa coefficient kappa was used to quantify inter-rater agreement. A total number of 50 lesions were described in 23 of the 25 patients. The two readers showed only moderate agreement in the interpretation of the planar findings (kappa = 0.593). Agreement improved to substantial by adding SPECT (kappa = 0.736) and to very good by adding SPECT/CT (kappa = 0.860). SPECT/CT resulted in up-staging of 18% of the lesions and down-staging of 12% compared to planar + SPECT (experienced reader). In addition, SPECT/CT tended to reduce the frequency of indefinite scores (equivocal, probably pathologic), from 18% in planar + SPECT to 6% (p = 0.065). Change of lesion localization by SPECT/CT tended to contribute to the change of lesion score (p = 0.055). The present results suggest that low-dose SPECT/CT stabilizes report quality in SRS by improving inter-rater agreement.

A human factors and survey methodology-based design of a web-based adverse event reporting system for families

Purpose Adverse event reporting systems allow healthcare institutions to detect and prevent recurrence of avoidable patient harm. It is known that standard reporting systems, which are initiated by clinicians, detect only a minority of chart-documented adverse events. The objective of the study was to develop a web-based system, the Family Reporting System (FRS), to elicit adverse event reports from families of children admitted to hospital through survey methodology and human factors engineering techniques. Measurements Face validity and usability were measured via standardized survey instruments. Utility was measured via the rate, typology, degree of harm, likelihood of recurrence, quality of information, and inter-rater agreement analysis of the reported events. Results The FRS has good face validity, excellent usability, and good clinical utility. Conclusion The application of survey and human factors methodologies to the design of an electronic system is an effective means of developing an electronic adverse event reporting system for the use of families of pediatric patients.

Does chair type influence outcome in the timed “Up and Go” test in older persons?

Objective To test the effects of the use of a collapsible, portable chair (chair B), as opposed to a ‘standard’ chair (chair A), on the outcome of the timed “Up and Go” (TUG) test. Design Cross-sectional. Setting Multipurpose senior centres. Participants Mobile older persons (N=118, mean age 77 years (range 62–99 years)). Outcome measures Time to complete the timed “Up and Go” test using chair A and chair B, and inter-rater agreement in the time scores. Results Time taken to complete the TUG test did not differ by chair type [median (interquartile range, IQR) = 12.3 (9.53–15.9) and 12.6 (9.7–16.6)] seconds for Chair A and B respectively, p-value=0.87. In multiple regression analyses, factors that impacted on time difference in test performance for the two chairs were use of a walking aid during the test [Odds ratio (OR) = 3.7 95%CI 1.1–11.9, p=0.031], observed difficulty with mobility (OR= 27.7 95%CI 2.6–290, p=0.006), and a history of arthritis in the knees (OR= 2.9 95%CI 1.0–8.7, P=0.05). In an inter-rater agreement analysis, no significant difference was found between time scores recorded by the two raters; median (IQR) = 12.4 (10.9–15.9) and 12.3 (7.2–59.1) seconds for the occupation therapist and for the research assistant, respectively (Wilcoxon matched pairs test, p=0.124, Spearman correlation coefficient = 0.99, p Conclusion The use of a portable canvas chair with standardised specifications offers an acceptable alternative to the use of a ‘standard’ chair in assessments of fall risk using the TUG test in field settings where field workers are reliant on public transport.

What is Relapse? A Contemporary Exploration of Treatment of Alcoholism

In clinical settings, clients treated for alcohol abuse or dependency often engage with multiple service providers. Nonstandardized definitions for treatment outcome obscure treatment success. Using content analysis and preestablished selection criteria on a set of alcohol treatment outcome studies conducted between 1990 and 2006, we evaluated the definition of relapse using a typology derived from an interrater agreement analysis. The interrater agreement was good and the modal category indicated that a multifaceted operationalization was most often used to indicate success. Key to standardizing treatment success might be inclusion of multiple domains of life function in defining relapse.

Inter-Rater Agreement for a Retrospective Exposure Assessment of Asbestos, Chromium, Nickel and Welding Fumes in a Study of Lung Cancer and Ionizing Radiation

A retrospective exposure assessment of asbestos, welding fumes, chromium and nickel (in welding fumes) was conducted at the Portsmouth Naval Shipyard for a nested case-control study of lung cancer risk from external ionizing radiation. These four contaminants were included because of their potential to confound or modify the effect of a lung cancer-radiation relationship. The exposure assessment included three experienced industrial hygienists from the shipyard who independently assessed exposures for 3519 shop/job/time period combinations. A consensus process was used to resolve estimates with large differences. Final exposure estimates were linked to employment histories of the 4388 study subjects to calculate their cumulative exposures. Inter-rater agreement analyses were performed on the original estimates to better understand the estimation process. Although concordance was good to excellent (78-99%) for intensity estimates and excellent (96-99%) for frequency estimates, overall simple kappa statistics indicated only slight agreement beyond chance (kappa < 0.2). Unbalanced distributions of exposure estimates partly contributed to the weak observed overall inter-rater agreement. Pairwise weighted kappa statistics revealed better agreement between two of the three panelists (kappa = 0.19-0.65). The final consensus estimates were similar to the estimates made by these same two panelists. Overall welding fume exposures were fairly stable across time at the shipyard while asbestos exposures were higher in the early years and fell in the mid-1970s. Mean cumulative exposure for all study subjects was 520 fiber-days cc(-1) for asbestos and 1000 mg-days m(-3) for welding fumes. Mean exposure was much lower for nickel (140 microg-days m(-3)) and chromium (45 microg-days m(-3)). Asbestos and welding fume exposure estimates were positively associated with lung cancer in the nested case-control study. The radiation-lung cancer relationship was attenuated by the inclusion of these two confounders. This exposure assessment provided exposure estimates that aided in understanding of the lung cancer-radiation relationship at the shipyard.

Assessment of Right Ventricular Size and Function: Echo Versus Magnetic Resonance Imaging

The assessment of right ventricular (RV) size and function is important in the management of many patients with heart disease. Although magnetic resonance imaging (MRI) is considered the gold standard for quantitation of ventricular volumes and systolic function, subjective assessment ("eyeball") by echocardiography is the modality most often used for the RV. The echocardiographic "eyeball" method of assessing RV size and systolic function was compared with quantitative MRI. Patients with right-sided congenital heart disease who underwent an echocardiogram within 6 months of MRI formed the study group. Four echocardiographers blinded to the MRI results reviewed the echocardiograms to subjectively assess RV size and systolic function. The reliability of an echocardiographer in accurately identifying a severely dilated RV and moderately to severely diminished RV systolic function was measured using the Kappa coefficient. Inter-rater agreement was also assessed using Kappa. The study group consisted of 22 patients aged 16.6 +/- 7.1 years, with interval between MRI and echocardiogram of 49 +/- 54 days. Using echocardiography, reliability for accurately identifying a severely dilated RV was "slight" with a prevalence-adjusted bias-adjusted Kappa (PABAK) of 0.25; and for identifying moderately to severely diminished RV systolic function was fair with a PABAK of 0.43. Inter-rater agreement analysis was poor for both with Kappas of 0.07 (P = .22) and 0.12 (P = .09), respectively. The usefulness of the echocardiographic "eyeball" method to estimate RV size and systolic function in patients with right heart disease has limitations when compared with MRI, specifically in regard to the variability between echocardiographers.

Barthel Index in a Middle-East Country: Translation, Validity and Reliability

Background: In Iran, a Middle-East country, no disability scale has been translated and validated for use in stroke clinical trials. This study was designed to translate the Barthel Index and make its Persian translated form valid and reliable. Methods: All items of the Barthel Index were translated into Persian. Also, the Modified Rankin Scale (MRS) was also translated to Persian. Telephone interview was used as the method of data acquisition. Two interviewers were chosen for this means in order to accelerate data gathering and measure interrater agreement. Samples were selected from Isfahan Cardiovascular Research Center’s Stroke Registry Unit, a WHO collaborating center in the center of Iran. All the patients were registered as stroke or intracerebral hemorrhage (ICH). These patients were inhabitants of Isfahan Province who had suffered from stroke or ICH between 12 and 24 months before data acquisition. Chronbach’s alpha, test-retest reliability, concurrent validity with MRS, interrater agreement and item analysis were done for the translated questionnaire. Results: Translated questionnaires were filled by interview from 459 stroke patients. Their mean age was 68.11 ± 11.59 years. 243 of them were male (52.9%). Cronbach’s alpha was 0.9354. Spearman’s correlation coefficient between translated Barthel Index scores and MRS scores was –0.912. Spearman’s correlation coefficient between 2 scores, to determine test-retest reliability was 0.989. Concordance correlation to determine interrater agreement was 0.994. All corrected item–total correlations were greater than 0.5. Conclusions: The Persian translated version of the Barthel Index is a reliable and valid questionnaire for use in stroke clinical trials.

Physician and Nurse Staffing in Nursing Homes

Objectives To assess nursing home staffing data reported in the Online Survey Certification and Reporting (OSCAR) system database for research and policy. Design Comparisons were made between OSCAR and a concurrent research survey of staffing data collected for the same facilities, using inter-rater agreement and correlation analyses. Setting Freestanding nursing homes from New York State (NYS) in 1997 (N = 327). Measurements Selected staffing variables were defined in comparable terms in both OSCAR and the NYS survey. Results The two data sources were in substantial agreement on the reported availability of a full-time physician (other than medical director) and of a physician assistant or nurse practitioner (Kappa >0.7), and they correlated well in the full-time equivalent (FTE) number of such staff (Spearman correlation >0.6). The correlation was 0.8 for FTE registered nurses (RNs), 0.7 for licensed practical nurses (LPNs), and 0.8 for certified nurse aides (CNAs). In terms of average nurse hours per patient day, separately for RNs, LPNs, CNAs, and all combined, the correlation was relatively weak (between 0.3 and 0.6). Overall staffing levels tended to be lower in OSCAR than in NYS. Conclusion The OSCAR data are useful for exploring relationships between staffing and various quality of care outcomes, but may not be accurate enough on a case-specific basis, or to determine policy regarding minimal staffing levels using average nurse hours per patient day measures. More systematic and timely efforts are needed to refine the OSCAR content and survey methodology to document nursing home staffing information. To assess nursing home staffing data reported in the Online Survey Certification and Reporting (OSCAR) system database for research and policy. Comparisons were made between OSCAR and a concurrent research survey of staffing data collected for the same facilities, using inter-rater agreement and correlation analyses. Freestanding nursing homes from New York State (NYS) in 1997 (N = 327). Selected staffing variables were defined in comparable terms in both OSCAR and the NYS survey. The two data sources were in substantial agreement on the reported availability of a full-time physician (other than medical director) and of a physician assistant or nurse practitioner (Kappa >0.7), and they correlated well in the full-time equivalent (FTE) number of such staff (Spearman correlation >0.6). The correlation was 0.8 for FTE registered nurses (RNs), 0.7 for licensed practical nurses (LPNs), and 0.8 for certified nurse aides (CNAs). In terms of average nurse hours per patient day, separately for RNs, LPNs, CNAs, and all combined, the correlation was relatively weak (between 0.3 and 0.6). Overall staffing levels tended to be lower in OSCAR than in NYS. The OSCAR data are useful for exploring relationships between staffing and various quality of care outcomes, but may not be accurate enough on a case-specific basis, or to determine policy regarding minimal staffing levels using average nurse hours per patient day measures. More systematic and timely efforts are needed to refine the OSCAR content and survey methodology to document nursing home staffing information.

Interrater agreement for the schedule for affective disorders and schizophrenia epidemiological version for school-age children (K-SADS-E).

The main objective of this study was to assess the interrater agreement for the Schedule for Affective Disorders and Schizophrenia Epidemiological version for School-Age Children (K-SADS-E). Four interviewers being trained with the K-SADS-E scored independently 29 videotaped interviews performed with psychiatric outpatients in the ADHD Outpatient Clinic at Hospital de Clínicas de Porto Alegre. Interrater agreement analysis was performed using the kappa coefficient (k). Kappa coefficients were.93 (p<.001) for affective disorders,.9 (p<.001) for anxiety disorders,.94 (p<.001) for attention-deficit/hyperactivity disorders and disruptive behavior disorders. These findings suggest an excellent interrater agreement for the diagnosis of several mental disorders in childhood and adolescence by the Brazilian Portuguese version of the K-SADS-E.

New LightCycler PCR for Rapid and Sensitive Quantification of Parvovirus B19 DNA Guides Therapeutic Decision-Making in Relapsing Infections

Detection of parvovirus B19 DNA offers diagnostic advantages over serology, particularly in persistent infections of immunocompromised patients. A rapid, novel method of B19 DNA detection and quantification is introduced. This method, a quantitative PCR assay, is based on real-time glass capillary thermocycling (LightCycler [LC]) and fluorescence resonance energy transfer (FRET). The PCR assay allowed quantification over a dynamic range of over 7 logs and could quantify as little as 250 B19 genome equivalents (geq) per ml as calculated for plasmid DNA (i.e., theoretically >or=5 geq per assay). Interrater agreement analysis demonstrated equivalence of LC-FRET PCR and conventional nested PCR in the diagnosis of an active B19 infection (kappa coefficient = 0.83). The benefit of the new method was demonstrated in an immunocompromised child with a relapsing infection, who required an attenuation of the immunosuppressive therapy in addition to repeated doses of immunoglobulin to eliminate the virus.

Confiabilidade da "Entrevista Clínica Estruturada para o DSM-IV - Versão Clínica" traduzida para o português

OBJETIVOS: Verificar a confiabilidade da "Entrevista Clínica Estruturada para o DSM-IV - Versão Clínica (SCID-CV)" traduzida para o português. MÉTODOS: Foram submetidos, a duas entrevistas independentes (teste-reteste), 45 pacientes psiquiátricos em seguimento no Hospital das Clínicas da Faculdade de Medicina de Ribeirão Preto da Universidade de São Paulo (HC-FMRP/USP). Os dados foram analisados pelo Coeficiente Kappa (K). RESULTADOS: O Kappa ponderado foi excelente (Kw=0,83). A confiabilidade foi estatisticamente significante em transtorno do humor (K=0,87); transtornos psicóticos (K=0,90); transtornos relacionados ao uso de substância (K=0,76); transtornos de ansiedade (K=0,61); e nas categorias diagnósticas específicas analisadas, exceto em agorafobia sem história de transtorno do pânico (K=-0,04). CONCLUSÕES: A SCID-CV traduzida e adaptada para o português apresenta, em geral, boa confiabilidade, mas a ausência de questões e critérios diagnósticos específicos no próprio instrumento em diagnósticos, como agorafobia sem história de transtorno de pânico, diminuiu sua confiabilidade.

A BAYESIAN ANALYSIS FOR INTER-RATER AGREEMENT

An analysis of inter-rater agreement is presented. We study the problem with several raters using a Bayesian model based on the Dirichlet distribution. Inter-rater agreement, including global and partial agreement, is studied by determining the joint posterior distribution of the raters. Posterior distributions are computed with a direct resampling technique. Our method is illustrated with an example involving four residents, who are diagnosing 12 psychiatric patients suspected of having a thought disorder. Initially employing analytical and resampling methods, total agreement between the four is examined with a Bayesian testing technique. Later, partial agreement is examined by determining the posterior probability of certain orderings among the rater means. The power of resampling is revealed by its ability to compute complex multiple integrals that represent various posterior probabilities of agreement and disagreement between several raters.

Validation study of injury surveillance data collected through Queensland hospital emergency departments

Objective: The objective of this study was to assess and validate the EDIS (Emergency Department Information System) as a means of acquiring Level 2 injury surveillance data as defined by the National Data Standards for Injury Surveillance in public hospital emergency departments in Queensland. Methods: This validation was based on interviews with patients presenting to selected emergency departments in Queensland for treatment of an injury which was then compared to data obtained from emergency department staff completing the routine medical record for these patients using EDIS. To assess the level of coding variability and set baseline levels of agreement the data were separately coded by three experienced coders and the level of agreement evaluated. At the time of the validation emergency department staff were also asked to complete a questionnaire relating to their attitudes towards EDIS and to injury surveillance. Results: A total of 572 cases of injury were collected during the period of the validation of which 509 (89.0%) corresponded to an injury recorded in EDIS. Demographic details collected in the emergency department data were coded correctly in almost all cases. The exception was Indigenous status where the emergency department systems recorded only 38% of the Indigenous cases correctly. Injury data fields were better than 85% complete for two sites but only 65% complete for the third. Coding agreement between Emergency Department Information System and the validation study for items relating to the injury event varied from 95% for Intent to 50% for Activity when injured. The interrater agreement analysis revealed that the highest agreement between the coders was for intent (94–96%) while the lowest was for main injury factor (63–78%). The results of the emergency department staff satisfaction survey showed a high acceptance of EDIS and injury surveillance in general. Conclusions: This analysis demonstrates that injury surveillance is a viable proposition for most emergency departments using computerized patient management systems designed with injury surveillance in mind. Emergency department-based injury surveillance provides a pro-active mechanism for monitoring injuries in specific areas of known concern, identifies previously unrecognized injury events and risk factors and permits the evaluation of preventive strategies.

Rapid and sensitive detection of immunoglobulin M (IgM) and IgG antibodies against canine distemper virus by a new recombinant nucleocapsid protein-based enzyme-linked immunosorbent assay.

Canine distemper morbillivirus (CDV) infection causes a frequently fatal systemic disease in a broad range of carnivore species, including domestic dogs. In CDV infection, classical serology provides data of diagnostic and prognostic values (kinetics of seroconversion) and is also used to predict the optimal vaccination age of pups. Routine CDV serology is still based on time- and cost-intensive virus neutralization assays (V-NA). Here, we describe a new capture-sandwich enzyme-linked immunosorbent assay (ELISA) that uses recombinant baculovirus-expressed nucleocapsid (N) protein of a recent CDV wild-type isolate (2544/Han95) for the detection of CDV-specific antibodies in canine sera. Recombinant antigen was produced with high efficacy in Heliothis virescens larvae. The capture-sandwich ELISA enabled a clear-cut qualitative evaluation of the CDV-specific immunoglobulin G (IgG) and IgM serostatuses of 196 and 35 dog sera, respectively. Inter-rater agreement analysis (kappa = 0.988) indicated that the ELISA can be used unrestrictedly as a substitute for the V-NA for the qualitative determination of CDV-specific IgG serostatus. In an attempt to semiquantify N-specific antibodies, a one-step-dilution (alpha method) IgG-specific ELISA was implemented. Alpha values of >/=50% showed very good inter-rater agreement (kappa = 0.968) with V-NA titers of >/=1/100 50% neutralizing dose (ND50) as measured against the central European CDV wild-type isolate 2544/Han95 in canine sera originating from northern Germany. An ND50 titer of 1/100 is considered a threshold, and titers of >/=1/100 indicate a resilient, protective immunity. CDV N-specific antibodies of the IgM class were detected by the newly developed ELISA in 9 of 15 sera obtained from dogs with symptoms of acute distemper. In leucocytes of 5 of the 15 dogs (all of which were also IgM positive) CDV RNA was detected by reverse transcription (RT)-PCR. The recombinant capture-sandwich ELISA detecting N-specific antibodies of the IgG class provided superior sensitivity and specificity and thus represents a rapid and cost-effective alternative to classical CDV V-NA. By detection of specific IgM antibodies, the ELISA will be complementary to RT-PCR and V-NA in the diagnosis of acute distemper infections.