Interrater Agreement Statistics Research Articles

Comparing tasks of 160 occupations across eight European countries

Purpose – Do similar job titles refer to the same work activities, as assumed often, but hardly empirically tested? The purpose of this paper is to analyze the similarity of 160 occupations within and across eight European countries using interrater agreement statistics (rWG). Design/methodology/approach – Using a multilingual web survey, experts and jobholders in the eight countries rated the frequency of ten tasks per occupation they had knowledge of (n=4,197 ratings). Three hypotheses are investigated: first, interrater agreements of occupations are similar regardless the country; second, interrater agreements of occupations are similar within countries; and third, experts and jobholders are similar in their ratings. Findings – Half of the occupations reveal no agreement across ratings, one-third shows a weak/moderate agreement and one in ten shows a strong agreement. H1 is rejected for task frequency but not for task importance. Within-country similarity of occupations is larger than across-country similarity. H2 is supported for two countries and rejected for two other countries. H3 is not supported. Jobholders demonstrate higher agreement than experts. Research limitations/implications – An empirical testing of occupation-specific tasks for a wide range of occupations across Europe seems a viable approach. Practical implications – The paper contributes to the challenges related to labor market mobility across borders. Originality/value – Work tasks for a wide range of occupations and countries, using job-specific work activities in combination with web surveys and the internet for recruitment of jobholders, have not been used before.

Molecular – genetic variance of RH blood group system within human population of Bosnia and Herzegovina

There are two major theories for inheritance of Rh blood group system: Fisher - Race theory and Wiener theory. Aim of this study was identifying frequency of RHDCE alleles in Bosnian - Herzegovinian population and introduction of this method in screening for Rh phenotype in B&H since this type of analysis was not used for blood typing in B&H before. Rh blood group was typed by Polymerase Chain Reaction, using the protocols and primers previously established by other authors, then carrying out electrophoresis in 2-3% agarose gel. Percentage of Rh positive individuals in our sample is 84.48%, while the percentage of Rh negative individuals is 15.52%. Inter-rater agreement statistic showed perfect agreement (K=1) between the results of Rh blood system detection based on serological and molecular-genetics methods. In conclusion, molecular - genetic methods are suitable for prenatal genotyping and specific cases while standard serological method is suitable for high-throughput of samples.

Clinical Application of Cine-MRI in the Visual Assessment of Mitral Regurgitation Compared to Echocardiography and Cardiac Catheterization

BackgroundDetecting and quantifying the severity of mitral regurgitation is essential for risk stratification and clinical decision-making regarding timing of surgery. Our objective was to assess specific visual parameters by cine-magnetic resonance imaging (MRI) in the determination of the severity of mitral regurgitation and to compare it to previously validated imaging modalities: echocardiography and cardiac ventriculography.MethodsThe study population consisted of 68 patients who underwent a cardiac MRI followed by an echocardiogram within a median time of 2.0 days and 49 of these patients who had a cardiac catheterization, median time of 2.0 days. The inter-rater agreement statistic (Kappa) was used to evaluate the agreement.ResultsThere was moderate agreement between cine MRI and Doppler echocardiography in assessing mitral regurgitation severity, with a kappa value of 0.47, confidence interval (CI) 0.29–0.65. There was also fair agreement between cine MRI and cardiac catheterization with a kappa value of 0.36, CI of 0.17–0.55.ConclusionCine MRI offers a reasonable alternative to both Doppler echocardiography and, to a lesser extent, cardiac catheterization for visually assessing the severity of mitral regurgitation with specific visual parameters during routine clinical cardiac MRI.

A decisional space for fMRI pattern separation using the principal component analysis-a comparative study of language networks in pediatric epilepsy

Atypical functional magnetic resonance imaging (fMRI) language patterns may be identified by visual inspection or by region of interest (ROI)-based laterality indices (LI) but are constrained by a priori assumptions. We compared a data-driven novel application of principal component analysis (PCA) to conventional methods. We studied 122 fMRI data sets from control and localization-related epilepsy patients provided by five children's hospitals. Each subject performed an auditory description decision task. The data sets, acquired with different scanners but similar acquisition parameters, were processed through fMRIB software library to obtain 3D activation maps in standard space. A PCA analysis was applied to generate the decisional space and the data cluster into three distinct activation patterns. The classified activation maps were interpreted by (1) blinded reader rating based on predefined language patterns and (2) by language area ROI-based LI (i.e., fixed threshold vs. bootstrap approaches). The different classification results were compared through κ inter-rater agreement statistics. The unique decisional space classified activation maps into three clusters (a) lower intensity typical language representation, (b) higher intensity typical, as well as (c) higher intensity atypical representation. Inter-rater agreements among the three raters were excellent (Fleiss κ = 0.85, P = 0.05). There was substantial to excellent agreement between the conventional visual rating and LI methods (κ = 0.69-0.82, P = 0.05). The PCA-based method yielded excellent agreement with conventional methods (κ = 0.82, P = 0.05). The automated and data-driven PCA decisional space segregates language-related activation patterns in excellent agreement with current clinical rating and ROI-based methods.

Preoperative Magnetic Resonance Angiography Detection of Septocutaneous Perforators in Fibula Free Flap Transfer

To investigate whether preoperative magnetic resonance angiography (MRA) is predictive of surgical findings in fibula free flap surgery for head and neck reconstruction. Retrospective review (April 2004 until September 2009) of 123 patients who underwent preoperative MRA as part of surgical planning for fibula free flap tissue transfer for head and neck reconstruction. Each MRA was reviewed by a board-certified radiologist masked to the intraoperative findings and to the number of septocutaneous perforators documented. Operative notes were reviewed and the number of septocutaneous perforators found during the operation was recorded. A κ interrater agreement statistic was calculated to compare these values. Two vascular anomalies found during the operation were undetected by MRA. Analysis of the entire cohort demonstrated that agreement between the number of perforators documented on MRA and the number found intraoperatively approached zero (unweighted κ = -0.088, P = .04). The agreement between the 2 values was 17.9% and the average percentage correctly classified was 10.9%. Contrary to previous reports, preoperative MRA does not accurately predict the presence and/or number of skin perforators found intraoperatively for a fibula free flap operation. The surgeon should not be dissuaded from planning a fibula free flap operation if skin perforators appear unfavorable on preoperative MRA because intraoperative observation is definitive. The surgeon should prepare for anomalous cases in which perforators may arise from the posterior tibial system. Further investigation is needed to achieve more accurate imaging modalities for evaluating septocutaneous perforators prior to free fibula flap transfer.

Clinical Utility of Predictors of Return-to-work Outcome Following Work-related Musculoskeletal Injury

Clinical expertise is one source of evidence that is generally under-utilised in the development of an evidence-base in rehabilitation. The current study aimed to incorporate this valuable clinical expertise in determining the utility of multiple predictors of return-to-work outcome following injury. Following systematic review of the rehabilitation literature and review, a total of 85 predictors were evaluated for clinical relevance by an expert panel of rehabilitation practitioners (n = 12). Each predictor was rated according the importance of the predictor in rehabilitation, its potential for modification and its classification into one of seven broad areas. In addition, practitioners were asked to provide a rationale as to why the predictor was important to rehabilitation. Analyses were conducted using inter-rater agreement statistics and text analysis. Predictors that were most commonly reviewed in the literature were not considered to be of greatest clinical utility, according to the current sample. From the total predictor set, only nine predictors were identified as clinically useful (i.e., both highly important and highly modifiable). Text analysis of the qualitative data revealed that these nine predictors highlighted the significance of time, context and engagement in rehabilitation practice. In the current study, predictors that were considered most clinically relevant were those that generally described workplace related processes. The findings confirmed the underlying supportive and collaborative processes that integrate predictors and account for their influence on outcome. Future rehabilitation efforts and indeed, individual outcomes, could benefit by incorporating these key predictors in targeted programs.

A New U.S. Adult Nonpatient Sample

A project to collect an American adult nonpatient sample for the Rorschach Comprehensive System (CS; Exner, 2001) took place between 1999 and 2005, and this article presents data for 450 participants in the project. Inclusion criteria, demographic information, and interrater agreement statistics are described. Findings for most of the CS variables are similar to earlier Rorschach Workshops samples, but some differences emerged and their implications for modifying interpretive guidelines are discussed.

Interrater Agreement Reconsidered: An Alternative to the rwg Indices

For continuous constructs, the most frequently used index of interrater agreement (r wg(1))can be problematic. Typically, rwg(1) is estimated with the assumption that a uniform distribution represents no agreement. The authors review the limitations of this uniform nullr wg(1) index and discuss alternative methods for measuring interrater agreement. A new interrater agreement statistic,a wg(1),is proposed. The authors derive thea wg(1)statistic and demonstrate thatawg(1) is an analogue to Cohen’s kappa, an interrater agreement index for nominal data. A comparison is made between agreement estimates based on the uniformr wg(1)and a wg(1), and issues such as minimum sample size and practical significance levels are discussed. The authors close with recommendations regarding the use ofr wg(1)/rwg(J) when a uniform null is assumed,r wg(1)/rwg(J) indices that do not assume a uniform null,awg(1) / a wg(J)indices, and generalizability estimates of interrater agreement.

Interrater agreement statistics with the microcomputer

The simple percentage of agreement measure of interrater reliability has retained its popularity (Kelly, 1977) despite its failure to take into account the pro­ portion of agreement due to chance. Consequently, misleading or insufficient reliability information has entered the professional literature. Demonstration of the role of chance in professional research was provided by Spitzer and Fleiss (1974) with a reanalysis of psy­ chiatric diagnostic studies and by Watkins (1979) in a reanalysis of manuscript-reviewer data. Chance-corrected solutions to the nominal scale classification problem have been developed by Cohen (1960) and Scott (1955). Both solutions utilize the formula Po - Pc/1 - Pc, where Po is the observed proportion of agreement and Pc is the probability of agreeing by chance alone. Scott's (1955) Pi and Cohen's (1960) Kappa are both widely used (Krippendorff, 1970), but they differ in the way expected frequencies (pc) are computed. A number of useful programs have been developed for computer applications of Kappa (Antonak, 1977; Berk & Campbell, 1976) and of Pi (Thornton & Croskey, 1975), but they are restricted to main-frame computers operating with the FORTRAN language. This paper presents a program that utilizes the BASIC language and a microcomputer to calculate both Pi and Kappa. Description. The program is written in Applesoft BASIC for the Apple II microcomputer. It occupies