Measure Of Inter-rater Agreement Research Articles

Kappa and AC1/2 statistics: beyond the paradox

Dear Editor, We thank Loef and colleagues for their interest in our article about the revised Cochrane risk of bias tool for randomized trials (RoB 2) inter-rater reliability and usability [[1]Minozzi S. Cinquini M. Gianola S. Gonzalez-Lorenzo M. Banzi R. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application.J Clin Epidemiol. 2020; 126: 37-44Abstract Full Text Full Text PDF PubMed Scopus (37) Google Scholar]. Their analysis and comments mainly focused on the choice of the more appropriate measure of inter-rater agreement. We acknowledge the paradoxes that may occur with the Kappa statistics, and referred to it as possible limitation of our results: “The expected agreement can exceed the observed agreement and then generate kappa values lower than 0. This is why in some cases Fleiss’ kappa may return low values even when agreement is actually high (Fleiss’ kappa paradox). Our results might be further affected by this paradox.”[[1]Minozzi S. Cinquini M. Gianola S. Gonzalez-Lorenzo M. Banzi R. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application.J Clin Epidemiol. 2020; 126: 37-44Abstract Full Text Full Text PDF PubMed Scopus (37) Google Scholar]. We agree that Gwet's AC1/2 statistics may overcome this paradoxes and be a valuable option. The two approaches differ in how the expected agreement by chance (“pe”) is calculated and this can produce an overestimation of the agreement with Gwet's AC1/2 statistics. As expected, the reanalysis of the data using this statistic led to a higher agreement, which would have been classified as moderate. We believe the main reason for this difference can be due to the sample size of our analysis (70 measurements). The two approaches are likely to estimate similar agreement values when a higher number of measurements are made, while results may be highly variable if samples are small [[2]Ohyama Tetsuji Statistical inference of Gwet's AC1 coefficient for multiple raters and binary outcomes.Commun in Statis - Theory and Meths. 2021; 50: 3564-3572https://doi.org/10.1080/03610926.2019.1708397Crossref Scopus (3) Google Scholar,[3]Wongpakaran N. Wongpakaran T. Wedding D. L Gwet K.L. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples.BMC Med Res Methodol. 2013; 13: 61https://doi.org/10.1186/1471-2288-13-61Crossref PubMed Scopus (417) Google Scholar]. We compared the number of studies where all the raters gave the same judgement and the interrater agreement values (Table 1) and found a good consistency. For instance, the raters gave the same overall judgement in 12 of 70 (17%) of the studies and this is consistent with the slight interrater agreement (IRR = 0.16) measured with Fleiss’ Kappa [[1]Minozzi S. Cinquini M. Gianola S. Gonzalez-Lorenzo M. Banzi R. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application.J Clin Epidemiol. 2020; 126: 37-44Abstract Full Text Full Text PDF PubMed Scopus (37) Google Scholar]. This seems to apply also to single domains.Table 1Interrater agreement measures for single domains and overall judgment compared with number studies where all the raters gave the same ROB judgementROB 2 domain% same ROB judgementIRR (95% CI)1470.45 (0.37 to 0.53)220Assignment 0.04 (-0.06 to 0.14)Adhering 0.21 (0.11 to 0.31)3580.22 (0.14 to 0.30)4500.27 (0.19 to 0.35)5310.30 (0.22 to 0.38)Overall170.16 (0.08 to 0.24) Open table in a new tab Simulation studies focused on the comparison between Gwet's AC1/2 statistics and Cohen's kappa for binary data. To our knowledge, robust comparison between Gwet's AC1/2 and Fleiss’ Kappa are lacking, making difficult any strong conclusion about the best approach. Both approaches rely on several assumptions about the experience of raters. The violation of these assumptions could contribute to produce paradoxical results. In practice, it is almost inevitable that the raters will vary in terms of their experience and expertise, and this was the case in our study. We believed Fliess Kappa was appropriate to our analysis given that our raters had different competences, expertise and training. Indeed, one of Kappa statistic's assumption is that raters’ judgment is independent. Regarding the second issue, we chose the Landis and Koch classification in a conservative way considering the point estimate only. However, cIMP may be a good alternative approach we will consider it in future analysis.

Bayesian approaches to the weighted kappa-like inter-rater agreement measures.

Inter-rater agreement measures are used to estimate the degree of agreement between two or more assessors. When the agreement table is ordinal, different weight functions that incorporate row and column scores are used along with the agreement measures. The selection of row and column scores is effectual on the estimated degree of agreement. The weighted measures are prone to the anomalies frequently seen in agreement tables such as unbalanced table structures or grey zones due to the assessment behaviour of the raters. In this study, Bayesian approaches for the estimation of inter-rater agreement measures are proposed. The Bayesian approaches make it possible to include prior information on the assessment behaviour of the raters in the analysis and impose order restrictions on the row and column scores. In this way, we improve the accuracy of the agreement measures and mitigate the impact of the anomalies in the estimation of the strength of agreement between the raters. The elicitation of prior distributions is described theoretically and practically for the Bayesian estimation of five agreement measures with three different weights using an agreement table having two grey zones. A Monte Carlo simulation study is conducted to assess the classification accuracy of the Bayesian and classical approaches for the considered agreement measures for a given level of agreement. Recommendations for the selection of the highest performing agreement measure and weight combination are made in the breakdown of the table structure and sample size.

Evaluation of Cohen's kappa and other measures of inter-rater agreement for genre analysis and other nominal data

Cohen's kappa (κ) is often recommended for nominal data as a measure of inter-rater (inter-coder) agreement or reliability. In this paper we ask which term is appropriate in genre analysis, what statistical measures are valid to measure it, and how much the choice of units affects the values obtained. We find that although both agreement and reliability may be of interest, only agreement can be measured with nominal data. Moreover, while kappa may be appropriate for macrostructure or corpus analysis, it is inappropriate for move or component analysis, due to the requirement of κ that the units be predetermined, fixed, and independent. κ further assumes that all disagreements in category assignment are equally likely, which may not be true. We also describe other measures, including correlation, chi square, and percent agreement, and demonstrate that despite its limitations, percent agreement is the only valid measure in many situations. Finally, we demonstrate why choice of unit has a large effect on the value calculated. These findings also apply to other studies in applied linguistics using nominal data. We conclude that the methodology used needs to be clearly explained to ensure that the requirements have been met, as in any other statistical testing.

The impact of grey zones on the accuracy of agreement measures for ordinal tables

BackgroundIn an inter-rater agreement study, if two raters tend to rate considering different aspects of the subject of interest or have different experience levels, a grey zone occurs among the levels of a square contingency table showing the inter-rater agreement. These grey zones distort the degree of agreement between raters and negatively impact the decisions based on the inter-rater agreement tables. In this sense, it is important to know how the existence of a grey zone impacts the inter-rater agreement coefficients to choose the most reliable agreement coefficient against the grey zones to reach out with more reliable decisions.MethodsIn this article, we propose two approaches to create grey zones in simulations setting and conduct an extensive Monte Carlo simulation study to figure out the impact of having grey zones on the weighted inter-rater agreement measures for ordinal tables over a comprehensive simulation space.ResultsThe weighted inter-rater agreement coefficients are not reliable against the existence of grey zones. Increasing sample size and the number of categories in the agreement table decreases the accuracy of weighted inter-rater agreement measures when there is a grey zone. When the degree of agreement between the raters is high, the agreement measures are not significantly impacted by the existence of grey zones. However, if there is a medium to low degree of inter-rater agreement, all the weighted coefficients are more or less impacted.ConclusionsIt is observed in this study that the existence of grey zones has a significant negative impact on the accuracy of agreement measures especially for a low degree of true agreement and high sample and tables sizes. In general, Gwet’s AC2 and Brennan-Prediger’s κ with quadratic or ordinal weights are reliable against the grey zones.

Distant Supervised Construction and Evaluation of a Novel Dataset of Emotion-Tagged Social Media Comments in Spanish

Tagged language resources are an essential requirement for developing machine-learning text-based classifiers. However, manual tagging is extremely time consuming and the resulting datasets are rather small, containing only a few thousand samples. Basic emotion datasets are particularly difficult to classify manually because categorization is prone to subjectivity, and thus, redundant classification is required to validate the assigned tag. Even though, in recent years, the amount of emotion-tagged text datasets in Spanish has been growing, it cannot be compared with the number, size, and quality of the datasets in English. Quality is a particularly concerning issue, as not many datasets in Spanish included a validation step in the construction process. In this article, a dataset of social media comments in Spanish is compiled, selected, filtered, and presented. A sample of the dataset is reclassified by a group of psychologists and validated using the Fleiss Kappa interrater agreement measure. Error analysis is performed by using the Sentic Computing tool BabelSenticNet. Results indicate that the agreement between the human raters and the automatically acquired tag is moderate, similar to other manually tagged datasets, with the advantages that the presented dataset contains several hundreds of thousands of tagged comments and it does not require extensive manual tagging. The agreement measured between human raters is very similar to the one between human raters and the original tag. Every measure presented is in the moderate agreement zone and, as such, suitable for training classification algorithms in sentiment analysis field.

A measure of interrater absolute agreement for ordinal categorical data

A measure of interrater absolute agreement for ordinal scales is proposed capitalizing on the dispersion index for ordinal variables proposed by Giuseppe Leti. The procedure allows to overcome the limits affecting traditional measures of interrater agreement in different fields of application. An unbiased estimator of the proposed measure is introduced and its sampling properties are investigated. In order to construct confidence intervals for interrater absolute agreement both asymptotic results and bootstrapping methods are used and their performance is evaluated. Simulated data are employed to demonstrate the accuracy and practical utility of the new procedure for assessing agreement. Finally, an application to a real case is provided.

Part I: Accuracy of Teledermatology in Inflammatory Dermatoses.

Teledermatology is assuming a progressively greater role as a healthcare delivery method, especially now, during this pandemic time. It is important to know how accurate this tool is for different skin diseases. Most of the studies have focused on skin neoplasms or general dermatology. Studies based on a large number of inflammatory dermatoses have not yet been performed. Such knowledge can help dermatologists to decide whether endorsing this method or not. Our objective was to determine the accuracy of teledermatology in inflammatory dermatoses in a robust number of cases. A retrospective cohort study was conducted in São Paulo, Brazil, from July 2017–18, where a store-and-forward Teledermatology project was implemented under primary-care attention to triage surgical, more complex, or severe dermatoses. A total of 30,976 patients presenting 55,012 lesions took part in the project. Thirteen participating teledermatologists had three options to refer the patients: directly to biopsy, to the in-person dermatologist or back to the general physician with most probable diagnosis and management. In the group referred to the in-person dermatologist, we looked for the 20 most frequent International Classification of Diseases and Related Health Problems- 10th revision (ICD-10) of inflammatory dermatoses, which resulted in 739 patients and 739 lesions. As patients had been triaged by teledermatology previously, we were able to compare ICD-10 codes filled both by teledermatogists and by in-person dermatologists. The proportion of complete, partial, and no agreement rates between the in-person dermatologist's and the teledermatologist's diagnoses was used for accuracy. We also calculated Cohen's kappa, a statistical measure of inter-rater agreement, for complete agreement. The mean complete agreement rate for all twenty dermatoses was 78% (31–100%) and kappa = 0.743; partial agreement 8%; and no agreement 14%, presenting variability according to the disease. Our study showed that teledermatology for inflammatory dermatoses has a high accuracy. This result reassures that it can be a proper option for patient care.

Rethinking Assessment Concepts in Dental Education.

Introduction Dental education involves teaching and assessing the acquisition of verifiable domains that require superior psychomotor, communication, and cognitive skills. Evolving technologies and methods of assessment could enhance student's learning environment and improve tutor assessment experience. The aim of this study was to review the current body of research and evaluate the effectiveness of various methods of assessments in improving learning and performance in preclinical and clinical dental practice. Materials and Methods A search strategy was implemented using electronic search in major databases. The following key terms, clinical skills, preclinical, dental students, and assessment, were included in the search. Two reviewers independently screened all the articles retrieved following very specific inclusion criteria. Results The initial search generated 5371 articles and 24 articles were selected for review and data extraction. Cohen's kappa coefficient was used to measure interrater agreement and a score of 94.7% was obtained. Conclusion Preclinical assessment is an effective tool for promoting skills transfer to clinical phase. Early psychomotor skills assessment is valuable. It allows early intervention in the learning process and assists in effective utilization of learning resources. Technology-enhanced assessment systems allow better patient simulation, enhance learning and self-assessment experiences, and improve performance evaluation. However, these systems serve as an adjunct to conventional assessment methods. Further research should aim at calibrating and integrating these systems to optimize students learning and performance.

The Pediatric-Specific American Society of Anesthesiologists Physical Status Score: A Multicenter Study.

When applied to the pediatric population, the American Society of Anesthesiologists physical status (ASA-PS) classification has exhibited poor reliability due to its subjective and adult-focused definitions. This study was done to measure interrater agreement of a pediatric-adapted ASA-PS classification and to solicit multicenter perspectives to optimize the pediatric ASA-PS classification. A prospective, mixed-methods study of 197 pediatric anesthesiologists from 13 academic pediatric hospitals in the United States, Europe, and Australia surveyed in May and July 2019. Participants assigned ASA-PS scores (I to V) for 15 pediatric cases with a heterogeneous mix of acute and chronic health conditions undergoing a variety of surgical and related procedures. Pediatric-adapted definitions of ASA-PS were provided. The intraclass correlation coefficient (ICC) was used to assess interrater reliability of ASA-PS scores. The ICC was estimated using 2-way mixed-effects modeling, accounting for multiple raters assigning scores for the same set of cases. Qualitative feedback on the pediatric-adapted ASA-PS classification was analyzed with line-by-line coding. The survey response rate was 83.8% (165 of 197). The ICC agreement among participants on ASA-PS scoring across all 15 clinical cases was 0.58 (95% confidence interval [CI], 0.42-0.77). ICC did not vary significantly by years of anesthesiology practice. ICC varied across hospitals (range: 0.34; 95% CI, 0.12-0.63 to 0.79; 95% CI, 0.66-0.91). The highest level of agreement occurred with cases most often scored as ASA-PS I, IV, and V; the lowest agreement occurred with cases most often scored ASA-PS II and III. Clarification of how well a chronic condition was controlled and presence of an acute illness were 2 common themes suggested to optimize the validity of the pediatric-adapted ASA-PS definitions. The pediatric-adapted ASA-PS classification had moderate interrater reliability among pediatric anesthesiologists. The lower reliability of scoring for ASA-PS II and III cases, in particular, supports the need for further ASA-PS definition refinement for pediatric populations.

Non-contrast-enhanced magnetic resonance imaging: Objective figures in differentiation between acute and chronic deep venous thrombosis in the lower extremities.

Deep vein thrombosis is a severe health problem. Treatment options may differ between acute and chronic deep vein thrombosis. Thus, distinguishing acute from chronic deep vein thrombosis is essential for patients with deep vein thrombosis.Triggered angiography non-contrast enhanced is an innovative magnetic resonance imaging protocol that may provide objective evidence in differentiating acute from chronic deep vein thrombosis. We prospectively collected information on consecutive patients who had been evaluated through triggered angiography non-contrast enhanced magnetic resonance imaging for venous pathology in their lower extremities at a vascular wound care center in a tertiary hospital between April 2017 and January 2020. Patients included were divided into two groups with the onset time cutoff point of 21 days. All were undergone non-contrast-enhanced magnetic resonance imaging evaluation. Non-contrast-enhanced magnetic resonance imaging images were evaluated by a radiologist, and lower extremity venous thrombosis, collateral-vein development, and subcutaneous honeycombing were emphasized. Cohen's kappa coefficient was used to measure interrater agreement between the development of collateral veins, subcutaneous honeycombing, and symptom onset over 21 days. Interrater agreement analysis revealed that the development of collateral veins was substantially correlated with the onset of symptoms over 21 days (Table 1). Additionally, the development of subcutaneous honeycombing detected through triggered angiography non-contrast enhanced magnetic resonance imaging also substantially agreed with the onset of symptoms over 21 days (Table 2). The diagnostic power of triggered angiography non-contrast enhanced magnetic resonance imaging in deep vein thrombosis is rival to current gold standard, color Doppler sonography. Triggered angiography non-contrast enhanced magnetic resonance imaging provides objective information on onset timing in patients with deep vein thrombosis that could differentiate acute from chronic deep vein thrombosis and provides guidance for treatment planning.

The DICA Endoscopic Classification for Diverticular Disease of the Colon Shows a Significant Interobserver Agreement among Community Endoscopists: an International Study.

The Diverticular Inflammation and Complication Assessment (DICA) endoscopic classification of diverticulosis and diverticular disease (DD) is currently available. It scores severity of the disease as DICA 1, DICA 2 and DICA 3. Our aim was to assess the agreement on this classification in an international endoscopists community setting. A total of 96 doctors (82.9% endoscopists) independently scored a set of DD endoscopic videos. The percentages of overall agreement on DICA score and a free-marginal multirater kappa (κ) coefficient were reported as statistical measures of interrater agreement. Overall agreement in using DICA was 91.8% with a free-marginal kappa of 88% (95% CI 80-95). The overall agreement levels were: DICA 1, 85.2%; DICA 2, 96.5%; DICA 3, 99.5%. The free marginal κ was: DICA 1 = 0.753, DICA 2 = 0.958, DICA 3 = 0.919. The agreement about the main endoscopic items was 83.4% (k 67%) for diverticular extension, 62.6% (k 65%) for number of diverticula for each district, 86.8% (k 82%) for presence of inflammation, and 98.5 (k 98%) for presence of complications. The overall interrater agreement in this study ranges from good to very good. DICA score is a simple and reproducible endoscopic scoring system for diverticulosis and DD.

The external elbow measure as surrogate of the anatomical width of cubital groove and possible risk factor of ulnar neuropathy at the elbow

Introduction: There are no studies on elbow anthropometry in ulnar neuropathy at the elbow (UNE). We aimed to test the interrater agreement of external elbow measurements with caliper, the matching of external width of cubital groove (WCG) measures with those obtained through conventional radiography (XR) and ultrasonography (US). The final aim was to evaluate the differences of anthropometric elbow and body measures between UNE cases and controls with multicenter prospective study.Materials and methods: After common training of five observers for external elbow and body anthropometric measurements, we assessed the interrater agreement of measures in a single blind measurement session in 16 healthy volunteers. Then we verified if external WCG measures in eight and four of the above 16 subjects matched with those obtained with US and XR. Finally, we enrolled 40 consecutive idiopathic UNE cases in four electromyographic labs matched for sex and age with 40 controls to evaluate the differences of anthropometric measures.Results: There was high interrater agreement of all anthropometric body and elbow measures (Kendal’s and interclass correlation coefficients between 0.84 and >0.9). We found high relations between WCG caliper measures and those obtained with US and XR (r > 0.9). WCG was smaller in cases than in controls (13.2 vs.15.7 mm, p < 0.001). There were no differences in body anthropometric measures (BMI and waist-to-hip ratio).Conclusion: The external measurement of WCG is reliable and reproducible and may be risk factor of UNE. Future studies should be performed in lager samples evaluating the relations with lifestyle and occupational factors.

RoB-SPEO: A tool for assessing risk of bias in studies estimating the prevalence of exposure to occupational risk factors from the WHO/ILO Joint Estimates of the Work-related Burden of Disease and Injury

BackgroundThe World Health Organization (WHO) and the International Labour Organization (ILO) are developing joint estimates of the work-related burden of disease and injury (WHO/ILO Joint Estimates). For this, systematic reviews of studies estimating the prevalence of exposure to selected occupational risk factors will be conducted to provide input data for estimations of the number of exposed workers. A critical part of systematic review methods is to assess risk of bias (RoB) of individual studies. In this article, we present and describe the development of such a tool, called the Risk of Bias in Studies estimating Prevalence of Exposure to Occupational risk factors (RoB-SPEO) tool; report results from RoB-SPEO's pilot testing; note RoB-SPEO's limitations; and suggest how the tool might be tested and developed further. MethodsSelected existing RoB tools used in environmental and occupational health systematic reviews were reviewed and analysed. From existing tools, we identified domains for the new tool and, if necessary, added new domains. For each domain, we then identified and integrated components from the existing tools (i.e. instructions, domains, guiding questions, considerations, ratings and rating criteria), and, if necessary, we developed new components. Finally, we elicited feedback from other systematic review methodologists and exposure scientists and agreed upon RoB-SPEO. Nine experts pilot tested RoB-SPEO, and we calculated a raw measure of inter-rater agreement (Pi) for each of its domain, rating Pi < 0.4 as poor, 0.4 ≤ Pi ≥ 0.8 as substantial and Pi > 0.80 as almost perfect agreement. ResultsOur review found no standard tool for assessing RoB in prevalence studies of exposure to occupational risk factors. We identified six existing tools for environmental and occupational health systematic reviews and found that their components for assessing RoB differ considerably. With the new RoB-SPEO tool, assessors judge RoB for each of eight domains: (1) bias in selection of participants into the study; (2) bias due to a lack of blinding of study personnel; (3) bias due to exposure misclassification; (4) bias due to incomplete exposure data; (5) bias due to conflict of interest; (6) bias due to selective reporting of exposures; (7) bias due to difference in numerator and denominator; and (8) other bias. The RoB-SPEO's ratings are low, probably low, probably high, high or no information. Pilot testing of the RoB-SPEO tool found substantial inter-rater agreement for six domains (range of Pi for these domains: 0.51–0.80), but poor agreement for two domains (i.e. Pi of 0.31 and 0.33 for biases due to incomplete exposure data and in selection of participants into the study, respectively). Limitations of RoB-SPEO include that it has not yet been fully performance-tested. ConclusionsWe developed the RoB-SPEO tool for assessing RoB in prevalence studies of exposure to occupational risk factors. The tool will be applied and its performance tested in the ongoing systematic reviews for the WHO/ILO Joint Estimates.

Exo-anal imaging of the anal sphincter: a comparison between introital and transperineal image acquisition.

Three-dimensional exoanal ultrasound imaging of the anal sphincter may be obtained transperineally with a convex probe, or at the introitus with a transvaginal probe. We hypothesised that introital acquisition would yield better quality and more reproducible evaluation. We acquired three 3D volumes of the anal sphincter (one transperineal transverse with a 4- to 8-MHz convex probe and two introital with a 5- to 9-MHz probe in transverse and mid-sagittal view) in 20 representative women attending the gynaecology clinic. Each 3D dataset was anonymised and hence blinded for clinical data and for acquisition method. Images were analysed off-line by two expert specifically trained ultrasonographers in a random order to assess image quality, sphincter integrity and sphincteric measurements. We assessed the intra- and interrater agreement by the Cohen's kappa (κ) and by the intraclass correlation coefficient for categorical and continuous variables respectively. The mid-sagittal introital acquisition had most inconclusive images owing to unsatisfactory quality, on which raters agreed (К = 0.80). Subsequently, agreement in the anal sphincter evaluation between transverse introital and transperineal acquisitions was compared. Agreement on internal anal sphincter gap was excellent for both transverse introital (К = 0.87) and transperineal acquisition (К = 0.93). Agreement on external anal sphincter discontinuity was excellent for the transperineal acquisition (К = 0.87) and good for the transverse introital acquisition (К = 0.73). Intra- and interrater agreement of external and internal anal sphincteric measurements were best for transperineal acquisitions. In our hands, transperineal acquisition with a 4- to 8-MHz probe performed better than introital acquisition with a 5- to 9-MHz probe in the assessment of the anal sphincter complex.

Development and validation of a survey instrument for measuring pre-service teachers’ pedagogical content knowledge

ABSTRACT In Australia, one of the key findings of the Teacher Education Ministerial Advisory Group was that not all graduating pre-service teachers possess adequate pedagogical content knowledge (PCK) to teach effectively. The concern is that higher education providers working with pre-service teachers are using pedagogical practices and assessments which are not informed by research. Due to its multifaceted nature, PCK is difficult to measure. In the case of mathematics PCK, some researchers have used multiple-choice questions while others use constructed-response formats depending on the specific attribute of PCK being measured. In either case, researchers need to use instruments that are appropriate to each task. This paper reports on the construction and reliability of a survey instrument used in a larger study to measure pre-service teachers’ mathematics PCK. Using four stages of survey development, in addition to inter-rater agreement measures and Rasch modelling techniques, data analyzes found high levels of validity and reliability for the Mathematics Pedagogical Content Knowledge Instrument (MPCKI).

Comparison of linear and angular changes assessed in digital dental models and cone-beam computed tomography.

To compare the three-dimensional (3D) linear displacements and the mesiodistal and buccolingual angulation changes after orthodontic treatment in digital dental models (DDMs) and 3D models derived from cone-beam computed tomography (CBCT). Digital dental model and CBCT scans were selected from 24 adults who had undergone orthodontic treatment for mandibular anterior crowding. 3D linear displacements and changes in angular measurements (mesiodistal and buccolingual angulation) were assessed in pre- and post-treatment DDM and CBCT images using the software ITK-snap and 3D SlicerCMF. Intra- and inter-rater agreement of measurements in DDM and CBCT were tested using the intraclass correlation coefficient (ICC). DDM and CBCT measurements were compared using the Wilcoxon test (P<.05), ICC and Bland-Altman plots. Intra- and inter-rater agreement varied from good (ICC>0.75) to excellent (ICC>0.90) for both DDM and CBCT measurements. Although no significant difference between DDM and CBCT methods was observed for linear measurements of tooth movement, the angular assessments were different for most measurements. The agreement between measurements from both assessments varied from poor to excellent. Longitudinal assessments of tooth movements including 3D linear displacements and mesiodistal and buccolingual angulation are reproducible when using both DDM and CBCT. Changes in angular measurements due to orthodontic treatment are discordant when measured in the digital models (clinical crown) and in the CBCT images (whole tooth).

Experimental Studies of Inter-Rater Agreement in Traditional Chinese Medicine: A Systematic Review.

Objectives: It has been recommended that clinical trials of Traditional Chinese Medicine (TCM) would be more ecologically valid if its characteristic mode of diagnostic reasoning were integrated into their design. In that context, however, it is also widely held that demonstrating a high level of agreement on initial TCM diagnoses is necessary for the replicability that the biomedical paradigm requires for the conclusions from such trials. Our aim was to review, summarize, and critique quantitative experimental studies of inter-rater agreement in TCM, and some of their underlying assumptions. Design: Systematic electronic searches were conducted for articles that reported a quantitative measure of inter-rater agreement across a number of rating choices based on examinations of human subjects in person by TCM practitioners, and published in English language peer-reviewed journals. Publications in languages other than English were not included, nor those appearing in other than peer-reviewed journals. Predefined categories of information were extracted from full texts by two investigators working independently. Each article was scored for methodological quality. Outcome measures: Design features across all studies and levels of inter-rater agreement across studies that reported the same type of outcome statistic were compared. Results: Twenty-one articles met inclusion criteria. Fourteen assessed inter-rater agreement on TCM diagnoses, two on diagnostic signs found upon traditional TCM examination, and five on novel rating schemes derived from TCM theory and practice. Raters were students of TCM colleges or graduates of TCM training programs with 3 or more years experience and licensure. Type of outcome statistic varied. Mean rates of pairwise agreement averaged 57% (median 65, range 19-96) across the 9 studies reporting them. Mean Cohen's kappa averaged 0.34 (median 0.34, range 0.07-0.59) across the seven studies reporting them. Meta-analysis was not possible due to variations in study design and outcome statistics. High risks of bias and confounding, and deficits in statistical reporting were common. Conclusions: With a few exceptions, the levels of agreement were low to moderate. Most studies had significant deficits of both methodology and reporting. Results overall suggest a few design features that might contribute to higher levels of agreement. These should be studied further with better experimental controls and more thorough reporting of outcomes. In addition, methods of complex systems analysis should be explored to more adequately model the relationship between clinical outcomes, and the series of diagnoses and treatments that are the norm in actual TCM practice.

Alternate Measure of Disability Intensity: Modified Rankin Scale

Objective: This study was aimed to propose an alternate method of scoring the intensity of disability in a patient, which will facilitate ranking and classifying a group of patients in mutually exclusive classes along with the quantification of progress made by a patient or the effect of interventions and also tracking the path of improvement of a patient and estimating the survival curve for drawing useful conclusions. Method: A nonparametric measure of disability intensity in terms of Cos θi for the ith patient is proposed to avoid limitations of the usual scoring of modified Rankin scale and satisfies many desired properties. Two measures of interrater agreement in terms of standard error of measurement and coefficient of variation are also proposed and their relationships with Cos θi are detailed. Illustration with hypothetical data involving 30 patients and 10 raters was undertaken. Results: Distribution of Cos θ was close to normal. Principal component analysis of Cos θ resulted in one factor explaining 64% of variance. Inconsistent raters were identified. Coefficient of variation appears to perform better than the standard error of measurement. Split-half reliability of raters in terms of angular association was higher. Similarly, split-half reliability of disability intensity (Cos θ) was higher. Average disability intensity [Formula: see text] showed similar empirical results with rigorous method of most preferred direction of a finite number of angles. Conclusions: The proposed simple measures will help the researchers and practitioners to make meaningful analysis and draw meaningful conclusions. Future studies may be undertaken to rescale Likert data to scales having properties of interval-level measurement and study the proposed measures in detail.

Translation and validation of the Indian Takayasu clinical activity score (ITAS2010) for the Brazilian Portuguese language

BackgroundThe Indian Takayasu Clinical Activity Score (ITAS2010) was developed in 2010 as an assessment tool for disease activity in patients with Takayasu arteritis (TA). It has since been widely used in different studies and in clinical practice for the management of patients with TA. The present study aims to translate the ITAS2010 into Brazilian Portuguese language and to validate it for use in clinical practice in Brazil.MethodsFor this cross-sectional study, the ITAS2010 was translated in accordance with the guidelines described by Beaton et al. and then applied with 27 patients with TA on three assessments by two rheumatologists working independently. To measure interrater agreement, the assessments were performed on the same day within approximately 1 hour. One of the rheumatologists performed a second evaluation of patients with TA within 7 to 14 days to measure intrarater agreement.ResultsThe correlation coefficient for the ITAS2010 score between the two raters was high (r = 0.916; p < 0.0001), as well as the intraclass correlation coefficient (ICC) [0.918 with a 95% confidence interval (95CI): 0.828–0.962]. The correlation coefficient and the ICC for intrarater agreement were moderate for ITAS2010 (r = 0.633; p < 0.0001 and ICC = 0.594; 95CI: 0.292–0.790). The ITAS2010 at baseline was compared with the physician’s global assessment (PGA) and with Kerr’s criteria for detecting disease activity in TA. Higher ITAS2010 scores were observed in patients with active and grumbling/persistent disease than in those presenting inactive disease according to the PGA [1.5 (0.0–3.0) vs. 0.0 (0.0–0.0); p = 0.0025]. Patients with active disease according to the Kerr’s criteria had also higher ITAS2010 scores than those considered in remission [3.0 (3.0–7.0) vs. 0.0 (0.0–0.0); p = 0.0068].ConclusionsThe Brazilian Portuguese version of the ITAS2010 is a valid and reproducible tool for the assessment of disease activity in TA and it is an additional tool for the routine evaluation of Brazilian patients with TA.

Adapting and validating the Hospital Survey on Patient Safety Culture (HSOPS) for nursing students (HSOPS-NS): A new measure of Patient Safety Climate

BackgroundPatient Safety Culture and Patient Safety Climate (PSC) are different factors. PSC is the shared perception that is held within a hospital's area or unit at a specific moment in time. This measure is necessary for designing activities for promoting and improving safety. It must include the perception of all the agents involved, including future nurses throughout their patient safety education. ObjectivesThe aim was to adapt and validate a new version of the Hospital Survey on Patient Safety Culture (HSOPS), targeted specifically at nursing students. It provides a new comprehensive and more complete measure of PSC that contributes to improving patient safety. MethodsData were obtained from 654 undergraduate and postgraduate nursing students. PSC was tested using factor analyses and structural equation modeling. In order to facilitate the improvement of PSC, we examined differences in climate strength across different academic groups using the Rwg(j) and ICC measures of inter-rater agreement. ResultsFactor analyses confirmed a five-factor solution that explained between 52.45% and 54.75% of the variance. The model was found to have adequate fit χ2 (5) = 14.333, p = .014; CFI = 0.99; RMSEA = 0.05. Cronbach's alphas for PSC were between 0.74 and 0.77. “Teamwork within units” was the highest rated dimension, and “Staffing” the lowest rated. Medium-to-high scores were obtained for PSC. The median of Rwg (j) was high in the five dimensions of the PSC survey, supporting the idea of shared climate perceptions (0.81–0.96) among undergraduate and postgraduate nursing students. ConclusionsHSOPS-NS is a useful and versatile tool for measuring the level and strength of PSC. It screens knowledge regarding patient safety in clinical practice placements and compares nursing students' perceptions of the strength of PSC. Weaknesses perceived in relation to PSC help implement changes in patient safety learning.