Median Interrater Reliability Research Articles

Experience of Chest Dysphoria and Masculinizing Chest Surgery in Transmasculine Youth.

Transmasculine individuals, those assigned female sex at birth but who identify as masculine, have high rates of suicidal behavior and often suffer from chest dysphoria (discomfort and distress from unwanted breast development). Growing numbers of transmasculine youth are pursuing definitive treatment with masculinizing chest surgery (MCS), and adult studies reveal marked benefits of MCS, although little is known about the impact of chest dysphoria on transmasculine youth or the optimal timing of MCS. In this study, we aimed to explore youth experiences of chest dysphoria and the impact of MCS. Transmasculine youth aged 13 to 21 were recruited from a pediatric hospital-based gender clinic. Participants completed a semistructured qualitative interview exploring the experience of chest dysphoria and thoughts about or experiences with MCS. Interview transcripts were coded by 3 investigators employing modified grounded theory, with the median interrater reliability at κ = 0.92. Subjects (N = 30) were a mean age of 17.5 years, and 47% had undergone MCS. Youth reported that chest dysphoria triggered strong negative emotions and suicidal ideation, caused a myriad of functional limitations, and was inadequately relieved by testosterone therapy alone. All post-MCS youth reported near or total resolution of chest dysphoria, lack of regret, and improved quality of life and functioning. We observed consensus that chest dysphoria is a major source of distress and can be functionally disabling to transmasculine youth. MCS performed during adolescence, including before age 18, can alleviate suffering and improve functioning. Additional research is needed to develop patient-reported outcome measures to assess the impact of chest dysphoria and MCS.

Inter-rater reliability and validity of good pharmacy practices measures in inspection of public sector health facility pharmacies in Uganda

BackgroundThe National Drug Authority (NDA) inspects and certifies private and public sector pharmacies in Uganda using an indicator-based inspection tool that measures adherence to good pharmacy practices (GPP). 67 measures identify the situation in the domains of premises, dispensing quality, stores management, and operating requirements. Although the GPP measures are well-recognized and used internationally, little is known about their validity and reliability. The study aimed to assess validity, which measures agreement of GPP measures between a gold standard inspector and NDA inspector and inter-rater reliability (IRR), which measures agreement among NDA inspectors, of GPP measures.MethodsWe assessed validity and IRR by four teams of inspectors in eight government health facilities that represent three levels of care. Each team inspected two facilities, resulting in 24 total inspections. Each team comprised one central-level inspector, one district-level inspector, and one gold-standard inspector (i.e., a very experienced central-level inspector). We calculated median validity and IRR for each GPP measure, overall, indicator categorized as either critical, major, or minor, by domains, by the inspection decision (i.e., certified or not certified) and by adequatevalidity and IRR score (i.e., score ≥ 75%).ResultsThe median validity for all GPP measures was 69%, with 29 (43%) measures having an adequate validity of ≥75%. The median IRR for all GPP measures was 71%, with 31 (46%) having an adequate IRR measure of ≥75%. Validity did not differ significantly by indicator category, domain or level of care. Adequate IRR and validity score (≥75%) was lowest for critical measures, which are key determinants of the certification decision, scoring 20 and 40% respectively. District inspectors had lower median validity for critical indicators and premises and higher validity for store management. Compared to central inspectors, the validity of district inspectors’certification decisions was lower; in the eight facilities, three district inspectors agreed with gold standard inspector vs. all eight central inspectors.ConclusionsOur findings question the validity and reliability of many GPP inspection measures, particularly critical measures that greatly impact certification decision. This study demonstrates the need for assessments of, and interventions to improve, validity and reproducibility of GPP measures and inspections.

Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT)

ObjectiveTo assess the interrater reliability (IRR) and usability of the Patient Education Materials Assessment Tool (PEMAT) and the relationship between PEMAT scores and readability levels. MethodsOne hundred ten materials (80 print, 30 audiovisual) were evaluated, each by two raters, using the PEMAT. IRR was calculated using Gwet’s AC1 and summarized across items in each PEMAT domain (understandability and actionability) and by material type. A survey was conducted to solicit raters’ experience using the PEMAT. Readability of each material was assessed using the SMOG Index. ResultsThe median IRR was 0.92 for understandability and 0.93 for actionability across all relevant items, indicating good IRR. Eight PEMAT items had Gwet’s AC1 values less than 0.81. PEMAT and SMOG Index scores were inversely correlated, with a Spearman’s rho of −0.20 (p=0.081) for understandability and −0.15 (p=0.194) for actionability. While 92% of raters agreed the PEMAT was easy to use, survey results suggested specific items for clarification. ConclusionWhile the PEMAT demonstrates moderate to excellent IRR overall, amendments to items with lower IRR may increase the usefulness of the tool. Practice implicationsThe PEMAT is a useful supplement to reading level alone in the assessment of educational materials.

Inter-rater agreement in evaluation of disability: systematic review of reproducibility studies

Objectives To explore agreement among healthcare professionals assessing eligibility for work disability benefits.Design Systematic review and narrative synthesis of reproducibility studies.Data sources Medline, Embase, and PsycINFO searched up to 16...

The objectivity of the Autism Diagnostic Observation Schedule (ADOS) in naturalistic clinical settings.

The Autism Diagnostic Observation Schedule (ADOS) is a first-choice diagnostic tool in autism spectrum disorder (ASD). Excellent interpersonal objectivity (interrater reliability) has been demonstrated for the ADOS under optimal conditions, i.e., within groups of highly trained "research reliable" examiners in research setting. We investigated the spontaneous interrater reliability among clinically trained ADOS users across multiple sites in clinical routine. Forty videotaped administrations of the ADOS modules 1-4 were rated by five different raters each from a pool of in total 15 raters affiliated to 13 different clinical sites. G(q,k) coefficients (analogous to intraclass correlations), kappas (ĸ) and percent agreement (PA) were calculated. The median interrater reliability for items across the four modules was G(q,k)=.74-.83, with the single ADOS items ranging from .23 to .94. G(q,k) for total scores was .85-.92. For diagnostic classification (ASD/non-spectrum), PA was 64-82% and Fleiss' ĸ .19-.55. Objectivity was lower for pervasive developmental disorder not otherwise specified and non-spectrum diagnoses as compared to autism. Interrater reliabilities of the ADOS items and domain totals among clinical users across multiple sites were in the same range as previously reported for research reliable users, while the one for diagnostic classification was lower. Differences in sample characteristics, rater skills and statistics compared with previous studies are discussed. Findings endorse the objectivity of the ADOS in naturalistic clinical settings, but also pinpoint its limitations and the need and value of adequate and continuous rater training.

A review of the agreement between clinicians’ personality disorder diagnoses and those from other methods and sources.

This review synthesizes a wide literature on the agreement of treating clinicians’ PD diagnoses with each other and their convergence with common research methods. Median interrater reliability between clinicians was moderate when calculated dimensionally (r = .46) or categorically (κ = .40). The agreement between clinicians’ diagnoses and those from research methods (e.g., self-report questionnaire) was more modest. Median dimensional agreement across 27 studies ranged from .05 to .36, with an overall median of .23. This overall value was moderated by several factors. First, clinicians’ diagnoses agreed more with semi-structured interviews than self-report questionnaires. Second, convergence increased slightly when clinicians utilized more systematic diagnostic methods. Results suggest relatively little overlap between PD diagnoses assigned in research versus naturalistic settings.

Inter- and Intrarater Reliability of Osteoarthritis Classification at the Trapeziometacarpal Joint

To assess the reliability of the Eaton and Glickel classification for base of thumb osteoarthritis. The interrater and intrarater reliability of this classification were assessed by comparing ratings from 6 raters using quadratic weighted kappa scores. Median inter-rater reliability ranged from kappa of .53 to .54; intrarater reliability ranged from kappa of .60 to .82. Using unweighted kappa interrater reliability was "slightly" reliable, and intrarater reliability was "fairly" reliable. Overall, the value of the intraclass correlation for all 6 raters was .56. This radiological classification does not describe all stages of carpometacarpal joint osteoarthritis accurately enough to permit reliable and consistent communication between clinicians. Therefore we believe it should be used with an understanding of its limitations when communicating disease severity between clinicians or as a tool to assist in clinical decision making.

Validation of Scores/Measures from a K-2 Developmental Assessment in Mathematics

This study describes psychometric investigations of a developmental assessment in mathematics composed of 19 performance tasks. Techniques from classical and the many-faceted Rasch approaches were combined to analyze field test data from a mixedage sample ( n = 110). Descriptive statistics on scores from the overall scale and subdomains indicated increased proficiency with age. Convergent validity coefficients of scores with scaled scores of the Stanford Achievement Test mathematics battery ranged from .22 to .57; internal consistency reliability of the total scores on proficiency and independence were .90 and .92, respectively; and median interrater reliability was .80. Classical item statistics and Rasch logit difficulties of tasks suggested an ordered scale structure, although eight tasks did not fit Rasch criteria. The original and calibrated task orderings were consistent at the extreme ends of the scale. Findings point to directions for future improvement of the scale and rater training programs.

The reliability of the functional independence measure: A quantitative review

Objective: The reliability of the Functional Independence Measure (FIM SM) for adults was examined using procedures of meta-analysis. Data Sources: Eleven published studies reporting estimates of reliability for the FIM were located using computer searches of Index Medicus, Psychological Abstracts, the Functional Assessment Information Service, and citation tracking. Study Selection: Studies were identified and coded based on type of reliability (interrater, test-retest, or equivalence), method of data analysis, size of sample, and training or experience of raters. Data Extraction: Information from the articles was coded by two independent raters. Interrater reliability for coding all elements included in the analysis ranged from .89 to 1.00. Data Synthesis: The 11 investigations included a total of 1,568 patients and produced 221 reliability coefficients. The majority of the reliability values (81%) were from interrater reliability studies, and the intraclass correlation coefficient (ICC) was the most commonly used statistical procedure to compute reliability. The reported reliability values were converted to a common correlation metric and aggregated across the 11 studies. The results revealed a median interrater reliability for the total FIM of .95 and median test-retest and equivalence reliability values of .95 and .92, respectively. The median reliability values for the six FIM subscales ranged from .95 for Self-Care to .78 for Social Cognition. For the individual FIM items, median reliability values varied from .90 for Toilet Transfer to .61 for Comprehension. Median and mean reliability coefficients for FIM motor items were generally higher than for items in the cognitive or communication subscales. Conclusions: Based on the 11 studies examined in this review the FIM demonstrated acceptable reliability across a wide variety of settings, raters, and patients.

Evaluation of a multicenter ethics objective structured clinical examination.

The purpose of this study was to evaluate a six-station ethics objective structured clinical examination (OSCE) on a volunteer sample of 66 medical students and 33 residents from three Ontario medical schools. The internal consistency reliability was 0.46 and the median interrater reliability was 0.675 (range 0.30 to 0.89). The residents' scores were higher than those of the medical students (F = 2.24, 0.046). Also, the scores differed among the three schools (F = 3.19, p = 0.0004). The ethics OSCE has adequate interrater reliability and construct validity, but low internal consistency reliability. There are differences among the schools that may assist in ethics curriculum evaluation and development.

Interrater reliability of the Handler Draw-a-Person scoring.

Draw-a-Person test scoring systems developed prior to 1967 had low levels of predictive validity and reliability estimates for these scales are rarely found in the literature. Then, in 1967, Handler developed a set of scales for scoring anxiety indices in the Draw-a-Person test. The Handler scoring manual may offer more promise for future studies designed to validate the clinical and research use of human figure drawings. Reliability estimates reported in the present investigation provide independent support that Handler's scoring procedures have high interrater reliabilities. The Draw-a-Person indices had a median interrater reliability of .90, this being congruent with the median coefficient of .87 reported by Handler.