Inter-rater Reliability Study Research Articles

Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters.

In undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales. The authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019. The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments. Fifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less- and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs. In a structured setting, both the Ottawa and Chen scales distinguished between preentrustable and entrustable learners; however, the Ottawa scale demonstrated more desirable characteristics. These findings represent a critical step forward in developing valid, reliable instruments to measure learner progression toward entrustment for the Core EPAs.

Dyspnea could be accurately assessed by a caregiver in hospitalized patients with respiratory diseases: Interrater reliability and agreement study

BackgroundThe perception of dyspnea is a subjective feeling typically self-assessed by the patient. However, the assessment by a caregiver is sometimes required. ObjectivesThe primary aim was to compare patient self-assessment and caregiver assessment of dyspnea (interrater reliability) using the modified Borg and visual analog scale (VAS) in hospitalized patients. The secondary aim was to compare dyspnea assessment between the two scales for patients and caregiver (inter-instrument reliability). MethodsSelf-assessment of dyspnea intensity of hospitalized patients with respiratory diseases was compared with caregiver's assessment. Dyspnea intensity was measured using two scales, the modified Borg scale (0–10 scale) and the 10 cm VAS. Mean difference and 95% confidence interval (CI) between assessors (i.e. patient versus caregiver) were calculated for each scale. Inter- and intra-rater reliability was calculated using intraclass correlation coefficients (ICCs). ResultsA total of 254 patients were recruited. The mean differences between patient and caregiver ratings were 0.31 (95% CI: 0.09, 0.53) for the modified Borg scale and 0.36 (95% CI: 0.06, 0.65) for the VAS scale. Interrater reliability was good for both scales with ICC of 0.79 (95% CI: 0.73, 0.84) for VAS and 0.82 (95% CI: 0.77, 0.86) for the modified Borg scale. The mean differences in scores between scales were 0.93 (95% CI 0.69, 1.17) for patients' ratings and 0.88 (95% CI 0.72, 1.04) for caregiver's rating. The inter-instrument reliability was moderate to good and similar for both assessors. ConclusionDyspnea can be accurately estimated by caregivers when patients with lung diseases cannot self-report. Scores on the VAS to rate dyspnea were higher than the scores on the Borg scale.

805 Towards a more perfect union: interrater reliability between telemedicine and in-person evaluators in sleep disordered breathing

Abstract Introduction Telemedicine has rapidly changed the landscape of all of clinical practice, and is now widely employed in sleep medicine. To date, the accuracy of telemedicine in identifying patients at risk for obstructive sleep apnea (OSA) is still unknown. Given differences in technique and fidelity, a perfect correlation between these two types of encounters cannot be assumed. We studied how providers using a telemedicine platform compared to providers using traditional in-person encounters in identifying risk for sleep disordered breathing. Methods 90 participants referred to a comprehensive university sleep program were randomized to this interrater reliability study. Subjects were representative of the gender and ethnic breakdown of the outlying community. The subjects were evaluated by an in-person clinician investigator, then randomized to a second clinician investigator who performed an evaluation online using a common teleconferencing platform. Both types of evaluations included a history and physical exam. The primary outcome was pretest probability for obstructive sleep apnea. Secondary comparators included level of daytime sleepiness, snoring volume, apneas witnessed by a third party, modified Mallampati score, presence/absence of tonsils, degree of overjet bite, and severity of apnea based on home sleep testing. Results Interrater reliability for pretest probability of OSA was reflected in a weighted kappa value of 0.414 (SE 0.090, p=0.002). This suggests moderate agreement between the an in-person and telemedicine evaluator. Kappa values of our secondary outcomes ranged from -0.044 (degree of maxillary overjet) to 0.702 (apneas witnessed by a third party), and were generally higher for historical elements and lower for physical exam findings. Conclusion Evaluation for pretest probability for sleep apnea via telemedicine has a moderate interrater correlation with in-person assessment. A relatively high degree of interrater reliability for historical elements suggests that the accuracy of telemedicine for OSA is tempered by a suboptimal physical exam. Telemedicine evaluations might become more accurate through standardization. For instance, using validated scales for OSA or templated encounter scripts may help with risk-stratification, and ultimately lead to more uniform management. Support (if any) This study was supported by an AASM Foundation Focused Project Award

Assessing the Quality of Mobile Health-Related Apps: Interrater Reliability Study of Two Guides.

BackgroundThere is a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high-quality apps and enable existing apps to be assessed, several guides have been developed.ObjectiveThe main aim of this study was to study the interrater reliability of a new guide — the Mobile App Development and Assessment Guide (MAG) — and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS). Moreover, we also focused on whether the interrater reliability of the measures is consistent across multiple types of apps and stakeholders.MethodsIn order to study the interrater reliability of the MAG and MARS, we evaluated the 4 most downloaded health apps for chronic health conditions in the medical category of IOS and Android devices (ie, App Store and Google Play). A group of 8 reviewers, representative of individuals that would be most knowledgeable and interested in the use and development of health-related apps and including different types of stakeholders such as clinical researchers, engineers, health care professionals, and end users as potential patients, independently evaluated the quality of the apps using the MAG and MARS. We calculated the Krippendorff alpha for every category in the 2 guides, for each type of reviewer and every app, separately and combined, to study the interrater reliability.ResultsOnly a few categories of the MAG and MARS demonstrated a high interrater reliability. Although the MAG was found to be superior, there was considerable variation in the scores between the different types of reviewers. The categories with the highest interrater reliability in MAG were “Security” (α=0.78) and “Privacy” (α=0.73). In addition, 2 other categories, “Usability” and “Safety,” were very close to compliance (health care professionals: α=0.62 and 0.61, respectively). The total interrater reliability of the MAG (ie, for all categories) was 0.45, whereas the total interrater reliability of the MARS was 0.29.ConclusionsThis study shows that some categories of MAG have significant interrater reliability. Importantly, the data show that the MAG scores are better than the ones provided by the MARS, which is the most commonly used guide in the area. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.

Flexor digitorum superficialis muscular activity is more reliable than mentalis muscular activity for rapid eye movement sleep without atonia quantification: A study of interrater reliability for artifact correction in the context of semiautomated scoring of rapid eye movement sleep without atonia.

To evaluate interrater reliability for artifact correction in the context of semiautomated quantification of rapid eye movement (REM) sleep without atonia (RWA) in the mentalis and flexor digitorum superficialis (FDS) muscles. We included video-polysomnographies of 14 subjects with apnea-hypopnea index in REM sleep (AHIREM) < 15/h and 11 subjects with AHIREM ≥ 15/h. Eight subjects had isolated REM sleep behavior disorder. A validated algorithm (www.osg.be) automatically scored phasic and "any" EMG activity in the mentalis muscle, and phasic EMG activity in the FDS muscles. Four independent expert scorers performed artifact correction according to the SINBAR (Sleep Innsbruck Barcelona) recommendations. Interrater reliability for artifact correction was computed with B-statistics. The variability across scorers of four RWA indices (phasic mentalis, "any" mentalis, phasic FDS and SINBAR-i.e. "any" mentalis and/or phasic FDS-EMG activity indices) was computed. With Friedman tests, we compared B-statistics obtained for mentalis and FDS muscles, and the variability of the RWA indices. Influence of AHIREM and REM sleep behavior disorder (RBD) diagnosis on the RWA indices variability was evaluated with linear regressions. Interrater reliability for artifact correction was higher in the FDS than in the mentalis muscle (p < 0.001). Phasic FDS activity was minimally affected by artifacts. Accordingly, the phasic FDS EMG activity index had the lowest variability across scorers (p < 0.001). Variability across scorers of the RWA indices including the mentalis muscle increased with AHIREM and was independent from RBD diagnosis. Due to the consistently found low number of artifacts, phasic FDS activity is a reliable measure of RWA.

Rating criteria of the causal nexus in Whiplash Associated Disorders: A study of interrater reliability

IntroductionThe diagnosis of Whiplash Associated Disorders is subject to various criteria, some of a subjective nature, which impedes a high percentage of agreement among raters of the causal nexus. It is necessary to develop an evidence-based methodology to maximize the uniformity of this diagnosis. Our objective is to assess the interrater reliability of four experts in legal medicine who establish the causal nexus of traffic accident patients. Material and method300 traffic accident patients evaluated by 4 legal medicine experts. They conducted two ratings, one without and one with a study of the biomechanics of the accident. The degree of concordance of the ratings was studied by the intraclass correlation coefficient (ICC) and the concordance index (CI). ResultsThe results of both the intraclass correlation coefficient and the concordance index of the rating prior to the biomechanical study of the injury yield moderate-high values (ICC=.70; CI=.60). Likewise, the values observed in the rating with the study of biomechanics also indicate moderate-high agreement (ICC=.78, CI=.67). ConclusionsThe study of biomechanics slightly improves concordance. However, the aforementioned study does not help to reduce the proportion of doubtful cases and the levels of improvement are not acceptable. There are significant differences between the classifications, suggesting that the proposed valuation methodology is not uniform enough to reach a consensus.

Categorisation of lumbar spine MRI referrals in Denmark as compliant or non-compliant to international imaging guidelines: an inter-rater reliability study

BackgroundManaging low back pain (LBP) often involves MRI despite the fact that international guidelines do not recommend routine imaging. To allow us to explore the topic and use this knowledge in further research, a reliable method to review the MRI referrals is needed. Consequently, this study aimed to assess the inter-rater reliability of a method evaluating lumbar spine MRI referrals’ appropriateness.MethodsFour inexperienced students (chiropractic master’s students) and a senior clinician (chiropractor) were included as independent raters in this inter-rater reliability study. Lumbar spine MRI referrals from primary care on patients (> 18 years) with LBP with or without leg pain were included. The referrals were classified using a modified version of the American College of Radiology (ACR) imaging appropriateness criteria for LBP. Categories of appropriate referrals included; fractures, cancer, previous surgery, candidate for surgery or suspicion of cauda equina. Inappropriate referrals included lacking information on previous non-surgical treatment, no word on non-surgical treatment duration, or “other reasons” for inappropriate referrals.After two rounds of training and consensus sessions, 50 lumbar spine MRI referrals were reviewed independently by the five raters. Inter-rater reliability was quantified using unweighted Kappa statistics, and the observed agreement was calculated with both a pairwise comparison and an overall five-rater comparison.ResultsInter-rater reliability was substantial, with a Kappa value for appropriate vs. inappropriate referrals of 0.76 (95% CI: 0.55–0.89). When six and eight subcategories were evaluated, the Kappa values were 0.77 (95% CI: 0.58–0.91) and 0.82 (95% CI: 0.72–0.92), respectively.The overall percentage of agreement for appropriate and inappropriate referrals was 92% and ranged from 88 to 98% for the pairwise comparisons of the five raters’ results. For the six and eight subcategories, the overall agreement was 92 and 88%, respectively, ranging from 88 to 98% and 84–92%, respectively, for the pairwise comparisons.ConclusionThe inter-rater reliability of the evaluation of the appropriateness of lumbar spine MRI referrals, according to the modified ACR-appropriateness criteria, was found to range from substantial to almost perfect and can be used for research and quality assurance purposes.

Interrater reliability between in-person and telemedicine evaluations in obstructive sleep apnea.

We examined how telemedicine evaluation compares to face-to-face evaluation in identifying risk for sleep-disordered breathing. This was a randomized interrater reliability study of 90 participants referred to a university sleep center. Participants were evaluated by a clinician investigator seeing the patient in-person, then randomized to a second clinician investigator who performed a patient evaluation online via audio-video conferencing. The primary comparator was pretest probability for obstructive sleep apnea. The primary outcome comparing pretest probability for obstructive sleep apnea showed a weighted kappa value of 0.414 (standard error 0.090, P = .002), suggesting moderate agreement between the 2 raters. Kappa values of our secondary outcomes varied widely, but the kappa values were lower for physical exam findings compared to historical elements. Evaluation for pretest probability for obstructive sleep apnea via telemedicine has a moderate interrater correlation with in-person assessment. A low degree of interrater reliability for physical exam elements suggests telemedicine assessment for obstructive sleep apnea could be hampered by a suboptimal physical exam. Employing standardized scales for obstructive sleep apnea when performing telemedicine evaluations may help with risk-stratification and ultimately lead to more tailored clinical management.

Thumb carpometacarpal palmar and radial abduction in adults with thumb carpometacarpal joint pain: Inter-rater reliability and precision of the inter-metacarpal distance method

Study DesignPsychometric study of inter-rater reliability. IntroductionFunctional use of the thumb can be limited in individuals with thumb carpometacarpal (CMC) osteoarthritis(OA), especially in the presence of a thumb adduction contracture. Goniometry is a common method of assessing palmar and radial abduction of the thumb base and can be used as a method of determining effectiveness of an intervention for adduction contracture. However, goniometry for the assessment of these motions has been shown to have low to moderate reliability. The intermetacarpal distance (IMD) measurement method has been shown to be the most reliable for measuring CMC palmar abduction in individuals with healthy hands but has not been studied in persons with thumb CMC OA. PurposesThe purpose of this study was to determine the inter-rater reliability and precision of the inter-metacarpal distance method for measuring palmar and radial abduction in persons with symptoms of thumb CMC OA. MethodsTwo trained hand therapists utilized the IMD method to measure palmar and radial abduction in the affected hands of 22 subjects (28 thumbs) with a physician-confirmed diagnosis or positive provocative test consistent with a diagnosis of thumb CMC OA. The intraclass correlation coefficient (ICC2,2) was used to assess inter-rater reliability of the IMD method. To determine the precision of the measurements, the standard error of measurement (SEM), minimal detectable change (MDC), and MDC percent were calculated. Findings were supplemented with descriptive data on the IMD values as well as descriptive data on the sample. ResultsIntraclass correlation coefficients for both radial and palmar abduction were found to be >.75, indicating excellent reliability. The precision of the IMD measurements were acceptable-to-excellent as evidenced by MDC% values of <30% and <10% for radial and palmar abduction respectively. ConclusionsWe present a new method for measuring thumb radial abduction. The inter-metacarpal distance method has excellent inter-rater reliability and acceptable-to-excellent precision when measuring palmar and radial abduction in individuals with or suspected to have thumb CMC OA. Currently, it is the most reliable tool for measuring thumb abduction.

Assessment of the reliability of the IHI Global Trigger Tool: new perspectives from a Brazilian study.

To assess the reliability of the Institute for Healthcare Improvement's Global Trigger Tool (IHI-GTT) between nurses and medical students as primary reviewers to measure adverse events (AEs). Interrater reliability study. A 500-bed general public hospital in Belo Horizonte, Brazil. A randomly selected sample of 220 hospital admissions of adults (≥18 years) from Oct-Nov, 2016. Two 4th-5th year-medical students and two experienced nurses applied a Portuguese-translated version of the IHI-GTT to medical records. The role of medical reviewer was performed by two senior physicians specialists in Internal Medicine. Ability to identify AEs was compared between pairs and against medical reviewer through percentage inter-examiner agreement and Kappa coefficient (K). Two outcomes -- "AE identification" and "category of harm" -- were evaluated according to two different denominators -- "admissions" (the total number of admissions evaluated in the sample; reflects the presence or not of at least one AE in each admission) and "all possibilities of agreement" (obtained by adding each identified AE to the admissions without events; allows agreement assessment to be performed for each AE individually). Were identified 199 adverse events in 90 hospitalizations, with rates of 40.9% of admissions with AEs, 76.1 AEs/1,000 patient-days and 90.5 AEs/100 admissions. Comparing student-pair and nurse-pair, we found K = 0.76 (95% IC 0.62-0.88) and K = 0.17 (95% IC 0.06-0.27) for "AE identification" outcome and K = 0.28 (95% IC 0.01-0.55) and K = 0.46 (95% IC 0.28-0.64) for "category of harm" outcome to denominators "admission" and "all possibilities of agreement", respectively. There was no significant difference between the performances of the different primary reviewers composed in any analyses. IHI-GTT reliability varies considerably depending on the denominator used to calculate agreement. As the purpose of the tool is, in addition to measuring, promoting opportunities for quality of care improvement, the individual analysis of the AEs seems more appropriate. Further studies are needed to assess the implications of the slight agreement reached between primary reviewers on the test's overall accuracy. Moreover, advanced medical students may be considered for primary review in settings where unavailability of staff is a barrier to IHI-GTT adoption.

Development of the Simulation-Positive Learning Evaluation (SIM-PLE) rubric for use in medical-surgical nursing high fidelity simulations

Clinical nurse educators in a BSN program developed the SIM-PLE (Simulation-Positive Learning Evaluation) rubric to evaluate student performance in simulation. A pilot study was conducted over two semesters with junior level medical-surgical classes made up of approximately 110–130 students each semester, and the tool was revised and re-evaluated. Inter-rater reliability testing utilized 69 SIM-PLE rubric student evaluations (N = 69). Inter-rater reliability was assessed with comparison of lead faculty with three other instructors utilizing two raters for each simulation experience. The intraclass correlation coefficient (ICC) suggested high inter-rater reliability. The t-test difference was not significant. One-way analysis of variance (ANOVA) showed a significant difference in the Collaboration subcategory score. Univariate ANOVA indicated both raters scored consistently with each other overall and that they both scored JR1 differently than JR2. The pilot study and testing of interrater reliability of the rubric revealed positive findings. The role of the rubric in providing formative feedback during debrief sessions empowered students to analyze their own performance and/or experience to self-correct or improve their future practice based on these experiences.

Reliability and validity study of the French-language version of the SCID-D semi-structured clinical interview for diagnosing DSM-5 and ICD-11 dissociative disorders

This paper presents the validation and inter-rater reliability study of the French translation of the new edition of the SCID-D, a semi-structured clinical interview for evaluating dissociative symptoms and diagnosing dissociative disorders based upon DSM-5 and ICD-11 criteria. The SCID-D evaluates five core dimensions of dissociation: amnesia, depersonalization, derealization, identity confusion and identity alteration. The SCID-D was administered to 20 subjects with a dissociative disorder, 19 subjects with a nondissociative psychiatric disorder, and 10 healthy control subjects. Results indicate that there was 96% agreement among raters on the SCID-D's ability to differentiate subjects with a dissociative disorder from subject with no dissociative disorder. The statistical analysis indicates that the overall diagnosis, the scores for each of the five dimensions, and the total SCID-D scores were homogenous amongst raters. The results indicate good to excellent inter-rater reliability. The SCID-D could efficiently discriminate dissociative subjects from nondissociative ones. Convergent validity was tested with the use of the Somatoform Dissociation Questionnaire SDQ-20 and the Dissociation Questionnaire DIS-Q. Correlation between the scores from these questionnaires and the SCID-D total score is high. The French translation of the SCID-D has good to excellent reliability, discriminant and convergent validity, and is similar to previous research of DSM-III and DSM-IV versions of the SCID-D in different languages. The SCID-D is a useful instrument for the assessment of dissociative symptom severity and the diagnosis of DSM-5 and ICD-11 dissociative disorders.

A new protocol for assessing action observation and imitation abilities in children with Developmental Coordination Disorder: A feasibility and reliability study

To develop a new protocol for the assessment of action observation (AO) abilities and imitation of meaningful and non-meaningful gestures, to examine its psychometric properties in children with DCD and typically developing (TD) children. For learning manual skills, AO and imitation are considered fundamental abilities. Knowledge about these modalities in children with DCD is scarce and an assessment protocol is lacking. The protocol consists of 2 tests. The AO test consists of two assembly tasks. The imitation test includes 12 meaningful and 20 non-meaningful gestures. Items of both tests are rated on a 4-point scale. Twelve children with DCD (mean age 8y3m, SD, 1.30) and 11 TD children (mean age 8y2m, SD 1.52) were enrolled. For inter-rater reliability, intraclass correlation coefficients (ICC) were calculated for the total score, weighted kappa and percentage agreement for single items. Known group validity was assessed by comparison of DCD and TD group (Wilcoxon rank sum test). For construct validity, the mABC-2 test was used. The protocol was adapted and confirmed by an intra and inter-rater reliability study (new sample of 11 DCD children, mean age 7y5m, SD 1.37). Excellent ICCs were reported for intra and inter-rater reliability for the final protocol. A significant difference between DCD and TD group was found for AO abilities (p<.01), for nonmeaningful gestures (p<.001). A significant correlation was reported between the AO test and the mABC-2 test (r=56;p≤0.0001). No significant correlations were revealed for the imitation tests. The results support the psychometric properties of this protocol. When fully validated, it may contribute to map the deficits in AO abilities and imitation, to evaluate treatment effects of imitation and AO interventions.

M-Mode Ultrasound Examination of Soleus Muscle in Healthy Subjects: Intra- and Inter-Rater Reliability Study.

Objective: M-mode ultrasound imaging (US) reflects the motion of connective tissue within muscles. The objectives of this study were to evaluate inter-rater and intra-rater reliability of soleus muscle measurements between examiners with different levels of US experience in asymptomatic subjects and to investigate the level of soleus muscle isometric activity in two positions (knee extended and knee flexed at 30°). Methods: Thirty volunteers without a history of ankle pain were evaluated with US examinations of the soleus muscle. Each muscle was scanned independently by two evaluators. Muscle at rest thickness, maximal isometric contraction thickness, time and velocity measures were detailed and blinded to the other examiner. Results: Intra- and inter-rater reliability at rest, in maximal isometric contraction thickness, contraction time and contraction velocity measures for both positions (extended and flexed knee) were reported from good to excellent for all outcome measurements. The position with the knee extended reported a statistically significant increase in thickness after motion showing 1.33 ± 0.27 mm for measurements at rest thickness with knee extended versus 1.50 ± 0.29 mm for measurements at end thickness with the knee in flexed position (p = 0.001), as well as 1.31 ± 0.23 mm for rest thickness with the knee in flexed position measurements with respect to 1.34 ± 0.24 mm for maximal isometric contraction thickness with extended knee measurements (p = 0.058). Conclusions: This study found that intra- and inter-examiner reliability of M-mode ultrasound imaging of the soleus muscle was excellent in asymptomatic subjects and the soleus muscle activity was different between the position with the knee extended and the position with the knee flexed.

Quantitative measurements of forward head in college-aged students: A conformational study of intra-rater and inter-rater reliability of a novel Posture Measuring Device

IntroductionMedical professionals frequently encounter forward head postures (FHP) in the treatment of cervical spine pain. In the clinical setting, FHP measurements are often described subjectively using the phrase, “patient demonstrates forward head posture.” There are reliable tools that can objectively measure FHP, but they are not frequently used in the clinical setting on a regular basis because the tools are unavailable, inconvenient or overly time-consuming. The objective of this study was to confirm if FHP can be reliably measured using a novel posture-measuring device. The Posture Measuring Device (PMD) was designed to quickly obtain objective measurements of FHP in the outpatient clinical setting. A pilot study demonstrated good to excellent reliability in intra-rater (test-retest) measurements and good interrater reliability using two first-year Doctor of Physical Therapy Students as raters. This study is to confirm that the PMD is able to provide intra-rater and interrater reliability using two physical therapists and two second-year Doctor of Physical Therapy students to obtain FHP measurements. MethodsTwenty-three college-aged subjects were measured individually using a PMD by a panel of four raters consisting of two physical therapists and two second-year physical therapy students to develop interrater reliability. Each rater measured each subject twice on two separate occasions to obtain a measure of intra-rater (relative) reliability. ResultsPearson Correlations demonstrate high intra-rater correlation with inconsistent interrater correlation. The average measured Intraclass Correlation Coefficient (ICC) was .822 with a 95% confidence interval indicating a high degree of reliability between the eight measurements performed on each subject. Cronbach's alpha results confirm the ICC findings of strong correlation among the eight raters (α = 0.87). Discussion and conclusionObjective measurement of forward head posture is valuable to document and monitor patient response to treatment. This study identifies a convenient and time-efficient method for measuring forward head in the clinical setting. The PMD demonstrated a high degree of reliability in intra-rater measurements of FHP.

Ultrasound imaging in patients with hip pain and suspected hip osteoarthritis: an inter-rater and intra-rater reliability study

ObjectivesThe objectives of this study were to asses (1) inter-rater and intrarater reliability of ultrasound imaging in patients with hip osteoarthritis, and (2) agreement between ultrasound and X-ray findings of...

Reliability and Feasibility of the First-Episode Psychosis Services Fidelity Scale-Revised for Remote Assessment.

The authors sought to evaluate the interrater reliability and feasibility of the First-Episode Psychosis Services Fidelity Scale-Revised (FEPS-FS-R) for remote assessment of first-episode psychosis programs according to the coordinated specialty care model. The authors used the FEPS-FS-R to assess the fidelity of 36 first-episode psychosis program sites in the United States with information from three sources: administrative data, health record review, and phone interviews with staff. Four raters independently conducted fidelity assessments of five program sites by listening to each of the staff interviews and independently rating the two other data sources from each site. To calculate interrater reliability, the authors used intraclass correlation coefficients (ICCs) for each of the five sites and across the total scores for each site. Total interrater reliability was in the good to excellent range, with a mean ICC of 0.91 (95% confidence interval = 0.72-0.99, p<0.001). Two first-episode psychosis program sites (6%) achieved excellent fidelity, 25 (69%) good fidelity, and nine (25%) fair fidelity. Of the 32 distinct items on the FEPS-FS-R, 23 (72%) were used with good or excellent fidelity. Most sites achieved high fidelity on most items, but five items received ratings indicating low-fidelity use at most sites. The fidelity assessment proved feasible, and sites required on average 10.5 hours for preparing and conducting the fidelity review. The FEPS-FS-R has high interrater reliability and can differentiate high-, moderate-, and low-fidelity sites. Most sites had good overall fidelity, but the FEPS-FS-R identified some services that were challenging to implement at many sites.

Evaluation of the Lower Trapezius Muscle Using Ultrasound Panoramic View (a Novel Approach): An Intra- and Inter-Rater Reliability Study.

The panoramic view ultrasound remains uncommon in clinical practice, probably because of its difficulty, high-cost, and lack of research. Morphological changes in muscles have been demonstrated to be related to symptomatology and provide data of interest for clinical assessment. Thus, the aim of this study was to evaluate the measurement reliability of the length of the lower trapezius muscle with the panoramic view ultrasound using a novel tool, SIG_VIP®. Twenty healthy volunteers were measured by two expert sonographers using the SIG_VIP® tool with a novel approach. Statistical analyses were performed with the R software. The intraclass correlation coefficient (ICC), standard error of measurement (SEM), minimal detectable change (MDC), and Bland-Altman plots were calculated. All the results indicated good intra-rater reliability (ICC3,1, 0.92 to 0.96; SEM, 0.59 to 0.85; MDC, 1.64 to 2.35) and inter-rater reliability (ICC3,2, 0.84 to 0.89; SEM, 1.22 to 1.53; MDC, 3.39 to 4.25). The novel system used with the described methodology can reliably measure the length of the inferior fibers of the trapezius muscle. Further research must be conducted to evaluate the reliability in patients and how pathology is related to the length of the lower trapezius muscle.

Prognostic value of histopathological DCIS features in a large-scale international interrater reliability study

PurposeFor optimal management of ductal carcinoma in situ (DCIS), reproducible histopathological assessment is essential to distinguish low-risk from high-risk DCIS. Therefore, we analyzed interrater reliability of histopathological DCIS features and assessed their associations with subsequent ipsilateral invasive breast cancer (iIBC) risk.MethodsUsing a case-cohort design, reliability was assessed in a population-based, nationwide cohort of 2767 women with screen-detected DCIS diagnosed between 1993 and 2004, treated by breast-conserving surgery with/without radiotherapy (BCS ± RT) using Krippendorff’s alpha (KA) and Gwet’s AC2 (GAC2). Thirty-eight raters scored histopathological DCIS features including grade (2-tiered and 3-tiered), growth pattern, mitotic activity, periductal fibrosis, and lymphocytic infiltrate in 342 women. Using majority opinion-based scores for each feature, their association with subsequent iIBC risk was assessed using Cox regression.ResultsInterrater reliability of grade using various classifications was fair to moderate, and only substantial for grade 1 versus 2 + 3 when using GAC2 (0.78). Reliability for growth pattern (KA 0.44, GAC2 0.78), calcifications (KA 0.49, GAC2 0.70) and necrosis (KA 0.47, GAC2 0.70) was moderate using KA and substantial using GAC2; for (type of) periductal fibrosis and lymphocytic infiltrate fair to moderate estimates were found and for mitotic activity reliability was substantial using GAC2 (0.70). Only in patients treated with BCS-RT, high mitotic activity was associated with a higher iIBC risk in univariable analysis (Hazard Ratio (HR) 2.53, 95% Confidence Interval (95% CI) 1.05–6.11); grade 3 versus 1 + 2 (HR 2.64, 95% CI 1.35–5.14) and a cribriform/solid versus flat epithelial atypia/clinging/(micro)papillary growth pattern (HR 3.70, 95% CI 1.34–10.23) were independently associated with a higher iIBC risk.ConclusionsUsing majority opinion-based scores, DCIS grade, growth pattern, and mitotic activity are associated with iIBC risk in patients treated with BCS-RT, but interrater variability is substantial. Semi-quantitative grading, incorporating and separately evaluating nuclear pleomorphism, growth pattern, and mitotic activity, may improve the reliability and prognostic value of these features.

Development and evaluation of the Basic Outcome Measure Protocol for Aphasia (BOMPA)

Purpose The Basic Outcome Measure Protocol for Aphasia (BOMPA) is a practical tool that allows for a quick self-report on quality of life from the perspective of the person with aphasia, as well as a clinical evaluation of aphasia severity and the ability to participate in conversation. The primary aim of this paper is to describe development of BOMPA and report on results of an inter-rater reliability study involving speech-language pathology raters. Method The inter-rater reliability study utilised a fully crossed design and included independent ratings of 12 videos by 20 speech-language pathologists. Result Results indicate moderate to strong inter-rater reliability among participant speech-language pathology raters (0.65–0.96), as well as when comparing these participant ratings with an expert rater’s gold standard (0.59–0.86). Conclusion BOMPA may be a useful outcome measurement tool for time-pressed clinicians in clinical settings.