Investigate Interrater Reliability Research Articles

Image quality and radiation doses in abdominal CT: A multicenter study

PurposeTo benchmark image quality and corresponding radiation doses for acute abdominal CT examination across different laboratories and CT manufacturers. MethodAn anthropomorphic phantom was scanned once with local abdominal CT protocols at 40 CT scanners, from four vendors, in thirty-three sites. Quantitative image quality was evaluated by CNR and SNR in the liver and kidney parenchyma. Qualitative image quality was assessed by visual grading analysis performed by three experienced radiologists using a five-point Likert scale to score thirteen image quality criteria. The CTDIvol was recorded for each scan. Pearson’s correlation coefficient was calculated for the continuous variables, and the intraclass correlation coefficient was used to investigate interrater reliability between the radiologists. ResultsCTDIvol ranged from 3.5 to 12 mGy (median 5.3 mGy, third quartile 6.7 mGy). SNR in liver parenchyma ranged from 4.4 to 14.4 (median 8.5), and CNR ranged from 2.7 to 11.2 (median 6.1). A weak correlation was found between CTDIvol and CNR (r = 0.270, p = 0.092). Variations in CNR across scanners at the same dose level CTDIvol were observed. No significant difference in CTDIvol or CNR was found based on scanner installation year. The oldest scanners had a 15 % higher median CTDIvol and a 12 % lower median CNR. The ICC showed acceptable agreement for all dose groups: low (ICC=0.889), medium (ICC=0.767), high (ICC=0.847), and in low (ICC=0.803) and medium (ICC=0.811) CNR groups. ConclusionThere was large variation in radiation dose and image quality across the different CT scanners. Interestingly, the weak correlation between CTDIvol and CNR indicates that higher doses do not consistently improve CNR, indicating a need for systematic assessment and optimization of image quality and radiation doses for the abdominal CT examination.

The Sensory Observation Autism Rating Scale (SOAR): Developed using the PROMIS® framework.

Autistic people experience the sensory world differently, impacting behavior. First-hand accounts and group-based research have found that sensory differences impact a range of things including family life, anxiety, participation, and daily living. Early sensory differences are widely reported to be associated with a cascade of developmental difference, suggesting that early autism diagnosis and sensory mapping could enable the provision of supports to facilitate flourishing. However, appropriate measurement tools are not available as all rely on proxy report or are observation measures which include limited modalities or domains and require the administration of stimuli. Therefore, following the gold-standard recommendations for measurement development outlined by the PROMIS® framework, we created the Sensory Observation Autism Rating scale (SOAR). We identified sensory behaviors across all primary domains and modalities through an extensive autism-sensory literature review and from focus groups with autism stakeholders. The initial item bank was then refined by an expert panel and through video coding five-minutes of free play from Autism Diagnostic Observation Schedule assessments of 105 age- and gender-matched autistic and developmentally delayed children (aged 13-36 months; 38 female). An additional 25% of the sample were double coded to investigate interrater reliability. Observational data and expert review supported the reduction of the item bank to 37 items. We propose that the refined SOAR has excellent face and ecological validity, along with interrater reliability (Intraclass correlation=0.87-0.99). Following further data collection and refinement, SOAR has promise to fully characterize sensory behaviors in autistic children and indicate useful supports.

Multi-Slice Computed Tomography Analysis in Patients Undergoing Transcatheter Aortic Valve Replacement - Impact of Workflows on Measurement of Virtual Aortic Annulus and Valve Size.

Exact and reliable measurements of anatomical dimensions in pre-procedural multi-slice computed tomography (MSCT) scans are crucial for optimal valve sizing and clinical results of transcatheter aortic valve replacement (TAVR). This study aimed to investigate interrater reliability between routinely used workflows for pre-procedural analysis. MSCT scans of 329 patients scheduled for TAVR were analyzed using both a 3mensio and SECTRA IDS7 platform. The results were retrospectively compared using the intraclass correlation coefficient, revealing excellent correlation in the analysis of simple diameters and poor correlation in the assessment of more complex structures with impact on calculated valve size.

Validity of patient self-reports and clinical signs in the assessment of sleep bruxism based on home-recorded electromyographic/electrocardiographic data.

Reliable and suitable bruxism assessment would be desirable, but available studies presented heterogeneous results. To determine the agreement of patient self-reports and clinical signs of sleep bruxism (SB) with electromyographic/electrocardiographic data. Two hundred individuals from a German dental clinic population (120 women and 80men) participated in the study. Participants completed different SB questionnaires, had a clinical examination to evaluate bruxism signs and used the Bruxoff® device to record electromyographic/electrocardiographic data for five nights. To investigate interrater reliability for clinical diagnosis of bruxism, 126 of the 200 participants were assessed for clinical signs of bruxism by two independent uncalibrated examiners. Statistical evaluation included calculation of sensitivity, specificity and accuracy and of Cohen's kappa. Based on the Bruxoff® data, 106 participants were identified as bruxers and 94 as non-bruxers. The 106 bruxers were further classified into 47moderate and 59 intense bruxers. The highest accuracy and sensitivity values were recorded for the overall score for clinical bruxism signs (accuracy: 72.0% and sensitivity: 70.8%). The best specificity (96.8%) was seen for the question regarding tooth grinding in the last two weeks reported by others, but concurrent sensitivity was very low (3.8%). Analysis of interrater reliability revealed a substantial agreement (Cohen's kappa of 0.6). The study results indicate that self-report questionnaires and clinical signs have moderate sensitivity, specificity and accuracy for diagnosing bruxism comparing with an ambulatory device for current SB (Bruxoff®). Regarding interrater reliability for clinical signs of SB, substantial agreement was found between the two examiners. NCT03039985.

Reliability and concurrent validity of mobile health technology for patient self-monitoring in physical rehabilitation

BackgroundForearm pronation and supination are important for everyday functional tasks and some recreational activities. Healthcare providers use reliable and valid tools during the physical rehabilitation process to measure joint range of motion (ROM), assess functional mobility, guide decisions for skilled interventions, and progress a plan of care. Since the onset of the COVID-19 pandemic, both healthcare providers and patients benefited from mobile health technologies that have emerged, which can be used by patients in the home to monitor ROM and assist the healthcare provider in guiding the rehabilitation process when utilizing telehealth.PurposeThe goal of this study was to investigate the reliability and concurrent validity of a smartphone application for obtaining goniometric measurements of forearm pronation and supination.MethodsThis study consisted of 83 participants that were recruited on a voluntary basis from an academic institution. An iPhone with the application Clinometer and a standard goniometer (SG) were utilized to obtain goniometric measurements of forearm pronation and supination. The intraclass correlation coefficient (ICC) was used to analyze intrarater reliability, and the Pearson correlation coefficient was used to analyze concurrent validity. Scatterplots with regression lines were created to visually display the results.ResultsThe smartphone demonstrated strong correlations for both pronation and supination (r = 0.71, P < .001; r = 0.73, P < .001). This study demonstrated overall good-excellent intrarater reliability and good concurrent validity for the smartphone application with a higher test-retest reliability in the measurement of forearm pronation compared to supination.ConclusionsThis study concludes that the reliability and concurrent validity of the smartphone was consistent with the SG for assessing forearm pronation and supination. It may be of value to further investigate interrater reliability between patient and healthcare practitioner, and report on the ease of use to assess ROM with a smartphone.

Neuroradiological differentiation of white matter lesions in patients with multiple sclerosis and Fabry disease

ObjectiveWhite matter lesions (WML) in multiple sclerosis (MS) differ from vascular WML caused by Fabry disease (FD). However, in atypical cases the discrimination can be difficult and may vary between individual raters. The aim of this study was to evaluate interrater reliability of WML differentiation between MS and FD patients.Materials and methodsBrain MRI scans of 21 patients with genetically confirmed FD were compared to 21 matched patients with MS. Pseudonymized axial FLAIR sequences were assessed by 6 blinded raters and attributed to either the MS or the FD group to investigate interrater reliability. Additionally, localization of WML was compared between the two groups.ResultsThe median age of patients was 46 years (IQR 35–58). Interrater reliability was moderate with a Fleiss' Kappa of 0.45 (95%CI 0.3–0.59). Overall, 85% of all ratings in the MS group and 75% in the FD group were correct. However, only 38% of patients with MS and 33% of patients with FD were correctly identified by all 6 raters. WML involving the corpus callosum (p < 0.001) as well as juxtacortical (p < 0.001) and infratentorial lesions (p = 0.03) were more frequently observed in MS patients.ConclusionInterrater reliability regarding visual differentiation of WML in MS from vascular WML in FD on standard axial FLAIR images alone is only moderate, despite the distinctive features of lesions in each group.

Interrater reliability and agreement between children with visual impairment and their parents on participation and quality of life.

ABSTRACTPurposeTo investigate interrater reliability and agreement between children with visual impairment (VI) and their parents on participation and quality of life and factors associated with disagreement.MethodsChildren 7–17 years and their parents completed the PAI‐CY 7–12 (n = 180) and 13–17 (n = 65), the KIDSCREEN‐27 (n = 250) and the CASP (n = 70). Mean scores of children and parents were compared, with effect sizes for the differences. Interrater reliability was evaluated using intraclass correlation coefficients (ICCs), whereas agreement was assessed using the Bland–Altman limits of agreement. Linear regression analyses examined child‐ and proxy‐related factors associated with discrepancies.ResultsOn average, children rated their participation and quality of life as significantly better than their parents on most (sub)scales, but with wide range of disagreement. Effect sizes were large for the PAI‐CY 7–12 (0.86) and 13–17 (0.86) and small for the CASP (0.36) and KIDSCREEN‐27 (0.18–0.28). Interrater reliability was poor for the PAI‐CY 7–12 (ICC = 0.29) and most KIDSCREEN‐27 subscales (ICC =0.18–0.32), moderate for the PAI‐CY 13–17 (ICC =0.43) and the KIDSCREEN‐27 Physical Wellbeing subscale (ICC = 0.46) and good for the CASP (ICC = 0.63). Comorbidity was significantly associated with greater discrepancies on participation scales.ConclusionChildren with VI and their parents have different perspectives on the child’s participation and quality of life. Disagreement was largest on participation scales and smallest on quality of life subscales, while opposite results were found for interrater reliability. Reports of children and parents seem to be complementary and are both relevant to obtain a complete picture of the burden of VI and relevant to inform healthcare decisions.

Development and psychometric evaluation of the Mandarin Chinese version of the Yale-Brown Obsessive-Compulsive Scale – Second Edition

Objective:To examine the psychometric properties of the Chinese version of the Yale-Brown Obsessive Compulsive Scale – Second Edition (Y-BOCS-II).Method:A total of 86 adults with a primary diagnosis of obsessive-compulsive disorder (OCD), ranging in age from 15 to 78 years, participated in the study. Participants were administered the Y-BOCS-II by a trained clinician who also rated overall illness severity on two additional measures. Patients completed the Obsessive Compulsive Inventory-Revised and Depression Anxiety Stress Scale-21.Results:Results indicated high internal consistency and fair 1-week test retest reliability. The Y-BOCS-II scales correlated strongly with clinician-rated obsessive-compulsive severity and modestly with self-reported obsessive-compulsive symptom frequency and distress. The relationship between Y-BOCS-II total score and depressive and anxiety symptoms was strong, which may reflect high rates of comorbid conditions in this sample or the linkage between obsessive-compulsive symptom severity and distress. Factor analysis demonstrated a two-factor structure consisting of obsession and compulsion factors, with interference due to obsessions cross-loading.Conclusions:Overall, these results support the use of the Y-BOCS-II among individuals from China. Future study by an independent group is necessary to replicate these findings, as well as investigate interrater reliability and treatment sensitivity.

Development of a new valid and reliable microsurgical skill assessment scale for ophthalmology residents

BackgroundMore and more concerns have been arisen about the ability of new medical graduates to meet the demands of today’s practice environment. In this study, we wanted to develop a valid, reliable and standardized assessment tool for evaluating the basic microsurgical skills of residents in a microsurgery laboratory, to get them well prepared before entering the surgical realm of ophthalmology.MethodsTwenty-three experts who have teaching experience reviewed the assessment scale. Constructive comments were incorporated to ensure face and content validity. Twenty-one attendings from different specialties then graded eight corneal rupture suturing videos with the scale to investigate interrater reliability. Fourteen of them graded the same videos 3 months later to investigate intrarater reliability (repeatability).ResultsA total of 280 assessment scales were completed. All the ICC values of interrater reliability were greater than 0.8 with 75% data greater than 0.9 (range 0.860–0.976). All the ICC values of intrarater reliability (repeatability) were also greater than 0.8 with 63% data greater than 0.9 (range 0.833–0.954).ConclusionsThe assessment scale we developed is valid and reliable. This tool could be useful to ensure that junior residents achieve a certain level of microsurgical technique in a laboratory environment before training in the operation room. Hopefully, this tool will provide a structured template for other residency programs to assess their residents for basic microsurgical skills.

Does the Lever Sign Test Have Added Value for Diagnosing Anterior Cruciate Ligament Ruptures?

Background:Diagnosing an anterior cruciate ligament (ACL) rupture based on a physical examination remains a challenge for both surgeons and physical therapists. The lever sign test was developed to overcome the practical limitations of other tests and to optimize diagnosis. An evaluation of the measurement properties of the lever sign test is needed to make adequate interpretations in practice.Purpose:To evaluate the reliability and diagnostic value of the lever sign test.Study Design:Cohort study (diagnosis); Level of evidence, 2.Methods:A total of 94 patients were recruited between November 2014 and July 2016. Patients were included if they were at least 16 years old, suffered from knee trauma, and had indications for knee arthroscopic surgery. Lever sign, anterior drawer, Lachman, and pivot-shift test outcomes were examined by an orthopaedic/trauma surgeon and a physical therapist. A test-retest design was used to investigate interrater reliability. Moreover, the lever sign test outcomes, alone and in combination with the other diagnostic tests, were compared with arthroscopic results, which served as the gold standard for the test’s diagnostic value.Results:The lever sign test and pivot-shift test had kappa values exceeding 0.80 for interrater reliability. The kappa values for the anterior drawer test and Lachman test were 0.80 and 0.77, respectively. The lever sign test showed the highest specificity (100%) and the lowest sensitivity (39%) when compared with the other 3 tests. Moreover, its positive and negative predictive values were 100% and 65%, respectively, while an accuracy of 71% was calculated. Clustering the lever sign test parallel with the other 3 tests resulted in the highest accuracy of 91%.Conclusion:The lever sign test appears to have high interrater reliability and is the most specific test, showing a maximal positive predictive value. A positive lever sign test result indicates an ACL rupture. These results support the added value of the lever sign test for diagnosing ACL ruptures.

Inter-rater Reliability of Needle Electromyographic Findings in Patients with Sciatica

Purpose: Investigate inter-rater reliability of needle electromyographic findings among experienced physical therapist electromyographers. Methods: Masked review of 24 electromyographic recordings from patients with sciatica referred to physical therapy was undertaken. An examiner unmasked to patient history and physical examination findings digitally recorded and stored insertional and resting electromyographic activity as de-identified digital audio-video files to be analyzed by 2 masked examiners. Examiners provided ratings for individual muscles and overall electrodiagnostic impression. Agreement was assessed using Cohen’s kappa (κ) statistics. Results: Examiner agreement for insertional and resting electromyographic activity for all muscles combined was substantial (κ ≥ 0.68, 95% CI: 0.50 to 0.89; P≤.001), ranging from fair (κ=0.33, 95% CI: -0.25 to 1.0; P>.05) to perfect (κ=1.0, 95% CI: 1.0 to 1.0; P≤.001) for individual muscles examined. Pairwise examiner comparisons revealed moderate (κ=0.43, 95% CI: 0.11 to 0.76; P=.01) to substantial (κ=0.75, 95% CI: 0.48 to 1.0; P<.0001) agreement for the final electrodiagnostic impression and fair (κw=0.31, 95% CI: 0.12 to 0.50; P=0.004) to substantial (κw=0.62, 95% CI: 0.37 to 0.87; P<.0001) agreement for the overall electrodiagnostic impression. Conclusions: Needle electromyographic activity can be reliably assessed among experienced physical therapist electromyographers in patients with sciatica referred to physical therapy.

Assessing Therapeutic Competence in Cognitive Therapy for Social Phobia: Psychometric Properties of the Cognitive Therapy Competence Scale for Social Phobia (CTCS-SP)

There has been considerable acknowledgement in treatment outcome research that, although the assessment of treatment integrity is essential in many respects, it requires great effort as well as resources and is therefore often neglected. In order to fill this gap, the Cognitive Therapy Competence Scale for Social Phobia (CTCS-SP) was developed, based on the Cognitive Therapy Scale, to measure therapist competence in delivering cognitive therapy for social phobia. The aim of the present study was to investigate interrater reliability, internal consistency and retest reliability of the scale. Raters evaluated therapist competence from 161 videotaped sessions (98 patients) selected from 234 cognitive treatments within a multi-centre study. Interrater-reliability was found to be high for the overall score (ICC = .81) and moderate for individual items (ICC = .62-.92). Internal consistency and retest reliability were also found to be high (Cronbach's alpha = .89; (ICCretest = .86). The results indicate that the CTCS-SP is highly reliable. As even individual items yield satisfactory reliability, the scale can be used in various fields of research, including the measurement of changes in skill acquisition and the impact of competence on outcome criteria.

High-Level Mobility Assessment Tool (HiMAT): Interrater Reliability, Retest Reliability, and Internal Consistency

The High-Level Mobility Assessment Tool (HiMAT) assesses high-level mobility in people who have sustained a traumatic brain injury (TBI). The purpose of this study was to investigate the interrater reliability, retest reliability, and internal consistency of data obtained with the HiMAT. Three physical therapists and 103 people with TBI were recruited from a rehabilitation hospital. Three physical therapists concurrently assessed a subset of 17 subjects with TBI to investigate interrater reliability. One physical therapist assessed a different subset of 20 subjects with TBI on 2 occasions, 2 days apart, to investigate retest reliability. Data from the entire sample of 103 subjects were used to investigate the internal consistency of this new scale. Both the interrater reliability (intraclass correlation coefficient [ICC]=.99) and the retest reliability (ICC=.99) of the HiMAT data were very high. For retest reliability, a small systematic change was detected (t=3.82, df=19), indicating a marginal improvement of 1 point at retest. Internal consistency also was very high (Cronbach alpha=.97). The HiMAT is a new tool specifically designed to measure high-level mobility, which currently is not a component of existing scales used in TBI. This study demonstrated that the HiMAT is a reliable tool for measuring high-level mobility.

Reliability of categorical and dimensional judgments of personality disorder

To investigate interrater reliability on categorical and dimensional judgments of personality disorder, five clinicians completed five different judgment tasks for each of 10 case vignettes. The reliability estimates support previously unconfirmed statements that dimensional judgments are substantially more reliable than categorical diagnoses.

Adaptive Functioning in Children Hospitalized for Psychiatric Disturbances

Abstract Fifty-two psychiatrically hospitalized children were assessed using the Vineland Adaptive Behavior Scale (VABS) to investigate interrater reliability of adaptive functioning ratings between parent and clinician. The VABS Survey Form was completed upon the child's admission and the VABS Classroom Edition, clinician rating, was completed after the first week of hospitalization. Parental scores were significantly lower (pairwise t test, p < 0.0000) in all VABS domains and the adaptive behavior composite score, suggesting that parents perceive their children as functioning at a lower level than do the clinical staff. Parental bias/distortion, clinical bias, and effect of hospitalization are discussed as possible sources of variance.

Preferences for Job Attributes of Blue-Collar Employees in the Public Sector

It is important to understand career motivations of employees. Such is the case for potential incumbents (e.g., career choice), incumbents (e.g., career choice), and organizations (e.g., recruiting, selection) (1, 4 ) . The present purpose was to describe values, in terms of preferences for job attributes of a group of blue-collar employees in the public sector. Subpurposes were to investigate interrater reliability of self-preferences and preferences attributed to others and the relationship between own and others' preferences. Participants were 90 individuals employed in Custodial Services Unit or Buildings and Grounds Maintenance Unit of the Physical Plant Department of a large public urban university in a major city. The dependent variable was job preferences ( 3 ) . Respondents ranked 10 job attributes under two conditions, respondent's preferences (self) and respondent's perception of others' preferences (ochers). The 10 job attributes were Advancement, Benefits, Company, Coworkers, Hours, Pay, Security, Supervisor, Type of Work, and Working Conditions. Order of self-preference was Pay (most preferred), Benefits, Advancement, Security, Company, Supervisor. Hours, Working Conditions, Type of Work, and Coworkers (least preferred). W was .21 ( p < .01). Order of others' preferences was Pay (most preferred), Benefits, Advancement, Hours, Security, Supervisor, Type of Work, Company, Working Conditions, and Coworkers (least preferred). W was .30 ( p < .01). Rho (self and others) was .86 ( p < .01). Implications for personnel management in the public sector vis-a-vis potential incumbents, incurnbenrs, and organizations are suggested-particularly regarding high ranking of Pay and low ranking of Type of Work and Coworkers (2 , 5, 6, 7 ) . Own and others' preferences showed adequate interrater reliability. There was a high oositive correlation between hierarchies of oreferences. For these 90 blue-collar employees in the public sector, there were rela;ively stable hierarchies of own and others' preferences; hierarchies were positively related. One caveat regarding generalization from these findings to other settings is emphasized. It is recommended the study be replicated in other organizations, public and private, with these and other categories of employees.