Reliability Of Composite Scores Research Articles

Multivariate Structural Equation Modeling Techniques for Estimating Reliability, Measurement Error, and Subscale Viability When Using Both Composite and Subscale Scores in Practice

We illustrate how structural equation models (SEMs) can be used to assess the reliability and generalizability of composite and subscale scores, proportions of multiple sources of measurement error, and subscale added value within multivariate designs using data from a popular inventory measuring hierarchically structured personality traits. We compare these techniques between standard SEMs representing congeneric relations between indicators and underlying factors versus SEM-based generalizability theory (GT) designs with simplified essential tau-equivalent constraints. Results strongly emphasized the importance of accounting for multiple sources of measurement error in both contexts and revealed that, in most but not all instances, congeneric designs yielded higher score accuracy, lower proportions of measurement error, greater average subscale score viability, stronger model fits, and differing magnitudes of disattenuated subscale intercorrelations. Extending the congeneric analyses to the item level further highlighted consistent weaknesses in the psychometric properties of negatively versus positively keyed items. Collectively, these findings demonstrate the practical value and advantages of applying GT-based principles to congeneric SEMs that are much more commonly encountered in the research literature and more directly linked to the specific measures being analyzed. We also provide prophecy formulas to estimate reliability and generalizability coefficients, proportions of individual sources of measurement error, and subscale added-value indices for changes made to measurement procedures and offer guidelines and examples for running all illustrated analyses using the lavaan (Version 0.6-17) and semTools (Version 0.5-6) packages in R. The methods described for the analyzed designs are applicable to any objectively or subjectively scored assessments for which both composite and subcomponent scores are reported.

Reliability and validity of composite scores from the timed subtests of the D-KEFS battery.

Many neuropsychological tests of executive functioning (EF) have test-retest reliabilities below what is considered acceptable. Combining several tests into a composite can improve reliability, although due to concerns about the multifaceted nature of the EF construct, this practice is rarely applied. The present study (a) examined internal consistencies and test-retest reliabilities of two composites of four EF tests (EF-Speed from timed scores and EF-Accuracy from error scores), and (b) preliminarily examined the criterion validity of the composites as a first step toward construct validation. Participants were 576 adults aged 18-93. Participants completed timed subtests of the Delis-Kaplan Executive Function System (D-KEFS) at baseline. A subset of participants was retested twice (at 30 min and 1 year), or once (at 6 weeks). An additional subsample completed Timed Instrumental Activities of Daily Living (TIADLs), which was used to preliminarily examine the composites' criterion validity. Internal consistency and test-retest reliabilities ranged from good to excellent for the EF-Speed composite (.809-.901) but were poor to acceptable for the EF-Accuracy composite (.510-.736). Both composites surpassed reliabilities of individual test or condition scores; one third of the examined individual speed-based scores and none of the individual error-based scores met the acceptable reliability threshold (i.e., < .7). The EF-Speed composite accounted for more variance in TIADL speed, and the EF-Accuracy in TIADL accuracy, than did any of the individual test scores, providing preliminary evidence of clinical utility and criterion validity. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Supplemental Material for Reliability and Validity of Composite Scores From the Timed Subtests of the D-KEFS Battery

Supplemental Material for Reliability and Validity of Composite Scores From the Timed Subtests of the D-KEFS Battery

Belgian rheumatologists' preferences regarding measures of disease activity in patients with rheumatoid arthritis: results from a mixed-methods study.

The reliability and clinical usefulness of the different composite disease activity scores and their individual components in Rheumatoid Arthritis (RA) are still debated. This study investigated which measures of disease activity were preferred by rheumatologists. A mixed-method study was performed. First, ten Belgian rheumatologists were invited for individual interviews on their current practice and preferences for measurement of RA disease activity. Results of this qualitative study and evidence from literature served as input for developing a survey. This survey asked rheumatologists to rate preferred standard disease activity score(s), their individual components, ultrasound and related patient-reported outcomes (PROs), by maximum difference scaling. The relative importance score (RIS) for each indicator was calculated using hierarchical Bayes modeling. The qualitative study included 6/10 invited rheumatologists. Composite scores and components were perceived as useful, while PROs were found subjective. Interestingly, ultrasound was used to mediate discrepancies between physician and patient. The survey based on this was sent to 244 Belgian rheumatologists, 83/244 (34%) responded, including 66/83 (80%) complete and 17/83 (20%) incomplete surveys (two missing essential information). Most rheumatologists (75/81, 93%) used a disease activity score and 68/81 (84%) preferred the DAS28-CRP. Swollen joint count obtained the highest mean ± SD RIS (22.54 ± 2.64), followed by DAS28 ESR/CRP (20.61 ± 4.06), ultrasound (16.47 ± 7.97), CRP (13.34 ± 6.11) and physician's global assessment (12.59 ± 7.83). PROs including fatigue, pain, and patient's global assessment, and Health Assessment Questionnaire, obtained the lowest mean RIS (0.34-2.54). Rheumatologists place more faith in self-assessed disease activity components or in laboratory tests. Trust in PROs to evaluate disease activity is low in clinical practice.

Item-level and Composite-level Interrater Reliability of Functional Movement Screen™ Scores Following Condensed Training in Novice Raters.

BACKGROUNDThe Functional Movement Screen™ (FMS™) is a clinical instrument designed to use movement behaviors to screen individuals for injury risk. Current rater certification programs focus on extensive, individualized training, which may not be appropriate in all screening contexts.PURPOSEThe purpose of this research was to examine the effect of a two-hour FMSTM training seminar on measures of reliability between previously untrained scorers.STUDY DESIGNRepeated measures, descriptive cohort study.METHODSFour novice raters completed a two-hour training course administered by an FMS™-certified, licensed physical therapist. The novices and the instructor then scored a group of 16 individuals on the seven FMS™ component tests on two separate occasions. Interrater reliability was assessed for FMS™ component scores using Fleiss’ kappa and Krippendorff’s α. Interrater reliability for the FMS™ composite score was assessed using a two-way ICC for agreement (a priori significance level=0.05).RESULTSReliability ranged from fair to almost perfect (kappa) for Deep Squat (0.61 Day 1, 0.79 Day 2), Shoulder Mobility (0.90 Day 1, 1.00 Day 2), Active Straight Leg Raise (0.53 Day 1, 0.69 Day 2), and Trunk Stability Push Up (0.48 Day 1, 0.49 Day 2) on both testing occurrences (p<0.05). Reliability (kappa) was fair for Inline Lunge (0.24 Day 1, 0.39 Day 2), and poor for Hurdle Step (Day 1 -0.01, Day 2 no result) and Rotary Stability (Day 1 -0.03, Day 2 -0.01). Results for Krippendorff’s α were similar, with unacceptable interrater reliability for Hurdle Step (Day 1 -0.01, Day 2 1.00), Inline Lunge (Day 1 0.31, Day 2 0.39), and Rotary Stability (Day 1 -0.02, Day 2 -0.01). Interrater composite score reliability (ICC) was good (0.79 Day 1, 0.84 Day 2; both p<0.05).CONCLUSIONSFindings suggest that a brief training seminar may be sufficient to ensure acceptable reliability in many, but not all, of the FMS™ component tests and composite score.Levels of EvidenceLevel 2b

Measuring Students' Curiosity Character Using Confirmatory Factor Analysis

&lt;p style="text-align: justify;"&gt;This study aims to test proving the scale of measuring the character of students' curiosity using confirmatory factor analysis. This study uses a quantitative approach. The sample used was 325 randomly selected participants. The variable examined in this study is the character of curiosity. The curiosity variable is focused on elementary school students. In this study, this variable was measured by four indicators and 16 measuring items. The four indicators include: paying attention, taking notes, asking questions, and comparing. Data collection was carried out by distributing curiosity questionnaires to participants. The data analysis technique used was confirmatory factor analysis. The reliability analysis used was the composite score reliability analysis. The results of the study concluded that the constituent indicators of the latent variables of students' curiosity show that all loading factor values have a significant effect on the latent variables in the first order confirmatory factor analysis (CFA). Meanwhile, in the second order CFA, the variable of curiosity, the biggest contribution was the indicators compare. This student curiosity assessment model has fulfilled the goodness of fit aspect of the model and has been reliable based on construct reliability. This study recommends a scale measuring elementary school students' curiosity.&lt;/p&gt;

Measuring Spinal Mobility Using an Inertial Measurement Unit System: A Reliability Study in Axial Spondyloarthritis.

The objectives of this study were to evaluate the reliability of wearable inertial motion unit (IMU) sensors in measuring spinal range of motion under supervised and unsupervised conditions in both laboratory and ambulatory settings. A secondary aim of the study was to evaluate the reliability of composite IMU metrology scores (IMU-ASMI (Amb)). Forty people with axSpA participated in this clinical measurement study. Participant spinal mobility was assessed by conventional metrology (Bath Ankylosing Spondylitis Metrology Index, linear version—BASMILin) and by a wireless IMU sensor-based system which measured lumbar flexion-extension, lateral flexion and rotation. Each sensor-based movement test was converted to a normalized index and used to calculate IMU-ASMI (Amb) scores. Test-retest reliability was evaluated using intra-class correlation coefficients (ICC). There was good to excellent agreement for all spinal range of movements (ICC > 0.85) and IMU-ASMI (Amb) scores (ICC > 0.87) across all conditions. Correlations between IMU-ASMI (Amb) scores and conventional metrology were strong (Pearson correlation ≥ 0.85). An IMU sensor-based system is a reliable way of measuring spinal lumbar mobility in axSpA under supervised and unsupervised conditions. While not a replacement for established clinical measures, composite IMU-ASMI (Amb) scores may be reliably used as a proxy measure of spinal mobility.

Composite reliability of multilevel data: It's about observed scores and construct meanings.

This article shows how the concept of reliability of composite scores, as defined in classical test theory, can be extended to the context of multilevel modeling. In particular, it discusses the contributions and limitations of the various level-specific reliability indices proposed by Geldhof, Preacher, and Zyphur (2014), denoted as ω̃b and ω̃w (and also α̃b and α̃w). One major limitation of those indices is that they are quantities for latent, unobserved level-specific composite scores, and are not suitable for observed composites at different levels. As illustrated using simulated data in this article, ω̃b can drastically overestimate the true reliability of between-level composite scores (i.e., observed cluster means). Another limitation is that the development of those indices did not consider the recent conceptual development on construct meanings in multilevel modeling (Stapleton & Johnson, 2019; Stapleton, Yang, & Hancock, 2016). To address the second limitation, this article defines reliability indices (ω2l, ωb, ωw, α2l, αb, αw) for three types of multilevel observed composite scores measuring various multilevel constructs: individual, configural, shared, and within-cluster. The article also shows how researchers can obtain sample point and interval estimates using the derived formulas and the provided R and Mplus code. In addition, a large-scale national data set was used to illustrate the proposed methods for estimating reliability for the three types of multilevel composite scores, and practical recommendations on when different indices should be reported are provided. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

An examination of reliable change methods for measuring cognitive change with the Cogstate Computerized Battery: Research and clinical implications.

To compare the performance of four reliable change (RC) methods with respect to measuring cognitive change on the Cogstate Computerized Battery (CCB). We assessed cognitive change in 57 healthy, urban, well-educated males on the CCB at baseline and 6months (Median age=50, 65% university-educated). The study CCB version comprised seven measures covering attention, processing speed, verbal learning, and memory. Raw scores were z-score transformed using age-corrected Cogstate norms (CN) or the sample mean and standard deviation (internal standardization [IS]), and then averaged to create composite z-scores. Composite scores were entered into four RC formulae. RC was defined based on a 90% two-tailed confidence interval. Change scores were compared as continuous (z-scores) and ordinal variables (RC outcomes). CCB composite score reliability (rXY=.78-.79) was replicated in an age- and sex-matched Cogstate database sample of similar size. There was good overall agreement between the four RC methods (Bland-Altman Mdiff=.00; 95% limits of agreement with the mean-CN: z=±.90; IS: z=±.93), with each model adhering closely to the 10% rate of RC expected by chance alone (largest χ2=.86, p=.99). Initial norming strategy (CN or IS) did not affect these outcomes. Norming strategy and RC method choice did not significantly impact cognitive change predictions on CCB composite scores. A series of example case data are provided to practically demonstrate the steps involved in applying the longitudinal norms generated in this study. Research in more diverse normative samples is warranted.

Test–retest reliability of RC21X: a web-based cognitive and neuromotor performance measurement tool

As computerized cognitive testing becomes increasingly popular in clinical and research settings, conducting studies on efficacy and psychometric properties is essential. One such program is RC21X, a web-based brain performance measurement tool. Based on empirically supported neurocognitive and neuromotor tasks, the 12-min test consists of 15 modules measuring memory, motor coordination, processing speed, and executive functioning. Because individuals may use RC21X repeatedly to track changes in cognitive performance, establishing reliability of the program is imperative. The current study examined test-retest reliability of RC21X within a 2-week period. The sample consisted of 222 individuals: 192 (86.5%) were male, and 30 (13.5%) were female. Average age was 44.06years (SD=17.76), with ages ranging from 7 to 82years. We computed Pearson's correlation coefficients for module and composite scores to determine reliability between performance at times 1 and 2. All correlations were statistically significant (p<.001). The 2-week test-retest reliability for composite score was 0.72, with subtest coefficients ranging from 0.54 on an auditory memory recognition task to 0.89 on a finger tapping task. We replicated these analyses with participants' (n=43) test sessions 3 and 4; we found similar results to those from test 1 and test 2 analyses, suggesting stability of results over multiple administrations. Results for RC21X were comparable to existing literature that supports moderate to high reliability of other computer-based tests. Although future research needs to investigate validity of RC21X, our findings support potential applications in research, clinical use, and personal brain performance measurement.

Synthesizing and Reporting Milestones-Based Learner Analytics: Validity Evidence From a Longitudinal Cohort of Internal Medicine Residents

Coordinating and operationalizing assessment systems that effectively streamline and measure fine-grained progression of residents at various stages of graduate medical training can be challenging. This article describes development, administration, and psychometric analyses of a learner analytics system to resolve challenges in implementation of milestones by introducing the Scoring Grid Model, operationalized in an internal medicine (IM) residency program. A 3-year longitudinal cohort of 34 residents at the University of Illinois at Chicago College of Medicine began using this learner analytics system, from entry (July 2013) to graduation (June 2016). Scores from 23 assessments used throughout the 3-year training were synthesized using the Scoring Grid Model learner analytics system, to generate scores corresponding to the 22 reportable IM subcompetencies. A consensus model was used to develop and pilot test the model using feedback from IM faculty members and residents. Scores from the scoring grid were used to inform promotion decisions and reporting of milestone levels. Descriptive statistics and mixed-effects regression were used to examine data trends and gather validity evidence. Initial validity evidence for content, internal structure, and relations to other variables that systematically integrate assessment scores aligned with the reportable milestones framework are presented, including composite score reliability of scores generated from the learner analytics system. The scoring grid provided fine-grained learner profiles and showed predictive utility in identifying low-performing residents. The Scoring Grid Model and associated learner analytics data platform may provide a practical, reasonable solution for generating fine-grained, milestones-based profiles supporting resident progress.

Test/Re-Test reliability of the readiness for interprofessional education learning scale and interdisciplinary education perception scale in health science students

ABSTRACT This study aimed to establish the test/re-test reliability of two common interprofessional education (IPE) instruments, the Readiness for Interprofessional Learning Scale (RIPLS) and the Interdisciplinary Education Perception Scale (IEPS), and to compare results based on previous IPE experience. The RIPLS and the IEPS were distributed to 251 students within five academic health science programs at one university. Both instruments were distributed at a second session 10–14 days later. Weighted Kappa, intraclass correlation coefficients (ICC), standard error of measurement, and minimal detectable change were calculated for each instrument’s composite scores and subscales. Assessments occurred for all subjects and between students with and without previous IPE experience. Overall and between-group composite score reliability for the RIPLS and IEPS were excellent (ICC≥0.81). RIPLS subscale ICCs were variable per previous IPE experience, ranging from fair-excellent (ICC = 0.56–0.86). IEPS subscale ICCs were excellent for students with previous IPE experience (ICC≥0.76), and fair-excellent for students without previous experience (ICC = 0.64–0.84). Students with previous IPE experience had significantly higher scores within and between sessions for the RIPLS (p ≤ 0.031) but not the IEPS. Both instruments have acceptable measurement consistency; however, the RIPLS varied in repeatability compared to the IEPS. Previous IPE experience should be accounted for when distributing/interpreting the results.

Assessing Online Viewing Practices among College Students

This article focuses on media literacy education for college students. First, we conducted psychometric analyses to verify the properties of the Critical Evaluation and Analysis of Media (CEAM) scale. CEAM measures college students’ self-reported practices for critically evaluating and analyzing the credibility, audience, and technical design elements of online media, such as news, advertisement, and entertainment media. Using CEAM, our second goal was to identify trends in critical viewing practices among first-year students enrolled in college. Results of confirmatory factor analysis (CFA) and item response theory (IRT) supported a three-factor structure for the CEAM scale. Composite score reliability for all items comprising the total scale displayed strong evidence for the internal consistency of the scale with a Coefficient Alpha (α) of .91. Score reliability estimates for each subscale follow: (a) Questioning Credibility (α = .80), (b) Recognizing Audience (α = .78), and (c) Recognizing Design (α = .81). Findings from the study indicate that while first-year college students generally perceive they have adequate practices in recognizing audience in media messages and questioning the credibility of news, there is room for improvement in questioning the credibility of advertisements, suggesting that college instructors should focus more on advertising literacy.

Variability in ImPACT Baseline Scores in Adolescent Athletes

Abstract Purpose The purpose of this study was to observe the test-retest-reliability of adolescent athlete Baseline ImPACT scores over a one and two-year interval. Methods Participants were selected from an archival de-identified sports medicine ImPACT database in Florida. The sample (N=4,044) consisted of primarily male (62.2%), adolescent athletes aged (Mage= 15.45, SD=1.291). Athletes were included in the study if they completed an initial and follow-up baseline ImPACT between 2016 and 2019, and if they had no history of concussion. One-Way ANOVA was conducted to determine the relationship between overall mean composite and total symptom scores between baseline tests. Results One-Way ANOVA revealed Verbal Memory (F[2,4041])=16.231, p&lt;.001), Visual Memory (F[2,4041])=23.528, p&lt;.001), and Visual Motor (F[2,4041)=46.111, p&lt;.001) composite scores significantly differed over time. While these results yielded statistical significance over time, further descriptive statistics showed that the mean differences for these composite scores did not fall outside of the ImPACT Reliable Change Index (RCI) criteria. Conclusion Assuming no significant physical or cognitive changes between baseline administrations, ImPACT composite and total symptom scores are expected to be consistent over time. Results indicate that ImPACT composite and total symptom scores are reliable in an adolescent sample across one and two-year time periods. Previous literature regarding test-retest reliability of composite and total symptom scores on ImPACT in youth and adolescent athletes has been inconsistent, which has caused researchers and clinicians to question the clinical utility of this assessment. Future research is needed to highlight the utilization of clinical judgement and clarify the psychometric properties of this measure.

At what sample size do latent variable correlations stabilize?

We conducted a Monte-Carlo simulation within a latent variable framework by varying the following characteristics: population correlation (ρ = 0.10, 0.20, 0.30, 0.40, 0.50, 0.60, 0.70, 0.80, 0.90, and 1.00) and composite score reliability (coefficient omega: ω = 0.40, 0.50, 0.60, 0.70, 0.80, and 0.90). The sample sizes required to estimate stable measurement-error-free correlations were found to approach N = 490 for typical research scenarios (population correlation ρ = 0.20; composite score reliability ω = 0.70) and as high as N = 1000+ for data associated with lower, but still sometimes observed, reliabilities (ω = 0.40–0.50). We encourage researchers to take into consideration reliability, when evaluating the sample sizes required to produce stable measurement-error-free correlations.

Motor adaption during repeated motor control testing: Attenuated muscle activation without changes in response latencies

With repeated exposure to postural perturbations the human postural control system can adapt and create efficient strategies to counteract these perturbations. The Motor Control Test (MCT) is commonly used to elicit reactionary postural movements. Though this device has been assessed for possible learning effects and reliability of composite scores, yet no study has evaluated possible neuromuscular alterations repeated bouts might elicit. Twenty participants (age: 25 ± 4.73 years; height: 183.8 ± 8.5 cm; mass: 85.2 ± 15.6 kg) volunteered and, following familiarization, performed five full-randomized MCTs over six testing sessions. The first five sessions occurred on consecutive days, with the sixth occurring two days later. Electromyography (EMG) was recorded on right lower extremity knee flexors and extensors, and ankle plantarflexors and dorsiflexors. Response latencies and Mean and RMS muscle activity were calculated and analyzed using 1 × 5 (within days) and 1 × 6 (across days) RM ANOVA. Decreases in muscle activation of proximal musculature were noted between days and trials within days, however these changes were not maintained after the two-day retention period. No differences were detected for MCT scores. These results suggest repeated MCT exposure modifies neuromuscular responses to maintain similar reaction time through a postural control strategy shift.

Resampling-Based Inference Methods for Comparing Two Coefficients Alpha.

The two-sample problem for Cronbach's coefficient [Formula: see text], as an estimate of test or composite score reliability, has attracted little attention compared to the extensive treatment of the one-sample case. It is necessary to compare the reliability of a test for different subgroups, for different tests or the short and long forms of a test. In this paper, we study statistical procedures of comparing two coefficients [Formula: see text] and [Formula: see text]. The null hypothesis of interest is [Formula: see text], which we test against one-or two-sided alternatives. For this purpose, resampling-based permutation and bootstrap tests are proposed for two-group multivariate non-normal models under the general asymptotically distribution-free (ADF) setting. These statistical tests ensure a better control of the type-I error, in finite or very small sample sizes, when the state-of-affairs ADF large-sample test may fail to properly attain the nominal significance level. By proper choice of a studentized test statistic, the resampling tests are modified in order to be valid asymptotically even in non-exchangeable data frameworks. Moreover, extensions of this approach to other designs and reliability measures are discussed as well. Finally, the usefulness of the proposed resampling-based testing strategies is demonstrated in an extensive simulation study and illustrated by real data applications.

Validity Evidence and Scoring Guidelines for Standardized Patient Encounters and Patient Notes From a Multisite Study of Clinical Performance Examinations in Seven Medical Schools.

To examine validity evidence of local graduation competency examination scores from seven medical schools using shared cases and to provide rater training protocols and guidelines for scoring patient notes (PNs). Between May and August 2016, clinical cases were developed, shared, and administered across seven medical schools (990 students participated). Raters were calibrated using training protocols, and guidelines were developed collaboratively across sites to standardize scoring. Data included scores from standardized patient encounters for history taking, physical examination, and PNs. Descriptive statistics were used to examine scores from the different assessment components. Generalizability studies (G-studies) using variance components were conducted to estimate reliability for composite scores. Validity evidence was collected for response process (rater perception), internal structure (variance components, reliability), relations to other variables (interassessment correlations), and consequences (composite score). Student performance varied by case and task. In the PNs, justification of differential diagnosis was the most discriminating task. G-studies showed that schools accounted for less than 1% of total variance; however, for the PNs, there were differences in scores for varying cases and tasks across schools, indicating a school effect. Composite score reliability was maximized when the PN was weighted between 30% and 40%. Raters preferred using case-specific scoring guidelines with clear point-scoring systems. This multisite study presents validity evidence for PN scores based on scoring rubric and case-specific scoring guidelines that offer rigor and feedback for learners. Variability in PN scores across participating sites may signal different approaches to teaching clinical reasoning among medical schools.

Intra-Rater reliability of the Functional Movement Screen (FMS) amongst NHS Physiotherapists

Objectives: The primary aim of this study was to evaluate the intra-rater reliability of the FMS amongst a group of novice National Health Service (NHS) Physiotherapists. The secondary objective was to establish if Intra-rater reliability differed between non-specialist rotational Physiotherapists and Physiotherapists working within the musculoskeletal (MSk) setting. Design: Reliability Study. Method: Forty participants with no recent MSk injury were video recorded completing the 7 component FMS tests. Six NHS Physiotherapists with no previous experience using the FMS attended a 2 hour training programme delivered by a certified FMS practitioner. Raters then viewed and scored videos of the 40 participants completing the FMS. Results: The intra-rater reliability of FMS composite scores was Excellent (mean ICC of 0.91 (95% CI 0.81-0.96)). The non-specialist rotational Physiotherapist group demonstrated Good-Excellent intra-rater reliability (ICC 0.9; 95% CI 0.79-0.95). Specialist MSk Physiotherapists demonstrated Excellent intra-rater reliability (ICC 0.92; 95% CI 0.84-0.96). Intra-rater reliability of the 7 component tests of the FMS ranged from Poor-Excellent (KW 95% CI 0.11-0.98). Conclusion: Among novice NHS Physiotherapists, the FMS composite score demonstrated Excellent intra-rater reliability. MSk specialists were found to demonstrate a marginally superior level of intra-rater agreement compared to non-specialist rotational Physiotherapists; however this is likely to be negligible in a clinical context. Clinical specialism also appears to have little impact on the intra-rater reliability of FMS components with both groups of raters achieving a Poor-Excellent level of agreement. Keywords: Intra-rater, reliability, functional movement screen, FMS

Differential Weighting for Subcomponent Measures of Integrated Clinical Encounter Scores Based on the USMLE Step 2 CS Examination: Effects on Composite Score Reliability and Pass-Fail Decisions.

Medical schools administer locally developed graduation competency examinations (GCEs) following the structure of the United States Medical Licensing Examination Step 2 Clinical Skills that combine standardized patient (SP)-based physical examination and the patient note (PN) to create integrated clinical encounter (ICE) scores. This study examines how different subcomponent scoring weights in a locally developed GCE affect composite score reliability and pass-fail decisions for ICE scores, contributing to internal structure and consequential validity evidence. Data from two M4 cohorts (2014: n = 177; 2015: n = 182) were used. The reliability of SP encounter (history taking and physical examination), PN, and communication and interpersonal skills scores were estimated with generalizability studies. Composite score reliability was estimated for varying weight combinations. Faculty were surveyed for preferred weights on the SP encounter and PN scores. Composite scores based on Kane's method were compared with weighted mean scores. Faculty suggested weighting PNs higher (60%-70%) than the SP encounter scores (30%-40%). Statistically, composite score reliability was maximized when PN scores were weighted at 40% to 50%. Composite score reliability of ICE scores increased by up to 0.20 points when SP-history taking (SP-Hx) scores were included; excluding SP-Hx only increased composite score reliability by 0.09 points. Classification accuracy for pass-fail decisions between composite and weighted mean scores was 0.77; misclassification was < 5%. Medical schools and certification agencies should consider implications of assigning weights with respect to composite score reliability and consequences on pass-fail decisions.