Excellent Intrarater Reliability Research Articles

Intrarater and Interrater Reliability of Quantitative Ultrasound Speed of Sound by Trained Raters at the Distal Radius in Postmenopausal Women.

Quantitative ultrasound (QUS) performed at the distal radius shows potential as an effective tool for evaluating fracture risk, especially in postmenopausal women, but the reliability of this tool has not been reported in this patient population. This study investigated the within-day intrarater and interrater reliability of trained raters performing QUS measures at the distal radius of postmenopausal community-dwelling women. Eighteen women between the ages of 56 and 87years were recruited from senior centers in the Greenville area of South Carolina and consented to participate. A QUS bone sonometer was used in this study; the trained raters performed 3 repeated speed of sound (SOS) measures at the nondominant radius of each participant on the same day. Intraclass correlation coefficients (ICCs) were calculated (model 3,1 for intrarater and 2,3 for interrater reliability) to determine the reliability within and between raters. In addition, we calculated the standard error of measurement (SEM) and the minimal detectable change at 95% confidence interval (MDC95). The within-day intrarater reliability was excellent for both physical therapist raters (ICC(3,1) ranging from 0.93 to 0.91). The SEM and MDC for the within-day intrarater reliability measures were small for both raters (SEM=36.79-38.75 or 0.92%-0.97% of the pooled mean; MDC95=101.99-107.40). The interrater reliability for SOS measurements using radial QUS was good with an ICC(2,3) of 0.77 (95% CI, 0.36-0.92) for 3 averaged measures. The SEM and MDC were larger for the interrater reliability measures (SEM=62.79 or 1.57% of the pooled mean; MDC95=174.05). Quantitative ultrasound SOS measurement at the distal radius showed excellent within-day intrarater and good interrater (when 3 measures were averaged) reliability in community-dwelling postmenopausal women. The SEM and MDC95 were also low for interrater reliability when a mean of 3 trials was used. This study is the first to demonstrate that this particular QUS device at the radius may be a reliable fracture risk screening tool for postmenopausal women.

A Modification to the Assessment of Problem-Solving in Play to Enhance Its Utility: Evaluation of Validity, Responsiveness, and Reliability.

This study investigated the psychometric properties of a 4-second interval scoring modification of the Assessment of Problem-Solving in Play (ie, Assessment of Problem-Solving in Play 4-second interval scoring [APSP-4]). A total of 95 children (3-48 months) with or at high risk for neuromotor delay were assessed with the APSP-4 and Bayley Scales of Infant and Toddler Development-III (Bayley). APSP-4 and Bayley cognitive raw scores were compared by age (construct validity) and over time (responsiveness). Twenty percent of videos were scored twice for intra- and interrater reliability. The APSP-4 demonstrated excellent interrater (intraclass correlations [ICCs]≥0.96) and intrarater (ICCs≥0.99) reliability and performed similar to Bayley cognitive raw scores at different ages (rs>0.5), and over time (r=0.81). The motor delay had similar effects on APSP-4 and Bayley scores. Results support the validity and reliability evidence of the APSP-4 for use over time in tracking problem-solving skills in young children aged 3 to 48months. Future research investigating clinical implementation of the APSP-4 is needed.

Development of the humeral head offset index for control of humeral torsion

BackgroundControl of humeral torsion can present a challenge, especially intraoperatively during closed reduction and fixation of humeral shaft fractures or 2-part surgical neck fractures of the proximal humerus. The objective of this study is to develop and validate an indirect method for the assessment of humeral torsion using an index that is linearly correlated with rotational arm position and can be derived from only a single plain radiographic image of the proximal humerus.MethodsThe Humeral Head Offset Index (HHOI) is calculated as the ratio of the medial and lateral offset of the humeral head measured from the outer cortices of the shaft on a plain radiographic or fluoroscopic image. The relationship of HHOI with humeral torsion was first verified on a sawbone model with radiopaque characteristics under fluoroscopic control. Different degrees of retroversion were simulated through manual rotation of the humerus with a digital protractor in 5° increments until 40° internally rotated and then in 5° increments until 40° externally rotated from the neutral position. The same procedure was subsequently performed digitally on Digitally Reconstructed Radiographs (DRRs) from computed tomography (CT) dataset of the sawbone. Next, the HHOI index was applied to eight randomly selected patients with total humerus CT using the same method. Spearman’s rho was calculated for the bivariate analysis of correlation between the simulated degree of retroversion and the HHOI. Strength of correlation was classified according to Koo and Li. Interrater and intrarater reliability of three blinded observers with repetition of measurement after three months were analyzed by assessing the intraclass correlation coefficient (ICC).ResultsBoth in the sawbone model and in DRRs, we demonstrated a high to very high significant linear correlation between simulated retroversion and the HHOI. ICC values demonstrated excellent interrater reliability and excellent intrarater reliability for measurement of the HHOI.ConclusionsThe HHOI is a new, simple, reliable index that has a linear relationship to the rotation of the humerus and can therefore allow an indirect control of humeral torsion in comparison to the contralateral side.

Comparison of Inter-Rater and Intra-Rater Reliability of Raters with Different Levels of Experience When Using Landing Error Scoring System (LESS) in Field-Based Screening of Professional Football Players.

It is essential for physical sports therapists to use reliable field-based tests to identify potential injury risk factors in athletes. The purpose of this study was to compare the inter- and intra-rater reliability of experienced and novice raters during use of the Landing Error Scoring System (LESS) in a field-based examination of professional football athletes. Thirty-seven male football athletes underwent pre-season LESS assessment. Two raters independently evaluated the recorded landing techniques at two separate intervals, two months apart, following the LESS standard protocol. Inter-and intra-rater values were calculated for the LESS total scores and individual scoring items. The overall LESS scores had excellent intra-rater reliability values for both the experienced (interclass correlation coefficient (ICC) = 0.95, 95% CI, 0.89-0.97; p < 0.001) and novice rater (ICC = 0.95, 95% CI, 0.90-0.97; p < 0.001), and very good to excellent inter-rater values for the first (ICC = 0.90, 95% CI, 0.77-0.95; p < 0.001) and second (ICC = 0.86, 95% CI, 0.71-0.93; p < 0.001) evaluation. Most of the individual scoring items ranged from moderate to perfect agreement. In conclusion, sports physical therapists, regardless of experience, can reliably use the LESS's total score, through video analysis of the regime. Individual scoring items can inform clinicians about impairments in the landing mechanism but data should be interpreted cautiously.

Reliability testing of sonographic techniques in detecting lateral displacement of patellar tendon

ObjectivesPatellofemoral pain syndrome (PFPS), a common knee condition, is characterized as presence of peripatellar pain during activities especially during loading of the lower extremity. Gonzalez-Suarez et al. (2021) developed a sonographic technique that assessed patellar tendon displacement within the trochlear groove. This study aims to determine the reliability of this sonographic technique. MethodsIntra-rater and inter-rater reliability study of three sonologist on the lateral displacement of the patellar tendon using ultrasound. Patellar tendon displacement was assessed using the three novel sonographic measurement techniques: 1) region of trochlear groove where the patellar tendon midpoint is located, 2) distance between deepest portion of the trochlear groove and patellar tendon midpoint and 3) percentage of lateral displacement of the patellar tendon. The measurements were done with the patients in supine and standing. ResultsTen participants with a mean age of 27.6 ± 1.56 years were included in the study. Pilot studies were done to improve the sonographic technique. Intraclass correlation coefficients yielded good to excellent inter-rater and excellent intra-rater reliability (>0.75) in the supine and standing positions except the inter-rater reliability of the region of the trochlear groove where patellar tendon midpoint is in supine position. ConclusionThe sonographic techniques for measurement of lateral patellar tendon displacement showed good to excellent intra-rater and inter-rater reliability between sonologists of different levels of expertise using a well-defined scanning protocol.

Reliability of Pressure Pain Threshold (PPT) and Conditioned Pain Modulation (CPM) in Participants with and without Chronic Shoulder Pain.

The objectives of this study were to estimate the intra-rater and inter-rater reliability of the Pressure Pain Threshold (PPT) and Conditioned Pain Modulation (CPM) in healthy participants and patients with chronic shoulder pain. Additionally, the Standard Error of Measurement (SEM) and Smallest Detectable Change (SDC) were calculated. Thirty-one healthy volunteers and twenty patients with chronic shoulder pain were assessed using the PPT and CPM by two raters, with a 24 h interval between sessions. Excellent intra-rater reliability was demonstrated for PPT, with similar SEM and SDC when assessed by the same rater. The inter-rater reliability for PPTs in patients was moderate to good (ICC = 0.59-0.89) with higher SEM (73.83-121.98 kPa) and SDC (61.58-97.59) values than the asymptomatic group (ICC = 0.92-0.96, SEM = 49.61-103.12 kPa, SDC = 42.01-56.30) respectively. CPM's intra-rater reliability was good (ICC = 0.82) in the patients and moderate (ICC = 0.67) in the asymptomatic group, while inter-rater reliability was low for the asymptomatic group (ICC = 0.37) and extremely low (ICC = 0.074) for the patients, with comparable SEM and SDC outcomes in both groups. PPT and CPM measurements are highly reliable when conducted by the same rater on the same day. Patients had lower inter-rater PPT reliability but better intra-rater CPM reliability. Clinicians need to be mindful of potential variability when interpreting these test results.

Diagnosing Sarcopenia with AI-Aided Ultrasound (DINOSAUR)-A Pilot Study.

Background: Sarcopenia has been recognized as a determining factor in surgical outcomes and is associated with an increased risk of postoperative complications and readmission. Diagnosis is currently based on clinical guidelines, which includes assessment of skeletal muscle mass but not quality. Ultrasound has been proposed as a useful point-of-care diagnostic tool to assess muscle quality, but no validated cut-offs for sarcopenia have been reported. Using novel automated artificial intelligence (AI) software to interpret ultrasound images may assist in mitigating the operator-dependent nature of the modality. Our study aims to evaluate the fidelity of AI-aided ultrasound as a reliable and reproducible modality to assess muscle quality and diagnose sarcopenia in surgical patients. Methods: Thirty-six adult participants from an outpatient clinic were recruited for this prospective cohort study. Sarcopenia was diagnosed according to Asian Working Group for Sarcopenia (AWGS) 2019 guidelines. Ultrasonography of the rectus femoris muscle was performed, and images were analyzed by an AI software (MuscleSound® (Version 5.69.0)) to derive muscle parameters including intramuscular adipose tissue (IMAT) as a proxy of muscle quality. A receiver operative characteristic (ROC) curve was used to assess the predictive capability of IMAT and its derivatives, with area under the curve (AUC) as a measure of overall diagnostic accuracy. To evaluate consistency between ultrasound users of different experience, intra- and inter-rater reliability of muscle ultrasound parameters was analyzed in a separate cohort using intraclass correlation coefficients (ICC) and Bland-Altman plots. Results: The median age was 69.5 years (range: 26-87), and the prevalence of sarcopenia in the cohort was 30.6%. The ROC curve plotted with IMAT index (IMAT% divided by muscle area) yielded an AUC of 0.727 (95% CI: 0.551-0.904). An optimal cut-off point of 4.827%/cm2 for IMAT index was determined with a Youden's Index of 0.498. We also demonstrated that IMAT index has excellent intra-rater reliability (ICC = 0.938, CI: 0.905-0.961) and good inter-rater reliability (ICC = 0.776, CI: 0.627-0.866). In Bland-Altman plots, the limits of agreement were from -1.489 to 1.566 and -2.107 to 4.562, respectively. Discussion: IMAT index obtained via ultrasound has the potential to act as a point-of-care evaluation for sarcopenia screening and diagnosis, with good intra- and inter-rater reliability. The proposed IMAT index cut-off maximizes sensitivity for case finding, supporting its use as an easily implementable point-of-care test in the community for sarcopenia screening. Further research incorporating other ultrasound parameters of muscle quality may provide the basis for a more robust diagnostic tool to help predict surgical risk and outcomes.

Assessment of Reliability, Agreement, and Accuracy of Masseter Muscle Ultrasound Thickness Measurement Using a New Standardized Protocol.

There is no validated method of assessing masseter muscle thickness (MMT) by ultrasound imaging (US). However, this is important to ensure study and measurement quality of MMT by US in future studies, as MMT differs depending on the examined area. Thus, this study's aim was to present a new standardized method for assessing the MMT by US and to evaluate the reliability, consistency, and accuracy of its measurements. We also compared the results of MMT measurements obtained by US and computer tomography (CT). The study included nine healthy adults. The US and CT scans were collected in a supine rest position with the mandible in relaxed position. US measurements were determined according to a new standardized protocol (with precise probe location). The MMT measured by CT and US over a seven-day interval showed excellent intra-rater reliability. The mean MMT measured by CT was 12.1 mm (1.74) on the right side and 11.9 mm (1.61) on the left side. The mean MMT measured by US was 12.7 mm (2.00) on the right side and 11.5 mm (1.37) on the left side. The mean percent error in MMT measurement between CT and US was below 6%. A strong linear relationship was found between the CT and US measurements of the MMT on both body sides (p < 0.001, r ≥ 0.93). The proposed method of MMT measurement using US demonstrated excellent reliability, yielding results similar to those obtained from CT images. We recommend the use of this standardization protocol in further studies where precise assessment of MMT by US is expected.

Reliability and accuracy of scoliotic parameters on using a wireless handheld 3D ultrasound for children with adolescent idiopathic scoliosis: a pilot study.

To report the accuracy and reliability of Cobb angle (CA), axial vertebral rotation (AVR), kyphotic and lordotic angles (KA and LA) measurements on using a new 3D ultrasound (US) system. Forty participants (34F, 6M, aged 14.0 ± 2.3 years) were recruited. The first 20 participants were scanned by the validated US system and the new US system. The other 20 participants were scanned with the new US system only. Two raters (R1 and R2) performed the measurements: R1 has 10 years of experience in radiology but is new in ultrasound scoliosis, while R2 has 30 years of scoliosis experience. All US images were measured twice by R1, and once by R2. Forty posteroanterior and 30 lateral standing radiographs were obtained and measured once by R1. Statistical analysis consisted of mean absolute difference (MAD), intraclass correlation coefficient (ICC(2,1)), and Bland-Altman plots. R1 showed excellent intra-rater and inter-rater reliability for US measurements with ICCs(2,1) ≥ 0.91. The inter-method reliability was good between the two US systems for all parameters with ICCs(2,1) ≥ 0.85 and maximum MAD of 3.4°. The new US showed good reliability and accuracy compared to radiographs for CA, AVR and KA with ICCs(2,1) ≥ 0.81 and maximum MAD of 5.8°, but poor results for LA with ICCs(2,1) of 0.27-0.35 and MADs of 14.0°-15.4°. The new 3D US system showed good reliability and accuracy for CA, AVR and KA measurements, but a large measurement discrepancy on LA. A new measurement method for US LA may need to investigate.

Reliability of Ultrasound Assessment of Hamstring Morphology, Quality, and Stiffness Among Healthy Adults and Athletes: A Systematic Review.

The incidence and recurrence rate of hamstring strain injuries remain persistently high, with recurrent injuries leading to increased time lost during play and extended recovery periods compared with initial injury. Ultrasound imaging assesses important factors such as hamstring fascicle length (FL), pennation angle (PA), cross-sectional area (CSA), muscle thickness (MT), echo intensity (EI), and shear wave elastography (SWE), all impacting athletic performance. However, its reliability must be established before employing any measurement tool in research or clinical settings. To determine the reliability and measurement error of ultrasound for assessing hamstring FL, PA, CSA, MT, EI, and SWE among healthy adults and athletes; to synthesize the information regarding the operationalization of ultrasound. A systematic literature search was done from January 1990 to February 5, 2023, to identify reliability and validity studies of hamstring ultrasound assessment published in peer-reviewed journals with identifiable methodology of outcome measures. Intraclass correlation coefficient measurement of 14 included studies reported moderate to excellent intrarater, interrater, and test-retest reliabilities of FL, PA, and MT regardless of the site of muscle testing, probe size, and setting, state of muscle, and use of different techniques in the extrapolation of FL. Good to excellent test-retest reliability rates for all hamstring anatomic CSA along midmuscle and different percentages of thigh length using panoramic imaging. Good intrarater reliability of EI regardless of gender and orientation of the probe but with excellent intrarater reliability in transverse scan using maximum region of interest. Good intrarater, interrater, and interday repeatability on SWE with the muscle in a stretched position. Evidence from studies with a predominantly low risk of bias shows that ultrasound is a reliable tool to measure hamstring FL, PA, CSA, MT, EI, and SWE in healthy adults and athletes under various experimental conditions.

Investigation of the reliability and validity of the Turkish version of the Comprehensive Coordination Scale in patients with Parkinson's disease

ObjectiveThe Comprehensive Coordination Scale (CCS) is a valid, reliable scale for evaluating motor coordination, described as the ability to produce context-dependent movements of multiple body parts in both spatial and temporal domains in patients with chronic stroke. The aim of this study was to develop a Turkish version of the CCS that can be used to evaluate coordination in Parkinson's disease patients and to test its validity and reliability. MethodsThirty-four patients with Parkinson’s disease (18 females, 16 males) with a mean age of 67.55± 10.93 years were included. The 9-Hole Peg Test (9 HPT), Timed Up and Go Test (TUG), Berg Balance Scale (BBS), Functional Reach Test (FRT), and Modified Hoehn and Yahr (H&Y) were administered to the patients. ResultsIn the correlation analysis for convergent validity, the CCS-TR showed a moderate negative correlation with both the 9 HPT and the TUG test (r=-0.684 and −0.641, respectively). There were also moderate positive correlations observed between the CCS-TR and the FRT and BBS (r=0.607 and 0.657, respectively). The total CCS-TR had excellent intrarater reliability (ICC=0.99; 95 % CI: 0.98–0.99) and interrater reliability (ICC=0.90; 95 % CI: 0.81–0.95). The CCS-TR subscales had excellent intrarater reliability (ICC: range 0.98 from 0.99). For intrarater agreement, the ICC was 0.99, and for interrater agreement, the ICC was 0.90 (excellent agreement). ConclusionThe results of the present study indicate that the Turkish version of the CCS is a reliable and valid coordination scale that can be used in patients with Parkinson’s disease.

Intra-rater and inter-rater reliability of robotic arthrometer DYNEELAX®.

The purpose of our study was to test the intra-rater and inter-rater reliability of the DYNEELAX® arthrometer in healthy subjects. Since rotational laxity will be measured for the first time in humans, indications for normative values will also be presented. Knee anterior laxity and rotational laxity using a DYNEELAX® arthrometer were assessed in 73 subjects (39 females and 34 males). An intraclass correlation coefficient was calculated to evaluate the intra-rater and inter-rater reliability of the DYNEELAX®. An intraclass correlation coefficient for intra-rater reliability ranges from 0.631 (95% confidence interval; [CI]: 0.47-0.75) to 0.699 (95% CI: 0.56-0.80) and from 0.916 (95% CI: 0.87-0.95) to 0.94 (95% CI: 0.91-0.96) for anterior knee laxity and rotational knee laxity, respectively. An intraclass correlation coefficient for inter-rater reliability ranges from 0.578 (95% CI: 0.40-0.71) to 0.646 (95% CI: 0.44-0.78) and from 0.822 (95% CI: 0.57-0.91) to 0.933 (95% CI: 0.89-0.96) for anterior knee laxity and rotational knee laxity, respectively. The DYNEELAX® arthrometer has good to excellent intra-rater and inter-rater reliability for rotational knee laxity and moderate intra-rater reliability for anterior knee laxity in healthy subjects. Future studies should investigate the clinical significance of anterior and rotational laxity measured with the DYNEELAX® arthrometer in patients with knee pathology, as both laxities are critical for assessing the integrity of the intra-articular structures of the knee in clinical practice. Level IV.

Effect of measurement procedure errors on assessing lung fluid via remote dielectric sensing system

The study assessed the impact of procedural errors on the remote dielectric sensing system (ReDS), a non-invasive lung fluid assessment technology, in an Asian cohort. Healthy volunteers underwent ReDS measurements following manufacturer’s instructions, with two consecutive measurements one minute apart. A subset of 20 participants had modified procedure settings. Reliability was measured using intraclass correlation coefficient (ICC). The study included 86 healthy volunteers, and all ReDS measurements fell within the recommended normal range. The intra-rater reliability of ReDS measurements was excellent, with an ICC of 0.968. Among the subset of 20 subjects, deviations in height and weight did not significantly affect ReDS values. However, deviations in chest size by ± 3 cm had a noticeable impact on ReDS measures, and incorrect station selection led to fluctuations in ReDS readings. In conclusion, the ReDS system demonstrated excellent intra-rater reliability and applicability in an Asian cohort. Procedural errors, such as chest size measurement and station selection, significantly influenced ReDS measurements. Adherence to standardized operating procedures is crucial to ensure accurate and consistent results. These findings highlight the importance of adherence to manufacturer instructions when utilizing ReDS for lung fluid assessment, thereby enhancing its reliability and clinical applicability.

Intrarater and interrater reliability of digital calipers in assessing Achilles tendon thickness in patients with knee osteoarthritis.

This study aimed to evaluate the intrarater and interrater reliability of measuring Achilles tendon (AT) thickness using a digital caliper in patients with knee osteoarthritis. A cross-sectional survey was conducted at the Physiotherapy Department of Rabia Moon Hospital, involving the recruitment of 61 patients with knee osteoarthritis. Measurements were taken in millimeters at a 90-degree angle, approximately 5cm from the attachment to the calcaneus, precisely where the ankle joint joins the medial malleolus. Two physical therapists conducted two testing sessions, separated by 7days, to assess both the intrarater and interrater reliability of the digital caliper. During the second session, two raters simultaneously assessed the patients' responses on the digital caliper. The study analyzed reliability indices, including the Intraclass Correlation Coefficient (ICC) and Bland-Altman plot. The study found high intrarater reliability for the digital caliper, with an ICC of 0.96 (95% confidence interval: 0.22, 0.99). For interrater reliability, the ICC was 0.98 (95% CI: 0.96, 0.98) in patients with knee OA. Additionally, both interrater and intrarater agreement for measuring AT thickness with the digital caliper fell within acceptable limits on 95% of occasions, as indicated by the Limits of Agreement values: 0.32 to -0.53mm for interrater agreement and -0.35 to -0.04mm for intrarater agreement. Digital Calipers have been found to provide excellent intrarater and interrater reliability when used to measure AT thickness in patients with knee osteoarthritis (OA).

Isometric muscle torque in Chilean children and adolescents evaluated by manual maintenance dynamometry: a reliability study

to determine the reliability of hand-held dynamometry to obtain the maximal isometric torque of upper and lower limb muscle groups in Chilean children and adolescents. Crosssectional study. Seventy-two participants aged between 7 and 15 years were selected from a school in Talca. Maximal isometric torque was recorded in 15 muscle groups of upper and lower limbs through hand-held dynamometry. Intra- and inter-rater evaluation was used, applying the intraclass correlation coefficient (ICC) to determine the reliability of the tests and Bland-Altman plots to evaluate concordance. The results demonstrated good to excellent inter-rater reliability (ICC = 0.850.98) and intra-rater reliability (ICC = 0.87-0.98). Only two groups, hip extensors and abductors, showed good inter-rater reliability (ICC = 0.85 and ICC = 0.88, respectively); and one group, the ankle dorsiflexors, showed good intra-rater reliability (ICC = 0.87). 100% of the tests presented at least 95.8% inter- and intra-rater agreement on the Bland-Altman plots. The evaluation of isometric muscle torque using hand-held dynamometry is a reliable procedure for use in different growth periods.

Development, validation, and psychometric analysis of Foot and Ankle Flexibility Index (FAFI)

AimTo develop a new tool for identifying joint hypermobility of the paediatric foot and ankle, based on a dichotomous scoring system utilising the Lower Limb Assessment Score (LLAS), to separate the foot and ankle items. Material and methodsA total of 205 children, aged between 5 and 10 years, participated in a cross-sectional study. The new tool Foot and Ankle Flexibility Index (FAFI) was predicated upon the last 7 items of LLAS, which are specific to the foot and ankle. The internal consistency was measured with Cronbach's test. Kappa statistics with 95% CI were calculated to verify the level of inter-rater and intra-rater agreement for the FAFI. ResultsCronbach's alpha returned 0.82. The correlations between items returned a mean of 0.59 (range: 0.43–0.74). The discrimination score on the ROC curve (4 points) showed that the model can be used to identify children with joint hypermobility of the foot and ankle. Inter-rater reliability was largely good (ICC = 0.89). Excellent intra-rater reliability was found (ICC = 0.96) ConclusionsThis study identified high reliability between evaluators, and high sensitivity and specificity, for a new reliable and valid tool for the identification of foot and ankle joint hypermobility.

Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability

Advancements in generative AI, such as large language models (LLMs), may serve as a potential solution to the burdensome task of essay grading often faced by language education teachers. Yet, the validity and reliability of leveraging LLMs for automatic essay scoring (AES) in language education is not well understood. To address this, we evaluated the cross-sectional and longitudinal validity and reliability of four prominent LLMs, Google's PaLM 2, Anthropic's Claude 2, and OpenAI's GPT-3.5 and GPT-4, for the AES of English language learners' writing. 119 essays taken from an English language placement test were assessed twice by each LLM, on two separate occasions, as well as by a pair of human raters. GPT-4 performed the best, demonstrating excellent intrarater reliability and good validity. All models, with the exception of GPT-3.5, improved over time in their intrarater reliability. The interrater reliability of GPT-3.5 and GPT-4, however, decreased slightly over time. These findings indicate that some models perform better than others in AES and that all models are subject to fluctuations in their performance. We discuss potential reasons for such variability, and offer suggestions for prospective avenues of research.

Reproducibility of freehand vs. foam cast as well as the intrarater reliability of foam cast ultrasound scans assessing the muscle architecture and tissue organization of the gastrocnemius medialis and vastus lateralis muscles.

This study compares the reproducibility of freehand (FH) vs. foam cast (FC) scans and investigates the intrarater reliability of the ultrasound FC muscle architecture and tissue organization measurements of the gastrocnemius medialis (GM) and vastus lateralis (VL) muscles with fixed and repositioning FC scans. Thirteen young adults (22 ± 3 years) underwent repeated sagittal B-mode ultrasound measurements of GM and VL. FH, FC, and repositioned FC scans were conducted. Muscle architecture measurements included muscle thickness (MT), pennation angle (PA), and fascicle length (FL). Spatial frequency analysis assessed muscle tissue organization. MT decreased from 2.1 to 1.8 cm in GM and from 2.4 to 2.2 cm in VL with the FC compared with the FH. Reproducibility between the FH and the FC showed poor to good intraclass correlation coefficients (ICCs) for MT (0.46-0.77) and PA (0.09-0.86) as well as poor to moderate ICCs for FL (0.41), with very low to moderate test-retest variability (TRV) (4%-18%). Tissue organization indicated low to good ICCs (0.21-0.80) with low to moderate TRV (4%-19.5%). The re-scanning results of fixed FC indicated excellent ICCs for MT (0.95-0.996), good for PA (0.77-0.90), and moderate for FL (0.73-0.76), with low TRV (5%-10%) for both muscles. Tissue organization displayed moderate to good ICCs (0.61-0.87) with very low to low TRV (4%-9%). For repositioned FC scans in GM and VL, MT showed good to excellent ICCs (0.86-0.98) with very low to low TRV (2%-8%). PA and FL demonstrated moderate to good ICCs (0.57-0.75), with very low to moderate TRV (2%-13%). Tissue organization revealed ICCs ranging from poor to good (0.13-0.87) for both muscles, with low to moderate TRV (5%-18%). The FC systematically reduced MT by 2-3 mm. Furthermore, reproducibility revealed low ICCs and high data variability for several muscle architecture and tissue organization parameters. Thus, switching methods within a single study is not recommended. Nevertheless, FC ultrasound scans demonstrated excellent intrarater reliability for assessing MT. In the case of fixed FC scans particularly, moderate to excellent ICCs were observed for all muscle architecture and tissue organization parameters, accompanied by very low to low variability. Therefore, FC scans are recommended for investigating acute effects on muscle architecture and tissue organization when the FC remains on the leg throughout the period of measurements.

Validity and reliability of the modified four square step test in total HIP arthroplasty

IntroductionThe Modified Four Square Step Test (mFSST) evaluates dynamic stabilization by using tape instead of canes to prevent people from tripping over obstacles and losing their balance. There was no study investigating reliability and validity in primary total hip arthroplasty (THA). This study aimed to evaluate the test-retest reliability, concurrent validity, and minimum detectable change in the mFSST for individuals with primary THA. MethodThis study was cross-sectional in design. A total of 29 individuals aged 56.86 ± 10.14 years participated. The study used the test-retest method to determine the reliability of the mFSST and concurrent validity. The Timed Up and Go Test (TUGT) and Berg Balance Scale (BBS) were applied for concurrent validity. The minimum detectable change (MDC95) and the standard error measurement (SEM) were calculated. ResultsThe mFSST showed excellent intra-rater reliability (ICC = 0.925). The SEM and MDC95 of the mFSST were 1.56 and 4.32. The test had a strong correlation with TUGT (r = 0.881) and BBS (r = −0.718) (p < 0.001). ConclusionThe mFSST is a valid and reliable assessment tool that can be used for individuals with primary THA.

DYNEELAX Robotic Arthrometer Reliability and Feasibility on Healthy and Anterior Cruciate Ligament Injured/Reconstructed Persons.

Anterior cruciate ligament (ACL) injuries are commonly assessed using clinical examination and magnetic resonance imaging, but these methods have limitations in reproducibility and quantification. Instrumented laxity measurements using devices, like the DYNEELAX®, offer an alternative approach. However, to date, there is no human data on the DYNEELAX® and the reliability of these devices remains a subject of debate, and there is no consensus on appropriate knee tightening levels for testing. We hypothesized that the DYNEELAX®, with standardized knee tightening, would provide reliable measurements of knee laxity in adult volunteers. This prospective cohort study involved 48 pain-free adult volunteers. Laxity measurements were taken using a robotic-type motorized instrument (DYNEELAX®) on two separate occasions, at least 1 h and no more than 8 h apart, with knee tightening forces of 90 N ± 5 N. Metrics of anterior tibial translation and internal/external tibial axial rotations were recorded. The device displayed excellent intrarater reliability for all the metrics, with intraclass correlation coefficients ranging from 0.91 to 0.96. Anterior translation exhibited the highest reliability (intraclass correlation coefficient = 0.96), with a minimum detectable change of 0.83 mm. DYNEELAX® is reliable in measuring knee laxity in adult volunteers when using standardized stabilizing knee tightening forces of 90 ± 5 N. The most sensitive measurement parameters (in terms of minimum detectable change as a proportion of the observed range) were anterior translation (in mm) at 150 N and secondary compliance.