Inter-rater Intraclass Correlation Research Articles

Reproducibility of Three-Dimensional Volumetric Measurement of Periocular Tumor Models.

In this study, we measured the volume of customized tumor models in the periocular area using three-dimensional (3D) stereophotogrammetry and evaluated the reproducibility of these measurements. Five tumor models of different colors and sizes were placed in different periocular positions, and 3D facial images were obtained from 68 healthy adult volunteers. Subsequently, the volumes of the tumor models were measured, and the intra- and interrater reproducibility was assessed. The gray 6mm model revealed the highest reliable measurements in both Caucasians (intra- and interrater intraclass correlation coefficients of 0.981 and 0.899, mean absolute difference of 1.446 and 3.327mm3, relative error measurement of 3.497% and 8.120%, technical error of measurement of 1.450 and 3.105mm3, and relative technical error of measurement of 3.506% and 7.580%) and Asians (0.968 and 0.844, 1.974 and 4.067mm3, 4.772% and 9.526%, 2.100 and 4.302mm3, and 5.076% and 10.076%, respectively). The highest reliability of measurements in the lateral upper eyelid (0.88 and 0.95, 4.042 and 3.626mm3, 9.730% and 9.020%, 5.714 and 3.358mm3, and 9.730% and 8.350%, respectively) and medial upper eyelid (0.81 and 0.89, 4.313 and 4.226mm3, 9.730% and 9.020%, 6.098 and 4.069mm3, and 9.730% and 8.350%, respectively) with eyes closed was evident in Caucasians, while the same trend (0.841 and 0.815, 2.828 and 3.757mm3, 9.860% and 9.840%, 4.052 and 4.308mm3, and 9.860% and 9.740%, respectively) was observed in Asians in the medial canthus with eyes closed. This study confirms, for the first time, the high reliability of periocular tumor volume measurements using 3D stereophotogrammetry, suggesting its feasibility for eyelid tumor measurement. Further trials are required to investigate its clinical use for documentation and follow-up of different eyelid tumors.

Reliability of point-of-care ultrasound for measuring quadriceps femoris muscle thickness.

Point-of-care ultrasound can be used to assess muscle thickness. However, its reliability has not been fully evaluated. This study aimed to assess the intrarater and inter-rater reliability of point-of-care ultrasound for the estimation of quadriceps and rectus femoris thickness in patients from a rehabilitation setting. This is a cross-sectional study. This study was conducted at the Department of Physical Medicine and Rehabilitation of a tertiary care hospital. Twenty-nine inpatients consecutively selected after admission. Four observers, two trained and two untrained, used point-of-care ultrasound to measure quadriceps femoris and rectus femoris thickness. Each observer performed two measurements followed by a second set of two measurements three hours later. Intraclass correlation coefficients (ICC) were then calculated. Both intrarater and inter-rater ICC were higher than 0.888 for both quadriceps and rectus femoris measurements. Reliability was highest when ICC were calculated based on the average of two measurements, with the intrarater ICC being of 0.956 (95% CI: 0.937-0.970) for rectus femoris and of 0.966 (95% CI: 0.951-0.976) for quadriceps femoris; and with the inter-rater ICC being of 0.919 (95% CI: 0.863-0.957) for rectus femoris and 0.945 (95% CI: 0.907- 0.971) for quadriceps femoris. Trained and untrained observers did not have significantly different ICC values. These results suggest that point-of-care ultrasound is a reliable option to measure muscle thickness of knee extensors by the same or different observers. Measuring knee extensors thickness may aid to adequately modulate treatment choices in patients with disability. This study suggests that quadriceps and rectus femoris muscle thickness measured after a short training course, by either an experienced or inexperienced clinician, presents high reliability. Reliability can be increased if the average of two measurements is used. Besides being inexpensive and portable, point-of-care ultrasound is a reliable tool for measuring knee extensors' thickness, rendering it potentially adequate to be used in clinical practice.

Point-of-care application of diaphragmatic ultrasonography in the emergency department for the prediction of development of respiratory failure in community-acquired pneumonia: A pilot study

BackgroundEarly recognition of patients with community-acquired pneumonia (CAP) at risk of poor outcomes is crucial. However, there is no effective assessment tool for predicting the development of respiratory failure in patients with CAP. Diaphragmatic ultrasonography (DUS) is a novel technique developed for evaluating diaphragmatic function via measurements of the diaphragm thickening fraction (DTF) and diaphragm excursion (DE). This study evaluated the accuracy of DUS in predicting the development of respiratory failure in patients with CAP, as well as the feasibility of its use in the emergency department (ED) setting.Materials and methodsThis was a single-center prospective cohort study. We invited all patients with ED aged ≥ 20 years who were diagnosed with CAP of pneumonia severity index (PSI) SIe diagnosed with CAP of pneumonia severe with respiratory failure or septic shock were excluded. Two emergency physicians performed DUS to obtain DTF and DE measurements. Data were collected to calculate PSI, CURB-65 score, and Infectious Diseases Society of America/American Thoracic Society severity criteria. Study endpoints were taken at the development of respiratory failure or 30 days post-ED presentation. Continuous variables were analyzed using T-tests, while categorical variables were analyzed using chi-square tests. Further logistic regression and receiver operating characteristic curve analyses were performed to examine the ability to predict the development of respiratory failure. Intra- and inter-rater reliability was examined with intraclass correlation coefficients (ICCs).ResultsIn this study, 13 of 50 patients with CAP enrolled developed respiratory failure. DTF was found to be an independent predictor (OR: 0.939, p = 0.0416). At the optimal cut-off point of 23.95%, DTF had 69.23% of sensitivity, 83.78% of specificity, 88.57% of negative predictive value, and 80% of accuracy. Intra- and inter-rater analysis demonstrated good consistency (intra-rater ICC 0.817, 0.789; inter-rater ICC 0.774, 0.781).ConclusionDUS assessment of DTF may reliably predict the development of respiratory failure in patients with CAP presenting to the ED. Patients with DTF > 23.95% may be considered for outpatient management.

Reliability and validity of proxy-SSPedi and mini-SSPedi in pediatric patients 2-7 years receiving cancer treatments

BackgroundSymptom Screening in Pediatrics Tool (SSPedi) was developed for symptom screening by children 8-18 years. Objectives were to evaluate the reliability and validity of proxy-SSPedi and self-report mini-SSPedi for younger children.MethodsThis multi-center study enrolled guardians of children 2-7 years receiving cancer treatments (proxy-SSPedi) and their children 4-7 years (mini-SSPedi). The two populations were: (1) More symptomatic group where children were receiving active cancer treatment and were in hospital or clinic for four consecutive days; and (2) Less symptomatic group where children were receiving maintenance therapy for acute lymphoblastic leukemia or had completed cancer therapy. Proxy-SSPedi or mini-SSPedi were completed with measures of mucositis, nausea, pain, quality of life and overall symptoms. Respondents in the more symptomatic group repeated proxy-SSPedi/mini-SSPedi and a global symptom change scale 3 days later.ResultsThere were 402 guardians and 326 children included in the analysis. Test re-test reliability of proxy-SSPedi showed intraclass correlation coefficient (ICC) 0.83 (95% confidence interval (CI) 0.72-0.90). Mean difference in proxy-SSPedi between more and less symptomatic groups was 9.7 (95% CI 8.3-11.1). Proxy-SSPedi was responsive to change and hypothesized relationships between measures were observed. With a priori threshold ≥0.6, inter-rater ICC among all dyads and those 6-7 years were 0.54 (95% CI 0.45-0.62) and 0.62 (95% CI 0.50-0.71) respectively. Among participating children, other hypothesized reliability and validity thresholds were generally met.ConclusionsProxy-SSPedi is reliable, valid and responsive in children 2-7 years old receiving cancer treatments. Mini-SSPedi can be used for children 6-7 years of age.

Diagnosis and Treatment of Children with a Radiological Fat Pad Sign without Visible Elbow Fracture Vary Widely: An International Online Survey and Development of an Objective Definition.

Children often present at the emergency department with a suspected elbow fracture. Sometimes, the only radiological finding is a ‘fat pad sign’ (FPS) as a result of hydrops or haemarthros. This sign could either be the result of a fracture, or be due to an intra-articular haematoma without a concomitant fracture. There are no uniform treatment guidelines for this common population. The aims of this study were (1) to obtain insight into FPS definition, diagnosis, and treatment amongst international colleagues, and (2) to identify a uniform definition based on radiographic measurements with optimal cut-off points via a receiver operating characteristic (ROC) curve. An online international survey was set up to assess the diagnostic and treatment strategies, criteria, and definitions of the FPS, the probability of an occult fracture, and the presence of an anterior and/or posterior FPS on 20 radiographs. Additionally, the research team performed radiographic measurements to identify cut-off values for a positive FPS, as well as test–retest reliability and inter-rater reliability via intraclass correlation coefficients (ICC). A total of 133 (paediatric) orthopaedic surgeons completed the survey. Definitions, further diagnostics, and treatments varied considerably amongst respondents. Angle measurements of the fat pad as related to the humeral axis line showed the highest reliability (test–retest ICC, 0.95 (95% CI 0.88–0.98); inter-rater ICC, 0.95 (95% CI 0.91–0.98)). A cut-off angle of 16° was defined a positive anterior FPS (sensitivity, 1.00; specificity, 0.87; accuracy, 99%), based on the respondents’ assessment of the radiographs in combination with the research team’s measurements. Any visible posterior fat pad was defined as a positive posterior FPS. This study provides insight into the current diagnosis and treatment of children with a radiological fat pad sign of the elbow. A clear, objective definition of a positive anterior FPS was identified as a ≥16° angle with respect to the anterior humeral line.

Validation of an instrument for the evaluation of exchange transfusion (INEXTUS) via an OSCE

IntroductionExchange transfusion is the treatment of choice for patients with severe hyperbilirubinemia who do not respond to phototherapy. This procedure is highly complex and requires substantial expertise to perform, however it´s not done frequently enough to guarantee adequate training. Traditional learning scenarios do not have a space reserved for teaching this procedure or an instrument that fully and objectively evaluates the skills that a professional must acquire. ObjectivesThe purpose was to construct and evaluate the INEXTUS instrument´s validity evidence relevant to internal structure, in a simulated scenario through the performance of an objective structured clinical exam (OSCE). Materials and methodsThe Delphi consensus methodology was utilized to design the instrument; six experts participated through three rounds using the Google Forms platform. The categories and items previously obtained were subjected to validation by nine experts through a dichotomous survey. Prior to data collection, the evaluators were trained through a pilot test with 10 medical students. Subsequently, all residents of a paediatric programme were evaluated through the OSCE methodology in a simulated scenario, with 6 stations, of a clinical case of a new-born with an explicit need for exchange transfusion. During their participation in the scenario, the residents were first evaluated with the instrument developed. Additionally, audio and video filming of all students who participated was performed with the aim of conducting a second evaluation two weeks after the first four evaluators participated.ResultsThe final INEXTUS instrument consists of 46 subitems grouped into 23 items divided into 6 categories, demonstrating an inter-rater intraclass correlation coefficient of 0.96 (95% CI 0.94, 0.98 p-value < 0.001). For the Fleiss Kappa of the 23 items evaluated, concordance was evaluated for 14 items but could not be determined for the 9 remaining items because all the ratings were equal, either because the items were not performed or they were all performed adequately. Of the 14 items, 9 good scores were obtained (95% CI 0.61 to 0.8; p value < 0.001), and 5 very good scores were obtained (95% CI 0.81 to 1; p value < 0.001).ConclusionsThe INEXTUS instrument evaluates exchange transfusion skills in medical personnel in training in simulated scenarios using the OSCE methodology; it has high validity and reliability and is a high-impact educational tool.

Accuracy, inter- and intrarater reliability, and user-experience of high tibial osteotomy angle measurements for preoperative planning: manual planning PACS versus semi-automatic software programs

PurposeTo compare the accuracy, inter- and intrarater reliability, and user-experience of manual and semi-automatic preoperative leg-alignment measurement planning software for high tibial osteotomy (HTO).MethodsThirty patients (31 lower limbs) who underwent a medial opening wedge HTO between 2017 and 2019 were retrospectively included. The mechanical lateral distal femur angle (mLDFA), mechanical medial proximal tibial angle (mMPTA), and planned correction angle were measured on preoperative long-leg full weight-bearing radiographs utilising PACS Jivex Review® v5.2 manual and TraumaCad® v2.4 semi-automatic planning software. Independent measurements were performed by four raters. Two raters repeated the measurements. Accuracy in the standard error of measurement (SEM), inter- and intrarater reliability, and user-experience were analysed. Additionally, measurements errors of more than 3° were remeasured and reanalysed.ResultsThe SEMs of all measured varus malalignment angles and planned correction angle were within 0.8° of accuracy for both software programs. Measurements utilising the manual software demonstrated moderate interrater intraclass correlation coefficient (ICC)-values for the mLDFA and mMPTA, and an excellent interrater ICC-value for the correction angle (0.810, 0.779, and 0.981, respectively). Measurements utilising the semi-automatic software indicated excellent interrater ICC-values for the mLDFA, mMPTA, and correction angle (0.980, 0.909, and 0.989, respectively). The intrarater reliability varied substantially per angle, presenting excellent intrarater agreements by both raters (ICC > 0.900) for the correction angle in each software program as well as poor-to-excellent ICC-values for the mLDFA (0.282–0.951 and 0.316–0.926) and mMPTA (0.893–0.934 and 0.594–0.941) in both the manual planning and semi-automatic software. Regarding user-experience, semi-automatic software was preferred by two raters, while the other two raters had no distinctive preference. After remeasurement of five outliers, excellent interrater ICC-values were found for the mLDFA (0.913) and mMPTA (0.957).ConclusionsSemi-automatic software outperforms the manual software when user-experience and outliers are considered. However, both software programs provide similar performance after remeasurement of the human-related erroneous outliers. For clinical practice, both programs can be utilised for HTO planning.Level of evidenceDiagnostic study, Level III.

Handheld dynamometer reliability to measure knee extension strength in rehabilitation patients-A cross-sectional study.

IntroductionThe Handheld Dynamometer (HHD) has the potential to overcome some of the logistic and economic limitations of isokinetic dynamometers for measuring knee extension muscle strength. However, its reliability has not been fully assessed. The purpose of this study is to measure intra and inter-rater reliability of HHD for knee extension strength in patients receiving rehabilitation treatment, as well as to understand in which conditions is the reliability higher.MethodsTwenty-nine patients admitted in an inpatient Physical Medicine and Rehabilitation unit were consecutively included in this cross-sectional study. Two experienced and two inexperienced physicians made two assessments of knee extension strength with HHD, separated by three hours. Intraclass Correlation Coefficients (ICC), absolute differences between assessments, and correlations between strength and functional variables were calculated.ResultsIntra and inter-rater ICC were overall high (≥ 0.950 and 0.927, respectively). Higher values were found when average of two measurements were made for estimating intra-rater ICC (ICC = 0.978; 95%CI = 0.969–0.985) but not for inter-rater ICC. ICC were not statistically significantly different when calculated based on measurements performed by inexperienced physicians and experienced ones. There was a moderate correlation between strength and functional variables.ConclusionHandheld Dynamometer seems to be a reliable option to measure knee extension muscle strength, particularly when two measurements are performed and their average is reported.

Glenoid Version Assessment When the CT Field of View Does Not Permit the Friedman Method: The Robertson Method.

Background:To improve spatial resolution, current clinical shoulder cross-sectional imaging studies reduce the field of view of the shoulder, excluding the medial scapula border and preventing glenoid version measurement according to the Friedman method.Purpose:To evaluate a method to accurately and reliably measure glenoid version on cross-sectional shoulder images when the medial scapula border is not included in the field of view, and to establish measurements equivalent to the Friedman method.Study Design:Controlled laboratory study.Methods:Sixty-five scapulae underwent computed tomography (CT) scanning with an optimal shoulder CT-positioning protocol. Glenoid version was measured on CT images of the full scapula using the Friedman method. We developed a measurement method (named the Robertson method) based on the glenoid vault version from partial scapula images, with a correction angle subtracted from the articular-surface-glenoid vault measurement. Comparison with the Friedman method defined the accuracy of the Robertson method. Three observers tested inter- and intraobserver reliability of the Robertson method. Accuracy was statistically evaluated with t tests and reliability with the intraclass correlation coefficient (ICC).Results:The statistical distribution of glenoid version was similar to published data,–0.5° ± 3° [mean ± SD]. The initial measurement using the Robertson method resulted in a more retroverted angle compared with the Friedman method, and a correction angle of 7° was then applied. After this adjustment, the difference between the 2 methods was nonsignificant (0.1° ± 4°; P > .65). Reliability of the Robertson method was excellent, as the interrater ICC was 0.77, the standard error of measurement (SEM) was 1.1° with P < .001. The intrarater ICC ranged between 0.84 and 0.92, the SEM ranged between 0.9° and 1.2° with P < .01.Conclusion:A validated glenoid version measurement method is now available for current clinical shoulder CT protocols that reliably create Friedman-equivalent values.Clinical Relevance:Friedman-equivalent values may be made from common clinical CTs of the shoulder and compared with prior and future Friedman measurements of the scapula.

The SParTA Score: A New Assessment System for Microsurgical Training.

We describe the Suture Parameters for Training Assessment (SParTA) tool, which is reliable and easily administered, to assess and guide both training and practice in microsurgery. It comprises six parameters that are easily measured from an image. [See Figure, Supplemental Digital Content 1, which demonstrates the six parameters of the SParTA scoring system: suture length (mm), yellow line; suture angle (degree), between the yellow line and blue line (the strip border); suture alignment (mm), distance between the green dot (midpoint of the suture) and blue dot (intersection point between the wound axis and that suture); suture interval (mm), distance between two adjacent green dots; area (mm2), formed by four red points (end points of the cut suture); and the missing suture, defined as having less than nine required sutures, https://links.lww.com/PRS/F92.] Twenty-six completed microsurgical suture strips (Digital Surgicals, Singapore) from our database were picked to represent the spectrum of performance. Participants were instructed to cut the strip perpendicularly and place nine interrupted sutures using 11-0 nylon. Images of the suture strips were captured and imported into ImageJ software version 2 (National Institutes of Health, Bethesda, Md.) for measurement. Five experts were provided with the same photographs of the suture strips. The experts independently ranked the suture strips using a four-point Likert scale, with 1 = very poor, 2 = poor, 3 = good, and 4 = very good. The average expert score was utilized as a standard for validation of the scoring systems. Subsequently, the samples were assessed using three scoring systems: our system, the DS Micro Trainer system (Digital Surgical Pte., Ltd., Singapore),1 and the End Product Intimal Assessment tool.2 The scores were compared against expert evaluations as a standard for validation. SParTA was used to score the strips, first by two different assessors and a second time by one of the two assessors after 7 days, for interrater and intrarater validity, respectively. All analyses were performed using SPSS version 23.0 software (SPSS, Inc., Chicago, Ill.). Statistical significance was defined as p less than 0.05. Intrarater and interrater reliability were determined using the intraclass correlation coefficient (95 percent confidence interval, two-way mixed effects model, absolute agreement, single rater, κ = 2). The mean differences of the intrarater and interrater scores were further evaluated using the two-tailed t test. SParTA scores ranged from 13 to 105, with a lower score indicating better performance (0 is the best possible score). The mean differences in intrarater and interrater scores were 1.27 (±7.36) and 1.31 (±9.02), respectively (p > 0.05). Both the intrarater and interrater scores were further validated using the intraclass correlation coefficient, which showed good to excellent reliability, with an intrarater intraclass correlation coefficient of 0.95 (95 percent CI, 0.89 to 0.98) and an interrater intraclass correlation coefficient of 0.93 (95 percent CI, 0.86 to 0.97). When the three scoring systems were compared to the expert assessment, there was strongest positive correlation between the expert score and SParTA (r = 0.89, p = 0.00) (Fig. 1), followed by the expert score and the DS Micro Trainer (r = 0.78, p = 0.00). A weaker positive correlation was observed for the End Product Intimal Assessment tool (r = 0.56, p = 0.03).Fig. 1.: Scatterplots and linear relationships between the SParTA scoring algorithm and expert assessment.SParTA’s parameters were chosen to reflect suturing characteristics that directly reflect the quality of work done. For example, SParTA assesses suture spacing rather than suture knot spacing, as done in the DS Micro Trainer.1,3 Beyond a score (and parameter subscores), SParTA provides useful detailed information that guides training and practice. Standardized microsurgery simulation tasks allow educators and trainees to benchmark their performance and learning.4 This is contingent on valid and reliable assessment of the training task. SParTA is a product scoring system that fulfils this requirement for microsurgical training. DISCLOSURE The authors have no financial disclosures to report. No funding was received for this article.

The Reliability of the Thai Version of the Toddlers’ Temperament Questionnaire

Objective: To evaluate the reliability of the Early Childhood Behavior Questionnaire (ECBQ)-very short form, Thai version; and to investigate Thai toddlers’ temperamentsMaterials and Methods: The English version of the very short form ECBQ was translated into Thai language. The primary caregivers of 360 healthy, 18-36 months old children, were asked to fill the questionnaire from the period of April, 2018 to June, 2019. The scores were calculated to determine the child’s temperament. The reliability of the very short form ECBQ- Thai version was assessed by Cronbach’s alpha coefficient for internal consistency and Intraclass correlation coefficient (ICC) for inter-rater reliability, two-week and six-month test-retest reliability.Results: The Cronbach’s alpha coefficients were 0.627-0.692, which indicated questionable internal consistency, but almost reach those of the original ECBQ. The inter-rater ICCs were 0.463-0.670, which were comparable to those in the original English version. The two-week ICCs were 0.602-0.750, which indicated moderate reliability, whereas the six-month ICCs decreased to 0.459-0.602. However, these values almost reached those of the original English version. Most toddlers were reported to have a surgency temperament. Boys significantly had more surgency temperament and had higher mean surgency scores, whereas girls exhibited a more effortful-control temperament.Conclusion: The very short-form ECBQ, Thai version can be implemented in assessing Thai toddlers’ temperament. The majority of Thai toddlers in this study demonstrated surgency temperament. Boys tended to be more surgency, while girls tended to be more effortful control.

A Novel MRI Mapping Technique for Evaluating Bone Bruising Patterns Associated With Noncontact ACL Ruptures

Background:Bone bruise patterns in the knee can aid in understanding the mechanism of injury in anterior cruciate ligament (ACL) ruptures. There is no universally accepted magnetic resonance imaging (MRI) mapping technique to describe the specific locations of bone bruises.Hypothesis:The authors hypothesized that (1) our novel mapping technique would show high interrater and intrarater reliability for the location of bone bruises in noncontact ACL-injured knees and (2) the bone bruise patterns reported from this technique would support the most common mechanisms of noncontact ACL injury, including valgus stress, anterior tibial translation, and internal tibial rotation.Study Design:Cross-sectional study; Level of evidence, 3.Methods:Included were 43 patients who underwent ACL reconstruction between 2018 and 2020, with MRI within 30 days of the injury on a 3.0-T scanner, documentation of a noncontact mechanism of injury, and no concomitant or previous knee injuries. Images were retrospectively reviewed by 2 radiologists blinded to all clinical data. The locations of bone bruises were mapped on fat-suppressed T2-weighted coronal and sagittal images using a novel technique that combined the International Cartilage Repair Society (ICRS) tibiofemoral articular cartilage surgical lesions diagram and the Whole-Organ Magnetic Resonance Imaging Scoring (WORMS) mapping system. Reliability between the reviewers was assessed using the intraclass correlation coefficient (ICC), where ICC >0.90 indicated excellent agreement.Results:The interrater and intrarater ICCs were 0.918 and 0.974, respectively, for femoral edema mapping and 0.979 and 0.978, respectively, for tibial edema mapping. Significantly more bone bruises were seen within the lateral femoral condyle compared with the medial femoral condyle (67% vs 33%; P < .0001), and more bruises were seen within the lateral tibial plateau compared with the medial tibial plateau (65% vs 35%; P < .0001). Femoral bruises were almost exclusively located in the anterior/central regions (98%) of the condyles as opposed to the posterior region (2%; P < .0001). Tibial bruises were localized to the posterior region (78%) of both plateaus as opposed to the anterior/central regions (22%; P < .0001).Conclusion:The combined mapping technique offered a standardized and reliable method for reporting bone bruises in noncontact ACL injuries. The contusion patterns identified using this technique were indicative of the most commonly reported mechanisms for noncontact ACL injuries.

Reliability of the Expanded Cutting Alignment Scoring Tool (E-CAST) to Assess Trunk and Limb Alignment During a 45-Degree Side-Step Cut.

BackgroundCurrent clinical screening tools assessing risky movements during cutting maneuvers do not adequately address sagittal plane foot and ankle evaluations. The Cutting Alignment Scoring Tool (CAST) is reliable in evaluating frontal plane trunk and lower extremity alignment during a 45-degree side-step cut. The Expanded Cutting Alignment Scoring Tool (E-CAST) includes two new sagittal plane variables, knee flexion and ankle plantarflexion angle.Hypothesis/PurposeTo assess the inter-and intra-rater reliability of the E-CAST to evaluate trunk and lower extremity alignment during a 45-degree side-step cut.Study DesignRepeated MeasuresMethodsParticipants included 25 healthy females (13.8 ± 1.4 years) regularly participating in cutting or pivoting sports. Participants were recorded performing a side-step cut in frontal and sagittal planes. One trial was randomly selected for analysis. Two physical therapists independently scored each video using the E-CAST on two separate occasions, with randomization and a two-week wash-out between rounds. Observed movement variables were awarded a score of “1”, with higher scores representing poorer technique. Intraclass correlation coefficients (ICC) and 95% confident intervals (95% CI) were calculated for the total score, and a kappa coefficient (k) was calculated for each variable.ResultsThe cumulative intra-rater reliability was good (ICC=0.78, 95% CI 0.59-0.96) and the cumulative inter-rater reliability was moderate (ICC=0.71, 95% CI 0.50-0.91). Intra-rater kappa coefficients ranged from moderate to excellent for all variables (k= 0.50-0.84) and inter-rater kappa coefficients ranged from slight to excellent for all variables (k=0.20-0.90).ConclusionThe addition of two sagittal plane variables resulted in lower inter-rater ICC compared to the CAST (ICC= 0.81, 95% CI 0.64-0.91). The E-CAST is a reliable tool to evaluate trunk and LE alignment during a 45-degree side-step cut, with good intra-rater and moderate inter-rater reliability.Level of EvidenceLevel 2, Diagnosis

Reliability of 2-Dimensional Video Analysis in Adolescent Runners.

The literature on the reliability of qualitative and quantitative measures for running video analysis in the adolescent population is limited. Reliability of 2-dimensional (2D) video analysis has been reported in adult runners, but these findings may not apply to youth runners. We sought to determine the intra-rater and inter-rater reliability of sagittal and frontal plane kinematics using 2D video analysis in healthy adolescent runners. High-definition (1080p) videos were recorded of 10 healthy runners between 14 and 18 years old running on a treadmill at self-selected speed with markers attached to the cervical spine, pelvis, and lower extremities. Kinematic variables in the sagittal and frontal planes were measured using Dartfish Motion Analysis Software by 3 raters (2 sports medicine physical therapists and a research assistant). Intra- and inter-rater reliability were calculated using intraclass correlation coefficients (ICCs). Of the 10 runners, 4 (40%) were male and the mean age was 16 ± 1.5 years. The intra-rater ICC for all kinematic variables ranged from 0.574 to 0.999 for the experienced physical therapist, and 0.367 to 0.973 for the inexperienced research assistant. The inter-rater ICC for all raters ranged from -0.01 to 0.941. Eleven kinematic variables showed substantial agreement and 4 showed almost perfect agreement. Step width and foot progression showed fair and poor agreement, respectively. Running analysis using 2D video can be performed reliably in adolescents on all kinematic variables except for step width and foot progression. Inexperienced raters can be properly trained in the video analysis of running kinematics to consistently assess the same runner.

Comparing reliability between 3D imaging and 2D photography for external nasal anthropometry

This study investigates and compares the reliability and reproducibility of two facial anthropometric methods about external nasal angles, 3D imaging and conventional 2D photography. 2D photograph images and 3D images about external nose of 30 volunteers were taken using digital camera and Morpheus 3D scanner. To evaluate intra-rater reliability, each images were taken over two different days for each subject by the same researcher. To evaluate inter-rater reliability, another researcher took each images for each subject on the first day. The reliability of each method for measuring 4 external nasal angle is obtained using intraclass correlation coefficient (ICC) and compared. Inter-rater and intra-rater reliability of both 3D imaging and 2D photography had excellent agreement in all 4 nasal angles. In the nasofacial angular parameter, Inter-rater ICC, 2D photography was significantly higher than 3D imaging. Result of intra-rater ICC also showed both 3D imaging and 2D photography had good reliability in all 4 nasal angles. Similar to those of inter-rater ICC, nasofacial angular parameter showed statistically significant differences between 3D imaging and 2D photography. In terms of reliability, both 2D and 3D showed appropriate anthropometric results and considering its own advantage, each methods can be used complementarily.

Assessing the inter-rater and intra-rater reliability of the Physical Therapy Competence Assessment for Airway Suctioning (PT-CAAS).

The Physical Therapy Competence Assessment for Airway Suctioning (PT-CAAS) is a recently developed measure to assess the clinical competence of physiotherapists who perform airway suctioning with adults. The purpose of this study was to assess the inter-rater and intra-rater reliability of the PT-CAAS. Scoring rules were developed through expert consultation. Reliability was then assessed using nine videos of suctioning performed in a simulated learning environment. A repeated measures design was used, with two replicate sets of measurements made by each participant for all videos. Data were analyzed using a repeated measures model for the concurrent assessment of inter-rater and intra-rater reliability. Participants were physiotherapists with suctioning experience. Twenty physiotherapists completed initial scoring and re-scoring for all nine videos; their data were included in the analysis. Intraclass correlation coefficients (ICCs) for inter-rater reliability ranged from 0.569 [lower one-sided 95% confidence interval (CI): 0.395; standard error of measurement (SEM): 0.963] for infection control to 0.759 (lower one-sided 95% CI: 0.612; SEM: 0.722) for post-suctioning assessment and care. The inter-rater ICC for overall performance was 0.752 (lower one-sided 95% CI: 0.602; SEM: 0.660). ICCs for intra-rater reliability ranged from 0.759 (lower one-sided 95% CI: 0.197; SEM 0.721) for infection control to 0.860 (lower one-sided 95% CI: 0.544; SEM: 0.550) for post-suctioning assessment and care. The intra-rater ICC for overall performance was 0.867 (lower one-sided 95% CI: 0.559; SEM: 0.483). Evidence of moderate to good inter-rater and good intra-rater reliability was found; however, the results should be interpreted with caution given the wide CIs and relatively large SEMs. Improved assessor training and assessments of reliability using a larger sample size are recommended.

Prioritizing Elective Surgical Cases During a Pandemic or Global Crisis: The Elective-Pediatric Orthopedic Surgical Timing (E-POST) Score.

As the first wave of the COVID-19 pandemic stabilized and resources became more readily available, elective surgery was reinitiated and hospitals realized that there was little guidance on how to prioritize elective cases. A prioritization tool was formulated based on clinically relevant elements and previous literature. Nine pediatric orthopaedic surgeons from North American institutions evaluated 25 clinical scenarios on 2 occasions separated in time. Intra-rater and inter-rater reliability were calculated [intraclass correlation coefficient (ICC)]. Surgeons also ranked the importance of each element and how confident they were with scoring each factor. Intra-rater ICC for total score showed good to excellent consistency; highest at 0.961 for length of stay (LOS) and lowest at 0.705 for acuity. Inter-rater ICC showed good to excellent agreement for American Society of Anesthesiologists score, LOS, duration of surgery, and transfusion risk and moderate agreement for surgical acuity and personal protective equipment (PPE) use. Transfusion risk and duration of surgery were deemed least important, and surgeons were least confident in scoring PPE and transfusion risk. Based on findings, the novel Elective-Pediatric Orthopedic Surgical Timing (E-POST) score for prioritizing elective cases was developed, consisting of 5 factors: surgical acuity, global health status, LOS, duration of surgery, and PPE requirement. The E-POST numeric total score or subscore may help objectively prioritize elective cases during a global crisis. Level V.

Psychometric properties of the School Participation Questionnaire: Testing a measure of participation-related constructs.

To explore concurrent validity, convergent validity, interrater reliability, test-retest reliability, and Rasch model analysis of the School Participation Questionnaire (SPQ), a tool for teachers to assess personal and environmental determinants of school participation. Teachers of children with additional support needs, including intellectual disability, autism, and learning difficulties completed measures. Data were collected using the SPQ and the Participation and Environment Measure for Children and Youth (PEM-CY). Test-retest and interrater reliability were assessed using intraclass correlation coefficients (ICCs). Internal consistency was assessed with Cronbach's alpha. Concurrent and convergent validity were explored via correlations with the PEM-CY. Further psychometrics were examined using a Rasch model. One hundred and eighty-seven children (136 [72.7%] male; mean age 9y [range 5y 6mo-12y 10mo, SD 2y]) were assessed by 67 teachers. Cronbach's alpha, test-retest, and interrater reliability were acceptable-excellent across each SPQ scale (alphas=0.89, 0.9, 0.94, 0.79; test-retest ICCs=0.64, 0.61, 0.78, 0.62; interrater ICCs=0.85, 0.71, 0.90, 0.81). Concurrent and convergent validity were confirmed with significant positive correlations between SPQ and PEM-CY. After Mokken and Rasch model analysis, person and item reliability were good, and unidimensionality was confirmed. Mean administration time was 8.2 minutes. The results suggest that the SPQ is a rapid, reliable, and valid tool for assessment of participation-related indicators in schools.

Validation of Inter-Rater and Intra-Rater Reliability of Remote Dielectric Sensing Measurement

Remote dielectric sensing (ReDS) is a recently introduced non-invasive electromagnetic-based device used to quantify lung fluid levels. Nevertheless, its inter-rater and intra-rater reliability remain uncertain. In 10 healthy volunteers, ReDS values were measured three times successively by the officially trained expert examiner to validate intra-rater reliability. Similar measures were performed by a total of three examiners to validate inter-rater reliability. Intra-class correlation (ICC) was applied to validate each reliability. Ten healthy volunteers [median 34 (32, 40) years old, 10 men, body mass index 23.0 (21.2, 23.9) ] were included. Median ReDS value was 28% (25%, 31%). For the intra-rater reliability, ICC (1, 1) and ICC (1, 3) were 0.966 and 0.988, respectively (P < 0.001). For the inter-rater reliability, ICC (2, 1) and ICC (2, 3) were 0.683 and 0.866, respectively (P < 0.001). Given almost perfect intra-rater reliability, an examiner does not need to repeat ReDS measurement. Given substantial inter-rater reliability, ReDS measurements had better be measured by multiple examiners if possible.

Do you see what I see? Investigating the validity of an equine personality questionnaire

Subjective equine personality questionnaires have the potential to predict a range of industry-relevant outcomes including fear reactivity, compliance with human cues, pain expression and susceptibility to stereotypies, in a time- and cost-efficient manner. However, to produce meaningful measures of target animals’ behavioural tendencies, subjective personality assessment tools must satisfy four criteria: internal consistency, predictive validity, inter-rater reliability, and test-retest reliability. The Equine Personality Test (EPT) has been developed to assess horses on five personality factors based on trait ratings from a familiar observer. While the EPT has been shown to have predictive validity, it has not been assessed for internal consistency, inter-rater reliability or test-retest reliability. To this end, three experienced primary caregivers and three riding instructors assessed 25 familiar horses using the EPT. The internal consistency, inter-rater reliability and test-retest reliability of the five subscales of the EPT were investigated using Cronbach’s α and intra-class correlation (ICC) analyses. The Agreeableness, Neuroticism, Extroversion and Gregariousness towards People subscales had high Cronbach α and inter-rater and test-retest ICC coefficients (α > 0.7; ICC>0.8). By contrast, the Gregariousness towards Horses subscale had low Cronbach α (α = 0.39) and inter-rater ICC coefficient (ICC=0.498), and an adequate test-retest ICC coefficient (ICC=0.784). Primary caregivers had higher ICC coefficients than instructors for most subscales and questionnaire items. The EPT therefore provides internally consistent and highly reliable measures of Agreeableness, Neuroticism, Extroversion, and Gregariousness towards People in equines, although measures of Gregariousness towards Horses should be interpreted with caution. The reliability of EPT scores can be further improved by targeting primary caregivers as raters. Taken together with previous findings demonstrating predictive validity for the questionnaire, these results contribute to making the EPT the only subjective equine personality questionnaire to have been checked against all four criteria of a valid and reliable personality assessment tool. This positions the EPT as a highly relevant equine personality assessment tool that may be used to predict behavioural tendencies in industry or research settings alike.