Interrater Reliability Of Measurements Research Articles

Reliability of and Correlation Between Measurements of Acetabular Morphology.

The authors sought to determine the intra- and interrater reliability of contemporary measures of acetabular morphology among a group of hip surgeons and to determine the correlations between measures of acetabular morphology. On 2 separate occasions, 3 hip surgeons independently performed blinded evaluations of anteroposterior pelvic radiographs of 40 patients. The lateral center-edge angle, medial center-edge angle, acetabular arc, extrusion index, acetabular index, Sharp's angle, posterior wall sign, crossover sign, femoro-epiphyseal acetabular roof index, acetabular wall indexes, and delta angle were assessed. A linear mixed model was used for variance estimation, and kappa and intra-class correlation coefficients were assessed for reliability. Pearson correlation coefficients were calculated for all possible pairs of radiographic measures. The acetabular index had the greatest interrater agreement (0.90; 95% confidence interval [CI], 0.84-0.93), whereas the lateral center-edge angle had the higher intrarater agreement (0.96; 95% CI, 0.92-0.98). The acetabular arc angle had the lowest interrater agreement (0.44; 95% CI, 0.29-0.57), and the Sharp's angle had the lowest intrarater agreement (0.66; 95% CI, 0.41-0.84). The posterior wall sign had an interrater agreement of 0.35 (95% CI, 0.11-0.54) and an intrarater agreement of 0.68 (95% CI, 0.49-0.86). The crossover sign had an interrater agreement of 0.66 (95% CI, 0.43-0.84) and an intrarater agreement of 0.85 (95% CI, 0.52-0.89). The acetabular index, lateral center-edge angle, and extrusion index presented with high coefficients of correlation. In addition, acetabular anteversion correlated with severity of dysplasia. Commonly used parameters such as the acetabular index, lateral center-edge angle, and extrusion index are reliable radiographic parameters to assess acetabular morphology. However, correlation between measures suggests that they may be redundant in quantifying acetabular morphology. [Orthopedics. 2018; 41(5):e629-e635.].

Systematic review of methods to quantify teamwork in the operating theatre.

Background Teamwork in the operating theatre is becoming increasingly recognized as a major factor in clinical outcomes. Many tools have been developed to measure teamwork. Most fall into two categories: self-assessment by theatre staff and assessment by observers. A critical and comparative analysis of the validity and reliability of these tools is lacking. Methods MEDLINE and Embase databases were searched following PRISMA guidelines. Content validity was assessed using measurements of inter-rater agreement, predictive validity and multisite reliability, and interobserver reliability using statistical measures of inter-rater agreement and reliability. Quantitative meta-analysis was deemed unsuitable. Results Forty-eight articles were selected for final inclusion; self-assessment tools were used in 18 and observational tools in 28, and there were two qualitative studies. Self-assessment of teamwork by profession varied with the profession of the assessor. The most robust self-assessment tool was the Safety Attitudes Questionnaire (SAQ), although this failed to demonstrate multisite reliability. The most robust observational tool was the Non-Technical Skills (NOTECHS) system, which demonstrated both test-retest reliability (P > 0·09) and interobserver reliability (Rwg = 0·96). Conclusion Self-assessment of teamwork by the theatre team was influenced by professional differences. Observational tools, when used by trained observers, circumvented this.

Models of assessing the extent of agreement beteween raters using kappa coefficients

In medical studies quality of assessment is of great importance. Typically it is characterized by reliability. A fundamental sense of interobserver reliability is to evaluate a degree of agreement between independent judges examining the same objects. The paper is an evaluation of some interrater reliability measures and their interpretation. Cohen’s kappa and Scott’s coefficient are considered. We describe models and connections between coefficients defined for dichotomous and politomous data. We show that abovementioned estimators of kappa for classification into more than 2 categories are weighted averages of kappas in binary models defined for each category separately.

Inter-rater reliability of ratings on the six-item Positive and Negative Syndrome Scale (PANSS-6) obtained using the Simplified Negative and Positive Symptoms Interview (SNAPSI)

Purpose: The six-item version of the Positive And Negative Syndrome Scale (PANSS-6) is a brief rating scale focusing on core symptoms of schizophrenia. In order to facilitate rating of PANSS-6 and selected items from other common psychiatric rating scales, we recently developed the Simplified Negative and Positive Symptoms Interview (SNAPSI). The objective of the present study was to test the inter-rater reliability of PANSS-6 ratings obtained using the SNAPSI.Materials and methods: Using the SNAPSI, seven raters (psychiatrists, first-year psychiatry residents and psychologists) performed a total of 56 PANSS-6 ratings of 12 in- or outpatients with schizophrenia. As a measure of inter-rater reliability, we calculated the intra-class correlation coefficient (ICC, ≥0.75 = excellent, 0.40–0.74 = fair to good, <0.40 = poor) for the PANSS-6 total score and individual item scores. Furthermore, for the PANSS-6 total scores obtained by the six noncertified PANSS raters, we calculated the median deviation from the PANSS-6 total scores obtained by the only certified PANSS rater.Results: The ICC for the PANSS-6 total score was 0.74 (F = 2.84, p = .03). The ICCs for the six individual PANSS-6 items ranged from 0.45 (N6 – Lack of spontaneity & flow of conversation) to 0.76 (P3 – Hallucinatory behavior). The PANSS-6 total scores obtained by the six noncertified PANSS raters deviated by a median of 12.7% (interquartile range: 6.2–20.0) from the PANSS-6 total scores obtained by the certified PANSS rater.Conclusions: We found a good level of inter-rater reliability of PANSS-6 ratings obtained using the SNAPSI for seven raters with varying levels of clinical and research experience.

Skin temperature of the foot: Reliability of infrared image analysis based in the angiosome concept

ObjectiveStudies reporting the reliability of image analysis when assessing skin temperature of the foot are scarce. The aim of this study was to assess the interrater and intrarater reliability of the analysis of foot skin temperature based on the angiosome concept and the association between skin temperature differences and the differences in size of the ROIs. MethodologyThermograms from 26 feet were analysed by two independent assessors and each assessor analysed the same images on different occasions. Mean temperature values of each of the six ROIs were extracted for analysis. Relative reliability was assessed by Intraclass Correlation Coefficient (ICC) measures and absolute reliability was assessed using Bland and Altman agreement measures and standard error of measurement (SEM). The Spearman correlation coefficient was used to assess the association between the skin temperature differences and the differences in size of the ROIs in the interrater and intrarater analysis. ResultsThe ICC values evidenced excellent interrater and intrarater reliability with the 95% confidence intervals (CI) ranging between 0.962 and 1.000 and the SEM ranged between 0.00 °C and 0.36 °C. The mean absolute difference (bias) between the measurements ranged between 0.002 °C and 0.117 °C and small to moderate associations between the differences in skin temperature and the difference in the number of pixels were identified. ConclusionThe excellent interrater and intrarater reliability measures suggest that the methodology of analysis was reliable and may be used in research and clinical settings. Although statistical significant associations between the skin temperature differences and the differences in size of the ROIs were found, the magnitude of the skin temperature differences between assessments and between assessors (0.02–0.17 °C) is not clinically relevant.

The identification and reliability of static and dynamic barefoot impression measurements: A systematic review

ObjectiveBarefoot impressions collected from crime scenes can be used in forensic analysis. The reliability of the measurement method employed during comparison of these foot impressions is paramount to prevent incorrect conclusions being made. A number of methods of obtaining measurements from barefoot impressions have been described in the research literature; however there has been no comprehensive review of their reliability. Therefore, the aim of this systematic review was to determine the reliability of measurements used to describe footprint morphology obtained from static and dynamic barefoot impressions. MethodsFour electronic databases were searched from inception to 23 November 2017. Eligible studies were required to report either the test–retest, intra or inter-rater reliability of measurements taken from barefoot impressions for the purposes of identification or classification of foot morphology. Methodological quality was assessed using the COSMIN checklist. ResultsEleven studies were identified that reported the reliability of 10 measurements (Reel method, arch index, Chippaux-Smirak index, footprint angle, Staheli index, contact area, Martirosov’s K index, toe score and metatarsal ridge-to-toe measurement). High intra-rater reliability has been established for the Reel method (ICC=0.98–0.99), arch index (ICC=0.96–0.99), Chippaux-Smirak index (ICC=0.98–0.99), footprint angle (ICC=0.97–0.98), Staheli index (ICC=0.98–0.99) and footprint index (ICC=0.96–0.97). High inter-rater reliability has been established for the Reel method (ICC=0.99) and footprint angle (ICC=0.99). Overall methodological quality was rated as ‘Poor’ to ‘Fair’. ConclusionThe measurement developed by Reel et al. has both its intra- and inter-rater reliability established to be high. However, the findings of this review were unable to inform a recommendation of one specific technique based on reliability data due to a small body of research at this time. Furthermore, there is a lack of data on the reliability of footprint measurement and comparison techniques in real-world scenarios. Overall, the findings regarding reliability of the techniques covered in this systematic review are to be interpreted with caution due to the methodological quality of reliability testing conducted within the included studies.

The approved Italian version of the comprehensive assessment of at-risk mental states (CAARMS-ITA): Field test and psychometric features.

The Comprehensive Assessment of At-Risk Mental States (CAARMS) was specifically developed to assess and detect young people at ultra-high risk (UHR) of developing psychosis. The current study was undertaken to test the reliability and validity of the authorized Italian version of the CAARMS (CAARMS-ITA) in a help-seeking population. Psychometric properties of the CAARMS-ITA were established using a sample of 223 Italian adolescents and young adults aged between 13 and 35 years, who were divided into 3 groups according to the CAARMS criteria: UHR-negative individuals (UHR [-]; n = 64), UHR-positive (UHR [+]; n = 55) and individuals with a first-episode psychosis (FEP; n = 104). The CAARMS-ITA's reliability was tested measuring interrater reliability and internal consistency. Construct validity was tested comparing the Positive and Negative Syndrome Scale (PANSS) and CAARMS-ITA subscale scores across groups (ie, UHR [-], UHR [+] and FEP). For concurrent validity, we studied correlations between symptoms of the CAARMS-ITA and their equivalents in the PANSS. Finally, the predictive validity was examined by following up with UHR [+] individuals. The 12-month transition rate to psychosis was calculated. The CAARMS-ITA showed good interrater reliability. The PANSS "Positive Symptoms" subscale scores in UHR [+] individuals were intermediate between FEP and UHR [-] groups. The positive and negative symptoms scores of the CAARMS-ITA significantly correlated with the corresponding scores of the PANSS. After 12 months, 4 of 41 (9.8%) UHR [+] individuals had transitioned to psychosis. The CAARMS-ITA is a reliable and valid instrument for assessing and detecting at-risk mental states in Italian clinical settings. It also appears to be helpful in the prediction of psychosis transition.

The Behavioral and Psychological Symptoms of Dementia in Down Syndrome (BPSD-DS) Scale: Comprehensive Assessment of Psychopathology in Down Syndrome.

People with Down syndrome (DS) are prone to develop Alzheimer’s disease (AD). Behavioral and psychological symptoms of dementia (BPSD) are core features, but have not been comprehensively evaluated in DS. In a European multidisciplinary study, the novel Behavioral and Psychological Symptoms of Dementia in Down Syndrome (BPSD-DS) scale was developed to identify frequency and severity of behavioral changes taking account of life-long characteristic behavior. 83 behavioral items in 12 clinically defined sections were evaluated. The central aim was to identify items that change in relation to the dementia status, and thus may differentiate between diagnostic groups. Structured interviews were conducted with informants of persons with DS without dementia (DS, n = 149), with questionable dementia (DS+Q, n = 65), and with diagnosed dementia (DS+AD, n = 67). First exploratory data suggest promising interrater, test-retest, and internal consistency reliability measures. Concerning item relevance, group comparisons revealed pronounced increases in frequency and severity in items of anxiety, sleep disturbances, agitation & stereotypical behavior, aggression, apathy, depressive symptoms, and eating/drinking behavior. The proportion of individuals presenting an increase was highest in DS+AD, intermediate in DS+Q, and lowest in DS. Interestingly, among DS+Q individuals, a substantial proportion already presented increased anxiety, sleep disturbances, apathy, and depressive symptoms, suggesting that these changes occur early in the course of AD. Future efforts should optimize the scale based on current results and clinical experiences, and further study applicability, reliability, and validity. Future application of the scale in daily care may aid caregivers to understand changes, and contribute to timely interventions and adaptation of caregiving.

Variation in tibial tuberosity lateralization and distance from the tibiofemoral joint line: An anatomic study

BackgroundTo describe variation in tibial tuberosity position in a normal adult population and inter-rater reliability of measurements of tibial tuberosity position. MethodsSurface models of 161 proximal tibia specimens (83 female, 78 male; 80 black, 81 white; age 28.7years, SD 7.5) were created with a three-dimensional laser scanner. Percent lateralization, tuberosity–eminence angle, and distance from joint surface were measured for each specimen. Variation in tuberosity position by sex, race, age, height, and BMI was calculated. Multivariate regression was used to assess for demographic factors independently associated with tuberosity positioning. ResultsMean percent lateralization was 57.9% (SD 2.4, range 52.4–64.9%). Tuberosity–eminence angle mean was 11.03° (SD 2.8, range 0–18.7°). Percent lateralization and tuberosity–eminence angle were not influenced by sex, race, age, height or, BMI (p>0.05). Mean tuberosity distance from joint surface was 29.2mm (SD 3.5, range 16.6–38.6mm) and larger in males than females (30.7mm (SD 2.9), 27.6mm (SD 3.3); p<0.001). Tuberosity distance from joint surface increased 0.18mm on average per 1.0cm increase in height (p<0.001). Inter-rater reliability was high for distance from joint surface (Cronbach alpha=0.99) and percent lateralization. ConclusionsTibial tuberosity percent lateralization falls in a narrow range for individuals, whereas tuberosity–eminence angle and distance from joint line are more variable. Inter-rater reliability is high for percent lateralization and distance for the joint surface. Distance of tibial tuberosity from joint surface is associated with sex and height.

Language Patterns Discriminate Mild Depression From Normal Sadness and Euthymic State.

Deviations from typical word use have been previously reported in clinical depression, but language patterns of mild depression (MD), as distinct from normal sadness (NS) and euthymic state, are unknown. In this study, we aimed to apply the linguistic approach as an additional diagnostic key for understanding clinical variability along the continuum of affective states. We studied 402 written reports from 124 Russian-speaking patients and 77 healthy controls (HC), including 35 cases of NS, using hand-coding procedures. The focus of our psycholinguistic methods was on lexico-semantic [e.g., rhetorical figures (metaphors, similes)], syntactic [e.g., predominant sentence type (single-clause and multi-clause)], and lexico-grammatical [e.g., pronouns (indefinite, personal)] variables. Statistical evaluations included Cohen's kappa for inter-rater reliability measures, a non-parametric approach (Mann-Whitney U-test and Pearson chi-square test), one-way ANOVA for between-group differences, Spearman's and point-biserial correlations to analyze relationships between linguistic and gender variables, discriminant analysis (Wilks' λ) of linguistic variables in relation to the affective diagnostic types, all using SPSS-22 (significant, p < 0.05). In MD, as compared with healthy individuals, written responses were longer, demonstrated descriptive rather than analytic style, showed signs of spoken and figurative language, single-clause sentences domination over multi-clause, atypical word order, increased use of personal and indefinite pronouns, and verb use in continuous/imperfective and past tenses. In NS, as compared with HC, we found greater use of lexical repetitions, omission of words, and verbs in continuous and present tenses. MD was significantly differentiated from NS and euthymic state by linguistic variables [98.6%; Wilks' λ(40) = 0.009; p < 0.001; r = 0.992]. The highest predictors in discrimination between MD, NS, and euthymic state groups were the variables of word order (typical/atypical) (r = -0.405), ellipses (omission of words) (r = 0.583), colloquialisms (informal words/phrases) (r = 0.534), verb tense (past/present/future) (r = -0.460), verbs form (continuous/perfect) (r = 0.345), amount of reflexive (e.g., myself)/personal (r = 0.344), and negative (e.g., nobody)/indefinite (r = 0.451) pronouns. The most significant between-group differences were observed in MD as compared with both NS and euthymic state. MD is characterized by patterns of atypical language use distinguishing depression from NS and euthymic state, which points to a potential role of linguistic indicators in diagnosing affective states.

S48. INTER-RATER RELIABILITY OF PANSS-6 SCHIZOPHRENIA SEVERITY RATINGS OBTAINED USING THE SIMPLIFIED NEGATIVE AND POSITIVE SYMPTOMS INTERVIEW (SNAPSI)

BackgroundSchizophrenia is a severe mental disorder requiring multimodal treatment. Monitoring the severity of schizophrenia during treatment is essential to a successful outcome. The most widely used measure of the severity of schizophrenia is the 30-item Positive And Negative Syndrome Scale (PANSS-30) obtained by the Structured Clinical Interview, SCI-PANSS, which takes approximately an hour to administer. This is too long for routine clinical use. Recently, our group extracted a 6-item scale (PANSS-6), which has shown promising psychometric properties. The scale consists of the following items: P1 - Delusions, P2 - Conceptual disorganization, P3 - Hallucinatory behavior, N1 - Blunted Affect, N4 - Passive/apathetic social withdrawal and N6 - Lack of spontaneity & flow of conversation. For now, it remains unknown whether it is possible to obtain sufficient information for PANSS-6 rating via a short and focused interview. Recently, our group developed an interview, the Simplified Negative and Positive Symptoms Interview (SNAPSI), which enables PANSS-6 rating. Field-testing at hospitals in the United States and Denmark has shown that the patient section of SNAPSI can be completed in approximately 15–25 minutes by raters who are unfamiliar with the interview and involving patients hearing the questions for the first time. The aim of the present study was to test the inter-rater reliability of PANSS-6 ratings obtained using the SNAPSI.MethodsThe team of raters (five medical doctors and two psychologists) attended training sessions prior to the inter-rater reliability test. At the training sessions one rater interviewed a patient with schizophrenia using the SNAPSI, while all raters conducted PANSS-6 ratings independently. After each interview the PANSS-6 ratings were discussed until an agreement was reached. Each rater participated in at least six SNAPSI/PANSS-6 training ratings.For the inter-rater reliability test, a total of 12 patients with a primary diagnosis of schizophrenia, currently undergoing in- or outpatient treatment at the Department for Psychosis, Aarhus University Hospital – Denmark, will be recruited. The team of raters will perform a total of at least 50 PANSS-6 ratings via SNAPSI. All raters will conduct the SNAPSI at least once. As a measure of inter-rater reliability, we will calculate the Intraclass Correlation Coefficient based on the 50 PANSS-6 ratings.ResultsThe results of the inter-rater reliability test will be available in January 2018 and presented at the SIRS 2018 conference.DiscussionIf the results of the inter-rater reliability test are satisfactory, we will conduct a clinical validation of PANSS-6. In this study we will test whether PANSS-6 ratings obtained using the SNAPSI correspond to PANSS-6 ratings extracted from independent PANSS-30 ratings obtained using the SCI-PANSS. If this is the case, PANSS-6 ratings obtained using the SNAPSI will facilitate valid measurement-based care of schizophrenia in clinical practice.

Building team agreement on large population surveys through inter-rater reliability among oral health survey examiners

Background: Oral health surveys conducted on a very large population involve many examiners who must be consistent in scoring different levels of an oral disease. Prior to the oral health survey implementation, a measurement of inter-rater reliability (IRR) is needed to know the level of agreement among examiners or raters. Purpose: This study aimed to assess the IRR using consensus and consistency estimates in large population oral health surveys. Methods: A total of 58 dentists participated as raters. The benchmarker showed the clinical sample for dental caries and community periodontal index (CPI) score, with the raters being trained to carry out a calibration exercise in dental phantom. The consensus estimate was measured by means of a percent agreement and Cohen’s Kappa statistic. The consistency estimate of IRR was measured by Cronbach’s alpha coefficient and intraclass correlation. Results: The percent agreement is 65.50% for photographic slides of dental caries, 73.13% for photographic slides of CPI and 78.78% for calibration of dental caries using phantom. There were statistically significant differences between dental caries calibration using photographic slides and phantom (p&lt;0.000), while the consistency of IRR between multiple raters is strong (Cronbrach’s Alpha: &gt;0.9). Conclusion: A percent agreement across multiple raters is acceptable for the diagnosis of dental caries. Consistency between multiple raters is reliable when diagnosing dental caries and CPI.

Spinal postural changes during the modified Matthiass test in healthy children : Interday and interrater reliability of dynamic rasterstereographic measurements.

The Matthiass test serves to identify postural insufficiencies by evaluating the sagittal posture but this subjective assessment may be responsible for different reports on the prevalence of postural insufficiencies in children. Previously, rasterstereography was used to objectify this clinical test. Until now, the coupling of the Matthiass test with rasterstereography has not yet been applied to children and the reliability is unknown; therefore, the objective of our study was to provide interday and interrater reliability of rasterstereographic measurements during dynamic (modified Matthiass test) and static trials (only interday) in healthy children. Trunk inclination, kyphotic, and lordotic angles were measured using rasterstereography during static and dynamic trials (modified Matthiass test: 90° shoulder flexion, hand-held load, 5% of body weight). Intraclass correlation coefficients (ICC) were calculated using atwo-way mixed model (absolute agreement, average measure). In this study 21healthy children were assessed (age range 10-12years). Dynamic rasterstereographic measurements showed fair to good interday and interrater reliability (ICC 0.46-0.70) and static measurements good to excellent interday reliability (ICC 0.63-0.91). Dynamic rasterstereography during the modified Matthiass test furnishes reliable data serving to objectify spinal changes of healthy children and detect postural insufficiencies. Additional efforts are needed to investigate how the early detection of postural insufficiencies can help to prevent back pain in children, adolescents and adults.

Qualitative Content Analysis of Coworkers' Safety Reports of Unprofessional Behavior by Physicians and Advanced Practice Professionals.

The aims of the study were to develop a valid and reliable taxonomy of coworker reports of alleged unprofessional behavior by physicians and advanced practice professionals and determine the prevalence of reports describing particular types of unprofessional conduct. We conducted qualitative content analysis of coworker reports of alleged unprofessional behavior by physicians and advanced practice professionals to create a standardized taxonomy. We conducted a focus group of experts in medical professionalism to assess the taxonomy's face validity. We randomly selected 120 reports (20%) of the 590 total reports submitted through the medical center's safety event reporting system between June 2015 and September 2016 to measure interrater reliability of taxonomy codes and estimate the prevalence of reports describing particular types of conduct. The initial taxonomy contained 22 codes organized into the following four domains: competent medical care, clear and respectful communication, integrity, and responsibility. All 10 experts agreed that the four domains reflected essential elements of medical professionalism. Interrater reliabilities for all codes and domains had a κ value greater than the 0.60 threshold for good reliability. Most reports (60%, 95% confidence interval = 51%-69%) described disrespectful or offensive communication. Nine codes had a prevalence of less than 1% and were folded into their respective domains resulting in a final taxonomy composed of 13 codes. The final taxonomy represents a useful tool with demonstrated validity and reliability, opening the door for reliable analysis and systems to promote accountability and behavior change. Given the safety implications of unprofessional behavior, understanding the typology of coworker observations of unprofessional behavior may inform organization strategies to address this threat to patient safety.

Reliability of a Market Basket Assessment Tool (MBAT) for Use in SNAP-Ed Healthy Retail Initiatives

ObjectiveTo evaluate the reliability of the Market Basket Assessment Tool (MBAT) for assessing the availability of fruits and vegetables, low-fat or nonfat dairy and eggs, lean meats, whole-grain products, and seeds, beans, and nuts in Supplemental Nutrition Assistance Program–authorized retail environments. MethodsDifferent trained raters used the MBAT simultaneously at 14 retail environments to measure interrater reliability. Raters returned to 12 retail environments (85.7%) 1 week later to measure test-retest reliability. Data were analyzed using paired-sample t tests and correlations. ResultsNo significant differences were found for interrater reliability or test-retest reliability for individual categories (mean differences, 0.0 to 0.3 ± 0.2 points) or total score (mean difference, 0.5 ± 0.4 points and (mean differences, 0.0 to 0.3 ± 0.3 points) or total score (mean difference, 0.8 ± 0.4 points), respectively. Conclusions and ImplicationsFuture steps include validation of the MBAT. A low-burden tool can facilitate evaluation of efforts to promote healthful foods in retail environments.

Systematic review of methods for quantifying teamwork in the operating theatre.

BackgroundTeamwork in the operating theatre is becoming increasingly recognized as a major factor in clinical outcomes. Many tools have been developed to measure teamwork. Most fall into two categories: self‐assessment by theatre staff and assessment by observers. A critical and comparative analysis of the validity and reliability of these tools is lacking.MethodsMEDLINE and Embase databases were searched following PRISMA guidelines. Content validity was assessed using measurements of inter‐rater agreement, predictive validity and multisite reliability, and interobserver reliability using statistical measures of inter‐rater agreement and reliability. Quantitative meta‐analysis was deemed unsuitable.ResultsForty‐eight articles were selected for final inclusion; self‐assessment tools were used in 18 and observational tools in 28, and there were two qualitative studies. Self‐assessment of teamwork by profession varied with the profession of the assessor. The most robust self‐assessment tool was the Safety Attitudes Questionnaire (SAQ), although this failed to demonstrate multisite reliability. The most robust observational tool was the Non‐Technical Skills (NOTECHS) system, which demonstrated both test–retest reliability (P > 0·09) and interobserver reliability (Rwg = 0·96).ConclusionSelf‐assessment of teamwork by the theatre team was influenced by professional differences. Observational tools, when used by trained observers, circumvented this.

Spine and Thoracic Height Measurements Have Excellent Interrater and Intrarater Reliability in Patients With Early Onset Scoliosis.

Reproducibility of measurements. This study investigates the reliability and standard error of measurement of spine and thoracic height radiographic measurements in patients with early onset scoliosis (EOS). Spine and thoracic height radiographic measurements are often used as a surrogate for pulmonary development in patients with EOS. There is limited literature validating the reliability of spine and thoracic height measurements in the EOS population. Using pilot data, we determined measuring 49 unique radiographs would provide 80% power to obtain a 95% confidence interval (CI) width of 0.05 for the interclass correlation coefficients (ICCs). A random sampling strategy, stratified by underlying diagnosis from the Classification of Early Onset Scoliosis (C-EOS), was used to distribute the diagnoses in the study sample. Two attending pediatric spine surgeons, two pediatric orthopedic fellows, and two research assistants measured coronal spine (T1-S1) and thoracic (T1-T12) height on digital radiographs using imaging software (Surgimap; Nemaris, Inc, New York) on two separate occasions at least 3 weeks apart. Order of images was randomized for the second iteration. Linear mixed model regression analyses were used to estimate interrater and intrarater reliability. The study sample included subjects (N = 48) with idiopathic (N = 17, 35%), congenital (N = 16, 33%, 1 patient excluded), neuromuscular (N = 11, 23%), and syndromic (N = 4, 8%) scoliosis. Overall interrater reliability estimates for spine height (ICC: 0.894, 95% CI: 0.847-0.932) and thoracic height (ICC: 0.890, 95% CI: 0.844-0.929) were excellent. Intrarater reliability estimates for spine height (ICC: 0.906, 95% CI: 0.830-0.943) and thoracic height (ICC: 0.898, 95% CI: 0.817-0.938) were also excellent. There is excellent interrater and intrarater reliability for radiographic measurements of spine and thoracic height in the EOS population at our institution. 2.

New Methods to Assess Forearm Torque and LiftingStrength: Reliability and Validity.

To determine the repeatability and validity of new methods designed to objectively measure forearm torque and lifting strength in a clinical setting. Twenty-eight healthy volunteers, 19 women and 9 men, were tested for lifting strength and forearm torque with the Kern and Baseline dynamometers. Two raters tested each participant on 3 occasions in the standing position. One of the raters also examined 15 subjects, 3 times, for forearm torque in the seated position and for lifting strength and forearm torque by the Work Simulator II, Baltimore Therapeutic Equipment (BTE II). Intraclass correlation coefficients (ICC) model 2,1 was used to calculate ICCs for intra- and interrater reliability. The same ICC model and Bland-Altman plots were used to analyze the validity and agreement between the new test methods and the BTE II equipment and for comparison between tests performed in the standing and seated positions. Intra- and interrater reliability for forearm torque measurements with both the Baseline and the BTE II demonstrated ICCs between 0.88 and 0.96. The comparison between the Baseline and the BTE equipment yielded lower ICCs of 0.74 to 0.88 but they were still substantial and in good agreement. The ICCs for torques recorded in the standing and seated position were 0.89 to 0.96. Lifting strength, measured in 3 different positions of forearm rotation, yielded ICC values between 0.84 and 0.96 for both raters and with both the Kern and the BTE II instruments. Similarly, comparisons between the Kern and the BTE II methods showed ICC values between 0.91 and 0.95. Both the Baseline and the Kern dynamometers demonstrated excellent intra- and interrater repeatability. Except for forearm torque test in direction of pronation, which had a slightly lower ICC of 0.74, our new methods were valid when the BTE II was used as the reference standard. Assessments in the standing or seated position for torque measurements made little difference. Thus, we found the quality of measurements performed with our new methods sufficient for future studies of forearm torque and lifting strength. Simple, yet reliable, methods to quantify torque and lifting strength in a clinical setting have the potential to improve evaluations of wrist and forearm disorders as well as their treatments.

THE RELIABILITY OF A NOVEL HEEL-RISE TEST VERSUS GONIOMETRY TO ASSESS PLANTARFLEXION RANGE OF MOTION

Ankle plantarflexion (PF) active range of motion (ROM) is traditionally assessed in a non-weight-bearing (NWB) position with a universal goniometer. However, a convenient, reliable, low-cost means of assessing functional PF active ROM in a weight-bearing (WB) position has yet to be established. To compare the intra- and interrater reliability of PF active ROM measurements obtained from a goniometric NWB assessment, and a functional heel-rise test (FHRT) performed in WB. Reliability study. Two physical therapy student examiners, blinded to each other's measurements, assessed PF active ROM through a NWB goniometric technique and a FHRT on all subjects within the same test session. Intra- and interrater reliability values were calculated using an intraclass correlation coefficient (ICC2,1, ICC2,k) and 95% confidence intervals. Standard error of measurement (SEM) and minimal detectable change (MDC) were recorded for each method. 43 healthy participants (mean ± SD, age: 22.7 ± 1.7 years, height: 1.7 ± 0.1 m, mass: 77.8 ± 17.2 kg) completed testing procedures. The within-session intrarater reliability (ICC2,1) estimates were observed for goniometry (right: 0.96, left: 0.95 - 0.97) and FHRT (right: 0.99, left: 0.99), as well as the interrater reliability (ICC2,k) of goniometry (right: 0.79, left: 0.79) and FHRT (right: 0.79, left: 0.87). Goniometry SEM (3.3 - 3.6 °) and MDC (9.2 - 9.8 °) were observed, in addition to FHRT SEM (0.6 cm) and MDC (1.6 - 1.7 cm). A weak correlation was found between FHRT and goniometric measurements (r = -0.03 - 0.13). The FHRT was found to have good to excellent intra- and interrater reliability, similar to goniometric measurement. The lack of agreement between these measurements requires further exploration of a WB assessment of ankle PF active ROM. 2b.

Using a Simulation of a Frustrated Faculty Member During Department Chair Searches: A Proof of Concept Project.

Vitae reviews, interviews, presentations, and reference checks are typical components of searches used to screen and select new department chairs/heads, but these strategies may fail to identify leaders who can communicate effectively with faculty in common, tense situations. Between May 2015 and November 2016, the authors piloted simulation scenarios in four department chair searches at Penn State College of Medicine/Penn State Health to assess candidates' skill at handling common, challenging situations with faculty members. In the scenarios, a frustrated faculty member complains that he/she has too little time for academic pursuits. Candidates were provided the scenario approximately two weeks in advance. They were asked to explain their goals prior to the 10-minute simulation, do the simulation, and then debrief with the search committees, who observed the interactions. Approximately two-thirds (20/29; 69.0%) of candidates were judged to have successfully passed the simulation and were ultimately advanced. In most cases, the simulations revealed wide variation in candidates' style, substance, and even underlying values that were not otherwise identified through the other parts of the recruitment and screening process. In some cases, candidates who performed well during group and individual interviews did poorly during simulations. The authors will build a larger pool of simulation scenario cases, create a rubric, and formally measure interrater reliability. They will study whether the strategy successfully identifies chairs who will be skilled at navigating common faculty challenges, and if this skill results in greater faculty satisfaction, engagement, and retention.