Level Of Inter-rater Agreement Research Articles

Interrater reliability of quantitative ultrasound measures of muscle in critically ill patients.

Quantitative neuromuscular ultrasound is increasingly used to study muscle pathology and changes in muscle mass during critical illness. Advantages of ultrasound are high axial resolution, low procedural risks, no ionizing radiation, and ease of use early in the course of disease. However, ultrasound is known to be an operator dependent imaging modality and the intensive care unit setting poses additional challenges to obtaining reliable measurements. There is limited evidence validating the feasibility and reliability of its application in this setting. To conduct a standardized protocol for measuring muscle linear depth and cross-sectional area in critically ill populations with a high degree of interrater agreement and feasibility. Prospective observational cohort study of interrater reliability. Medical intensive care unit at an academic medical center and a level one trauma and burn center. Fifteen critically ill patients were evaluated using a standardized ultrasound protocol measuring total elbow flexor, knee extensor, and tibialis anterior depth, as well as rectus femoris cross-sectional area. Each site was independently scanned by two investigators. Reliability of measurements between observers was determined by calculating intraclass correlation coefficients (ICCs) using a two-way random effects model and absolute agreement. An ICC > 0.75 was considered good and >0.90 was considered excellent. In critically ill patients, interrater reliability of linear depth measured at elbow flexor, knee extensor, tibialis anterior, and cross-sectional area of rectus femoris sites was good to excellent with ICC between 0.87 (0.54-0.97) and 0.99 (0.97-1.00). Interrater reliability was improved by creating a summary index of measures resulting in an ICC of 0.99 (0.98-1.00). Feasibility, as definite by the percentage of each measure that was obtainable, ranged from 75%-100%. Using a standardized protocol, ultrasound measures obtained in critically ill patients demonstrated high levels of interrater agreement with good to excellent feasibility.

Evaluating Surgical Complexity of Endoscopic Hysterectomy: An Inter-rater and Intra-rater Agreement Study of Novel Scoring Tool

Evaluate inter-rater and intrarater reliability of a novel scoring tool for surgical complexity assessment of endoscopic hysterectomy. Validation study. Academic medical center. Total of 11 academic obstetrician-gynecologists with varying years of postresidency training, clinical practice, and surgical volumes. Application of a novel scoring tool to evaluate surgical complexity of 150 sets of images taken in a standardized fashion before surgical intervention (global pelvis, anterior cul-de-sac, posterior cul-de-sac, right adnexa, left adnexa). Using only these images, raters were asked to assess uterine size, number, and location of myomas, adnexal and uterine mobility, need for ureterolysis, and presence of endometriosis or adhesions in relevant locations. Surgical complexity was staged on a scale of 1 to 4 (low to high complexity). Number of postresidency years in practice for participating surgeons ranged from 2 to 15, with an average of 8 years. A total of 8 obstetrician-gynecologists (72.7%) had completed a fellowship in minimally invasive gynecologic surgery. Six (54.6%) reported an annual volume of >50 hysterectomies. Raters reported that 95.4% of the images were satisfactory for assessment. Of the 150 sets of images, most were found to be stage 1 to 2 complexity (stage 1: 23.8%, stage 2: 41.6%, stage 3: 32.8%, stage 4: 1.8%). The level of inter-rater agreement regarding stage 1 to 2 vs 3 to 4 complexity was moderate (κ=0.49; 95%confidence interval [CI], 0.42-0.56). Moderate inter-rater agreement was also found between surgeon raters with an annual hysterectomy volume >50 (κ=0.49; 95%CI, 0.40-0.57) as well as between surgeon raters with fellowship experience (κ=0.50; 95%CI, 0.42-0.58). Intrarater agreement averaged 80.2% among all raters and also achieved moderate agreement (mean weighted κ=0.53; range, 0.38-0.72). This novel scoring tool uses clinical assessment of preintervention anatomic images to stratify the surgical complexity of endoscopic hysterectomy. It has rich and comprehensive evaluation capabilities and achieved moderate inter-rater and intrarater agreement. The tool can be used in conjunction with or instead of traditional markers of surgical complexity such as uterine weight, estimated blood loss, and operative time.

Parenchymal echotexture changes as a predictor of viability in testicular torsion.

Preoperative test that can predict the salvageability of the torsed testis may add essential information to the surgeon managing testicular torsion (TT), this can assist with patients' and parents' expectations, particularly with nonviable testes. We aimed to examine if parenchymal echotexture changes in preoperative ultrasound can predict irreversible hemorrhagic necrosis. Preoperative ultrasound studies of 154 patients with TT were reviewed by 3 raters (2 radiologists and 1 urologist). The raters were asked to categorize the affected testicular parenchymal echotexture into one of the following categories: (1) normal (identical to the contra-lateral testis), (2) homogenous hypoechoic, or (3) focal heterogeneous echotexture. Testis non-viability was defined macroscopically during surgical exploration and correlated with the US results. Sensitivity, specificity, and positive and negative predicting values of the proposed diagnostic test were calculated. Cohen's kappa coefficient was used to determine inter-rater agreement. A total of 54/154 patients had a nonviable testis. Mean of 59.5% cases was classified as category 1, 27.3% cases as category 2, and 13.2% cases as category 3. Testicular necrosis was 12%, 34%, and 92% in each category, respectively. Category 3 classified non-viability with a mean specificity of 99.3% and with a high inter-rater agreement level (Cohen's kappa coefficient of 0.830). Mean positive predictive value of 97% and mean negative predictive value of 74.3%. The mean sensitivity of this test however was low 39.7%. Ultrasound finding of focal parenchymal echotexture heterogeneous changes is highly specific although not sensitive, for nonviable testis. The presence of this finding reassures non-viability in over 99%.

Translation, Cross-Cultural Adaptation, and Validation of the Portuguese Version of the Rotterdam Elderly Pain Observation Scale

Introduction: This study reports on the translation, cultural adaptation, and validation of a Portuguese version of the Rotterdam Elderly Pain Observation Scale (REPOS), a Dutch scale to assess pain in patients who cannot communicate, with or without dementia. Methods: This is a multicenter study in pain and neurological units involving Brazil (clinical phase) and the Netherlands (training phase). We performed a retrospective cross-sectional, 2-staged analysis, translating and culturally adapting the REPOS to a Portuguese version (REPOS-P) and evaluating its psychometric properties. Eight health professionals were trained to observe patients with low back pain. REPOS consists of 10 behavioral items scored as present or absent after a 2-min observation. The REPOS score of ≥3 in combination with the Numerical Rating Scale (NRS) of ≥4 indicated pain. The Content Validity Index (CVI) in all items and instructions showed CVI values at their maximum. According to the higher correlation coefficient found between NRS and REPOS-P, it may be suggested that there was an adequate convergent validity. Results: The REPOS-P was administered to 80 patients with a mean age of 60 years (SD 11.5). Cronbach’s alpha coefficient showed a moderate internal consistency of REPOS-P (α = 0.62), which is compatible with the original study of REPOS. All health professionals reached high levels of interrater agreement within a median of 10 weeks of training, assuring reproducibility. Cohen’s kappa was 0.96 (SD 0.03), and the intraclass correlation coefficient was 0.98 (SD 0.02), showing high reliability of REPOS-P scores between the trainer (researcher) and the trainees (healthcare professionals). The Pearson correlation coefficient was 0.95 (95% confidence interval 0.94–0.97), showing a significant correlation between the total scores of REPOS-P and NRS. Conclusion: The REPOS-P was a valuable scale for assessing elderly patients with low back pain by different healthcare professionals. Short application time, ease of use, clear instructions, and the brief training required for application were essential characteristics of REPOS-P.

Assessment of catatonia and inter-rater reliability of three instruments: a descriptive study

BackgroundClinical assessment of catatonia includes the use of diagnostic systems, such as the Diagnostic and Statistical Manual, Fifth Edition (DSM-5) and the International Classification of Disease, Tenth Revision (ICD-10), or screening tools such as the Bush Francis Catatonia Screening Instrument (BFCSI)/Bush Francis Catatonia Rating Scale (BFCRS) and the Braunig Catatonia Rating Scale. In this study, we describe the inter-rater reliability (IRR), utilizing the BFCSI, BFCRS, and DSM-5 to screen for catatonia.MethodsData from 10 participants recruited as part of a larger prevalence study (of 135 participants) were used to determine the IRR by five assessors after they were trained in the application of the 14-item BFCSI, 23-item BFCRS, and DSM-5 to assess catatonia in new admissions. Krippendorff’s α was used to compute the IRR, and Spearman’s correlation was used to determine the concordance between screening tools. The study site was a 35-bed acute mental health unit in Dora Nginza Hospital, Nelson Mandela Bay Metro. Participants were mostly involuntary admissions under the Mental Health Care Act of 2002 and between the ages of 13 and 65 years.ResultsOf the 135 participants, 16 (11.9%) had catatonia. The majority (92 [68.1%]) were between 16 and 35 years old, with 126 (93.3%) of them being Black and 89 (66.4%) being male. The BFCRS (complete 23-item scale) had the greatest level of inter-rater agreement with α = 0.798, while the DSM-5 had the lowest level of inter-rater agreement with α = 0.565. The highest correlation coefficients were observed between the BFCRS and the BFCSI.ConclusionThe prevalence rate of catatonia was 11.9%, with the BFCSI and BFCRS showing the highest pick-up rate and a high IRR with high correlation coefficients, while the DSM-5 had deficiencies in screening for catatonia with low IRR and the lowest correlation with the other two tools.

Evaluating Surgical Complexity of Endoscopic Hysterectomy: An Inter-Rater Agreement Study for Novel Scoring Tool

<h3>Study Objective</h3> Evaluate inter-rater reliability of a novel scoring tool for surgical complexity assessment of endoscopic hysterectomy. <h3>Design</h3> Validation study. <h3>Setting</h3> Academic medical center. <h3>Patients or Participants</h3> 11 academic OB/GYNS with varying years of post-training practice and surgical volumes <h3>Interventions</h3> Application of a novel scoring tool to evaluate surgical complexity of 150 sets of surgical images taken in a standardized fashion (global pelvis, anterior cul-de-sac, posterior cul-de-sac, right adnexa, left adnexa). Surgical complexity was staged on a scale of I-IV (low to high complexity). Raters were asked to assess uterine size, number and location of fibroids, adnexal and uterine mobility, need for ureterolysis, and presence of endometriosis or adhesions. <h3>Measurements and Main Results</h3> Number of post-residency years in practice for participating surgeons ranged from 2 – 15, with an average of 8 years. 8 (72.73%) OB/GYNs had completed a fellowship in minimally invasive gynecologic surgery. 6 (54.55%) reported annual volume of hysterectomy >50 cases. Raters reported 95.4% of images were satisfactory for assessment. Of the 150 sets of images, most were found to be Stage I-II complexity (Stage I: 46.4%, Stage II: 30.6%, Stage III: 16.4%, Stage IV: 6.6%). Level of inter-rater agreement regarding Stage I-II vs. III-IV complexity was moderate (k=0.493, CI [0.43-0.56]). Moderate inter-rater agreement persisted between surgeon raters with annual hysterectomy volume > 50 (k=0.484, CI [0.42-0.56]) as well as between surgeon raters with fellowship experience (k=0.498, CI [0.434-0.57]). <h3>Conclusion</h3> Current methods for assessment of surgical complexity for clinical and research purposes rely on seemingly objective markers like uterine weight, estimated blood loss, and operative time, which can be limited or flawed. In contrast, this novel scoring tool has rich and comprehensive evaluation capabilities and achieved moderate inter-rater agreement. The complexity scoring tool can be used in conjunction with or instead of traditional methods for assessment of surgical complexity of endoscopic hysterectomy.

Inter-rater reliability of occupational exposure assessment in a case-control study of female breast cancer

The objective of this paper was to estimate the inter-rater reliability of expert assessments of occupational exposures. An inter-rater reliability sub-study was conducted within a population-based case-control study of postmenopausal breast cancer. Detailed information on lifetime occupational histories was obtained from participants and two industrial hygienists assigned exposures to 185 jobs using a checklist of 293 agents. Experts rated exposure for each job–agent combination according to exposure status (unexposed/exposed), confidence that the exposure occurred (possible/probable/definite), intensity (low/medium/high), and frequency (% time per week). The statistical unit of observation was each job–agent assessment (185 jobs × 293 agents = 54,205 assessments per expert). Crude agreement, Gwet AC1/2 statistics, and Cohen’s Kappa were used to estimate inter-rater agreement for confidence and intensity; for frequency, the intra-class correlation coefficient (ICC) was used. The majority of job–agent combinations were evaluated by the two experts to be not exposed (crude agreement >98% of decisions). The degree of agreement between the experts for the confidence of exposure status was Gwet AC1/2 = 0.99 (95% CI: 0.99–0.99), and for intensity, a Gwet AC2 = 0.99 (95% CI: 0.99–0.99). For frequency, an ICC of 0.31 (95% CI: 0.26–0.35) was found. A sub-analysis restricted to job–agent combinations for which the two experts agreed on exposure status revealed a moderate agreement for confidence of exposure (Gwet AC2 = 0.66) and high agreement for intensity (Gwet AC2 = 0.96). For frequency, the ICC was 0.52 (95% CI: 0.47–0.57). A high level of inter-rater agreement was found for identifying exposures and for coding intensity, but agreement was lower for the coding of frequency of exposure.

App-Based Feedback for Rehabilitation Exercise Correction in Patients With Knee or Hip Osteoarthritis: Prospective Cohort Study.

BackgroundThe use of digital therapeutic solutions for rehabilitation of conditions such as osteoarthritis provides scalable access to rehabilitation. Few validated technological solutions exist to ensure supervision of users while they exercise at home. Motion Coach (Kaia Health GmbH) provides audiovisual feedback on exercise execution in real time on conventional smartphones.ObjectiveWe hypothesized that the interrater agreement between physiotherapists and Motion Coach would be noninferior to physiotherapists’ interrater agreement for exercise evaluations in a cohort with osteoarthritis.MethodsPatients diagnosed with osteoarthritis of the knee or hip were recruited at a university hospital to perform a set of 6 exercises. Agreement between Motion Coach and 2 physiotherapists’ corrections for segments of the exercises were compared using Cohen κ and percent agreement.ResultsParticipants (n=24) were enrolled and evaluated. There were no significant differences between interrater agreements (Motion Coach app vs physiotherapists: percent agreement 0.828; physiotherapist 1 vs physiotherapist 2: percent agreement 0.833; P<.001). Age (70 years or under, older than 70 years), gender (male, female), or BMI (30 kg/m2 or under, greater than 30 kg/m2) subgroup analysis revealed no detectable difference in interrater agreement. There was no detectable difference in levels of interrater agreement between Motion Coach vs physiotherapists and between physiotherapists in any of the 6 exercises.ConclusionsThe results demonstrated that Motion Coach is noninferior to physiotherapist evaluations. Interrater agreement did not differ between 2 physiotherapists or between physiotherapists and the Motion Coach app. This finding was valid for all investigated exercises and subgroups. These results confirm the ability of Motion Coach to detect user form during exercise and provide valid feedback to users with musculoskeletal disorders.

Validity and reliability of the RAC adult infection risk scale: A new instrument to measure healthcare‐associated infection risk

Healthcare-associated infections represent a public health problem, and they have repercussions for patient safety. The aim of this study was to determine the psychometric properties of the Rodríguez-Almeida-Cañon (RAC) adult infection risk scale, focusing on the construct and predictive validity and reliability. The study enrolled 278 patients at a large hospital in southern Brazil. The research process involved the following three phases: construct validation, assessing predictive validity, and assessing reliability. Confirmatory factor analysis showed a good fit using a two-factor model with 15 items. The logistic regression analysis showed an association between the scale score and prediction of developing healthcare-associated infections (odds ratio: 1.18; 95% confidence interval: 1.08-1.28). The Cronbach's alpha was 0.72 for intrinsic factors subscale and 0.71 for extrinsic factors subscale. A high level of inter-rater agreement (intraclass correlation coefficient ≥0.97) was found for both subscales. The Bland and Altman method showed narrow agreement limits, demonstrating good agreement between evaluators. The findings of this study showed that the RAC adult infection risk scale is a new, reliable, and psychometrically valid instrument to assess healthcare-associated infections risk. Future research using this scale may lead to a better understanding of the healthcare-associated infections risk and assist health professionals in decision-making for interventions to improve patient safety.

Utility of the Emergency Severity Index by Accuracy of Interrater Agreement by Expert Triage Nurses in a Simulated Scenario in Japan: A Randomized Controlled Trial

ObjectiveThe Emergency Severity Index (ESI) is a highly reliable and valid triage scale that is widely used in emergency departments in not only English language regions but also other countries. The Japan Triage and Acuity Scale (JTAS) is frequently used for emergency patients, and the ESI has not been evaluated against the JTAS in Japan. This study aimed to examine the decision accuracy of the ESI for simulated clinical scenarios among nursing specialists in Japan compared with the JTAS. MethodA parallel group randomized trial was conducted. In total, 23 JTAS–trained triage nurses from 10 Japanese emergency departments were randomly assigned to the ESI or the JTAS group. Nurses independently assigned triage categories to 80 emergency cases for the assessment of interrater agreement. ResultsInterrater agreement between the expert and triage nurses was κ = 0.82 (excellent) in the ESI group and κ = 0.74 (substantial) in the JTAS group. In addition, interrater agreement by acuity was level 2 = 0.42 (moderate) in the ESI group and level 2 = 0.31 (fair) in the JTAS group. Interrater agreement for triage decisions was classified in a higher category in the ESI group than in the JTAS Scale group at level 2. Triage decisions based on the ESI in Japan maintained the same level of interrater agreement and sensitivity as those in other countries. ConclusionThese findings suggest that the ESI can be introduced in Japan, despite its different emergency medical background compared with other countries.

A Pilot Study Investigating the Effect of the Supervision-Questioning-Feedback Model of Supervision on Stimulating Critical Thinking in Speech-Language Pathology Graduate Students

Purpose The purpose of this study was to investigate the effect of the supervision-questioning-feedback (SQF) model of supervision on critical thinking in graduate students studying speech-language pathology. The researchers hypothesized that students who were provided with the SQF model of supervision would score higher than students who received the non-SQF (NSQF) style of supervision on the selected critical thinking measures. Method Seventeen out of 24 first semester graduate students in speech-language pathology completing their on-site university-based clinical practicum experience consented to participate in the study. Of the 17 participating first semester students, 9 were randomly assigned to 1 of 3 SQF trained supervisors, and the other 8 were randomly assigned to 1 of 2 NSQF trained supervisors for the duration of 1 semester. Additionally, 3 out of 24 fourth semester graduate students completing their off-site externship experience and their supervisors consented to participate in the study. Four additional study participants served as independent SQF-trained raters charged with the task of analyzing video recorded student-supervisor conferences to determine whether the SQF model of supervision was being implemented. Prior to and at the conclusion of the clinical experience, all participating students completed two measures of critical thinking: (1) California Critical Thinking Skills Test (CCTST) and (2) two Simucase® clinical simulations. At the conclusion of the clinical experience, seventeen out of 20 participating students (11/12 SQF students and 6/8 NSQF students) completed a post-survey rating their supervisory experience . Results For participating first semester students, there were no overall statistically significant differences between SQF and NSQF groups as measured by pre to post completion of (1) CCTST (p=.544) and (2) two Simucase® clinical simulations (p=.781). The 3 participating fourth semester students who received the SQF model of supervision also showed no statistically significant differences on pre to post completion of the (1) CCTST (p=.827) and (2) two Simucase® virtual cases (p=.879). Results from SQF ratings revealed variability in the implementation of the SQF model across supervisors with a moderate level of inter-rater agreement. Results from post surveys completed by students showed that students preferred the SQF model of supervision over the NSQF model (p=.044). Conclusion Results from this preliminary study indicated that the SQF model did not influence the overall outcomes on the selected critical thinking measures. Student preference for the SQF model may support existing evidence that learning clinicians want to be actively engaged in the supervisory process. There were several limitations to this study including the small sample size, variability in the implementation of the SQF model across supervisors, sensitivity of the selected critical thinking measures, and timing of post-intervention procedures. Further investigation of the effects of SQF on students’ critical thinking is warranted.

The Effects of Tablet-Based Electronic Grading on Airline Evaluator Performance

ABSTRACT Objective: This experimental study assessed the effects of an Apple iPad (Apple and iPad are trademarks of Apple Inc., registered in the U.S. and other countries) grading instrument on airline evaluator performance. Background: Extant research exploring the effects of grading formats on evaluator performance focused on pen and paper modalities. This research investigated how a contemporary iPad-based grading format affected airline evaluator performance. Method: Forty-five evaluators from a major U.S. airline graded a Boeing 767 (Boeing is a wordmark of The Boeing Company, registered in the U.S. and other countries) flight crew experiencing an engine failure during takeoff. Three grading formats were used: pen and paper, a paper-based equivalent of the iPad grading format designed to replicate the function of the iPad, and the iPad. The effects of the grading formats were measured using the following dependent variables: recorded technical criteria, recorded non-technical criteria, and correlation and inter-rater agreement with a referent score. Results: Members of the iPad paper-based equivalent and iPad groups recorded significantly more technical and non-technical criteria compared to participants using pen and paper (p < .001). Members of the iPad paper-based equivalent and iPad groups had significantly higher correlation (p = .01) and inter-rater agreement levels (p < .01) compared to participants using pen and paper. Conclusion: The results of the study failed to support the iPad as a superior grading format. Research outcomes suggested structured grading may be a more important factor in predicting airline evaluator performance than the type of grading format used.

Reproducibility analysis of multi-institutional paired expert annotations and radiomic features of the Ivy Glioblastoma Atlas Project (Ivy GAP) dataset.

The availability of radiographic magnetic resonance imaging (MRI) scans for the Ivy Glioblastoma Atlas Project (Ivy GAP) has opened up opportunities for development of radiomic markers for prognostic/predictive applications in glioblastoma (GBM). In this work, we address two critical challenges with regard to developing robust radiomic approaches: (a) the lack of availability of reliable segmentation labels for glioblastoma tumor sub-compartments (i.e., enhancing tumor, non-enhancing tumor core, peritumoral edematous/infiltrated tissue) and (b) identifying "reproducible" radiomic features that are robust to segmentation variability across readers/sites. From TCIA's Ivy GAP cohort, we obtained a paired set (n=31) of expert annotations approved by two board-certified neuroradiologists at the Hospital of the University of Pennsylvania (UPenn) and at Case Western Reserve University (CWRU). For these studies, we performed a reproducibility study that assessed the variability in (a) segmentation labels and (b) radiomic features, between these paired annotations. The radiomic variability was assessed on a comprehensive panel of 11700 radiomic features including intensity, volumetric, morphologic, histogram-based, and textural parameters, extracted for each of the paired sets of annotations. Our results demonstrated (a) a high level of inter-rater agreement (median value of DICE≥0.8 for all sub-compartments), and (b) ≈24% of the extracted radiomic features being highly correlated (based on Spearman's rank correlation coefficient) to annotation variations. These robust features largely belonged to morphology (describing shape characteristics), intensity (capturing intensity profile statistics), and COLLAGE (capturing heterogeneity in gradient orientations) feature families. We make publicly available on TCIA's Analysis Results Directory (https://doi.org/10.7937/9j41-7d44), the complete set of (a) multi-institutional expert annotations for the tumor sub-compartments, (b) 11700 radiomic features, and (c) the associated reproducibility meta-analysis. The annotations and the associated meta-data for Ivy GAP are released with the purpose of enabling researchers toward developing image-based biomarkers for prognostic/predictive applications in GBM.

Role of placenta accreta index in prediction of morbidly adherent placenta: A reliability study.

To estimate the level of interobserver agreement in the calculation of placenta accreta index (PAI) as well as to evaluate the accuracy of PAI in prediction of morbidly adherent placenta. This was a prospective study where 45 pregnant women (from 28 to 37 weeks of gestational age) with at least one previous Caesarean section and ultrasound-proven placenta previa were included. A known and previously published scoring system, the PAI, was evaluated independently by two radiologists and the cases were followed for the delivery and histopathology outcome. The accuracy of the PAI and the level of interrater agreement was analysed using cross-table analysis, intraclass correlation efficient and Cohen's kappa as statistical variables. Adherent placenta was found in 15 patients accounting for 33% of cases. The PAI showed nearly 90% sensitivity, specificity and the predictive values. Interrater agreement in calculation of PAI by the two radiologists was perfect with an intraclass correlation efficient of 0.959. An easy-to-use morbid adherent placenta score was also predicted to simplify the results of PAI, which showed moderate agreement (κ = 0.746). The PAI can be helpful in stratifying the individual risk of placental invasion above the baseline risk. The PAI-derived, simplified scoring system called morbid adherent placenta score can be used as a simple tool to interpret and convey the results of PAI.

Psychological trait inferences from women\u2019s clothing: human and machine prediction

People use clothing to make personality inferences about others, and these inferences steer social behaviors. The current work makes four contributions to the measurement and prediction of clothing-based person perception: first, we integrate published research and open-ended responses to identify common psychological inferences made from clothes (Study 1). We find that people use clothes to make inferences about happiness, sexual interest, intelligence, trustworthiness, and confidence. Second, we examine consensus (i.e., interrater agreement) for clothing-based inferences (Study 2). We observe that characteristics of the inferring observer contribute more to the drawn inferences than the observed clothes, which entails low to medium levels of interrater agreement. Third, the current work examines whether a computer vision model can use image properties (i.e., pixels alone) to replicate human inferences (Study 3). While our best model outperforms a single human rater, its absolute performance falls short of reliability conventions in psychological research. Finally, we introduce a large database of clothing images with psychological labels and demonstrate its use for exploration and replication of psychological research. The database consists of 5000 images of (western) women’s clothing items with psychological inferences annotated by 25 participants per clothing item.

Development of an observational tool to assess health coaching fidelity

ObjectiveThis study describes the development of the Health Coaching Index (HCI), an observational tool for assessing fidelity to implementing health coaching practical skills. MethodsInitial HCI items were developed, adapted following cognitive interviews, and refined during coding training. Participants (n = 42) were trainees who completed a National Board for Health and Wellness Coaching (NBHWC)-approved training program and coached a standardized patient. Interrater reliability for the HCI was determined by calculating interclass correlations from ten videos coded by three raters. Construct validity was evaluated from 42 recordings using Spearman’s Rho between HCI and Roter Interaction Analysis System (RIAS) codes. ResultsThe interclass correlation (ICC) for HCI total score was 0.81, considered an excellent level of inter-rater agreement. Some significant correlations between HCI and RIAS codes supported construct validity (e.g., patient activation: Rho = 0.32; empathy: Rho = 0.36). ConclusionThe HCI total score can reliably be used to assess fidelity to health coaching skills, and the HCI has construct validity similar to the RIAS as a measure of patient activation. Practice ImplicationsAdoption and further study of the HCI tool will allow for a more consistent implementation of health coaching skills, and may facilitate more robust training of health coaches for clinical practice and research.

Brief Montreal-Toulouse Language Assessment Battery: adaptation and content validity

BackgroundEvaluating patients in the acute phase of brain damage allows for the early detection of cognitive and linguistic impairments and the implementation of more effective interventions. However, few cross-cultural instruments are available for the bedside assessment of language abilities. The aim of this study was to develop a brief assessment instrument and evaluate its content validity.MethodsStimuli for the new assessment instrument were selected from the M1-Alpha and MTL-BR batteries (Stage 1). Sixty-five images were redesigned and analyzed by non-expert judges (Stage 2). This was followed by the analysis of expert judges (Stage 3), where nine speech pathologists with doctoral training and experience in aphasiology and/or linguistics evaluated the images, words, nonwords, and phrases for inclusion in the instrument. Two pilot studies (Stage 4) were then conducted in order to identify any remaining errors in the instrument and scoring instructions.ResultsSixty of the 65 figures examined by the judges achieved inter-rater agreement rates of at least 80%. Modifications were suggested to 22 images, which were therefore reanalyzed by the judges, who reached high levels of inter-rater agreement (AC1 = 0.98 [CI = 0.96–1]). New types of stimuli such as nonwords and irregular words were also inserted in the Brief Battery and favorably evaluated by the expert judges. Optional tasks were also developed for specific diagnostic situations. After the correction of errors detected in Stage 4, the final version of the instrument was obtained.ConclusionThis study confirmed the content validity of the Brief MTL-BR Battery. The method used in this investigation was effective and can be used in future studies to develop brief instruments based on preexisting assessment batteries.

Inter-Rater Agreement of Biofield Tuning: Testing a Novel Health Assessment Procedure.

Objectives: Practitioners of Biofield Tuning assess health status of their clients by detecting off-the-body biofield perturbations using tuning fork (TF) vibrations. This study tested inter-rater agreement (IRA) on location of these perturbations. Design: Three Biofield Tuning practitioners, in randomized order, identified locations of the 4-5 "strongest" perturbations along each of 4 sites for the same series of 10 research subjects. Setting/Location: An Integrative Health and Medicine Center in La Jolla, CA. Subjects: Adult volunteers with no serious current illness and no prior experience of a Biofield Tuning session. Interventions: Practitioners used an activated 174 Hz unweighted TF to "comb" the same four sites per subject, located on the left and right sides of the base of the spine and the heart. Outcome Measures: Practitioners identified and vocalized the distance from the body of perturbations along each site. Distances were recorded by a research assistant in the clinic room. No health information related to perturbation sites was discussed with the subjects. Results: Practitioners reported 6.3 ± 0.6 (mean ± standard deviation) perturbations per combed site per subject, with no significant difference among the raters. The overall level of IRA was low based initially on a first-pass, nonstatistical, analysis of results, with "agreement" defined within a tolerance of ±2 inches. In this approach agreement was 33%. More rigorous statistical analysis, including a statistical test using a Monte Carlo approach, strongly supported the conclusion of poor IRA. Conclusions: IRA was low despite attempts to balance the real-world practice of Biofield Tuning with the constraints of research. For example, while IRA necessitates multiple assessments of the same subject, no information exists as to whether an initial assessment may affect subsequent assessments. Our study exemplifies the challenges faced when attempting to fit interventions with incompletely understood procedures and mechanisms into conventional research designs.

Reliability and Similarity of the Stuttering Severity Instrument-Fourth Edition and a Global Severity Rating Scale

ABSTRACT Purpose: The present study compared reliability and similarity for two measures of stuttering severity: The Stuttering Severity Instrument – Fourth Edition (SSI-4) Final Severity Rating and a 5-point global severity rating (GSR) scale. Method: Twenty-four graduate students rated four videos of persons who stutter using either the SSI-4 procedures or a 5-point GSR scale. Intra-rater and inter-rater agreement values, and obtained severity ratings, were compared across the two methods. Results: The SSI-4 produced better intra-rater and inter-rater agreement, although neither measure produced an acceptable level of inter-rater agreement of 80%. Severity scores produced from the GSR scale were significantly higher compared to the SSI-4 Final Severity Ratings. Conclusions: The 5-point GSR scale used in the present study is not a sufficient replacement for the SSI-4 as a measure of stuttering severity. Rater agreement for both methods may benefit from training and the use of exemplars to standardized levels of stuttering severity.

Machine Learning to Analyze Single-Case Data: A Proof of Concept

Visual analysis is the most commonly used method for interpreting data from single-case designs, but levels of interrater agreement remain a concern. Although structured aids to visual analysis such as the dual-criteria (DC) method may increase interrater agreement, the accuracy of the analyses may still benefit from improvements. Thus, the purpose of our study was to (a) examine correspondence between visual analysis and models derived from different machine learning algorithms, and (b) compare the accuracy, Type I error rate and power of each of our models with those produced by the DC method. We trained our models on a previously published dataset and then conducted analyses on both nonsimulated and simulated graphs. All our models derived from machine learning algorithms matched the interpretation of the visual analysts more frequently than the DC method. Furthermore, the machine learning algorithms outperformed the DC method on accuracy, Type I error rate, and power. Our results support the somewhat unorthodox proposition that behavior analysts may use machine learning algorithms to supplement their visual analysis of single-case data, but more research is needed to examine the potential benefits and drawbacks of such an approach.