Assess Interrater Reliability Research Articles

Case detection delay in leprosy: Testing tool reliability and measurement consistency in Ethiopia, Mozambique, and Tanzania.

Case detection delay (CDD) in leprosy is defined as the period between the onset of the first signs and symptoms and the time of diagnosis. A tool, consisting of a questionnaire and a detailed guide for researchers, which includes photos of typical skin signs and notes on establishing the timing of events, was developed to determine this period of delay in months in recently diagnosed leprosy patients. The aims of the study were to determine the reliability and consistency of this CDD assessment tool. This study was conducted in Ethiopia, Mozambique and Tanzania. Two types of consistency were considered: over time (test-retest reliability) and across different researchers (interrater reliability). A CDD questionnaire was administered to 167 leprosy patients who were diagnosed within 6 months prior to their inclusion. One month later, the same or another researcher re-administered the CDD questionnaire to the same patients. Both test-retest and interrater reliability were assessed using the intraclass correlation coefficient (ICC), where a value greater than or equal to 0.7 is considered acceptable. In this study, 10 participants (6.0%) were under 15 years of age, and 56 (33.5%) were women. In the test-retest assessment, the mean CDD from the first and second interviews was 23.7 months (95% CI 14.4-34.8) and 24.0 months (95% CI 14.8-33.2), respectively. The ICC for test-retest reliability was 0.99 (95% CI 0.994-0.997). For the interrater reliability assessment, the first and second interviews revealed a mean CDD of 24.7 months (95% CI 18.2-31.1) and 24.6 months (95% CI 18.7-30.5), respectively, with an ICC of 0.90 (95% CI 0.85-0.94). A standard error of measurement of 0.46 months was found in the test-retest and 1.03 months in the interrater measurement. Most answers given by participants during the first and second interviews were matching (≥86%). Most non-matching answers were in the 0-2 month delay category (≥46%). The tool, including a questionnaire to determine the CDD of newly diagnosed leprosy patients, was validated in three African countries. The test-retest and interrater measurements demonstrated that the instrument is reliable and measures consistently. The tool can be used in routine leprosy programmes as well as in research settings.

Validity and Reliability of the Sri Lankan Version of the Patient-Centered Questionnaire-Infertility (PCQ-Infertility)

Background: Reproductive healthcare of the highest calibre should encompass not only safety and effectiveness but also a patient-centered approach. Patients demand patient-centered subfertility treatment in addition to efficient medical management. Offering patient-centered care is crucial for both patients and healthcare professionals. The absence of a suitable instrument for measuring patient-centeredness poses a significant challenge in evaluating this essential aspect of care in Sri Lanka. The present study resulted in a culturally adapted and validated tool in Sinhala language to measure patient-centredness of care received by female partners of subfertile couples. Methods: The Patient Centredness Questionnaire-Infertility (PCQ-Infertility) was culturally adopted using a modified Delphi process, translated into Sinhala and assessed for its judgmental validity. The study was conducted September to December 2019. A cross-sectional study among 251 females attending the Subfertility Clinic at Teaching Hospital Mahamodara assessed the factor structure through exploratory factor analysis. The factor structure was confirmed with another cross-sectional study at the same setting among 250 females through confirmatory factor analysis. Test re-test reliability, internal consistency and inter-rater reliability were also assessed. Results: The validated PCQ-Infertility had 44 items with a seven-factor structure identified after exploratory factor analysis and confirmatory factor analysis with a combination of acceptable fit indices (SRMR of 0.079, CFI 0.862 and RMSEA 0.069). A Cronbach’s alpha value 0.761 assured high internal consistency. Test- retest and inter-rater reliability assessment gave satisfactory results with values exceeding 0.7 for all domains. Conclusion: Modified PCQ-Infertility is a valid and reliable instrument which can be used to assess patient centredness care received of subfertile females in Sri Lanka.

Simulation-Based Cardiopulmonary Examination Skills Checklist: Assessment of Inter-rater Reliability

Purpose: Experiential learning using high-fidelity simulation is a valuable tool for teaching Doctor of Physical Therapy (DPT) students the cardiopulmonary examination. There are currently no standardized assessment tools to evaluate student performance and provide feedback on these skills. The cardiovascular and pulmonary section of the American Physical Therapy Association recently published recommended competency levels for entry-level physical therapists including examination skills. The primary aim of this study was to create and assess inter-rater reliability of a cardiopulmonary examination skills checklist that aligns with these recommendations and can be used in the simulation environment. Methods: A skills checklist with 6 sections was developed to assess student performance on the cardiopulmonary examination within a high-fidelity simulation experience. Training videos on the cardiopulmonary examination and checklist use were created and given to 3 graders. Seventy-three DPT students performed the cardiopulmonary examination within simulation while being recorded. Three graders then assessed the same 25 randomly selected videos using the checklist. Inter-rater reliability, item % agreement and student performance were analyzed. Results: The overall interclass correlation coefficient (ICC) of the checklist was calculated to be 0.858, demonstrating strong agreement between graders. The ICC for individualized checklist sections ranged from 0.659 to 0.960, demonstrating good to near perfect agreement. The average percent correct achieved by students was 80.48 (6.91%), R = 65% to 93.33%. There were >80% agreement for 55/60 items and >90% agreement for 44/60 items. Conclusion: The checklist provided an effective tool to assess student performance on the cardiopulmonary examination and demonstrated consistent scoring between graders.

Interrater Agreement of Height Assessment by Rigid Proctoscopy/Rectoscopy for Rectal Carcinoma.

Some guidelines for rectal carcinoma consider 12 cm, measured by rigid endoscopy, to be the cutoff tumor height for optional neoadjuvant chemoradiation therapy. Measuring differences of only a few centimeters may predetermine the choice of further therapy. However, rigid endoscopy may exhibit similar operator dependence to most other clinical examination methods. Evaluation of concordance of rigid rectoscopic tumor height measurements performed by 4 experienced examiners, 2 measuring with patients in the lithotomy position and 2 in the left lateral position. Assessment of tumor palpability and distance of the anal verge to the anocutaneous line were also evaluated. This study used a prospective observational design. This study was conducted at an academic teaching hospital that is a referral center for colorectal surgery. There were 50 patients, of whom 35 were men (70%). The median age was 72.5 years (53-88 years). Interrater agreement of tumor height assessment and tumor height of less than or greater than the 12-cm height limit. With an intraclass correlation coefficient of 0.947 (95% CI, 0.918-0.967, p < 0.001), interrater reliability of tumor height assessment was statistically rated "excellent." Despite this, in 26% of patients, there was no agreement regarding the allocation of the tumor <12- or >12-cm height limit. Furthermore, there was also considerable disagreement concerning tumor palpability and the distance of the anal verge to the anocutaneous line. Patient positioning was not found to influence results. Single-center study. Rigid rectal endoscopy may not be a sound pivotal basis for the consideration of optional chemoradiation therapy in rectal carcinoma. Application of a universally valid height limit ignores biological variability in body frame, gender, and acquired pelvic descent. Eligibility for neoadjuvant therapy should not rely on height measurements alone. Uniform MRI or CT imaging protocols, based on agreed upon terminology, including factors such as tumor height relative to the pelvic frame and peritoneal reflection, may be an important diagnostic addition to such a decision. See Video Abstract .Clinical trial registration: DRKS00012758 (German National Study Registry), ST-D 406 (German Cancer Society). ANTECEDENTES:Algunas guías para el carcinoma de recto consideran que 12 cm, medidos mediante endoscopia rígida, es la altura de corte del tumor para la quimiorradiación neoadyuvante opcional. Por lo tanto, una diferencia de medición de sólo unos pocos centímetros puede predeterminar la elección de una terapia adicional. Sin embargo, la endoscopia rígida puede presentar una dependencia del operador similar a la de la mayoría de los demás métodos de examen clínico.OBJETIVOS:Evaluación de la concordancia de las mediciones de la altura del tumor rectoscópico rígido realizadas por cuatro examinadores experimentados, dos en litotomía y dos en posición lateral izquierda. También se evaluó la evaluación de la palpabilidad del tumor y la distancia del borde anal a la línea anocutánea.DISEÑO:Estudio observacional prospectivo.LUGAR:Hospital universitario, centro de referencia para cirugía colorrectal.PACIENTES:50 pacientes, 35 varones (70%), mediana de edad 72,5 años (53-88 años).PRINCIPALES MEDIDAS DE RESULTADOS:Acuerdo entre evaluadores en la evaluación de la altura del tumor y la asignación del tumor por debajo o más allá del límite de altura de 12 cm.RESULTADOS:Con un coeficiente de correlación intraclase de 0,947 (IC del 95%: 0,918-0,967, p < 0,001), la confiabilidad entre evaluadores de la evaluación de la altura del tumor se calificó estadísticamente como "excelente". A pesar de esto, en el 26% de los pacientes no hubo acuerdo sobre la asignación del tumor por debajo o por encima del límite de 12 cm de altura. Además, también hubo un considerable desacuerdo con respecto a la palpabilidad del tumor y la distancia del borde anal a la línea anocutánea. No se encontró que la posición del paciente influyera en los resultados.LIMITACIONES:Estudio unicéntrico.CONCLUSIONES:La endoscopia rectal rígida puede no ser una base sólida y fundamental para considerar la quimiorradiación opcional en el carcinoma de recto. La aplicación de un límite de altura universalmente válido obviamente ignora la variabilidad biológica en la constitución corporal, el género y el descenso pélvico adquirido. La elegibilidad para la terapia neoadyuvante no debe depender únicamente de las mediciones de altura. Los protocolos uniformes de imágenes por resonancia magnética o tomografía computarizada, basados en una terminología acordada, incluidos factores como la altura del tumor en relación con la estructura pélvica y la reflexión peritoneal, pueden ser una adición diagnóstica importante para tal decisión. (Traducción-Yesenia Rojas-Khalil )Clinical trial registration: DRKS00012758 (German National Study Registry), ST-D 406 (German Cancer Society).

Using computer vision libraries to streamline nuclei quantification: A validation study

Valid analyses of cell function-based experiments depend on accuracy and reproducibility of the experimental protocol, often requiring normalization to cell count or other measures. The process of counting cells manually is laborious and susceptible to human error. Although cell counting programs exist, they are not preferred by all users, may be cost-prohibitive, and validation is minimal. Therefore, we aimed to validate an object visualization effciency tool that we developed to provide users with an alternative means of automated cell counting for the very specific, necessary, and widely performed task: quantifying nuclei to assess cell count in images. We hypothesized strong agreement between manual and automated counts. To ensure validation across cell densities, C2C12 myoblasts were seeded at 7.3x104 cells per cm2, proliferated for various timeframes (24, 48, and 72 hours; 6 wells across 2 plates per time point), and fixed with ice-cold methanol for 10 minutes. The nuclei were then stained using the fluorescent DNA stain, 4’,6-diamidino-2-phenylindole (DAPI), and 5 images per well were captured. The number of nuclei in each image were manually quantified by 2 separate raters (MC1 and MC2) using ImageJ, and with our automated program developed using computer vision libraries in Python (AI). To minimize risk for observer bias, AI counts and counts completed by each rater were kept separately until all 3 counts for each image were complete. To validate the program, we assessed interrater reliability between 1) MC1 and MC2 to ensure manual count agreement and 2) the average of MC1 and MC2 (MCavg) with AI to assess agreement between manual and AI counts. Spearman’s Rho (rs) was utilized to assess the strength of the association between counts across all images and separately for each quartile (Q) where Q1 had the fewest and Q4 had the most nuclei by manual counts. Nuclei per image ranged from 44 to 1166. Interrater reliability was nearly perfect between MC1 and MC2 (ICC=0.999) and between MCavg and AI (ICC=0.991). A significant, strong correlation was observed between AI and MCavg (rs=0.998, p&lt;0.001). This strong correlation between persisted regardless of cell density (MCavg vs AI: Q1: rs=.998, p&lt;0.001; Q2: rs=0.999, p&lt;0.001; Q3: rs=1.000, p&lt;0.001; Q4: rs=1.000, p&lt;0.001). The results indicate strong agreement between nuclei counts obtained manually and using our automated object recognition program, which remained consistent regardless of cell quantity, demonstrating the validity of this program. Therefore, the automated object recognition program analyzed in the present study can be considered an accurate, rapid, user-friendly, and inexpensive resource for quantifying nuclei to assess cell count across a range of cell densities. Use of this software will save the user valuable time during the critically important but often time-consuming task of quantifying nuclei. Danielle Levitt's start-up funding through Texas Tech University. This is the full abstract presented at the American Physiology Summit 2024 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.

Interrater Agreement of BT-RADS for Evaluation of Follow-Up MRI in Treated Primary Brain Tumor Patients.

The Brain Tumor Reporting and Data System (BT-RADS) is a structured radiology reporting algorithm that was introduced to provide uniformity in post-treatment primary brain tumor follow-up and reporting, but its interrater reliability (IRR) assessment has not been widely studied. Our goal is to evaluate the IRR among neuroradiologists and radiology residents in the use of BT-RADS. This retrospective study reviewed 103 consecutive MR studies in 98 adult patients previously diagnosed with and treated for primary brain tumor (January 2019 to February 2019). Six readers with varied experience (4 neuroradiologists and 2 radiology residents) independently evaluated each case and assigned a BT-RADS score. Readers were blinded to the original score reports and the reports from other readers. Cases in which at least one neuroradiologist scored differently were subjected to consensus scoring. After the study, a post-hoc reference score was also assigned by 2 readers using future imaging and clinical information previously unavailable to readers. The interrater reliabilities were assessed using Gwet's AC2 index with ordinal weights and percent agreement. Of the 98 patients evaluated (median age, 53 years; interquartile range, 41-66 years), 53% were males. The most common tumor type was astrocytoma (77%) of which 56% were grade 4 glioblastoma. Gwet's index for interrater reliability among all six readers was 0.83 (95% CI: 0.78, 0.87). The Gwet's index for the neuroradiologists' group (0.84 [95% CI: 0.79, 0.89]) was not statistically different from that for the residents' group (0.79 [95% CI: 0.72, 0.86]) (χ2 = 0.85; p = 0.36). All four neuroradiologists agreed on the same BT-RADS score in 57 of the 103 studies, three neuroradiologists agreed in 21 of the 103 studies, and two neuroradiologists agreed in 21 of the 103 studies. Percent agreement between neuroradiologist blinded scores and post-hoc reference scores ranged from 41%-52%. A very good interrater agreement was found when tumor reports were interpreted by independent blinded readers using BT-RADS criteria. Further study is needed to determine if this high overall agreement can translate into greater consistency in clinical care. BI-RADS = Breast Imaging Reporting and Data System; BT-RADS = Brain Tumor Reporting and Data System; IQR = interquartile range; IRR = interrater reliability; NI-RADS = Neck Imaging Reporting and Data System.

Pressure pain mapping of equine distal joints: feasibility and reliability.

Osteoarthritis is a prevalent degenerative joint disease initiating chronic pain and lameness in horses. While several objective gait analysis systems have been developed and validated to quantify lameness severity in horses, methods to evaluate whether peripheral sensitization contributes to the pain experienced are missing. To evaluate whether periarticular pressure pain mapping could be proposed as an auxiliary assessment tool in horses. Specific aims were to evaluate the feasibility and intra- and inter-rater reliability of pressure pain thresholds (PPT) determination at sites overlying the distal thoracic limb joints of clinically healthy horses. Prospective, randomized validation study. For feasibility assessment, PPT were measured with a hand-held digital algometer at six periarticular landmarks (2 sites per joint, 3 joints) bilaterally on the distal thoracic limb of 40 healthy horses (20 warmblood and 20 Freiberger). The joints tested were the metacarpophalangeal, on the latero-palmar and dorsal aspects (L-MCP and D-MCP), the proximal interphalangeal, on the dorsal and palmar aspect (D-PIP and P-PIP) and the distal interphalangeal, on the dorsal and lateral aspect (D-DIP and L-DIP). A feasibility score, ranging from 0 to 5, was attributed to each testing session. For intra- and inter-rater reliability assessment, L-MCP and D-MCP were selected to be tested again at 2 weeks intervals in 20 out of the 40 horses. Data were analyzed using a mixed-effect linear model to test differences in threshold per site and limb. Intra- and inter-rater correlation was calculated. Bland-Altman plots were performed to evaluate the variability of the measures. The procedure was considered feasible (score <2) in 95% of horses (95% CI 88%-100%). Overall, median [interquartile range (IQR)] PPT was 9.4 (7.5-11.3) N. No significant side differences were found. P-PIP and D-DIP recorded significantly lower PPT (p < 0.001 and p = 0.002, respectively) than L-MCP. Median (IQR) were 9.9 (7.3-12.4) N, 8.4 (6.1-10.5) N and 9.0 (7.4-10.6) N for L-MCP, P-PIP and D-DIP, respectively. The intra-rater agreement was 0.68 (95% CI 0.35-0.86) for L-MCP, and 0.50 (95% CI 0.08-0.76) for D-MCP. Inter-rater agreement was 0.85 (95% CI 0.66-0.94) for L-MCP and 0.81 (0.57, 0.92) for D-MCP. Evaluation of feasibility was performed only for distal thoracic limbs joints; no data are provided for hind limbs or proximal joints. Only warmblood and Freiberger horses were included. Intra- and inter-rater reliability assessments were performed exclusively on data collected at the MCP joint. Pressure pain mapping of distal thoracic limb joints was feasible in horses. Local sensitivity differed among sites and no side differences were noticed. Data collected from the MCP joint suggest highly variable, subject dependent intra-rater reliability, ranging from poor to good, and good to excellent inter-rater reliability. Further studies evaluating pathologic vs. healthy joints are needed before recommendations can be made about clinical usability and diagnostic validity.

Interrater reliability of the violence risk assessment checklist for youth: a case vignette study

BackgroundFacilities providing health- and social services for youth are commonly faced with the need for assessment and management of violent behavior. These providers often experience shortage of resources, compromising the feasibility of conducting comprehensive violence risk assessments. The Violence Risk Assessment Checklist for Youth aged 12–18 (V-RISK-Y) is a 12-item violence risk screening instrument developed to rapidly identify youth at high risk for violent behavior in situations requiring expedient evaluation of violence risk. The V-RISK-Y instrument was piloted in acute psychiatric units for youth, yielding positive results of predictive validity. The aim of the present study was to assess the interrater reliability of V-RISK-Y in child and adolescent psychiatric units and acute child protective services institutions.MethodsA case vignette study design was utilized to assess interrater reliability of V-RISK-Y. Staff at youth facilities (N = 163) in Norway and Sweden scored V-RISK-Y for three vignettes, and interrater reliability was assessed with the intraclass correlation coefficient (ICC).ResultsResults indicate good interrater reliability for the sum score and Low-Moderate-High risk level appraisal across staff from the different facilities and professions. For single items, interrater reliability ranged from poor to excellent.ConclusionsThis study is an important step in establishing the psychometric properties of V-RISK-Y. Findings support the structured professional judgment tradition the instrument is based on, with high agreement on the overall risk assessment. This study had a case vignette design, and the next step is to assess the reliability and validity of V-RISK-Y in naturalistic settings.

Constructing a descriptive sensory panel for tactile comfort evaluations: Effect of demographic variables and panel size

Sensory tests are essential components of comfort studies, and constructing a sensory panel is a crucial step of this process. In the current study, Total Hand (TH) scores of 41 woven fabrics were determined by assessors having different demographic characteristics. Assessment accuracy and inter-rater reliability of panel members were investigated via correlation and concordance analyses. Effect of demographic variables (gender, age, and level of expertise), panel size and sampling method on sensory evaluation results were discussed based on statistical measures. Findings of the study certified that sensory evaluations carried out with female panel members represent overall TH scores more successfully than males and assessment of female participants are in a better agreement with each other. It was also observed that assessment accuracy and inter-rater agreement improved with increasing levels of expertise. Investigations revealed that small panel sizes were sufficient to accurately evaluate fabric hand. Therefore, it was concluded that increasing the number of participants may not necessarily provide further information on comfort preferences and perceptions of potential customers.

The Intrarater and Interrater Reliability and Validity of Universal Goniometer, Digital Inclinometer, and Smartphone Application Measuring Range of Motion in Patients with Total Knee Arthroplasty.

The aim of this study was to investigate the intrarater and interrater reliability and validity of range of motion measurements obtained with a universal goniometer, digital inclinometer, and smartphone application in patients with total knee arthroplasty. Range of motion of the knee joint was measured by two examiners with a universal goniometer, digital inclinometer, and a smartphone application. Data were obtained from 51 knees of 27 patients at postoperative 6months. Two measurements made by the first examiner were compared to assess interrater reliability, and measurements from both examiners were compared to assess intrarater reliability. Statistical analysis was performed using intraclass correlation coefficient (ICC) and Spearman's rho values. With all three methods, active and passive knee flexion range of motion measurements showed high intrarater and interrater reliability (ICC = 0.749-0.949). Concurrent validity analysis also demonstrated statistically significant, moderate to strong correlation among the three methods (r = 0.775-0.941). The universal goniometer, digital inclinometer, and smartphone application were all found to be reliable and valid assessment tools in clinical practice for patients with total knee arthroplasty.

Impact of the Split Teaching Practice Model on Undergraduate Pre-Service Teachers’ Performance in Science Content Delivery

The study adopted an observational research design to assess the impact of the Split Teaching Practice Model on undergraduate pre-service teachers' performance in science content delivery within classroom settings, without researchers’ intervention. The population consisted of 119 second-year and 102 third-year science education students at Federal University Gusau - Nigeria, engaged in First and Second-phase teaching practice during the 2022/2023 academic session. A convenience sample of 96 pre-service teachers (54 in First-phase, 42 in Second-phase) was obtained. Data collection involved structured classroom observations using instrument tagged Pre-service Teachers' Performance in Science Content Delivery. Validity and reliability of the instrument were established through expert validation and inter-rater reliability assessment, yielding a correlation coefficient of 0.86. Four null hypotheses were tested using t-tests, focusing on lesson planning, presentation, class management, and attainment of objectives. The findings of the study revealed that, significant differences in favour of Second-phase pre-service teachers in aspects like sequential presentation, subject matter mastery, instructional materials usage, time management, and student participation. No significant differences were found in lesson introduction and questioning techniques. Second-phase pre-service teachers outperformed in class management and objective attainment. The study recommended continued adoption of the Split Teaching Practice Model, emphasizing refinement and customization to address specific improvement areas through feedback in teacher training institutions.

Adapting the Multiple Sclerosis Functional Composite for Telehealth Administration Using Videoconference Delivery: Methodological Considerations and Interrater Reliability

ObjectiveTo describe the adaptations made and to examine interrater reliability and feasibility of administering a telehealth version of the Multiple Sclerosis Functional Composite (tele-MSFC). DesignThe Multiple Sclerosis Functional Composite (MSFC) is a commonly used, in-person clinical outcome assessment. It is composed of the timed 25-Foot Walk Test (T25FWT), Nine-Hole Peg Test (NHPT), and Paced Auditory Serial Addition Test (PASAT). The MSFC was adapted for videoconference administration as part of a larger clinical trial. One of the adaptations included administering a timed 12.5-Foot Walk Test (T12.5FWT) for participants who did not have adequate space in their homes for the T25FWT. Participants, examiners, and raters completed surveys online about their satisfaction and experience with tele-MSFC. SettingParticipants underwent the tele-MSFC in their homes using a laptop or smartphone while examiners scored the tele-MSFC in real-time at a remote location. ParticipantsCommunity-dwelling adults (n=61) with mild-to-moderate multiple sclerosis (MS) symptoms. InterventionsNot applicable. Main Outcome MeasureTele-MSFC. ResultsIntraclass correlation coefficients (ICC) assessed interrater reliability between the examiner and 2 independent raters who later scored a recording of the tele-MSFC. Interrater reliability was excellent (ICC>0.90) for all tests, including the T12.5FWT. Participants were highly satisfied with tele-MSFC. However, challenges included adequate space for T25FWT, technical difficulties, and safety and privacy considerations of individuals with moderate impairments who were requested to have their caregivers present during testing. ConclusionThe tele-MSFC is reliable and feasible to administer with adaptations for community-dwelling adults with mild to moderate MS symptoms. Further validation of T12.5FWT is needed.

Testing the validity of three acute care assessment tools for assessing residents' performance during in situ simulation: the ACAT-SimSit study.

The assessment of technical and nontechnical skills in emergency medicine requires reliable and usable tools. Three Acute Care Assessment Tools (ACATs) have been developed to assess medical learners in their management of cardiac arrest (ACAT-CA), coma (ACAT-coma) and acute respiratory failure (ACAT-ARF). This study aims to analyze the reliability and usability of the three ACATs when used for in situ (bedside) simulation. This prospective multicenter validation study tested ACATs using interprofessional in situ simulations in seven emergency departments and invited training residents to participate in them. Each session was rated by two independent raters using ACAT. Intraclass correlation coefficients (ICC) were used to assess interrater reliability, and Cronbach's alpha coefficient was used to assess internal consistency for each ACAT. The correlation between ACATs' scores and the learners' level of performance was also assessed. Finally, a questionnaire and two focus groups were used to assess the usability of the ACATs. A total of 104 in situ simulation sessions, including 85 residents, were evaluated by 37 raters. The ICC for ACAT-CA, ACAT-coma and ACAT-ARF were 0.95 [95% confidence interval (CI), 0.93-0.98], 0.89 (95% CI, 0.77-0.95) and 0.92 (95%CI 0.83-0.96), respectively. The Cronbach's alphas were 0.79, 0.80 and 0.73, respectively. The ACAT-CA and ARF showed good construct validity, as third-year residents obtained significantly higher scores than first-year residents ( P < 0.001; P < 0.019). The raters supported the usability of the tools, even though they expressed concerns regarding the use of simulations in a summative way. This study reported that the three ACATs showed good external validity and usability.

Validity and Reliability of the Sri Lankan Version of the Patient-Centered Questionnaire-Infertility (PCQ-Infertility)

Background: Reproductive healthcare of the highest calibre should encompass not only safety and effectiveness but also a patient-centered approach. Patients demand patient-centered subfertility treatment in addition to efficient medical management. Offering patient-centered care is crucial for both patients and healthcare professionals. The absence of a suitable instrument for measuring patient-centeredness poses a significant challenge in evaluating this essential aspect of care in Sri Lanka. The present study resulted in a culturally adapted and validated tool in Sinhala language to measure patient-centredness of care received by female partners of subfertile couples.&#x0D; Methods: The Patient Centredness Questionnaire-Infertility (PCQ-Infertility) was culturally adopted using a modified Delphi process, translated into Sinhala and assessed for its judgmental validity. The study was conducted September to December 2019. A cross-sectional study among 251 females attending the Subfertility Clinic at Teaching Hospital Mahamodara assessed the factor structure through exploratory factor analysis. The factor structure was confirmed with another cross-sectional study at the same setting among 250 females through confirmatory factor analysis. Test re-test reliability, internal consistency and inter-rater reliability were also assessed.&#x0D; Results: The validated PCQ-Infertility had 44 items with a seven-factor structure identified after exploratory factor analysis and confirmatory factor analysis with a combination of acceptable fit indices (SRMR of 0.079, CFI 0.862 and RMSEA 0.069). A Cronbach’s alpha value 0.761 assured high internal consistency. Test- retest and inter-rater reliability assessment gave satisfactory results with values exceeding 0.7 for all domains.&#x0D; Conclusion: Modified PCQ-Infertility is a valid and reliable instrument which can be used to assess patient centredness care received of subfertile females in Sri Lanka.

O21: ASSIST: Validation of a novel hybrid clinician-patient remote outcome measure for diagnosis of surgical site infection

Abstract Introduction Current remote methods of surgical site infection diagnosis have high specificity but lack sensitivity in diagnosis. This study evaluates the acceptability, reliability and validity of a novel hybrid clinician-patient remote outcome measure (ASSIST) for detecting SSI. Methods Patients completed the Bluebelle wound healing questionnaire (WHQ) at 30 days post-surgery and additionally submitted wound image(s). Images were reviewed two independent clinicians. Internal consistency of the unidimensional scale was assessed. Inter-rater and intra-rater reliability assessments were performed. Sensitivity and specificity for SSI discrimination against face-to-face CDC diagnosis were evaluated and receiver operator characteristic (ROC) curve plotted with calculation of the area under the curve. Results 69 participants were included and no questionnaires contained missing items. The single scale showed strong Internal consistency (α = 0.9). Intra-rater reliability was good with intraclass correlation (ICC) of 0.818 (95% CI, 0.672-0.903, p&amp;lt;0.001). Inter-rater agreement was excellent (ICC 0.915, 95% CI 0.733-0.903, p&amp;lt;0.001). Sensitivity and specificity for SSI diagnosis were excellent at 91.7% and 97.6% respectively. Discrimination for SSI was high with an area under the receiver operating characteristic curve of 0.966 (95%CI 0.913-1.000). Discussion The ASSIST measure is acceptable, reliable, and valid for diagnosis of SSI. Implementation can be safely utilised in both patient-initiated follow-up or at pre-determined time points for safe post discharge surveillance.

Managing vestibular schwannomas with radiosurgery and radiotherapy: AGREE II appraisal of clinical practice guidelines.

Vestibular schwannomas (VSs) are rare, benign intracranial tumours that have prompted clinical practice guideline (CPG) creation given their complex management. Our aim was to utilize the Appraisal of Guidelines for Research and Evaluation (AGREE II) instrument to assess if such CPGs on the management of VSs with radiosurgery and radiotherapy are of acceptable quality. Relevant CPGs were identified following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) protocols. Experienced reviewers then extracted general CPG properties and rated their quality via the AGREE II instrument. Intraclass correlation coefficients (ICCs) were quantified to assess interrater reliability. Nine CPGs on the management of VSs with radiosurgery and radiotherapy were identified. All CPGs were created in the past six years and developed recommendations based on literature review and expert consensus. One guideline was deemed as high quality with seven others being moderate and one being low in quality. The clarity of the presentation domain had the highest mean scaled domain score of 96.0%. The domains of stakeholder involvement and applicability had the lowest means of 49.2% and 47.2%, respectively. ICCs were either good or excellent across all domains. Current CPGs on the management of VSs with radiosurgery and radiotherapy are of acceptable quality but would greatly benefit from improvements in applicability, stakeholder involvement, editorial independence and rigour of development. We recommend CPG authors reference the European Association of Neuro-Oncology (EANO) guideline as a developmental framework with the Congress of Neurological Surgeons/American Association of Neurological Surgeons (CNS/AANS) CPG being a valid alternative.

More Evidence of Low Inter-Rater Reliability of a High-Stakes Performance Assessment of Teacher Candidates

From 2010 to 2015, our school of education used the Performance Assessment for California Teachers (PACT), a summative assessment designed to assess preservice teacher competence. Candidate portfolios were uploaded to an evaluation portal, and trained evaluators assigned a final score of Pass or Fail to the work samples. Three consensus estimates of inter-rater reliability of 181 candidate portfolios that were either double- or triple-scored were computed. Two chance-corrected estimates of inter-rater reliability (Cohen’s kappa and Gwet’s AC1) and percent agreement were computed and calculated within five content areas: elementary math, secondary history/social science, math, science, and English language arts. An initial Pass or Fail score was not more likely to be followed by either a Pass or Fail score given by a subsequent evaluator. Inter-rater reliability was interpreted as being low across all content areas that were examined. None of the percent agreement coefficients attained the minimum standard of 0.700 for consensus agreement. Increasing research access to proprietary double-scored data would lead to an increased understanding of, and perhaps improvement in, teacher performance assessments.

Dental Practice Websites in Germany-How Do They Inform about Fluoridation?

Fluoridation (Fl) is effective in preventing caries; however, it is unclear to what extent its use is counteracted by misinformation on the internet. This study aimed to evaluate the information provided on professional websites of German dental practices regarding fluoridation. A systematic search was performed by two independent examiners, utilizing three search engines, from 10 September 2021 to 11 December 2021. Modified, validated questionnaires (LIDA, DISCERN) were used to evaluate technical and functional aspects, generic quality, and risk of bias. Demographic information and statements about Fl were also collected. The intra- and inter-rater reliability assessments were excellent. Of the 81 websites analyzed, 64 (79%) mentioned Fl, and 31 (38%) indicated it as a primary focus. Most websites met at least 50% of the LIDA (90%) and DISCERN criteria (99%), indicating that the general quality was good. Thirty (37%) of the websites explained the impact of Fl, and forty-five (56%) indicated an opinion (for/against) on Fl. The practice location and the clinical focus were not associated with the overall quality of websites. Only a minority of websites explained the effects of Fl. Taken together, this study highlights that there is a distinct lack of good-quality information on FL.

Development and validation of an artificial intelligence model to accurately predict spinopelvic parameters.

Achieving appropriate spinopelvic alignment has been shown to be associated with improved clinical symptoms. However, measurement of spinopelvic radiographic parameters is time-intensive and interobserver reliability is a concern. Automated measurement tools have the promise of rapid and consistent measurements, but existing tools are still limited to some degree by manual user-entry requirements. This study presents a novel artificial intelligence (AI) tool called SpinePose that automatically predicts spinopelvic parameters with high accuracy without the need for manual entry. SpinePose was trained and validated on 761 sagittal whole-spine radiographs to predict the sagittal vertical axis (SVA), pelvic tilt (PT), pelvic incidence (PI), sacral slope (SS), lumbar lordosis (LL), T1 pelvic angle (T1PA), and L1 pelvic angle (L1PA). A separate test set of 40 radiographs was labeled by four reviewers, including fellowship-trained spine surgeons and a fellowship-trained radiologist with neuroradiology subspecialty certification. Median errors relative to the most senior reviewer were calculated to determine model accuracy on test images. Intraclass correlation coefficients (ICCs) were used to assess interrater reliability. SpinePose exhibited the following median (interquartile range) parameter errors: SVA 2.2 mm (2.3 mm) (p = 0.93), PT 1.3° (1.2°) (p = 0.48), SS 1.7° (2.2°) (p = 0.64), PI 2.2° (2.1°) (p = 0.24), LL 2.6° (4.0°) (p = 0.89), T1PA 1.1° (0.9°) (p = 0.42), and L1PA 1.4° (1.6°) (p = 0.49). Model predictions also exhibited excellent reliability at all parameters (ICC 0.91-1.0). SpinePose accurately predicted spinopelvic parameters with excellent reliability comparable to that of fellowship-trained spine surgeons and neuroradiologists. Utilization of predictive AI tools in spinal imaging can substantially aid in patient selection and surgical planning.

Validity and reliability evidence support task-specific metrics for laparoscopic fundoplication.

Laparoscopic hiatal hernia repair (LHHR) is a complex operation requiring advanced surgical training. Surgical simulation offers a potential solution for learning complex operations without the need for high surgical volume. Our goal is to develop a virtual reality (VR) simulator for LHHR; however, data supporting task-specific metrics for this procedure are lacking. The purpose of this study was to develop and assess validity and reliability evidence of task-specific metrics for the fundoplication phase of LHHR. In phase I, structured interviews with expert foregut surgeons were conducted to develop task-specific metrics (TSM). In phase II, participants with varying levels of surgical expertise performed a laparoscopic Nissen fundoplication procedure on a porcine stomach explant. Video recordings were independently assessed by two blinded graders using global and TSM. An intraclass correlation coefficient (ICC) was used to assess interrater reliability (IRR). Performance scores were compared using a Kruskal-Wallis test. Spearman's rank correlation was used to evaluate the association between global and TSM. Phase I of the study consisted of 12 interviews with expert foregut surgeons. Phase II engaged 31 surgery residents, a fellow, and 6 attendings in the simulation. Phase II results showed high IRR for both global (ICC = 0.84, p < 0.001) and TSM (ICC = 0.75, p < 0.001). Significant between-group differences were detected for both global (χ2 = 24.01, p < 0.001) and TSM (χ2 = 18.4, p < 0.001). Post hoc analysis showed significant differences in performance between the three groups for both metrics (p < 0.05). There was a strong positive correlation between the global and TSM (rs = 0.86, p < 0.001). We developed task-specific metrics for LHHR and using a fundoplication model, we documented significant reliability and validity evidence. We anticipate that these LHHR task-specific metrics will be useful in our planned VR simulator.