Intraclass Correlation Coefficient Scores Research Articles

Utilizing large language models for EFL essay grading: An examination of reliability and validity in rubric‐based assessments

AbstractThis study investigates the validity and reliability of generative large language models (LLMs), specifically ChatGPT and Google's Bard, in grading student essays in higher education based on an analytical grading rubric. A total of 15 experienced English as a foreign language (EFL) instructors and two LLMs were asked to evaluate three student essays of varying quality. The grading scale comprised five domains: grammar, content, organization, style & expression and mechanics. The results revealed that fine‐tuned ChatGPT model demonstrated a very high level of reliability with an intraclass correlation (ICC) score of 0.972, Default ChatGPT model exhibited an ICC score of 0.947 and Bard showed a substantial level of reliability with an ICC score of 0.919. Additionally, a significant overlap was observed in certain domains when comparing the grades assigned by LLMs and human raters. In conclusion, the findings suggest that while LLMs demonstrated a notable consistency and potential for grading competency, further fine‐tuning and adjustment are needed for a more nuanced understanding of non‐objective essay criteria. The study not only offers insights into the potential use of LLMs in grading student essays but also highlights the need for continued development and research. Practitioner notesWhat is already known about this topic Large language models (LLMs), such as OpenAI's ChatGPT and Google's Bard, are known for their ability to generate text that mimics human‐like conversation and writing. LLMs can perform various tasks, including essay grading. Intraclass correlation (ICC) is a statistical measure used to assess the reliability of ratings given by different raters (in this case, EFL instructors and LLMs). What this paper adds The study makes a unique contribution by directly comparing the grading performance of expert EFL instructors with two LLMs—ChatGPT and Bard—using an analytical grading scale. It provides robust empirical evidence showing high reliability of LLMs in grading essays, supported by high ICC scores. It specifically highlights that the overall efficacy of LLMs extends to certain domains of essay grading. Implications for practice and/or policyThe findings open up potential new avenues for utilizing LLMs in academic settings, particularly for grading student essays, thereby possibly alleviating workload of educators.The paper's insistence on the need for further fine‐tuning of LLMs underlines the continual interplay between technological advancement and its practical applications.The results lay down a footprint for future research in advancing the use of AI in essay grading.

Test-Retest Reliability of the Verran Professional Governance Scale©.

Background and Purpose: Stability testing, conducted using a test-retest protocol, measures an instrument's reliability by evaluating the consistency of participant responses to survey questions with repeated testing within a short interval. No studies have measured the stability of the Verran Professional Governance Scale© (VPGS). The purpose of this study was to evaluate the test-retest reliability of the VPGS. Methods: Volunteers from a parent study using the VPGS were sent a link to a retest version of the survey 14 days after taking the initial survey with a reminder email sent 5 days after the first request. Item-level and subscale comparisons were made between participants' initial and retest responses using intraclass correlation coefficients (ICCs) applying a two-way random-effects model. Results: VPGS subscales had ICC scores of 0.71 for decision-making, 0.73 for collateral relationships, and 0.86 for professional obligation. Conclusions: Findings suggest that the VPGS demonstrates test-retest reliability. Future research should evaluate the instrument's responsiveness.

Surgical quality assessment of critical view of safety in 283 laparoscopic cholecystectomy videos by surgical residents and surgeons.

Surgical quality assessment has improved the efficacy and efficiency of surgical training and has the potential to optimize the surgical learning curve. In laparoscopic cholecystectomy (LC), the critical view of safety (CVS) can be assessed with a 6-point competency assessment tool (CAT), a task commonly performed by experienced surgeons. The aim of this study is to determine the capability of surgical residents to perform this assessment. Both surgeons and surgical residents assessed unedited LC videos using a 6-point CVS, a CAT, using an online video assessment platform. The CAT consists of the following three criteria: 1. clearance of hepatocystic triangle, 2. cystic plate, and 3. two structures connect to the gallbladder, with a maximum of 2 points available for each criterion. A higher score indicates superior surgical performance. The intraclass correlation coefficient (ICC) was employed to assess the inter-rater reliability between surgeons and surgical residents. In total, 283 LC videos were assessed by 19 surgeons and 31 surgical residents. The overall ICC for all criteria was 0.628. Specifically, the ICC scores were 0.504 for criterion 1, 0.639 for criterion 2, and 0.719 for the criterion involving the two structures connected to the gallbladder. Consequently, only the criterion regarding clearance of the hepatocystic triangle exhibited fair agreement, whereas the other two criteria, as well as the overall scores, demonstrated good agreement. In 71% of cases, both surgeons and surgical residents scored a total score either ranging from 0 to 4 or from 5 to 6. Compared to the gold standard, i.e., the surgeons' assessments, surgical residents are equally skilled at assessing critical view of safety (CVS) in laparoscopic cholecystectomy (LC) videos. By incorporating video-based assessments of surgical procedures into their training, residents could potentially enhance their learning pace, which may result in better clinical outcomes.

Reproducibility of Assessment of Lepidic (Noninvasive) Patterns in Lung Adenocarcinoma With Cytokeratin Immunostain Compared With Hematoxylin and Eosin and the Proposed New International Association for the Study of Lung Cancer Algorithm

IntroductionLepidic growth is considered noninvasive in lung nonmucinous adenocarcinoma, whereas other patterns are invasive. Considerable interobserver variability in assessing “invasion” has been reported. We assessed the utility of cytokeratin 7 (CK7) stain and recently proposed International Association for the Study of Lung Cancer criteria to improve assessment of noninvasion in lung adenocarcinoma. MethodsFour pathologists (two staff, two trainees) assessed 158 hematoxylin and eosin (HE)- and CK7-stained slides of 108 pT1N0-2 nonmucinous lung adenocarcinoma cases. Scoring took place in four rounds. First, sections were independently scored for percentage of noninvasive/probable noninvasive (definite/probable) and invasive/probable invasive (definite/probable) patterns. Second, a consensus scoring algorithm for CK7 was formulated and slides were rescored. Subsequent third-round scoring was conducted on HE slides using the 2023 International Association for the Study of Lung Cancer (IASLC) proposed criteria, and fourth-round scoring on both HE and CK7 slides simultaneously. Intraclass correlation coefficient (ICC) was calculated for each round. Recurrence-free survival was assessed using Cox proportional hazards regression methods. ResultsIn the first two rounds, interobserver concordance was consistently higher with CK7 (ICC range = 0.44–0.6) than HE (range = 0.24–0.49) scores. The IASLC proposed algorithm improved ICC of HE scores to 0.60 (95% confidence interval: 0.52–0.67), and round 4 HE and CK7 combined improved ICC to 0.75 (95% confidence interval: 0.70–0.80). Continuous measures of averaged noninvasive and probable noninvasive scores on HE were associated with improved recurrence-free survival (hazard ratio: 0.83–0.86). ConclusionsCK7 staining consistently increased interobserver concordance in assessment of invasive versus noninvasive patterns than HE. Combining CK7 with the 2023 IASLC criteria for morphologic features of invasion may further improve the interobservers’ concordance for the recognition of lepidic growth in nonmucinous lung adenocarcinoma.

Assessing Advance Care Planning Fidelity within the Context of Cognitive Impairment: The SHARE Trial

ContextAdvance care planning (ACP) is critical among primary care patients with cognitive impairment, but few interventions have tested ACP with this population. ObjectiveDescribe the development and evaluation of a tool for assessing ACP fidelity within the context of cognitive impairment, including inter-rater reliability, convergent validity, and overall fidelity using clinical trial data. DesignSHARE is a multicomponent intervention inclusive of facilitated ACP conversations. From a two group, single blind, randomized controlled trial, recorded ACP conversations were rated for fidelity. 145 primary care patients and their care partners were randomized to receive the intervention. Participating patients were 80+ years, had a care partner, and indications of cognitive impairment. An ACP Fidelity Checklist was developed with three subscales: Meeting Set-Up; ACP Meeting Topics; and Communication Skills. Scores were converted to percentages (100% = perfect fidelity) with a target of ≥80% fidelity. A post-ACP meeting report completed by ACP facilitators was used to assess convergent validity of the checklist. Intra-class correlation (ICC) was to evaluate inter-rater reliability. ResultsACP conversations averaged 33.6 minutes (SD = 14.1). The mean fidelity score across N = 91 rated meetings was 82.9%, with a range of 77.3%-90.6% for subscales. 63.7% of meetings achieved a rating of ≥80%. Cognitive function was positively associated with patient participation (rho = .59, P < 0.001). For checklist items, ICC scores ranged from 0.43–0.96. Post-ACP meeting form scores were correlated with the checklist Meeting Topics subscale (r = 0.36, P = 0.001). ConclusionsAssessing the fidelity of ACP conversations involving primary care patients living with cognitive impairment and their care partners is feasible.

Shoulder MRI parameters in anticipating anterior shoulder dislocation: Are they a reliable and reproducible tool in clinical practice?

BackgroundAnterior shoulder dislocation (ASD) is a frequently observed musculoskeletal injury that is often encountered in the context of sports activities or as a result of trauma. Several magnetic resonance imaging (MRI) parameters have been previously investigated for the purpose of characterizing the anatomical features, which could potentially be responsible for the episodes of instability. These measurements have the potential to identify patients who are susceptible to dislocation. Consequently, ensuring the reliability and consistency of these measurements is crucial in the diagnosis and the management of athletic or traumatic shoulder injuries. MethodsA group of four students, who had no previous experience in reading MRI series, were selected to perform radiographic measurements on specific parameters of MRI scans. These parameters were glenoid version, glenoid depth, glenoid width, humeral head diameter, humeral containing angle, and the ratio of humeral head diameter to glenoid diameter. The four participants conducted two distinct readings on a total of 28 sets of shoulder MRI scans. Simultaneously, the aforementioned measures were assessed by a consultant shoulder surgeon. ResultsA total of 1512 measurements were categorized into nine sets: eight from students’ measurements (two per student) and one from the consultant. Intra-rater reliability assessed by the intra-class correlation (ICC) test indicated excellent or good reliability for all parameters (p < 0.05), with glenoid depth showing the highest (0.925) and humeral-containing angles the lowest (0.675) ICC value. Inter-rater correlation, also evaluated using ICC, demonstrated strong correlation (p < 0.05), with glenoid diameter having the highest ICC score (0.935) and glenoid depth the lowest (0.849). Agreement analysis, expressed by Cohen's Kappa test, revealed substantial agreement (p < 0.05) for all parameters, with humeral head diameter having the highest agreement (0.90) and humeral-containing angle the lowest (0.73). ConclusionIn this study, intra- and inter-rater MRI parameters are substantially concordant. Credibility comes from these reliability and agreement analyses' statistical significance. Glenoid diameter and depth are the most reliable intrarater and interrater, respectively. Best agreement was with the humeral-containing angle. These data demonstrate repeatability and clinical relevance. Level of EvidenceLevel IV

Automated analysis of knee joint alignment using detailed angular values in long leg radiographs based on deep learning

Malalignment in the lower limb structure occurs due to various causes. Accurately evaluating limb alignment in situations where malalignment needs correction is necessary. To create an automated support system to evaluate lower limb alignment by quantifying mechanical tibiofemoral angle (mTFA), mechanical lateral distal femoral angle (mLDFA), medial proximal tibial angle (MPTA), and joint line convergence angle (JLCA) on full-length weight-bearing radiographs of both lower extremities. In this retrospective study, we analysed 404 radiographs from one hospital for algorithm development and testing and 30 radiographs from another hospital for external validation. The performance of segmentation algorithm was compared to that of manual segmentation using the dice similarity coefficient (DSC). The agreement of alignment parameters was assessed using the intraclass correlation coefficient (ICC) for internal and external validation. The time taken to load the data and measure the four alignment parameters was recorded. The segmentation algorithm demonstrated excellent agreement with human-annotated segmentation for all anatomical regions (average similarity: 89–97%). Internal validation yielded good to very good agreement for all the alignment parameters (ICC ranges: 0.7213–0.9865). Interobserver correlations between manual and automatic measurements in external validation were good to very good (ICC scores: 0.7126–0.9695). The computer-aided measurement was 3.44 times faster than was the manual measurement. Our deep learning-based automated measurement algorithm accurately quantified lower limb alignment from radiographs and was faster than manual measurement.

Development and validation of the Japanese version of the Lesbian, Gay, Bisexual, and Transgender Development of Clinical Skills Scale.

The Lesbian, Gay, Bisexual, and Transgender Development of Clinical Skills Scale (LGBT-DOCSS) is a validated self-assessment tool for health and mental health professionals who provide healthcare for sexual and gender minority patients. This study aimed to develop and validate a Japanese version of LGBT-DOCSS (LGBT-DOCSS-JP) and examine its psychometric properties. LGBT-DOCSS was translated into Japanese and cross-culturally validated using cognitive debriefing. We then evaluated the structural validity, convergent and discriminant validity, internal consistency, and test-retest reliability of LGBT-DOCSS-JP using an online survey. Data were analyzed for 381 health and mental health professionals aged 20 years or older from three suburban medical institutions. The confirmatory factor analysis indicated that the original three-factor model did not fit well with LGBT-DOCSS-JP. Exploratory factor analysis revealed four new factors: Attitudinal Awareness, Basic Knowledge, Clinical Preparedness, and Clinical Training. Convergent and discriminant validity were supported using four established scales that measured attitudes toward lesbians and gay men, genderism and transphobia, authoritarianism and conventionalism, and social desirability. The internal consistency of LGBT-DOCSS-JP was supported by the Cronbach's alpha values for the overall scale (0.84), and for each of its subscales (Attitudinal Awareness and Basic Knowledge both 0.87, Clinical Preparedness 0.78, and Clinical Training 0.97). The test-retest reliability for the overall LGBT-DOCSS-JP was supported by an intraclass correlation coefficient score of 0.86. LGBT-DOCSS-JP has the potential to serve as a valuable tool in the development and assessment of effective curricula for LGBT healthcare education, as well as a means to promote self-reflection among trainees and professionals.

The Validity and Reliability of the Turkish Version of Preschool Hearing Environments and Reflection on Quality of Life Questionnaire.

The purpose of this study was to assess preschool children's hearing-related quality of life and to introduce the scale, originally titled "Preschool Hearing Environments and Reflections on Quality of Life Questionnaire" (Preschool HEAR-QL), to the literature through Turkish adaption, validity, and reliability analyses. Our study included 210 children aged 2-6 years: 110 with hearing loss (56 cochlear implant users and 54 hearing aid users) and 100 with normal hearing. Demographic data were collected and then the Preschool HEAR-QL, which was translated into Turkish, was administered twice at 15-day intervals. Confirmatory factor analysis was used to determine whether the factor structure found in the original scale development study was confirmed in our sample (construct validity). Cronbach's alpha, intraclass correlation scores, and the test-retest method were used to assess the scale's reliability as the findings were analyzed using paired samples t tests. The children with hearing loss had lower scores than their normal-hearing peers, and the difference was significant in the subdomains of behavior and attention, hearing environments, and communication. The subscale of the Preschool Period Listening Environments and their Reflections on Quality of Life Scale consists of 23 items, and Cronbach's alpha value was found as 0.922 in the first application and 0.926 in the second application (high reliability). In the model established for validity analysis, χ2/df = 2.156, root-mean-square error of approximation = 0.074, standard root-mean-square residual = 0.078, goodness-of-fit index = 0.830, adjusted goodness-of-fit index = 0.789, comparative fit index = 0.895, and Tucker-Lewis index = 0.881 (good model fit). In terms of test-retest reliability, the correlation between the two measurements was 0.837, and there was no statistical difference between the test-retest scores (p = .15). The Turkish version of the Preschool HEAR-QL scale was proven to be a valid and reliable scale for assessing the hearing-related quality of life of children aged 2-6 years. https://doi.org/10.23641/asha.25236949.

Assessing the reliability of biomechanical variables during a horizontal deceleration task in healthy adults

ABSTRACT Horizontal deceleration technique is an underpinning factor to musculoskeletal injury risk and performance in multidirectional sport. This study primarily assessed within- and between-session reliability of biomechanical and performance-based aspects of a horizontal deceleration technique and secondarily investigated the effects of limb dominance on reliability. Fifteen participants completed four horizontal decelerations on each leg during test and retest sessions. A three-dimensional motion analysis system was used to collect kinetic and kinematic data. Completion time, ground contact time, rate of horizontal deceleration, minimum centre of mass height, peak eccentric force, impulse ratio, touchdown distance, sagittal plane foot and knee angles at initial contact, maximum sagittal plane thorax angle, and maximum knee flexion moment were assessed. Coefficients of variation (COV) and intraclass correlation coefficients (ICC) were used to assess within- and between-session reliability, respectively. Seven variables showed “great” within-session reliability bilaterally (COV ≤9.13%). ICC scores were ‘excellent’ (≥0.91; n = 4), or ‘good’ (0.76-0.89; n = 7), bilaterally. Limb dominance affected five variables; three were more reliable for the dominant leg. This horizontal deceleration task was reliable for most variables, with little effect of limb dominance on reliability. This deceleration task may be reliably used to assess and track changes in deceleration technique in healthy adults.

Cosmetic Contact Lenses in the United Kingdom: Assessment of Online Regulation and Quality of Consumer Health Information.

To determine the compliance of online vendors to the UK Opticians Act 1989 Section 27 requirements and safety regulations for cosmetic contact lens (CCL) sales and the quality of online CCL health information. The top 50 websites selling CCLs on each three search engines, namely Google, Yahoo, and Bing, were selected. Duplicates were removed, and the remaining websites were systematically analyzed in February 2023. UK legal authorization for CCL sales was assessed using the Opticians Act Section 27 and safety regulations determined by the presence of Conformité Européene (CE) marking. The quality and reliability of online information was graded using the DISCERN (16-80) and JAMA (0-4) scores by two independent reviewers. Forty-seven eligible websites were analyzed. Only six (12.7%) met the UK legal authorization for CCL sales. Forty-nine different brands of CCLs were sold on these websites, of which 13 (26.5%) had no CE marking. The mean DISCERN and JAMA benchmark scores were 26 ± 12.2 and 1.3 ± 0.6, respectively (intraclass correlation scores: 0.99 for both). A significant number of websites provide consumers with easy, unsafe, and unregulated access to CCLs. Most online stores do not meet the requirements set out in the Opticians Act for CCL sales in the United Kingdom. A significant number of CCLs lack CE marking, while the average quality of information on websites selling CCLs is poor. Together, these pose a risk to consumers purchasing CCLs from unregulated websites, and therefore, further stringent regulations on the online sales of these products are needed.

Quality analysis of the clinical practice guidelines for management of impacted maxillary central incisors: a systematic review.

To assess the quality of clinical practice guidelines (CPG) for management of impacted central incisors. Search was performed in PubMed, LILACS, Web of Science, EMBASE, Scopus, and Cochrane databases, and guideline-focused databases/repositories on 15-09-2022 without any limitations and was updated on 15-07-2023. Grey literature search was also performed. Two independent reviewers were involved in the study selection and data extraction. Quality assessment of the included CPG was performed by four independent appraisers using the AGREE-II instrument. The degree of agreement among the appraisers was calculated using the intraclass correlation coefficient (ICC). Five CPG were included in the review. The Ministry of Health, Malaysia (MHM) guideline obtained the highest scores in all six domains of AGREE-II and an overall score of 73% demonstrating the "highest" quality. The remaining four guidelines obtained overall "low-quality" scores ranging from 34.57-37.52%. The ICC scores ranged from 0.530 to 0.990 for various domains of AGREE-II. MHM guidelines demonstrated high-quality scores in domains of 'scope and purpose', 'clarity of presentation', 'applicability domain', and 'editorial independence', while others were found to have moderate or low quality. This review identified areas that can be addressed by future guideline developers to avoid these discrepancies.

Transcultural adaptation of Odia version of health-related kidney disease and quality of life-36 instrument

Objective: The study objective was to translate, validate, and test the reliability of the original kidney disease and quality of life-36 (KDQOL-36™) instruments in Odia. Materials and Methods: A cross-sectional design with a purposive sampling technique was used. According to RAND Corporation guidelines, initially, the items of the KDQOL-36™ questionnaires were translated into Odia by two independent, bilingual, professional translators, and then back-translated to English, followed by tryout and field testing. The experts validated the KDQOL-36™ instrument review committee for review related to kidney health conditions. The tool was implemented among 180 patients undergoing “maintenance” hemodialysis. The following tests evaluated reliability and validity: test–retest reliability with Cronbach’s alpha correlation (stability), (reliability) internal consistency, and contents validity index. Results: The Cronbach’s alpha value and intraclass correlation coefficient (ICC) score of all five domains, namely “physical component summary, mental component summary (MCS), the burden of kidney disease, symptoms and problems of kidney disease, and effects of kidney disease” of both KDQOL-36™ English and Odia (KDQOL-36-E™ and KDQOL-36-O™) version, recommended excellent homogeneity. A high positive correlation (r = 0.998) was found between the Odia version of KDQOL-36™ and the English version KDQOL-36™ questionnaire. The ICC score ranges from 0.889 to 0.997 at a 95% confidence interval for test–retest reliability, and Cronbach’s alpha was 0.832. Conclusion: This study explores the Odia version of KDQOL-36™ psychometric properties, depicted at an acceptable level of internal consistency. The KDQOL-36-O™ instrument is a valid and reliable tool for assessing the kidney disease-related quality of life in Odia-speaking hemodialysis patients.

Magnetic resonance imaging preprocessing and radiomic features for classification of autosomal dominant polycystic kidney disease genotype.

Our study aims to investigate the impact of preprocessing on magnetic resonance imaging (MRI) radiomic features extracted from the noncystic kidney parenchyma of patients with autosomal dominant polycystic kidney disease (ADPKD) in the task of classifying PKD1 versus PKD2 genotypes, which differ with regard to cyst burden and disease outcome. The effect of preprocessing on radiomic features was investigated using a single T2-weighted fat saturated (T2W-FS) MRI scan from PKD1 and PKD2 subjects (29 kidneys in total) from the Consortium for Radiologic Imaging Studies of Polycystic Kidney Disease study. Radiomic feature reproducibility using the intraclass correlation coefficient (ICC) was computed across MRI normalizations (-score, reference-tissue, and original image), gray-level discretization, and upsampling and downsampling pixel schemes. A second dataset for genotype classification from 136 subjects T2W-FS MRI images previously enrolled in the HALT Progression of Polycystic Kidney Disease study was matched for age, gender, and Mayo imaging classification class. Genotype classification was performed using a logistic regression classifier and radiomic features extracted from (1)the noncystic kidney parenchyma and (2)the entire kidney. The area under the receiver operating characteristic curve (AUC) was used to evaluate the classification performance across preprocessing methods. Radiomic features extracted from the noncystic kidney parenchyma were sensitive to preprocessing parameters, with varying reproducibility depending on the parameter. The percentage of features with good-to-excellent ICC scores ranged from 14% to 58%. AUC values ranged between 0.47 to 0.68 and 0.56 to 0.73 for the noncystic kidney parenchyma and entire kidney, respectively. Reproducibility of radiomic features extracted from the noncystic kidney parenchyma was dependent on the preprocessing parameters used, and the effect on genotype classification was sensitive to preprocessing parameters. The results suggest that texture features may be indicative of genotype expression in ADPKD.

Inter-observer variance of examiner scoring in urology Objective Structured Clinical Examinations.

The Objective Structured Clinical Examination (OSCE) is an attractive tool of competency assessment in a high-stakes summative exam. An advantage of the OSCE is the ability to assess more realistic context, content, and procedures. Each year, the Queen's Urology Exam Skills Training (QUEST) is attended by graduating Canadian urology residents to simulate their upcoming board exams. The exam consists of a written component and an OSCE. The aim of this study was to determine the inter-observer consistency of scoring between two examiners of an OSCE for a given candidate. Thirty-nine participants in 2020 and 37 participants in 2021 completed four stations of OSCEs virtually over the Zoom platform. Each candidate was examined and scored independently by two different faculty urologists in a blinded fashion at each station. The OSCE scoring consisted of a checklist rating scale for each question. An intra-class correlation (ICC) analysis was conducted to determine the inter-rater reliability of the two examiners for each of the four OSCE stations in both the 2020 and 2021 OSCEs. For the 2020 data, the prostate cancer station scores were most strongly correlated (ICC 0.746, 95% confidence interval [CI] 0.556-0.862, p<0.001). This was followed by the general urology station (ICC 0.688, 95% CI 0.464-0.829, p<0.001), the urinary incontinence station (ICC 0.638, 95% CI 0.403-0.794, p<0.001), and finally the nephrolithiasis station (ICC 0.472, 95% CI 0.183-0.686, p<0.001). For the 2021 data, the renal cancer station had the highest ICC at 0.866 (95% CI 0.754-0.930, p<0.001). This was followed by the nephrolithiasis station (ICC 0.817, 95% CI 0.673-0.901, p<0.001), the pediatric station (ICC 0.809, 95% CI 0.660-0.897, p<0.001), and finally the andrology station (ICC 0.804, 95% CI 0.649-0.895, p<0.001). A Pearson correlation coefficient was calculated for all stations, and all show a positive correlation with global exam scores. It is noteworthy that some stations were more predictive of overall performance, but this did not necessarily mean better ICC scores for these stations. Given a specific clinical scenario in an OSCE exam, inter-rater reliability of scoring can be compromised on occasion. Care should be taken when high-stakes decisions about promotion are made based on OSCEs with limited standardization.

The Significance of Selecting an Appropriate Patient-Reported Outcome Measure (PROM): A Cross-Cultural Adaptation of the Specific Paediatric International Documentation Committee Subjective (Pedi-IKDC) Knee Form.

The selection of an appropriate PROM is a crucial aspect in assessing outcomes. Questionnaires that have not been designed or validated for a paediatric population are routinely used. Using a questionnaire requires translation, cultural adaptation, and testing the psychometric properties of the translated questionnaire. There is no applicable questionnaire in our country for children with knee-specific conditions in sports orthopaedics. Therefore, this study aims to translate, culturally adapt, and assess the psychometric properties of the Paediatric IKDC (Pedi-IKDC) questionnaire within the Lithuanian paediatric population. The translation was conducted in accordance with international standards. Patients aged 11-17 years with various knee disorders participated in three surveys and completed the Pedi-IKDC, Lysholm, and PedsQL questionnaires. Interviews with patients following the translation process, in addition to floor and ceiling effects, were used to assess content validity. Cronbach alpha (α) statistics and the intraclass correlation coefficient (ICC) were applied to measure internal consistency and reproducibility, respectively. The standard error of measurement (SEM) and smallest detectable change (SDC) were calculated to assess reliability. Pearson correlations were calculated between Pedi-IKDC and Lysholm PedsQL scores to determine criteria validity. The effect size (ES) and standardised response mean (SRM) were calculated to assess the responsiveness to change. Cronbach's alpha (α) was 0.91 for the total score, 0.75 for symptoms, and 0.92 for the sport/function component. The ICC for overall scores was 0.98, with each question ranging from 0.87 to 0.98. The SEM was 2.97, and the SDC was 8.23. Lysholm and PedsQL physical functioning domain scores had moderate correlations (0.8 > r > 0.5), and the overall PedsQL score had a weak correlation (0.5 > r > 0.2) to the Pedi-IKDC score. The floor and ceiling effects were 3.3% and 1.6%, respectively. The SRM was 1.72 and the ES was 1.98. The Lithuanian Pedi-IKDC version is an appropriate evaluation instrument for assessing outcomes in children with knee disorders. All of the psychometric features produced acceptable results.

Development and Evaluation of a Natural Language Processing System for Curating a Trans-Thoracic Echocardiogram (TTE) Database.

Although electronic health records (EHR) provide useful insights into disease patterns and patient treatment optimisation, their reliance on unstructured data presents a difficulty. Echocardiography reports, which provide extensive pathology information for cardiovascular patients, are particularly challenging to extract and analyse, because of their narrative structure. Although natural language processing (NLP) has been utilised successfully in a variety of medical fields, it is not commonly used in echocardiography analysis. To develop an NLP-based approach for extracting and categorising data from echocardiography reports by accurately converting continuous (e.g., LVOT VTI, AV VTI and TR Vmax) and discrete (e.g., regurgitation severity) outcomes in a semi-structured narrative format into a structured and categorised format, allowing for future research or clinical use. 135,062 Trans-Thoracic Echocardiogram (TTE) reports were derived from 146967 baseline echocardiogram reports and split into three cohorts: Training and Validation (n = 1075), Test Dataset (n = 98) and Application Dataset (n = 133,889). The NLP system was developed and was iteratively refined using medical expert knowledge. The system was used to curate a moderate-fidelity database from extractions of 133,889 reports. A hold-out validation set of 98 reports was blindly annotated and extracted by two clinicians for comparison with the NLP extraction. Agreement, discrimination, accuracy and calibration of outcome measure extractions were evaluated. Continuous outcomes including LVOT VTI, AV VTI and TR Vmax exhibited perfect inter-rater reliability using intra-class correlation scores (ICC = 1.00, p < 0.05) alongside high R2 values, demonstrating an ideal alignment between the NLP system and clinicians. A good level (ICC = 0.75-0.9, p < 0.05) of inter-rater reliability was observed for outcomes such as LVOT Diam, Lateral MAPSE, Peak E Velocity, Lateral E' Velocity, PV Vmax, Sinuses of Valsalva and Ascending Aorta diameters. Furthermore, the accuracy rate for discrete outcome measures was 91.38% in the confusion matrix analysis, indicating effective performance. The NLP-based technique yielded good results when it came to extracting and categorising data from echocardiography reports. The system demonstrated a high degree of agreement and concordance with clinician extractions. This study contributes to the effective use of semi-structured data by providing a useful tool for converting semi-structured text to a structured echo report that can be used for data management. Additional validation and implementation in healthcare settings can improve data availability and support research and clinical decision-making.

Evaluation of the international Ki67 working group cut point recommendations for early breast cancer: comparison with 21-gene assay results in a large integrated health care system

PurposeThe International Ki67 Working Group (IKWG) has developed training for immunohistochemistry (IHC) scoring reproducibility and recommends cut points of ≤ 5% and ≥ 30% for prognosis in ER+, HER2−, stage I/II breast cancer. We examined scoring reproducibility following IKWG training and evaluated these cut points for selecting patients for further testing with the 21-gene Recurrence Score (RS) assay.MethodsWe included 307 women aged 50+ years with node-negative, ER+PR+HER2− breast cancer and with available RS results. Slides from the diagnostic biopsy were stained for Ki67 and scored using digital image analysis (IA). Two IHC pathologists underwent IKWG training and visually scored slides, blinded to each other and IA readings. Interobserver reproducibility was examined using intraclass correlation (ICC) and Kappa statistics.ResultsDepending on reader, 8.8–16.0% of our cohort had Ki67 ≤ 5% and 11.4–22.5% had scores ≥ 30%. The ICC for Ki67 scores by the two pathologists was 0.82 (95% CI 0.78–0.85); it was 0.79 (95% CI 0.74–0.83) for pathologist 1 and IA and 0.76 (95% CI 0.71–0.80) for pathologist 2 and IA. For Ki67 scores ≤ 5%, the percentages with RS < 26 were 92.6%, 91.8%, and 90.9% for pathologist 1, pathologist 2, and IA, respectively. For Ki67 scores ≥ 30%, the percentages with RS ≥ 26 were 41.5%, 51.4%, and 27.5%, respectively.ConclusionThe IKWG’s Ki67 training resulted in moderate to strong reproducibility across readers but cut points had only moderate overlap with RS cut points, especially for Ki67 ≥ 30% and RS ≥ 26; thus, their clinical utility for a 21-gene assay testing pathway remains unclear.

Predicted and achieved overjet and overbite measurements with the Invisalign appliance: a retrospective study.

To determine whether achieved outcome regarding overjet and overbite matched the predicted outcome following treatment with Invisalign (Align Technology, San Jose, Calif) aligner appliances. Data including pretreatment, predicted and achieved overjet, and overbite measurements provided by Align's digital treatment facility, ClinCheck Pro, were evaluated. Descriptive statistics, Mann-Whitney U-test, and Wilcoxon rank-sum intraclass correlation (ICC) test results were calculated. From an initial sample of 600, 355 adult patients satisfied the inclusion and exclusion criteria. ICC scores for data input were excellent. Median (interquartile range [IQR]) age was 30.14 (23.33, 39.92) years and most (n = 259; 72.95%) were women. Almost one-third (n = 101; 28.45%) had undergone extraction as part of their treatment. More aligners were prescribed in the initial digital treatment plan for patients (median, 44; IQR: 35, 51.5; minimum, 17; maximum, 92) undergoing extractions as part of their orthodontic treatment than those who were not (median, 24; IQR: 18.25, 32; minimum, 13; maximum, 85) (P < .0001). Planned changes in overjet differed significantly from achieved outcomes (P < .001). Planned increases in overbite resulted in up to more than twice (222.72%) as much increase than predicted. Planned reduction of overbite achieved 8.69% of its predicted reduction in extraction cases. Achieved overjet and overbite measurements differed significantly from the predicted outcomes at the end of an initial sequence of aligners. Planned increases in overbite resulted in greater overbite correction than predicted, particularly in extraction cases. Planned overbite reduction was challenging, especially in patients with extractions.

Do metrics derived from self-reported and clinician-reported pain drawings agree for individuals with chronic low back pain?

BackgroundClinicians commonly use pain drawings to define the spatial extent and location of a person's pain, but limited research has investigated who should perform the drawing. ObjectivesTo establish the inter-rater reliability of pain extent and location derived from three sets of pain drawings for people with chronic low back pain: one self-reported and two clinician-reported. Additionally, convergent validity of pain extent was assessed using the same dataset. DesignRepeated-measures cross-sectional study. MethodFifteen patients with chronic low back pain and a pool of eight clinicians were involved to assess the reliability of pain extent and location extracted by self-report and clinician-reported pain drawings. Inter-rater reliability of pain extent was computed using intraclass correlation coefficients (ICC) and Bland Altman analysis. Convergent validity of pain extent was assessed using Spearman's rank correlation. Inter-rater reliability of pain location was assessed using the Jaccard similarity index. ResultsThe inter-reliability analysis for pain extent, derived from self-reported and clinician-reported pain drawings, revealed ICC scores ranging from 0.39 to 0.51, all with wide confidence intervals. The mean Jaccard similarity indexes computed for pain location ranged from 0.60 to 0.65. Moderate to good correlation was found for pain extent derived by the sets of pain drawings. ConclusionsInter-rater reliability of pain extent and pain location derived from self-reported and clinician-reported pain drawings is poor in patients with chronic low back pain. The lack of reliability is also confirmed when considering only clinician-reported PDs. The convergent validity analysis of pain extent revealed that the two pain drawing approaches measure a similar construct.