Interrater Reliability Coefficients Research Articles

A sentiment corpus for the cryptocurrency financial domain: the CryptoLin corpus

AbstractThe objective of this paper is to describe Cryptocurrency Linguo (CryptoLin), a novel corpus containing 2683 cryptocurrency-related news articles covering more than a three-year period. CryptoLin was human-annotated with discrete values representing negative, neutral, and positive news respectively. Eighty-three people participated in the annotation process; each news title was randomly assigned and blindly annotated by three human annotators, one in each different cohort, followed by a consensus mechanism using simple voting. The selection of the annotators was intentionally made using three cohorts with students from a very diverse set of nationalities and educational backgrounds to minimize bias as much as possible. In case one of the annotators was in total disagreement with the other two (e.g., one negative vs two positive or one positive vs two negative), we considered this minority report and defaulted the labeling to neutral. Fleiss’s Kappa, Krippendorff’s Alpha, and Gwet’s AC1 inter-rater reliability coefficients demonstrate CryptoLin’s acceptable quality of inter-annotator agreement. The dataset also includes a text span with the three manual label annotations for further auditing of the annotation mechanism. To further assess the quality of the labeling and the usefulness of CryptoLin dataset, it incorporates four pretrained Sentiment Analysis models: Vader, Textblob, Flair, and FinBERT. Vader and FinBERT demonstrate reasonable performance in the CryptoLin dataset, indicating that the data was not annotated randomly and is therefore useful for further research1. FinBERT (negative) presents the best performance, indicating an advantage of being trained with financial news. Both the CryptoLin dataset and the Jupyter Notebook with the analysis, for reproducibility, are available at the project’s Github. Overall, CryptoLin aims to complement the current knowledge by providing a novel and publicly available Gadi and Ángel Sicilia (Cryptolin dataset and python jupyter notebooks reproducibility codes, 2022) cryptocurrency sentiment corpus and fostering research on the topic of cryptocurrency sentiment analysis and potential applications in behavioral science. This can be useful for businesses and policymakers who want to understand how cryptocurrencies are being used and how they might be regulated. Finally, the rules for selecting and assigning annotators make CryptoLin unique and interesting for new research in annotator selection, assignment, and biases.

Development of a diagnostic prediction model for giant cell arteritis by sequential application of Southend Giant Cell Arteritis Probability Score and ultrasonography: a prospective multicentre study

Giant cell arteritis is a critically ischaemic disease with protean manifestations that require urgent diagnosis and treatment. European Alliance of Associations for Rheumatology (EULAR) recommendations advocate ultrasonography as the first investigation for suspected giant cell arteritis. We developed a prediction tool that sequentially combines clinical assessment, as determined by the Southend Giant Cell Arteritis Probability Score (SGCAPS), with results of quantitative ultrasonography. This prospective, multicentre, inception cohort study included consecutive patients with suspected new onset giant cell arteritis referred to fast-track clinics (seven centres in Italy, the Netherlands, Spain, and UK). Final clinical diagnosis was established at 6 months. SGCAPS and quantitative ultrasonography of temporal and axillary arteries with three scores (ie, halo count, halo score, and OMERACT GCA Score [OGUS]) were performed at diagnosis. We developed prediction models for diagnosis of giant cell arteritis by multivariable logistic regression analysis with SGCAPS and each of the three ultrasonographic scores as predicting variables. We obtained intraclass correlation coefficient for inter-rater and intra-rater reliability in a separate patient-based reliability exercise with five patients and five observers. Between Oct 1, 2019, and June 30, 2022, we recruited and followed up 229 patients (150 [66%] women and 79 [34%] men; mean age 71 years [SD 10]), of whom 84 were diagnosed with giant cell arteritis and 145 with giant cell arteritis mimics (controls) at 6 months. SGCAPS and all three ultrasonographic scores discriminated well between patients with and without giant cell arteritis. A reliability exercise showed that the inter-rater and intra-rater reliability was high for all three ultrasonographic scores. The prediction model combining SGCAPS with the halo count, which was termed HAS-GCA score, was the most accurate model, with an optimism-adjusted C statistic of 0·969 (95% CI 0·952 to 0·990). The HAS-GCA score could classify 169 (74%) of 229 patients into either the low or high probability groups, with misclassification observed in two (2%) of 105 patients in the low probability group and two (3%) of 64 of patients in the high probability group. A nomogram for easy application of the score in daily practice was created. A prediction tool for giant cell arteritis (the HAS-GCA score), combining SGCAPS and the halo count, reliably confirms and excludes giant cell arteritis from giant cell arteritis mimics in fast-track clinics. These findings require confirmation in an independent, multicentre study. Royal College of Physicians of Ireland, FOREUM.

Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability.

Inter-rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the applicant is either selected or not. This final outcome is not considered in IRR, which instead focuses on the ratings of the individual subjects or objects. We outline the connection between the ratings' measurement model (used for IRR) and a binary classification framework. We develop a simple way of approximating the probability of correctly selecting the best applicants which allows us to compute error probabilities of the selection procedure (i.e., false positive and false negative rate) or their lower bounds. We draw connections between the IRR and the binary classification metrics, showing that binary classification metrics depend solely on the IRR coefficient and proportion of selected applicants. We assess the performance of the approximation in a simulation study and apply it in an example comparing the reliability of multiple grant peer review selection procedures. We also discuss other possible uses of the explored connections in other contexts, such as educational testing, psychological assessment, and health-related measurement, and implement the computations in the R package IRR2FPR.

Intra- and Inter-Rater Reliability of Magnetic Resonance Imaging Measurements of Supraspinatus Muscle Thickness, Acromiohumeral Distance, and Coracohumeral Distance in Patients with Shoulder Pain

Aim: The aim of this study was to evaluate the intra- and inter-rater reliability coefficients of the supraspinatus muscle thickness (SMT ), acromiohumeral distance (AHD), and coracohumeral distance (CHD) in patients with shoulder pain. Materials and Methods: This retrospective study included 80 patients who presented with complaints of shoulder pain at a single centre between January 2023 and January 2024. The supraspinatus muscle thickness, acromiohumeral distance, and coracohumeral distance measurements were obtained one week apart by two observers. The intraclass correlation coefficient (ICC), minimum detectable change, and standard error of measurement were subsequently calculated. Results: AHD intra-rater reliability results evaluated on coronal sections were excellent (Rater 1, ICC=0.96; 95% CI, 0.94-0.97; Rater 2 ICC=0.75; 95% CI, 0.61-0.84). AHD intra-rater reliability results evaluated on sagittal sections were excellent (Rater 1, ICC=0.94; 95% CI; 0.91-0.96; Rater 2 ICC=0.77; 95% CI, 0.64-0.85). The results of CHD measured using coronal sections were good (Rater 1, ICC=0.85; 95% CI, 0.76-0.90; Rater 2 ICC=0.82; 95% CI, 0.71-0.88). SMT measurement results measured using coronal sections were excellent (Rater 1, ICC=0.98; CI, 0.98-0.99; Rater 2, ICC=0.89; 95% CI, 0.82-0.93).The inter-rater reliability values were good for AHD-Coronal (ICC= 0.75; 95% CI, 0.61-0.84) and AHD-Sagittal (ICC=0.86; 95% CI, 0.79-0.91), were fair for CHD evaluated on coronal sections (ICC=0.74; %95 CI, 0.58-0.83), and were excellent for SMT evaluated on coronal sections (ICC=0.92; 95% CI, 0.87-0.95). Conclusion: These results suggest that magnetic resonance imaging measurements of the supraspinatus muscle thickness and acromiohumeral distance can be reliable and consistent for the clinical evaluation of patients with shoulder pain. Keywords: Reliability, shoulder measurement, magnetic resonance imaging

Meta-analytical estimates of interrater reliability for direct supervisor performance ratings: Optimism under optimal measurement designs.

Performance appraisal (PA) is used for various organizational purposes and is vital to human resources practices. Despite this, current estimates of PA reliability are low, leading to decades of criticism regarding the use of PA in organizational contexts. In this article, we argue that current meta-analytical interrater reliability (IRR) coefficients are underestimates and do not reflect the reliability of interest to most practitioners and researchers-the reliability of an employee's direct supervisor. To establish the reliability of direct supervisor ratings, those making PA ratings must directly supervise employee job performance instead of nonparallel rater designs (e.g., direct supervisor ratings correlated with ratings from a more senior leader). The current meta-analysis identified 22 independent samples that met this more restrictive study inclusion criterion, finding an average observed IRR of .65. We also report reliability estimates for several important contextual moderators, including whether ratings were completed in operational settings (.60) or for research purposes (.67). In sum, we argue that this study's meta-analytical IRR estimates are the best available estimates of direct supervisor reliability and should be used to guide future research and practice. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Preoperative validation of edema-corrected tractography in neurosurgical practice: translating surgeon insights into novel software implementation.

Peritumoral edema alters diffusion anisotropy, resulting in false negatives in tractography reconstructions negatively impacting surgical decision-making. With supratotal resections tied to survival benefit in glioma patients, advanced diffusion modeling is critical to visualize fibers within the peritumoral zone to prevent eloquent fiber transection thereafter. A preoperative assessment paradigm is therefore warranted to systematically evaluate multi-subject tractograms along clinically meaningful parameters. We propose a novel noninvasive surgically-focused survey to evaluate the benefits of a tractography algorithm for preoperative planning, subsequently applied to Synaptive Medical's free-water correction algorithm developed for clinically feasible single-shell DTI data. Ten neurosurgeons participated in the study and were presented with patient datasets containing histological lesions of varying degrees of edema. They were asked to compare standard (uncorrected) tractography reconstructions overlaid onto anatomical images with enhanced (corrected) reconstructions. The raters assessed the datasets in terms of overall data quality, tract alteration patterns, and the impact of the correction on lesion definition, brain-tumor interface, and optimal surgical pathway. Inter-rater reliability coefficients were calculated, and statistical comparisons were made. Standard tractography was perceived as problematic in areas proximal to the lesion, presenting with significant tract reduction that challenged assessment of the brain-tumor interface and of tract infiltration. With correction applied, significant reduction in false negatives were reported along with additional insight into tract infiltration. Significant positive correlations were shown between favorable responses to the correction algorithm and the lesion-to-edema ratio, such that the correction offered further clarification in increasingly edematous and malignant lesions. Lastly, the correction was perceived to introduce false tracts in CSF spaces and - to a lesser degree - the grey-white matter interface, highlighting the need for noise mitigation. As a result, the algorithm was modified by free-water-parameterizing the tractography dataset and introducing a novel adaptive thresholding tool for customizable correction guided by the surgeon's discretion. Here we translate surgeon insights into a clinically deployable software implementation capable of recovering peritumoral tracts in edematous zones while mitigating artifacts through the introduction of a novel and adaptive case-specific correction tool. Together, these advances maximize tractography's clinical potential to personalize surgical decisions when faced with complex pathologies.

K-Alpha Calculator–Krippendorff's Alpha Calculator: A user-friendly tool for computing Krippendorff's Alpha inter-rater reliability coefficient

Krippendorff's Alpha is a measure for assessing inter-rater reliability, facilitating triangulated evaluations among multiple raters. Indeed, Krippendorff's Alpha is key in validating the dependability of human assessments, thereby reinforcing the robustness of human-based choices in contexts where interpretive variability could otherwise undermine the research outcomes. Despite its versatility across various data types, the procedure for computing this coefficient might limit its applicability for researchers unfamiliar with specialised statistical software. Addressing this issue, this paper introduces the “K-Alpha Calculator”—Krippendorff's Alpha Calculator—a freely accessible, user-friendly, web application available at https://www.k-alpha.org designed for easy computation of Krippendorff's Alpha. By offering a web interface without any software dependency, the K-Alpha Calculator seeks to promote the broader integration of such a reliability metric in research processes. We envisage that the tool's contribution could support researchers in enhancing methodological rigour, fostering conclusions are rooted in triangulated interpretations and assessments. The K-Alpha Calculator is both a computational tool and an educational resource, encouraging more researchers to include reliability measures in their studies.•Development of the K-Alpha Calculator.The study centres on the introduction of a web-based application: the K-Alpha Calculator. The tool is developed to simplify the calculation of Krippendorff's Alpha without any required software dependency.•User-friendly interface for diverse data types.The K-Alpha Calculator is characterised by its accessibility and user-friendliness, designed to simplify the procedures for computing the Krippendorff's Alpha. Its utility extends to various data types, which aligns with the intrinsic versatility of this coefficient.

Scoring Methods for Progress Monitoring Using CBM in Early Writing

This initial exploratory study assessed the evidence of reliability, criterion validity, and growth of scores obtained from total correct word sequences (CWS), correct word sequences per response (CWSR), and a trait-based rubric used with a curriculum-based measure of writing (CBM-W) for progress monitoring. Participants were 26 writers with or at risk for a learning disability (LD) in grades 2-3. Participants were given the CBM-W task twice weekly over 6 weeks and were given the Spelling and Writing Samples subtests of the Woodcock-Johnson Tests of Achievement-III at pre- and posttest. Results indicated large inter-rater reliability coefficients but mixed alternate form reliability and validity evidence. CWS was the only metric to demonstrate significant growth (r = .60). CWS has the strongest evidence for use as a progress monitoring scoring metric for students at risk for LD, although information from CWSR and rubric scores could be instructionally useful.

Measurement properties of Brief-BESTest scores from children, adolescents, and youth with visual impairments

Purpose: This study aimed to vet the measurement properties of Brief-BESTest scores in children, adolescents, and youth with visual impairment and blindness (YVI). Methods: A cross-sectional sample of YVI (N = 101) completed the Brief-BESTest, a modified version of the Y-Balance Test, the 360-degree turn test, bipedal quiet stance, and Activity-Specific Balance Confidence Scales. Thirty-seven YVI completed test-retest of the Brief-BESTest using a one-year interval. Using classical test theory, various forms of total and item-level Brief-BESTest score reliability and validity were investigated in YVI. Results: All inter-rater reliability coefficients were ≥ .80. When considering the eight items of the Brief-BESTest, 27 of the 28 possible correlations were statistically significant (p<.05). Various internal consistency and item difficulty results were strong. When taking total Brief-BESTest scores and their association with the complementary balance tasks/metrics into account, 11 of 13 associations were statistically significant (p<.05) providing strong convergent validity evidence. Being multimorbid and degree of vision significantly predicted total Brief-BESTest scores (p<.001) suggesting construct (i.e. known groups) validity. Numerous test-retest results (e.g. coefficients, limits of agreement) following the one-year interval were indicative of score stability. Conclusion: Practitioners and researchers should have confidence in, and consider adopting, the Brief-BESTest to examine multidimensional balance in YVI.

Indonesian emotion regulation scale for students based on reappraisal and suppression factor: The Rasch analysis

&lt;span lang="EN-US"&gt;Every human being has emotional turmoil, which will be a serious problem if they cannot control it. The importance of emotion regulation has become one of the bases for developing a measure of emotion regulation for students, the Indonesian emotion regulation scale (IERS). No research yet describes an instrument that measures students’ level of emotion regulation in Indonesia. IERS consists of nine items in two aspects: reappraisal and suppression factor. The content validity test involves two experts in the field of psychometrics. In comparison, the construct validity test involved 354 high school students in Yogyakarta, Indonesia. Data analysis using inter-rater reliability (IRR) coefficient of Cohen’s kappa and Rasch analysis. Based on the study of the IRR coefficient of Cohen’s kappa, two experts agree on the acceptability of the IERS statement items. Besides that, the results of the application of Rasch analysis show that IERS is good, precise, and conforms with the model. IERS is a reliable and valid tool to measure students’ level of emotion regulation accurately. This paper discusses the implications and recommendations for further research for the implementation of guidance and counseling containing the value of emotion regulation as a follow-up to the performance of IERS.&lt;/span&gt;

Assessing the reliability and feasibility of frailty screening tools among hospitalised older adults

Background The 29-item Frail-Physical, Psychological and Social (Frail-PPS) and the 14-item Frailty Assessment Measure (FAM) were developed in Singapore to identify risk of frailty among community dwelling older adults and validated for use among hospitalised older adults. Objectives This study aimed to establish the interrater reliability and feasibility of the two screening tools ‒ FAM and Frail-PPS, for assessing frailty among hospitalized older adults. Methods The FAM and Frail-PPS were administered during the initial nursing assessment by nurses to 62 patients aged 65 years and older within 24 h of admission. Interrater reliability, convergent validity and intraclass correlation coefficients (ICCs) were established. Feasibility was determined by the time of administration. Results ICCs for Frail-PPS and FAM were 0.95 and 0.95 respectively. A positive correlation was established (r = 0.97). The administration time for Frail-PPS averaged 6.7 min, and 3.3 min for FAM. Conclusions The FAM, with its high reliability and convergent validity, as well as shorter administration time, may be the preferred screening tool for use in acute care settings.

Adaptation and Validation of the Standardized Swallowing Assessment Tool for Patients With Moderate-Severe Traumatic Brain Injury and Cervical Spinal Cord Injury.

Oropharyngeal dysphagia is common in moderate-severe traumatic brain injury (TBI) and cervical spinal cord injury (SCI) patients and can have serious consequences. Delaying feeding in these patients can also be detrimental. Nonetheless, the psychometric properties of screening tools that can promptly identify dysphagia have never been tested in these neurotrauma populations. This study aimed to: (1) adapt, translate, and validate the content of the French-Canadian version of the the Standardized Swallowing Assessment (SSA) tool to meet the needs of moderate-severe TBI and cervical SCI patients, (2) examine its inter-rater reliability and criterion-concurrent validation, and (3) evaluate its clinical utility from the perspectives of critical care nurses. The SSA tool was adapted and translated using an integrated method for the cultural adaptation and translation of tools. Eleven experts participated in the adaptation of the SSA tool, which led to the clarification of one item, as well as a new step and instructions for the screening procedure. Content validation (i.e., item and scale relevance) was evaluated by multidisciplinary team members (n = 17). The mean content validity index (CVI) score was 0.97 for the entire scale, while the mean CVI scores for individual items ranged from 0.82 to 1.0. A total of 60 neurotrauma patients were enrolled for inter-rater reliability and criterion-concurrent validation. Interrater reliability was determined by comparing two scores: one score from nurses responsible for the care of enrolled patients and one score from the research nurse. The weighted kappa coefficients for inter-rater reliability were 0.86 for moderate-severe TBI patients (n = 30) and 0.73 for cervical SCI patients (n = 30). A speech language therapist (SLT) also assessed dysphagia and results were used as the standard clinical reference criterion to determine concurrent validity (sensibility and specificity) of the adapted SSA tool. The sensitivity and specificity were 92% and 50% for moderate-severe TBI, and 77% and 75% for cervical SCI, respectively. The positive predictive value (PPV) and the negative predictive value (NPV) were 65% and 87% in TBI patients, and 75% and 76% in SCI patients. Test accuracy was 71% and 77% for these same groups. The clinical utility of the tool was evaluated according to the following domains: appropriateness, accessibility of the required material, applicability, perceived effectiveness, and acceptability. Acceptability was the only domain with a level of agreement <80% (74%) among trauma critical care nurses (n = 49). Findings support the content validation and inter-rater reliability of the adapted French-Canadian version of the SSA tool in moderate-severe TBI and cervical SCI patients. Sensitivity was acceptable in both groups, but the specificity was lower, especially in moderate-severe TBI patients. Further validation of the adapted French-Canadian version of the SSA tool is needed in neurotrauma patients to confirm these results and to ensure safe dysphagia screening while avoiding oral feeding deferrals.

Six-Lead Electrocardiography Enables Identification of Rhythm and Conduction Anomalies of Patients in the Telemedicine-Based, Hospital-at-Home Setting: A Prospective Validation Study.

The hospital-at-home (HAH) model is a viable alternative for conventional in-hospital stays worldwide. Serum electrolyte abnormalities are common in acute patients, especially in those with many comorbidities. Pathologic changes in cardiac electrophysiology pose a potential risk during HAH stays. Periodical electrocardiogram (ECG) tracing is therefore advised, but few studies have evaluated the accuracy and efficiency of compact, self-activated ECG devices in HAH settings. This study aimed to evaluate the reliability of such a device in comparison with a standard 12-lead ECG. We prospectively recruited consecutive patients admitted to the Sheba Beyond Virtual Hospital, in the HAH department, during a 3-month duration. Each patient underwent a 12-lead ECG recording using the legacy device and a consecutive recording by a compact six-lead device. Baseline patient characteristics during hospitalization were collected. The level of agreement between devices was measured by Cohen's kappa coefficient for inter-rater reliability (Ϗ). Fifty patients were included in the study (median age 80 years, IQR 14). In total, 26 (52%) had electrolyte disturbances. Abnormal D-dimer values were observed in 33 (66%) patients, and 12 (24%) patients had elevated troponin values. We found a level of 94.5% raw agreement between devices with regards to nine of the options included in the automatic read-out of the legacy device. The calculated Ϗ was 0.72, classified as a substantial consensus. The rate of raw consensus regarding the ECG intervals' measurement (PR, RR, and QT) was 78.5%, and the calculated Ϗ was 0.42, corresponding to a moderate level of agreement. This is the first report to our knowledge regarding the feasibility of using a compact, six-lead ECG device in the setting of an HAH to be safe and bearing satisfying agreement level with a legacy, 12-lead ECG device, enabling quick, accessible arrythmia detection in this setting. Our findings bear a promise to the future development of telemedicine-based hospital-at-home methodology.

Auditory-Perceptual Assessments of Cough: Characterizing Rater Reliability and the Effects of a Standardized Training Protocol

Introduction: Auditory-perceptual assessments of cough are commonly used by speech-language pathologists working with people with swallowing disorders with emerging evidence beginning to demonstrate their validity; however, their reliability among novice clinicians is unknown. Therefore, the primary aim of this study was to characterize the reliability of auditory-perceptual assessments of cough among a group of novice clinicians. As a secondary aim, we assessed the effects of a standardized training protocol on the reliability of auditory-perceptual assessments of cough. Methods: Twelve novice clinicians blindly rated ten auditory-perceptual cough descriptors for 120 cough audio clips. Standardized training was then completed by the group of clinicians. The same cough audio clips were then re-randomized and blindly rated. Reliability was analyzed pre- and post-training within each clinician (intra-rater), between each unique pair of raters (dyad-level inter-rater), and for the entire group of raters (group-level inter-rater) using intraclass correlation coefficients and Cohen’s Kappa. Results: Pre-training reliability was greatest for measures of strength, effectiveness, and normality and lowest when judging the type of expiratory maneuver (cough, throat clear, huff, other). The measures that improved the most with training were ratings of perceived crispness, amount of voicing, and type of expiratory maneuver. Intra-rater reliability coefficients ranged from 0.580 to 0.903 pre-training and 0.756–0.904 post-training. Dyad-level inter-rater reliability coefficients ranged from 0.295 to 0.745 pre-training and 0.450–0.804 post-training. Group-level inter-rater reliability coefficients ranged from 0.454 to 0.919 pre-training and 0.558–0.948 post-training. Conclusion: Reliability of auditory-perceptual assessments varied across perceptual cough descriptors, but all appeared within the range of what has been historically reported for auditory-perceptual assessments of voice and visual-perceptual assessments of swallowing and cough airflow. Reliability improved for most cough descriptors following 30–60 min of standardized training. Future research is needed to examine the validity of auditory-perceptual assessments of cough by assessing the relationship between perceptual cough descriptors and instrumental measures of cough effectiveness to better understand the role of perceptual assessments in clinical practice.

ChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results.

People living with dementia or other cognitive decline and their caregivers (PLWD) increasingly rely on the web to find information about their condition and available resources and services. The recent advancements in large language models (LLMs), such as ChatGPT, provide a new alternative to the more traditional web search engines, such as Google. This study compared the quality of the results of ChatGPT and Google for a collection of PLWD-related queries. A set of 30 informational and 30 service delivery (transactional) PLWD-related queries were selected and submitted to both Google and ChatGPT. Three domain experts assessed the results for their currency of information, reliability of the source, objectivity, relevance to the query, and similarity of their response. The readability of the results was also analyzed. Interrater reliability coefficients were calculated for all outcomes. Google had superior currency and higher reliability. ChatGPT results were evaluated as more objective. ChatGPT had a significantly higher response relevance, while Google often drew upon sources that were referral services for dementia care or service providers themselves. The readability was low for both platforms, especially for ChatGPT (mean grade level 12.17, SD 1.94) compared to Google (mean grade level 9.86, SD 3.47). The similarity between the content of ChatGPT and Google responses was rated as high for 13 (21.7%) responses, medium for 16 (26.7%) responses, and low for 31 (51.6%) responses. Both Google and ChatGPT have strengths and weaknesses. ChatGPT rarely includes the source of a result. Google more often provides a date for and a known reliable source of the response compared to ChatGPT, whereas ChatGPT supplies more relevant responses to queries. The results of ChatGPT may be out of date and often do not specify a validity time stamp. Google sometimes returns results based on commercial entities. The readability scores for both indicate that responses are often not appropriate for persons with low health literacy skills. In the future, the addition of both the source and the date of health-related information and availability in other languages may increase the value of these platforms for both nonmedical and medical professionals.

P-556 Digital platform for remote pre-treatment assessment: Perceived usefulness and quality by patients and doctors

Abstract Study question Is a digital platform for remote pre-treatment assessment perceived as useful and able to fulfil the needs of patients and clinicians? Summary answer The digital platform was found to be of high quality, useful, and answer the expressed needs of the targeted audiences in cross-cultural samples. What is known already Infertility is a life crisis. It takes on average 3.2 years to be diagnosed, 1.6 years to access a specialist, and 2.2 years to complete treatment. The pre-assessment phase is time-consuming, costly, often disruptive to the patient’s life, and may require traveling abroad. Patients need organized care where they can schedule/order tests, access information about their specific health conditions, and reach out to professionals. On the other hand, clinicians find themselves trying to assess a large number of patients before initiating treatment abroad without having a unified platform where they can review the patient’s clinical history and treatment plan. Study design, size, duration Study I explored patients’ experience with pre-treatment care and their perceived usefulness of the Patient’s Portal. An online questionnaire of 25 questions was used. Study II explored clinicians’ perceived usefulness and quality of the Doctor’s Portal. A semi-structured interview was conducted and an online questionnaire of 17 items was used. Both questionnaires had multiple options, Likert-scale, and open-ended items. The studies followed a cross-sectional design. The data were analyzed using IBM SPSS Statistics, Version 28.0. Participants/materials, setting, methods Study I had a cross-cultural sample of 59 women (M = 37.2, SD = 4.9 years). 41% are trying to conceive for more than 24 months, 89.8% were in a heterosexual relationship, and all were undergoing treatment/pursuing a clinic. The internal consistency of the questionnaire is α=.97. Study II had a cross-cultural sample (N = 8) of clinic leaders, all from different clinics. The interview questions were open-ended to capture unprompted opinions. The internal consistency of the questionnaire is α=.94. Main results and the role of chance Study I: People who took &amp;gt;4 weeks to have tests done/get the results scored significantly higher values than people who took less time in terms of how much they had to pay, struggled to find where to have the tests done, had difficulty managing emotions, and would’ve preferred their clinic had scheduled the testing for them. Overall, 61.02% reported the testing had been disruptive to their lives. Regarding the perceived usefulness of the platform, means &amp;gt;4 and a mode of 5 on a scale of 1-5 were obtained. We also found that regardless of the woman’s age or how long she’s been trying to conceive, she’d benefit from having this platform available. Study II: 75% made mention the high-quality of the platform, 50% said it would improve their current practice, covered information comprehensively, and was user-friendly, and 37.5% said it would make the process of diagnosing patients faster. Suggestions were used to adapt the platform. Regarding the quality questionnaire, the means varied between 7.86 and 9.14 (SD between .69 and 1.11) and the mode between 8 and 9 on a scale of 1-10. The inter-rater reliability coefficient was .94. This means the perceived high-quality and usefulness of the platform were consensual. Limitations, reasons for caution Small sample sizes condition the extrapolation of the findings. Further studies of the perceived usefulness and quality of the Patients’ Portal and Doctors’ Portal after the implementation of the digital platform are required. Wider implications of the findings The digital platform for remote pre-treatment assessment is currently available and being used by clinicians and patients. In the future it, can be further developed to develop and train an Artificial Intelligence algorithm to Diagnose &amp; Predict Fertility Treatment Outcomes with Personalised Patient Pathways. Trial registration number not applicable

Project-based Learning at a Saudi University: Faculty and Student Feedback

Abstract Education for sustainable development (ESD) flourishes with student-centered learning methods, including project-based learning (PBL). These methods are still new to Saudi Arabian university students who tend to experience teacher-centered approaches. Saudi Arabia’s recent policy decision to shift to a knowledge-based economy that respects sustainability will benefit from PBL-enhanced learning. This small-scale qualitative study investigated faculty and students’ feedback on their experience with PBL. Four faculty members (three female, one male in accounting, sports management, and information management) and six female students (financial sciences or accounting) from an Eastern Province university college were interviewed (45 minutes) in 2020–2021 using Zoom. Without exception, the instructors were in favor of PBL as a teaching strategy, and students’ overall experience could be described as positive. A thematic analysis (89 % inter-rater reliability coefficient) generated three themes: (a) beneficial learning strategy, (b) beneficial but challenging learning strategy with room for improvement, and (c) sustained student support over the semester with exceptions. Saudi universities are encouraged to employ PBL in all courses and use assessment methods conducive to student-centered learning. Universities are encouraged to streamline the project approval process, which was quite involved herein. Communications with community partnerships are important, and faculty and students should be thoroughly in-serviced and oriented to PBL.

The use of slice encoding for metal artifact correction (SEMAC) sequencing improves the diagnostic evaluation of graft integrity following anterior cruciate ligament reconstruction

ObjectivesTo determine whether magnetic resonance imaging (MRI) with metal artifact reduction sequencing is superior to conventional knee MRI in the evaluation of an injured anterior cruciate ligament (ACL) graft, where visualisation on conventional MRI can be limited by the metal artifact from fixation devices. MethodsEighteen patients underwent conventional MRI sequence (proton density fat saturated [PDFS]) and two types of metal artifact reduction sequencing MRI (WARP, slice encoding for metal artifact correction (SEMAC); Siemens) following a secondary injury to their ACL reconstructed knee. Six raters with experience in knee MRI evaluation reviewed sagittal PDFS, WARP, and SEMAC sequences, providing semi-quantitative grades for visualisation and diagnostic confidence assessing the ACL, posterior cruciate ligament , menisci, tibial and femoral tunnel margins, and articular cartilage. Intra-class correlation coefficients for inter-rater reliability were evaluated. The 6-rater mean scores for the visualisation and diagnostic confidence derived from each sequence were compared using the Friedman test for multiple paired samples. ResultsNo statistically significant difference in the ACL visualisation among the sequences was found (p ​= ​0.193). Further, a subgroup analysis was performed in cases evaluated as “moderately blurry” or “indistinct ACL visualisation” on PDFS (58% of cases). SEMAC significantly improved diagnostic confidence in ACL visualisation (p ​= ​0.041) and ACL graft rupture (p ​= ​0.044) compared to PDFS.There was no statistically significant difference in the inter-observer reliability between sequences. The WARP sequence added 2.84 ​± ​0.69 ​min, while SEMAC added 2.95 ​± ​0.40 ​min to the standard knee MRI scan time. Conclusionuse of the SEMAC metal reduction sequence significantly improved diagnostic accuracy and confidence in the detection of ACL graft rupture in cases where the ACL was moderately blurry or indistinct on the PDFS sequence. This sequence should be considered as an adjunct to conventional PDFS in cases where graft visualisation is limited by the metal artifact from fixation devices. Level of evidenceIII.

Evaluation of vincristine-induced peripheral neuropathy in children with cancer: Turkish validity and reliability study

BackgroundThe evaluation of peripheral neuropathy in children receiving Vincristine treatment is challenging. This study examined the Turkish validity and reliability of the Total Neuropathy Score-Pediatric Vincristine (TNS-PV) measurement tool, which can measure Vincristine-induced peripheral neuropathy symptoms in children with cancer. MethodsA total of 53 children aged 5–17 years who received Vincristine treatment in two pediatric hematology-oncology centers participated in the study. Data was collected using the Total Neuropathy Score-Pediatric Vincristine (TNS-PV), the Common Terminology Criteria for Adverse Events (CTCAE), the Wong-Baker FACES Pain Scale, and the Adolescent Pediatric Pain Tool (APPT). The correlation between the TNS-PV total score and other scales and the inter-rater reliability coefficient was evaluated. FindingsOf the children, 81.1% were diagnosed with ALL and 13.2% with Ewing Sarcoma. Cronbach's alpha values of form A and B of the TNS-PV scale were 0.628 and 0.639, respectively. As the cumulative Vincristine dose increased, the children's scores on TNS-PV were higher. A moderate and significant positive correlation was found between the TNS-PV form A total score and the worst subjective symptoms a, b (A), strength, tendon reflexes, and autonomic / constipation (r = 0.441, r = 0.545, r = 0.472, r = 0.536, p < 0.01). DiscussionThe TNS-PV form B total score was found to have a moderate level, significant correlation with CTCAE sensory neuropathy score and Wong-Baker FACES Pain Scale, and a high level, significant positive correlation with CTCAE motor neuropathy score. Application to practiceThe TNS-PV is valid and reliable for measuring Vincristine-induced peripheral neuropathy in practice in Turkish children 5 years and older.

The Curriculum–TIMSS Alignment for Grade 8 Science from 2007 to 2019: A Case of South Africa and Singapore

The present study explores the validity of inferences that may be drawn from the Trends in International Mathematics and Science Study (TIMSS) learner achievement data by measuring the South African curriculum–TIMSS alignment for the Grade 8 science assessments from 2007 to 2019. The Singapore curriculum–TIMSS alignment indices are a fair benchmark for comparison, as Singapore is the best-performing country in the TIMSS. The research questions that guide the present study included the calculation and comparison of the South African curriculum–TIMSS and Singapore curriculum–TIMSS alignment indices. The study analysed 585 South African curriculum items, 659 Singapore curriculum items and 1025 TIMSS assessment items from 2007 to 2019. The present study employed four raters and achieved an inter-rater reliability coefficient of 0.85. The South African curriculum–TIMSS analysis achieved an average content alignment index of 0.34, and an average cognitive alignment index of 0.54. As a benchmark, the Singapore curriculum–TIMSS analysis achieved an average content alignment index of 0.36 and an average cognitive alignment index of 0.74. The literature reviewed supports acceptable content and cognitive alignment indices of 0.67 and 0.65, respectively. The present study concludes that valid inferences from learner performance in the TIMSS may not be drawn for the case of South Africa, and even for Singapore valid inferences are based on cognitive alignment only. However, South Africa’s continued participation in the TIMSS is encouraged as the literature supports many benefits associated with the TIMSS non-achievement data.