Reliability Of Scores Research Articles

The response readiness tool: An instrument to measure knowledge, skills, and attitude after taking a stop the Bleed® course.

Stop the Bleed® is an emergency response course that offers a certificate of completion but no ending assessment. The purpose of this educational study was to develop and test the Response Readiness Tool (RRT) that measures learning of participants after taking Stop the Bleed®. The study used a pre-/post-test design to measure knowledge and attitudes, and a post-test only for skills. Participants were recruited from existing Stop the Bleed® courses. Knowledge was measured with a 10-item questionnaire, attitude was measured with five Likert-style questions, and skills were measured by observing simulated tourniquet placement. Ninety-five participants were recruited over two semesters. There was a significant difference in the knowledge and attitude scores indicating participants' improvement in learning and attitude toward responding. Internal consistency reliability of scores showed moderate reliability with Cronbach's Alpha of 0.73 and McDonald's Omega of 0.75. A positive correlation was found between expected proficiency and pre-test indicating the tool had construct validity. RRT provides Stop the Bleed® instructors with an instrument to measure knowledge, skills, and attitude of participants after taking Stop the Bleed®. Statistics show moderate reliability and validity; however, larger samples are needed for full psychometric testing. This publication presents the revised tool after the completion of this study.

Bayesian Maximal Reliability Evaluation Using Latent Variable Modeling

ABSTRACT We discuss an application of Bayesian factor analysis for estimation of the optimal linear combination and associated maximal reliability of a multi-component measuring instrument. The described procedure yields point and credibility interval estimates of this reliability coefficient, which are readily obtained in educational and behavioral measurement research. In addition, the outlined method permits evaluation of the gain in measurement consistency resulting from utilizing the maximal reliability coefficient instead of the traditionally used overall sum score reliability. The discussed Bayesian inference approach is applicable with widely available software in empirical studies, and is illustrated using a data example.

Individual differences in emotion-induced blindness: Are they reliable and what do they measure?

The emotion-induced-blindness (EIB) paradigm has been extensively used to investigate attentional biases to emotionally salient stimuli. However, the low reliability of EIB scores (the difference in performance between the neutral and emotionally salient condition) limits the effectiveness of the paradigm for investigating individual differences. Here, across two studies, we investigated whether we could improve the reliability of EIB scores. In Experiment 1, we introduced a mid-intensity emotionally salient stimuli condition, with the goal of obtaining a wider range of EIB magnitudes to promote reliability. In Experiment 2, we sought to reduce the attentional oddball effect, so we created a modified EIB paradigm by removing the filler images. Neither of these approaches improved the reliability of the EIB scores. Reliability for the high- and mid-intensity EIB difference scores were low, while reliability of the scores for absolute performance (neutral, high-, and mid-intensity) were high and the scores werealso highly correlated, even though overall performance in the emotionally salient conditions were significantly worse than in the neutral conditions. Given these results, we can conclude that while emotionally salient stimuli impair performance in the EIB task compared with the neutral condition, the strong correlation between the emotionally salient and neutral conditions means that while EIB can be used to investigate individual differences in attentional control, it is not selective to individual differences in attentional biases to emotionally salient stimuli.

Spanish Version of the Netherlands Empowerment List: Adaptation, Psychometric Validation, and Gender Differential Item Functioning Analysis

AbstractThere is a lack of valid instruments to measure empowerment in Spanish-speaking populations. This study aimed to adapt the Netherlands Empowerment List (NEL), a 40-item scale, into Spanish and to test its psychometric properties examining its dimensional structure, internal consistency, temporal stability, relationships with other variables, and differential item functioning by gender. Participants were 406 users of mental health community rehabilitation services (52.5% male), with a mean age of 47.8 years. Results confirmed a six first-order factor structure of the scale. Internal consistency was excellent for the total score and ranged from excellent to adequate for subscale scores. Temporal stability was excellent for four subscales and good for the remaining two. The analysis of relationships between the Spanish NEL with other variables (i.e., empowerment, recovery, hope, perceived social support) provided additional evidence of its validity. Although four items showed uniform differential item functioning by gender, the effect size was negligible. The Spanish NEL yields valid and reliable scores, and it may be used to assess empowerment in Spanish-speaking countries.

Breast density prediction from low and standard dose mammograms using deep learning: effect of image resolution and model training approach on prediction quality

Purpose. To improve breast cancer risk prediction for young women, we have developed deep learning methods to estimate mammographic density from low dose mammograms taken at approximately 1/10th of the usual dose. We investigate the quality and reliability of the density scores produced on low dose mammograms focussing on how image resolution and levels of training affect the low dose predictions. Methods. Deep learning models are developed and tested, with two feature extraction methods and an end-to-end trained method, on five different resolutions of 15,290 standard dose and simulated low dose mammograms with known labels. The models are further tested on a dataset with 296 matching standard and real low dose images allowing performance on the low dose images to be ascertained. Results. Prediction quality on standard and simulated low dose images compared to labels is similar for all equivalent model training and image resolution versions. Increasing resolution results in improved performance of both feature extraction methods for standard and simulated low dose images, while the trained models show high performance across the resolutions. For the trained models the Spearman rank correlation coefficient between predictions of standard and low dose images at low resolution is 0.951 (0.937 to 0.960) and at the highest resolution 0.956 (0.942 to 0.965). If pairs of model predictions are averaged, similarity increases. Conclusions. Deep learning mammographic density predictions on low dose mammograms are highly correlated with standard dose equivalents for feature extraction and end-to-end approaches across multiple image resolutions. Deep learning models can reliably make high quality mammographic density predictions on low dose mammograms.

Evaluation of Ema, Töllner and Rodwell scores in the diagnosis of neonatal sepsis

Introduction: There are no specific signs, symptoms and rapid laboratory tests to definitively diagnose sepsis in the neonatal period. This makes it difficult to diagnose sepsis in newborns in a timely manner and initiate appropriate treatment, or causes babies without sepsis to receive unnecessary antibiotics. Purpose: Therefore, in this study, we planned to investigate the clinical adequacy and reliability of EMA (European Medicines Agency), Töllner and Rodwell hematological scoring in the early diagnosis of neonatal sepsis. Method: EMA, Töllner and Rodwell hematological scoring was performed on each patient. Complete blood count, peripheral smear, C-reactive protein, procalcitonin, blood gas and blood sugar values of newborn babies with suspected sepsis were recorded, blood culture, urine culture and cerebrospinal fluid examination and cultures taken when necessary were evaluated. Using statistical analysis, the 'Positive Expected Value and Negative Expected Values' ratios of the scores were obtained, and the performance results were examined. Results: 95 newborns with a preliminary clinical diagnosis of sepsis were included in the study. These babies were divided into two groups: clinical (n:71) and proven sepsis (n:24) according to blood culture results. Positive and negative predictive values of scoring systems in definitive sepsis diagnosis; for EMA respectively; 21.5%, 56.3% for Töllner; It was determined as 31.3%, 77.8%, and 100%, 77.8% for Rodwell. Conclusion: Our study showed that clinician opinion and standard laboratory tests are limited in the diagnosis of neonatal sepsis, and Rodwell hematological scoring is more prominent in recognizing proven sepsis compared to the other two scores .

Pan-enteric Crohn's capsule (Eliakim) score reliability and responsiveness to change in active Crohn's disease.

Pan-enteric capsule endoscopy (PillCam Crohn's capsule [PCC]) is a useful tool in diagnosing and monitoring Crohn's disease (CD). Eliakim score [ES] reliability and strong correlation to Lewis score (LS) and inflammatory biomarkers have been previously demonstrated using PCC in quiescent CD. We aimed to examine ES performance in active CD and its responsiveness to clinical/biochemical change over time. Patients with CD who have started biologics were included, and were prospectively followed with clinical visits, biomarkers, and PCC at baseline, after 14 and 52 weeks. Crohn's disease activity index (CDAI), C-reactive protein (CRP) and fecal-calprotectin (FC) levels were collected, and LS and ES were calculated (independently reviewed by two experienced readers). Inter-class classification (ICC), Spearman's baseline correlation, and repeated-measures correlation (RMC) analyses were performed. 74 patients were included (age: 30.5 [23.3-45.0] years-old, male-50%). 142 PCCs were read (baseline-62, week-14-58, week-52-22). Inter-rater agreement was high for both LS and ES (ICC: 0.872 [p<0.001] and 0.925 [<0.001], respectively). Baseline correlations between FC&ES (r=0.509 [p<0.001]) and FC&LS (r=0.467 [p<0.001]) were comparable (p=0.56). RMC between the inflammatory biomarkers and ES were higher than between the former and LS (Reader-1: CRP- r=0.306 vs. r=0.138 [p=0.057], FC- r=0.479 vs. r=0.297 [p=0.034]; Reader-2 CRP- r=0.376 vs. r=0.204 [p=0.035], FC- r=0.549 vs. r=0.412 [p=0.075]). Moreover, ES was better correlated to CDAI than LS (p=0.036). ES is a reliable scoring system in assessing pan-enteric mucosal inflammation in active CD, and might have a better responsiveness to clinical/biochemical change over time compared to LS.

Neuropsychiatric symptoms and metamemory across the life span: psychometric properties of the German Multifactorial Memory Questionnaire (MMQ).

We assessed the psychometric properties, established normative data for the German Multifactorial Memory Questionnaire (MMQ), and analyzed its association with neuropsychiatric factors across the life span to provide a validated metamemory assessment for a German-speaking population. The three MMQ scales (memory satisfaction, self-rated ability, and strategy application) were translated into German, considering cultural, linguistic, and conceptual aspects. To validate the MMQ and assess associations with neuropsychiatric factors, the Complainer Profile Identification, Geriatric Depression Scale, Beck Anxiety Inventory, Pittsburgh Sleep Quality Index, and Short-Form-Health Survey were applied in an online study in 336 healthy participants with follow-up after 8months. Psychometric evaluation of the German MMQ showed normal distribution of all scales and good to excellent validity, internal consistency, and retest reliability. We provide percentiles and normative data for z-score conversion. Importantly, even subclinically elevated scores in depressiveness and anxiety were associated with decreased memory satisfaction and self-rated ability. Furthermore, although the influence of age on the German MMQ scales was minimal, effects of neuropsychiatric factors such as sleep quality, anxiety, and depressiveness on MMQ Satisfaction and Ability varied across the life span. Our study provides a validated German translation of the MMQ with normative data and reliability measures, including reliable change scores. We show the impact of neuropsychiatric factors on the MMQ scales across the life span and emphasize the relevance of a multifactorial approach to metamemory as a measure of individualized everyday functionality and the importance of including neuropsychiatric factors into both research and clinical assessments of metamemory.

Evaluating the factor structure, reliability and validity of the Copenhagen Burnout Inventory-Student Survey (CBI-SS) among faculty of arts students of Ekiti State University, Ado-Ekiti, Nigeria

The Copenhagen burnout inventory-student survey (CBI-SS) has shown promising psychometric properties in diverse student populations. This study aims to investigate the psychometric properties of the Nigerian version of the CBI-SS. This was a cross-sectional study of 635 students from Ekiti State University, Ado-Ekiti, Nigeria. Confirmatory factor analysis (CFA) was utilized to assess the CBI-SS validity. The reliability score of the CBI-SS was 0.957, ranging from 0.862 to 0.914 for the subscales. Correlation coefficients among the four CBI-SS factors ranged from 0.507 to 0.713. The CFA indicated an adequate goodness-of-fit for the four-factor model of the CBI-SS with the sample data. However, Item 10 was removed due to unacceptably low Average Variance Extracted score. The four factors demonstrated a negative correlation with both General Academic Self-Efficacy Scale and Cumulative Grade Point Average. Furthermore, both self-reported burnout and perceived course stress showed associations with the CBI-SS, where lower levels of burnout corresponded with lower median scores on the CBI-SS scales. This study underscores the significance of the CBI-SS in evaluating student burnout within our student population. The findings indicate that the CBI-SS is a highly reliable and valid instrument for assessing student burnout, suggesting its potential for effective utilization in the Nigerian academic context.

Towards a Culturally Adapted JVQ: Rasch Model Validation among Adolescents in India

Purpose: This study evaluates the psychometric properties of Juvenile Victimization Questionnaire (JVQ) based on the lifetime victimization and traumatic experiences reported by Indian adolescents. Method: A cohort of 350 adolescents, aged 12 to 18 years, from various schools in Aligarh District of India participated in this study. Result: The JVQ’s content validity using Fleiss Kappa analysis, Unidimensionality, and item and person reliability were assessed through rasch measurement model. Moreover, the analysis yielded promising result for the JVQ, indicating high reliability, the Fleiss Kappa score of 0.92 suggest an almost perfect agreement, while the person and item reliability scores were recorded at 0.70 and 0.90, respectively. Conclusion: This initial study supports the JVQ’s potential reliability and validity, suggesting it could be foundational tool for broader research across multiple locations.

YouTube Videos Are a Moderately Comprehensive, Reliable, and Quality Option to Learn About “Multiple Sclerosis and Sexuality”

Objectives This study aimed to evaluate the performance, comprehensiveness, reliability, and quality of English-language YouTube videos addressing the subject of multiple sclerosis and sexuality. Methods In August 2023, a search was conducted on a computer using the keywords “multiple sclerosis and sexuality,” “multiple sclerosis and sexual health,” “multiple sclerosis and sexual health problems,” and “multiple sclerosis and sexual dysfunction” for this descriptive study. According to predetermined inclusion and exclusion criteria, 38 videos that met the research purpose were examined. The related URLs were recorded. For each video, the following information was collected: content producers, performance with YouTube statistics, comprehensiveness with a form developed by researchers, reliability with Singh’s Reliability Evaluation Form, and quality with Global Quality Scale. Two researchers independently evaluated the videos. Results Eighty-nine and a half percent of the videos contained information presented by professionals. The average number of views was 2699.132 ± 3382.848, the comprehensiveness score was 4.2 ± 1.711, the reliability score was 3.184 ± 1.182, and the quality score was 3.421 ± 1.2. Nearly half (42.2%) contained good and useful information for viewers, and half (50%) had high video quality. The reliability and quality scores of videos containing each item in terms of comprehensiveness were higher compared to videos that did not include that item (p < 0.05). In addition, the videos with higher comprehensiveness scores had higher quality and reliability scores (p < 0.001). Conclusion These results underscore the constrained performance attributes of YouTube videos addressing multiple sclerosis and sexuality, with their content exhibiting a moderate level of comprehensiveness, reliability, and quality. These results may provide a basis for increasing the effectiveness of YouTube videos on multiple sclerosis and sexuality.

“Quasi-Metacognitive Machines: Why We Don’t Need Morally Trustworthy AI and Communicating Reliability is Enough”

Many policies and ethical guidelines recommend developing “trustworthy AI”. We argue that developing morally trustworthy AI is not only unethical, as it promotes trust in an entity that cannot be trustworthy, but it is also unnecessary for optimal calibration. Instead, we show that reliability, exclusive of moral trust, entails the appropriate normative constraints that enable optimal calibration and mitigate the vulnerability that arises in high-stakes hybrid decision-making environments, without also demanding, as moral trust would, the anthropomorphization of AI and thus epistemically dubious behavior. The normative demands of reliability for inter-agential action are argued to be met by an analogue to procedural metacognitive competence (i.e., the ability to evaluate the quality of one’s own informational states to regulate subsequent action). Drawing on recent empirical findings that suggest providing reliability scores (e.g., F1-scores) to human decision-makers improves calibration in the AI system, we argue that reliability scores provide a good index of competence and enable humans to determine how much they wish to rely on the system.

Reliability of the Hand-Eye Coordination Test among Adults Using a Software: A Longitudinal Follow-up Study

This study aims to determine the reliability of the hand-eye coordination Software. The integrated control of hand and eye movements, as well as the processing of visual information to support attainment and acquisition and the use of hand proprioception to guide the eyes, are known as hand-eye coordination. Thus, this study aims to determine the reliability of the tests, so the research was conducted thrice with an interval of 6-7 days (test-retest). 28 Young Adults (18- 35 years of age) have participated in the study. The analysis includes finger-pointing tasks performed only with the use of Hand-eye coordination software prepared by the Software engineer.In this Software, Visual targets in the form of a ball then appeared on the screen on the left, on the right, or in the center in random order., and participants were asked to point an index finger of the leading hand as rapidly and precisely as possible at all targets on the screen. After touching the target (indicated by its disappearance), the participant immediately placed his/her hand back in the designated starting position (touchpad).Each participant was required to repeat this finger-pointing task. The accuracy value after 1 minute was recorded and the average accuracy value was calculated and used in the analysis A smaller value indicates greater touch accuracy. In addition, the average reaction time (in milliseconds) was also measured. Based on the data analysis, the reliability score is 0.806 (Cronbach’s Alpha), which indicates a high level of internal Consistency.

Are serious games an alternative to traditional personality questionnaires? Initial analysis of a gamified assessment.

Personality questionnaires stand as crucial instruments in personnel selection but their limitations turn the interest towards alternatives like game-related assessments (GRAs). GRAs developed for goals other than fun are called serious games. Within them, gamified assessments are serious games that share similarities with traditional assessments (questionnaires, situational judgment tests, etc.) but they incorporate game elements like story, music, and game dynamics. This paper aims to contribute to the research on serious games as an alternative to traditional personality questionnaires by analyzing the characteristics of a gamified assessment called VASSIP. This gamified assessment, based on an existing measure of the Big Five personality traits, incorporates game elements such as storyfication, immersion, and non-evaluable gamified dynamics. The study performed included 98 university students (77.6% with job experience) as participants. They completed the original personality measure (BFI-2-S), the gamified evaluation of personality (VASSIP), a self-report measure of the main dimensions of job performance (task performance, contextual performance, and counterproductive work behaviors), and measures of applicant reactions to BFI-2-S and VASSIP. Results showed that the gamified assessment behaved similarly to the original personality measure in terms of reliability and participants' scores, although the scores in Conscientiousness were substantially higher in VASSIP. Focusing on self-reports of the three dimensions of job performance, regression models showed that the gamified assessment could explain all of them. Regarding applicant reactions, the gamified assessment obtained higher scores in perceptions of comfort, predictive validity, and attractiveness, although the effect size was small except for the latter. Finally, all applicant reactions except for attractiveness were related to age and personality traits. In conclusion, gamified assessments have the potential to be an alternative to traditional personality questionnaires but VASSIP needs more research before its application in actual selection processes.

Improving clinical reasoning and communication during handover: An intervention study of the BRIEF-C tool

BackgroundExisting handover communication tools often lack a clear theoretical foundation, have limited psychometric evidence, and overlook effective communication strategies for enhancing diagnostic reasoning. This oversight becomes critical as communication breakdowns...

Comprehensive evaluation of a new automatic scoring system for cleanliness assessment in video capsule endoscopy

AbstractA reliable, quick‐to‐assess, and automatic scoring system for cleanliness assessment in video capsule endoscopy (VCE) is presently not available. The present study proposes an approach to automatically assess the cleanliness in VCE frames as per the latest scoring system, that is, Korea‐Canada (KODA). First, a new multi‐label frame dataset containing medical scores of 28 VCE videos was generated through the proposed mobile‐based application called Artificial Intelligence‐KODA (AI‐KODA) score. The scores were saved automatically in real‐time through the application. The generated dataset was transformed into three datasets based on the scores, and each of the dataset was then randomly split into train:validate:test ratio of 60:20:20. Second, a comprehensive evaluation, interpretation, and benchmarking of the three classification tasks were performed with the help of eight transfer learning algorithms on NVIDIA RTX A5000 workstation. Thorough analysis indicates that AI‐KODA utilized with AI is reliable, quick‐to‐access, and free from observer bias. It promotes automatic scoring system for cleanliness assessment in VCE. The meta‐data is available here (link).

Utility of the ACD-GENE-CLI Score in Asian Patients with Critical Limb Ischemia Undergoing Endovascular Interventions.

Critical limb ischemia (CLI) is an emerging public health threat and lacks a reliable score for predicting the outcomes. The Age, Body Mass Index, Chronic Kidney Disease, Diabetes, and Genotyping (ABCD-GENE) risk score helps identify patients with coronary artery disease who have cytochrome P450 2C19 (CYP2C19) polymorphism-related drug resistance and are at risk for cardiovascular adverse events. However, its application to CLI remains unknown. In this study, we aim to validate a modified ACD-GENE-CLI score to improve the prediction of major adverse limb events (MALEs) in patients with CLI receiving clopidogrel. Patients with CLI receiving clopidogrel post-endovascular intervention were enrolled prospectively in two medical centers. Amputation and revascularization as MALEs were regarded as the outcomes. A total of 473 patients were recruited, with a mean follow-up duration of 25 months. Except for obesity, old age, diabetes, chronic kidney disease (CKD), and CYP2C19 polymorphisms were significantly associated with MALEs. Using bootstrap regression analysis, we established a modified risk score (ACD-GENE-CLI) that included old age (≥ 65 years), diabetes, CKD, and CYP2C19 polymorphisms. At a cutoff value of 8, the ACD-GENE-CLI score was superior to the CYP2C19 deficiency only, and the conventional ABCD-GENE score in predicting MALEs (area under the curve: 0.69 vs. 0.59 vs. 0.67, p=0.01). The diagnostic ability of the ACD-GENE-CLI score was consistent in the external validation. Also, Kaplan-Meier curves showed that in CYP2C19 deficiency, the ABCD-GENE and ACD-GENE-CLI scores could all differentiate patients with CLI who are free from MALEs. The modified ACD-GENE-CLI score could differentiate patients with CLI receiving clopidogrel who are at risk of MALEs. Further studies are required to generalize the utility of the score.

Validity evidence for a team-leading assessment tool in pediatric emergency resuscitations using video review.

Effective leadership of health care action teams has demonstrated positive influence on team performance and patient care, but there is no consensus on how to assess these skills. We developed a novel team leadership assessment tool for leaders of interprofessional pediatric resuscitation teams and collected validity evidence for this tool using video review. This was a prospective cohort study from November 2021 to October 2022. A novel team leadership assessment tool was developed using literature review and local expertise and then piloted and refined using medical simulation. Pediatric emergency medicine (PEM) fellows from a single tertiary care pediatric medical center were enrolled, and videos of one medical resuscitation and one trauma resuscitation were collected per fellow each month. Three reviewers underwent reviewer training and then scored the videos using the assessment tool. Raters provided feedback on feasibility and ease of use using a 5-point Likert scale. Inter-rater reliability for the assessment tool using Gwet's agreement coefficient and the association between performance and clinical level of training using generalized linear mixed model were calculated. Twelve PEM fellows enrolled and 146 videos were reviewed. The inter-rater reliability for each domain ranged from 0.45 (p < 0.0001) to 0.59 (p < 0.0001), with the inter-rater reliability of the total score being 0.49 (p < 0.0001). The reviewers' mean ratings of the elements of the tool were as follows: clarity of the domains (4.6/5), the independence of each domain from each other (3.9/5), the ease of use of the 5-point Likert scale (4.5/5), the usefulness of the provided examples for each domain (4.6/5), and the ability to assess each domain without having to rewatch (4.5/5). The tool differentiated between levels of clinical training for two of the six domains (p < 0.02). We developed a novel team leadership assessment tool for pediatric resuscitation team leaders that demonstrated moderate inter-rater reliability. The tool was easy to use and feasible for educators to assess the performance of PEM trainees in complex high-stakes clinical situations.

Endoscopic assessment of the J pouch in ulcerative colitis: A narrative review.

Patients with ulcerative colitis sometimes need a total colectomy with ileal pouch-anal anastomosis due to medically refractory disease or colitis-associated neoplasia. Up to 50% of patients with ulcerative colitis postoperatively develop pouchitis and the rate of chronic inflammatory pouch conditions requiring pouch excision or diverting ileostomy is reported to be 10%. In order to diagnose and monitor pouchitis, pouchoscopy is essential to assess endoscopic inflammatory findings of the J pouch and to survey neoplasia development, particularly in the remnant distal rectum. However, endoscopic protocols for the evaluation of the pouch may not be standardized worldwide and the reliability of existing disease activity indices for pouchitis has been questioned due to the lack of validation. Recently, reliable endoscopic scoring systems based on an observation of the anatomical location of the J pouch were reported and a significant association between the distribution pattern of endoscopic inflammation (i.e., endoscopic phenotype) and pouch outcomes was also uncovered. In this review, we discuss how to survey the J pouch using pouchoscopy, endoscopic indices for pouchitis disease activity, endoscopic phenotypes and classification, and the pathological mechanisms of pouchitis phenotype in patients with ulcerative colitis.

Validation of the Learner Engagement Instrument for Continuing Professional Development.

Learner engagement is the energy learners exert to remain focused and motivated to learn. The Learner Engagement Instrument (LEI) was developed to measure learner engagement in a short continuing professional development (CPD) activity. The authors validated LEI scores using validity evidence of internal structure and relationships with other variables. Participants attended 1 of 4 CPD courses (1 in-person, 2 online livestreamed, and 1 either in-person or online) in 2018, 2020, 2021, and 2022. Confirmatory factor analysis was used to examine model fit for several alternative structural models, separately for each course. The authors also conducted a generalizability study to estimate score reliability. Associations were evaluated between LEI scores and Continuing Medical Education Teaching Effectiveness (CMETE) scores and participant demographics. All statistical methods accounted for repeated measures by participants. 415 unique participants attended 203 different CPD presentations and completed the LEI 11,567 times. The originally hypothesized 4-domain model of learner engagement (domains: emotional, behavioral, cognitive in-class, cognitive out-of-class) demonstrated best model fit in all 4 courses, with Comparative Fit Index ≥0.99, Standardized Root Mean Square Residual ≤0.031, and Root Mean Square Error of Approximation ≤0.047. The reliability for overall scores and domain scores were all acceptable (50-rater G-coefficient ≥ 0.74) except for the cognitive-in-class domain (50-rater G-coefficient 0.55 to 0.66). All findings were similar for both in-person and online delivery modalities. Correlation of LEI scores with teaching effectiveness was confirmed (rho 0.58), and a small correlation was found with participant age (rho 0.19); other associations were small and not statistically significant. Using these findings, we generated a shortened 4-item instrument, the LEI Short Form. This study confirms a 4-domain model of learner engagement, and provides validity evidence that supports using LEI scores to measure learner engagement in both in-person and livestreamed CPD activities.