Evidence Of Validity Research Articles

Debriefing Is Germane to Simulation-Based Learning: Parsing Cognitive Load Components and the Effect of Debriefing.

Cognitive load (CL) theory provides a framework for optimizing learning in simulation. Measures of CL components (intrinsic [IL], extraneous [EL] and germane [GL]) may inform simulation design but lack validity evidence. The optimal timing for CL assessment and contributions of debriefing to CL are not established. This prospective observational study assessed self-reported CL for first-year anesthesiology residents during 10 individual-learner simulations. Following each simulation and before debriefing, participants completed 4 CL measures: Paas scale, National Aeronautics and Space Administration-Task Load Index (NASA-TLX), Cognitive Load Component questionnaire (CLC) and Cognitive Load Assessment Scales in Simulation (CLAS-Sim). After debriefing, participants repeated the Paas and CLAS-Sim. Twenty-nine first-year anesthesiology residents participated. Correlations were significant among all total CL measures (r range = 0.51-0.69) and between CLC and CLAS-Sim IL (r = 0.66), EL (r = 0.41), and GL (r = 0.61) (all P < 0.01). We observed a significant interaction between total CL measures and case complexity, and a significant main effect of case complexity for CLC and CLAS-Sim IL, with no main effect for IL measure. The CLAS-Sim EL was higher (P = 0.001) than respective CLC scales across cases, with no difference for GL. Participants reported higher CLAS-Sim GL after (versus before) debriefing (P < 0.001), with no difference in IL, EL, or Paas scores. This study provides further validity evidence for the CLAS-Sim and demonstrates generalizability in a different population of medical trainees. The CLAS-Sim GL increases following debriefing, reflecting expected learning, demonstrating initial GL scale validity evidence.

Development and psychometric evaluation of a patient-reported symptom index for patients with non-muscle invasive bladder cancer: the NMIBC-SI.

Non-muscle invasive bladder cancer (NMIBC) is a chronic condition requiring frequent follow-up with endoscopic examinations, tumour resections and intravesical treatments. In this clinical context, patient-reported outcomes (PROs) have enormous potential to inform treatment assessment and recommendations for NMIBC. We aimed to develop and evaluate a patient-reported NMIBC Symptom Index (NMIBC-SI) to facilitate clinical research and enhance care. NMIBC-SI items were developed based on existing literature and qualitative interviews with patients and clinicians, and evaluated in two field tests: item reduction, using NMIBC-SI data from 220 patients on active treatment from nine Australian centres; reliability and validity evaluation of item-reduced version using NMIBC-SI data from 232 patients from five countries. NMIBC-SI assesses disease and treatment-related symptom burden and two treatment-specific side-effects (cystoscopy, intravesical BCG/Chemotherapy). Composite analysis supported a single composite model including core symptom and cystoscopy index items (Intravesical index items were not tested due to small sample). Test-retest reliability was strong (range 0.894-0.91). As expected, the NMIBC-SI was able to discriminate between no treatment and any treatment groups, and no treatment and chemo/BCG groups, providing evidence towards validity. NMIBC-SI assesses patients' self-reported symptom burden and can be used to evaluate NMIBC treatments from the perspective of patients. The NMIBC-SI is acceptable to patients and has evidence for reliability and validity. Future validation work with patients with greater symptom burden is warranted.

Assessment of expert decisions in graduate quantum mechanics

One of the greatest weaknesses of physics education research is the paucity of research on graduate education. While there are a growing number of investigations of graduate student degree progress and admissions, there are very few investigations of at the graduate level. Additionally, existing studies of learning in physics graduate programs frequently focus on content knowledge rather than professional skills such as problem solving. Given that over 90% of physics Ph.D. graduates report solving technical problems regularly in the workplace, we sought to develop an assessment to measure how well graduate programs are training students to solve problems. Using a framework that characterizes expert-like problem-solving skills as a set of decisions to be made, we developed and validated such an assessment in graduate quantum mechanics (QM) following recently developed design frameworks for measuring problem solving and best practices for assessment validation. We collected validity evidence through think-aloud interviews with practicing physicists and physics graduate students, as well as written solutions provided by physics graduate and undergraduate students. The assessment shows strong potential in differentiating novice and expert problem solving in QM and showed reliability in repeated testing with similar populations. These results show the promise of measuring expert decision making in graduate QM and provide baseline measurements for future educational interventions to more effectively teach these skills. Published by the American Physical Society 2025

Measures of adolescents’ emotional security and exposure to violence in relation to the community, police, and gangs: Evidence for validity in a sample of Honduran youth.

Measures of adolescents’ emotional security and exposure to violence in relation to the community, police, and gangs: Evidence for validity in a sample of Honduran youth.

Stressors and normalized stress in undergraduate engineering education culture: development of the Engineering Stress Culture Scale and Undergraduate Engineering Stressors Questionnaire

BackgroundUndergraduate engineering students report increased rates of mental health distress. Evidence suggests that these students experience high stress, which can perpetuate mental health challenges. Further, engineering students may engage in help-seeking and self-care activities more rarely than students in other disciplines. We hypothesize the existence of a culture of high stress that is normalized and expected by undergraduate engineering students. We leveraged interviews previously conducted by our team to develop two survey measures: (1) the Engineering Stress Culture (ESC) Scale, which measures cultural norms and expectations surrounding stress, and (2) the Undergraduate Engineering Stressors Questionnaire (U-ESQ), which measures factors that contribute to undergraduate engineering student stress experiences (stressors).ResultsWe developed a novel, exploratory battery of items to measure students’ experiences with stress in undergraduate engineering education. After administering the new 81-item survey (N = 624) comprising both the U-ESQ and ESC Scale, we performed exploratory factor analysis and tested the internal consistency for each factor subscale. We then refined the survey to 58 items and administered it to a second sample of participants (N = 561) at a second institution for confirmatory factor analyses. We present evidence that the ESC Scale is a valid and reliable instrument for measuring an engineering stress culture. The final factor structure of the U-ESQ revealed eight distinct factors, each representing different stressors such as competition or identity-related experiences. We provide validity evidence for the U-ESQ. While the confirmatory evidence of the reliability of the U-ESQ instrument is less robust compared to the ESC Scale, there is acceptable evidence of reliability. This may be partially attributed to the inclusion of career-related and faculty-interaction-related stressors that may become more for salient later-stage undergraduate students, who are undersampled in our results.ConclusionsWe present validity and reliability evidence for the developed measures that quantify engineering student stressors and the associated high-stress culture. These measures may apply to students in other non-engineering STEM disciplines, warranting future investigations. The developed measure is a first step towards identifying and dismantling a culture of stress in engineering, which will promote student well-being and thriving.

The Development and Validation of the Postgraduate School Application Outcome Expectations Scale

This study sought to develop and validate the Postgraduate School Application Outcome Expectations Scale (PSAOE), which measures an individual’s perceived consequences of completing self-regulatory tasks inherent to the postgraduate school application process. Items were generated from an extensive review of the literature, and a panel of experts and a pilot sample of undergraduate students provided feedback on items. Exploratory factor analysis of responses from 314 participants yielded a conceptually interpretable, 15-item, three-factor solution, which explained 57.08% of the variance. Confirmatory factor analysis of responses from 377 participants provided support for the three-factor model. Three conceptually similar measures, including the VOE-R, CEOES, and PSASE, were used to provide evidence of convergent and discriminant validity. Implications for practice and future directions were also discussed.

Evaluating the construct validity of the health promotion literacy scale: a confirmatory factor analysis in Taiwan’s university social responsibility context

IntroductionThe development of a Health Promotion Literacy Scale within the USR Curriculum is crucial for addressing the existing gap in measuring the impact of USR on students’ citizenship and social engagement. This study aimed to develop and assess the psychometric properties of the Health Promotion Literacy-based Scale in USR (HPLS-USR) scale on medical university students through confirmatory factor analysis (CFA) in Taiwan.MethodsWe conducted a cross-sectional study and recruited a convenience sample of 405 students in central Taiwan. The HPLS-USR scale, developed by Hung et al. in 2022, comprises four factors—personal growth, responsibility of citizenship, social interaction, and intellectual development—that together accounted for 61.83% of the total variance. Data analysis was performed using SPSS version 26 and AMOS (version 22.0) to perform a CFA to verify the model’s goodness of fit.ResultsThe modified model derived 17 items in 3 factors, with 51.02% of the total variance explained. The structural equation model displayed a good fit, suggesting that personal growth, responsibility for citizenship, and social interaction were latent variables of the Health Promotion Literacy-based Scale on a USR curriculum. The Cronbach’s alphas for the overall scale and the three subscales ranged from 0.82 to 0.93. The psychometric properties of the scale demonstrated good to excellent model fit in the CMIN/DF (χ2/df) = 3.149, GFI = 0.91, AGFI = 0.88, SRMR = 0.04, RMSEA = 0.07, NFI = 0.90, NNFI = 0.92, RFI = 0.89, IFI = 0.93, CFI = 0.93, PNFI = 0.78, and PCFI = 0.81. The Cronbach’s alphas ranged between.81 and.89; the composite alphas ranged between.87 and.94.ConclusionThis study provides preliminary construct validity evidence for the HPLS-USR scale, a robust and psychometrically sound instrument for assessing health promotion literacy among USR students.

Reliability and validity of simulation-based Electrocardiogram assessment rubrics for cardiac life support skills among medical students using generalizability theory

ABSTRACT Simulation-based learning (SBL) is effective for EKG interpretation training in the advanced cardiac life support (ACLS) context, enhancing motivation, confidence, and learning outcomes. However, research on the psychometrics of assessment rubrics for ACLS skills among pre-clinical students is limited. This study investigates the validity and reliability of assessment rubrics for ACLS skills, including EKG interpretation, scenario and pharmacological management, and teamwork. An SBL course that integrates basic EKG interpretation into ACLS Stations was conducted at Phramongkutklao College of Medicine, utilizing high-fidelity mannequins to simulate realistic scenarios, enrolling 96 medical students. The course consisted of five independent stations, and each student was assessed once by two raters using ten-item assessment rubrics. The rubrics included three domains: (1) EKG and ACLS algorithm skills, (2) management and mechanisms of action, and (3) affective domains. Validity evidence on the content was gathered, and construct validity was confirmed with confirmatory factor analysis (CFA). Inter-rater and internal consistency reliability were calculated. Generalizability theory was utilized to analyse the data. Three expert reviews yielded an item-objective congruence index of 0.67–1.00, with iterative validation through alpha and beta tests. The CFA demonstrated a good fit, but two questions with loading factors below 0.30 were removed, resulting in an eight-item assessment form. An inter-rater correlation of 0.70 (p < 0.001) and a Cronbach’s alpha of 0.76 was demonstrated. To achieve a Phi-coefficient ≥0.80, three raters and at least 10 items are required in a p×i×r crossed design. With eight items, r:(p×i) nested design reliability was 0.69, 0.79, and 0.83 for one, two, and three raters, respectively. While a single rater with 10 items achieved a Phi-coefficient of 0.74. The rubrics for assessing ACLS skills among pre-clinical students demonstrated acceptable validity and reliability. A condensed eight-item rubric with acceptable reliability is proposed as a practical tool for optimizing assessment in future evaluations relevant to the pre-clinical context.

Testing Comparability of Gender Bullying Victimization Assessments Through a Latent Class Approach.

Bullying is a social and health problem that requires appropriate interventions based on valid and fair evaluations of bullying experiences. The validity of interpretations of bullying victimization scores can be compromised by measurement artifacts or biases that may arise during the assessment process. Boys' and girls' bullying experiences could lead to differences in their response processes when they answer the bullying scale items and compromise validity if such differences come from measurement artifacts. The study is intended to illustrate how to obtain validity evidence of response processes for the Students' Experience of Being Bullied Scale from the Programme for International Student Assessment (PISA) 2018 Student Questionnaire and test measurement invariance across gender by latent class analysis (LCA). The sample was taken from the PISA 2018 study and consisted of 11,599 Spanish high school students (50.3% female, 49.7% male). Response patterns were examined through LCA. Four profiles were found: (a) Not Bullied, (b) Bullied (All Types), (c) Relational Bullying, and (d) Potential Friendly Teasing. Measurement invariance across gender was analyzed by a multigroup LCA. LCA results do not guarantee equivalence of measurements. Class prevalence per group and a multinomial logistic regression were calculated to further examine gender differences across latent classes. Boys were more likely to belong to the Bullied (All Types) class and the Potential Friendly Teasing class, whereas girls were more likely to belong to the Relational Bullying class. These findings illustrate the possibilities LCA can offer to provide validity evidence of response processes and suggest that different bullying experiences of girls and boys could compromise the fairness and validity of comparative interpretations of test scores, which is an important issue to address to develop bias-free classroom interventions.

Supplemental Material for Measures of Adolescents’ Emotional Security and Exposure to Violence in Relation to the Community, Police, and Gangs: Evidence for Validity in a Sample of Honduran Youth

Supplemental Material for Measures of Adolescents’ Emotional Security and Exposure to Violence in Relation to the Community, Police, and Gangs: Evidence for Validity in a Sample of Honduran Youth

Patient-reported assessment of bulbar function in spinal muscular atrophy (SMA): Validation of a self-report scale.

Background: Bulbar dysfunction is a well-recognized burden experienced by individuals with spinal muscular atrophy (SMA). Metrics that capture the impact of these problems are lacking. Objectives: To develop and validate an SMA-Bulbar Scale that captures and quantifies patient-reported experiences with bulbar dysfunction. Methods: Cure SMA database members were invited to complete online surveys and a 31-item Bulbar Scale developed by the authors with reference to both bulbar dysfunction in the literature and consultations with SMA-dedicated health care providers, pharmaceutical companies, and persons affected by SMA. Results: 166 adults with SMA reported a range of bulbar dysfunctions. The most common problems, occurring more than 80% of the time, were prolonged meal times, difficulty with mouth opening, and swallowing pills. In addition, 10% of the respondents reported a worsening feeding function over the course of their lives. Across the diverse array of bulbar functions, exploratory and confirmatory factor analyses of responses identified three coherent dimensions of bulbar dysfunction: Swallowing; Mealtimes and Communication; and Breath Sounds, Speech, Voice, and Secretion Management. Higher scores of overall bulbar dysfunction, (p < .001) and each factor (p < .001), were reported by respondents with any degree of feeding restrictions. Sex, age, or use of SMA disease-modifying treatment did not correlate with bulbar scores. Conclusions: This patient-reported scale of bulbar function in adults with a wide range of SMA severity captures and quantifies the variable manifestations of experienced bulbar impairment. With preliminary evidence of validity, the scale supports efforts to standardize accurate identification of bulbar dysfunction, incorporate the perspectives of people with SMA on key areas of their daily functioning, provide metrics essential for meaningful endpoints in clinical trials, inform practice guidelines, and promote advancement of the regulatory science needed for the evaluation and development of therapeutic interventions. Identification of three coherent dimensions of bulbar dysfunction may improve further investigations.

Effects of visual art observation on technical skills in novice healthcare learners: A scoping review

Background Recently, health professional education uses visual art observation to promote various observation-related technical skills. This article maps the studies on such interventions, scrutinizes what they measured as observational skills, and discusses their effectiveness. Methods Following the PRISMA Extension for Scoping Reviews, a scoping review was conducted. Publications from 2001 on were identified by searching four databases and by hand searching. The author screened each publication using the pre-designed eligibility criteria: participants were novice healthcare learners enrolled in visual art observation training; the study aimed to evaluate the effect of the intervention on technical skills related to observation; the skills were objectively measured. The author extracted relevant information from the included papers without additional inquiry into the study authors. The extracted information was illustrated in both a tabular and descriptive format. Results 3,157 publications were identified, of which 18 articles were included. Few studies had valid and reliable experiments. The relatively valid evidence is that the participants listed more elements or signs for artistic or medical images. Conclusions Sound evidence is lacking for all the technical skills intended to be fostered. Observation skills for artistic images have not been demonstrated to transfer to technical skills. Nor do the studies show that they promoted accurate diagnoses and reduced misdiagnoses. Additionally, the evidence on verbalizing skills is not isolated from the impact of discussions and is unclear regarding its transfer to actual communication. For the others, there are not enough valid studies on technical skills. This is true for studies that directly examine promoting accurate diagnosis or reducing misdiagnosis. Moreover, there may be promising alternatives to visual art observations for cultivating such technical skills, but no comparative studies were conducted.

Theory and Psychometric Development of the Self-Comforting and Coping Scale (SCCS): A Novel Measure of Self-Comforting Behaviors

The ability to cope with stress and adversity is crucial for psychological wellbeing. Self-comforting behaviours, which involve actions taken to soothe oneself during distress, are increasingly recognized as vital components of adaptive coping. However, systematic assessment of these behaviours remains underexplored. This study introduces the Self-Comforting and Coping Scale (SCCS), a novel tool designed to measure self-comforting behaviours across diverse populations. The SCCS was developed through a multi-stage process, including literature review, expert consultations, focus group feedback, and pretesting, resulting in a comprehensive 13-domain framework. Findings indicate that the SCCS exhibits robust psychometric properties, including evidence of content and face validity. These results highlight the scale’s potential as a valuable measure for assessing self-comforting behaviours and their role in resilience, mental health, and wellbeing. The SCCS addresses a critical gap in coping research, offering a holistic approach to understanding self-comforting behaviours and their contributions to adaptive coping strategies. Additionally, the constructs of the SCCS closely align with the Self-Comforting and Coping Theory (SCCT), ensuring that the scale accurately reflects the theoretical framework it is designed to test. Future research is encouraged to further validate the SCCS through longitudinal studies, cross-cultural comparisons, and investigations into its applications in clinical and organizational settings. This work provides a valuable foundation for advancing both theoretical insights and practical interventions in stress management and resilience-building, making the SCCS a significant contribution to the field of psychological assessment.

Association between endocrine disrupting chemicals exposure and diabetic kidney disease in adults: A national cross-sectional NHANES study.

Association between endocrine disrupting chemicals exposure and diabetic kidney disease in adults: A national cross-sectional NHANES study.

Visum et Repertum as criminal evidence within the Indonesian criminal justice system

This research discusses the use of Visum et Repertum (VeR), an expert testimony, as evidence in criminal justice system in Indonesia. The main issue identified in this research is the legal vacuum regarding the use of VeR before the investigation, which results in VeR made during the preliminary investigation stage not having the same legal force as VeR made during the investigation stage. Hence, this study used normative legal approach with analysis of applicable legislation. The analysis shows that VeR has a strong legal position as valid evidence according to the Criminal Procedure Code but there is an urgent need for regulations that state VeR requested during the preliminary investigation stage can be used as evidence with the same probative force as VeR requested during the investigation stage. This study proposes the issuance of regulations stating that VeR made before the investigation remains valid, both investigators and interrogators are authorized to request VeR in writing, and the probative force of VeR requested during the preliminary investigation stage should be considered equivalent to VeR requested during the investigation stage.

Is dharma the same as calling? Considering religio-cultural context in cross-cultural calling research

ABSTRACT A growing body of calling literature demonstrates evidence of cross-cultural validity of scales to measure perceiving and living a calling, pointing to salience of the construct and useful application of calling interventions across populations. However, there has been little inquiry into the relationship of calling to similar concepts in cultures with roots in religions other than Christianity. In this paper, I examine the Hindu concept of dharma as a vocational concept similar to calling. By tracing its history and the lived experience of dharma through caste, I argue that sufficient differences in conceptualisation of dharma and calling warrant further considerations of dharma as a factor in cross-cultural calling research. After reviewing existing research literature on calling in Indian and Indian American contexts, I outline approaches future researchers could use to integrate the potential role of dharma in observed cross-cultural differences in perceiving and living a calling.

Predictive validity evidence of Yes-No Angoff standard setting in a Pre-Clinical medical school curriculum

BackgroundStandard setting plays a critical role in determining student outcomes by defining the level of biomedical knowledge required to be considered competent. This is especially important for accurately classifying medical students as ready (or not) to progress through the pre-clinical curriculum. In multiple-choice medical knowledge exams, the Yes-No Angoff method may be used for setting passing scores. This method relies on faculty experts’ judgments about whether students with borderline but adequate competence would answer each question correctly. Using a construct validity framework with the construct of academic success defined as the ability of a student to progress without obstacles, we examined the predictive validity of the passing standards set by this method.MethodsWe analyzed academic success for four pre-clinical semesters across three student cohorts. First, we identified passing standards for pre-clinical courses using the Yes-No Angoff method. Then, we applied binary logistic regression and receiver-operator characteristic (ROC) analyses with area under the curve (AUC) to evaluate passing standards. For binary outcomes, we defined academic success in terms of unimpeded progress through the curriculum and students’ first-attempt passage of the United States Medical Licensing Examination (USMLE) Step 1. Model predictors for ROC analyses included Yes-No Angoff passing standards, Medical College Admissions test scores, and grade point averages for math and science courses.ResultsROC analyses showed a low but acceptable area under the curve for a single semester in one cohort and excellent or outstanding AUCs for the remaining 11 semesters. Rates of overall classification accuracy for the Yes-No Angoff passing scores ranged between 89% and 96% for predicting academic success for all pre-clinical semesters across all cohorts.ConclusionsThe Yes-No Angoff method yielded passing standards that aided in accurately predicting academic success, providing predictive validity evidence for our school’s passing standards in a pre-clinical medical curriculum.

Evidence of structural validity for the Turkish version of solution-focused inventory: a study of measurement invariance

Evidence of structural validity for the Turkish version of solution-focused inventory: a study of measurement invariance

Exploring the Role of Immersive Virtual Reality Simulation in Health Professions Education: Thematic Analysis.

Although technology is rapidly advancing in immersive virtual reality (VR) simulation, there is a paucity of literature to guide its implementation into health professions education, and there are no described best practices for the development of this evolving technology. We conducted a qualitative study using semistructured interviews with early adopters of immersive VR simulation technology to investigate use and motivations behind using this technology in educational practice, and to identify the educational needs that this technology can address. We conducted 16 interviews with VR early adopters. Data were analyzed via directed content analysis through the lens of the Unified Theory of Acceptance and Use of Technology. The main themes that emerged included focus on cognitive skills, access to education, resource investment, and balancing immersion. These findings help to clarify the intended role of VR simulation in health professions education. Based on our data, we synthesized a set of research questions that may help define best practices for future VR development and implementation. Immersive VR simulation technology primarily serves to teach cognitive skills, expand access to educational experiences, act as a collaborative repository of widely relevant and diverse simulation scenarios, and foster learning through deep immersion. By applying the Unified Theory of Acceptance and Use of Technology theoretical framework to the context of VR simulation, we not only collected validation evidence for this established theory, but also proposed several modifications to better explain use behavior in this specific setting.

Psychometric evaluation of the unmet needs instrument for carers of people with dementia (UNI-C)

BackgroundCarers play an important role in providing practical and emotional support to people with dementia. There is a need to ensure carers are supported in this role to maximise their ability to provide care while maintaining their own wellbeing. This study aimed to develop a psychometrically rigorous self-report instrument to assess the unmet needs of carers of people with dementia.MethodsThe Unmet Needs Instrument for Carers of People with Dementia (UNI-C) was developed using a multi-methods approach including a comprehensive literature review, in-depth interviews and focus groups, expert opinion, cognitive interviews and pilot testing. A cross-sectional survey of 169 carers was used to examine the psychometric characteristics of the instrument. Specifically, internal reliability, and structural and convergent validity were assessed and the number of items reduced.ResultsThe original 80 item instrument was reduced to 46 items across four domains based on prevalence, item-total and pairwise correlations, factor loadings and clinical relevance. The resulting instrument, named the UNI-C46, showed acceptable evidence of structural and convergent validity, and good internal reliability (α = 0.94) and acceptability. The four domains measure ‘Your own wellbeing’ (16 items), ‘Finding information’ (12 items), ‘Managing practical needs’ (10 items), and ‘Managing dementia symptoms’ (8 items).ConclusionsThis study provides the first demonstration of reliability and validity of the UNI-C46. Further testing of these properties with a larger and more representative sample of carers is recommended.