Response Process Validity Research Articles

Knowledge of development during pregnancy and the transition to parenthood: Psychometric properties of the Domains of Development Instrument

Researchers and practitioners need robust measurement tools for evaluating knowledge of child development to better support parents and their children during pregnancy and the transition to parenthood. We addressed this need by evaluating the psychometric properties of the Domains of Development Instrument (DoDI) for measuring knowledge of developmental milestones from birth to 3 years. We evaluated four types of validity evidence for the DoDI: test content, response processes, internal structure, and relations to other variables. We convened an expert panel to evaluate test content and conducted cognitive interviews with mothers to evaluate response processes. We also collected responses from a sample of 418 English-speaking pregnant women to evaluate internal structure and relations to other variables. We observed content validity and response process validity, as well as the predicted internal structure, internal consistency, test–retest reliability, and convergent validity. We conclude with recommendations for future research with the DoDI.

Psychometric properties of the newly developed self-report environmental determinants of health questionnaire (EDH-Q): development and validation

BackgroundThe environmental determinants of health (EDH) have a significant impact on people’s physical, mental, and social wellbeing. Everyone needs access to environmental resources of all types, including food, materials, and energy, to survive. Currently, no valid and reliable instrument exists for evaluating individuals’ perceived levels of EDH. Hence, the purpose of this study was to develop and validate the environmental determinants of health questionnaire (EDH-Q) among undergraduate students in Nigeria.MethodWe conducted a cross-sectional survey among undergraduate students in Nigeria to assess the psychometric properties of the newly developed Environmental Determinants of Health Questionnaire (EDH-Q). Respondents were selected using a convenience sampling approach to evaluate their perceptions of environmental determinants of health. The Content Validity Index (CVI) and Face Validity Index (FVI) were calculated to ascertain the scale’s content validity and response process validity, respectively. Additionally, exploratory factor analysis (EFA), confirmatory factor analysis (CFA), composite reliability (CR), average variance extracted (AVE), Cronbach’s alpha, and intraclass correlation coefficient (ICC) were computed to assess the scale’s construct validity.ResultsThe study involved 300 respondents in the EFA (males 55.7%, females 44.3%) and 430 respondents in the CFA (males 54.0%, females 46.0%). In the EFA, two constructs were identified (the natural environment and the built environment). The EFA model was able to explain 63.57% of the total cumulative variance, and the factor correlation was 0.671. The whole scale Cronbach’s alpha value was 0.947, while the two constructs’ Cronbach’s alpha values were 0.918 (natural environment) and 0.935 (built environment). In the CFA, six pairs of error covariances were included between items within the same construct to improve the fit indices of the initial proposed measurement model. The final re-specified measurement model showed that the EDH-Q, which has two constructs and 18 items, has adequate construct validity (CFI = 0.948, TLI = 0.938, SRMR = 0.046, RMSEA = 0.052, and RMSEA p-value = 0.344). The CRs were 0.845 (natural environment) and 0.854 (built environment). The ICCs were 0.976 (natural environment) and 0.970 (built environment).ConclusionThe results show that the newly created EDH-Q has sufficient construct validity and may be utilized to assess participants’ perceptions of their level of EDH. Researchers should examine this instrument in populations with different age ranges and other demographic characteristics, as the present study only applied it to undergraduate students who may share similar characteristics.

Dysphagia Screening in Brazilian Older Adults with Dementia: Content Development and Validation of a Questionnaire for Caregivers - RaDID-QC.

This study aims to develop and validate the content and response processes of a questionnaire intended for caregivers to screen for dysphagia in Brazilian older adults with dementia due to Alzheimer's disease and/or vascular dementia. The instrument items were developed in Brazilian Portuguese language based on the theoretical framework. A committee of speech-language-hearing therapists analyzed the relevance, objectivity, clarity, and understandability of the items with the Delphi method. The content validity index cutoff agreement score for experts' answers to validate each item in the questionnaire was 0.78; in the intraclass correlation coefficient, it was 0.75 for all items. For response process validity evidence, the questionnaire was applied to 30 caregivers of older adults with dementia, who judged the clarity and understandability of the items. Each item was validated when understood by at least 95% of participants. The first version of the instrument had 29 items. After two expert assessments, the last version had 24 items. The intraclass correlation coefficient was 0.85. Only one item needed semantic adjustments in the pre-test. The dysphagia screening instrument applied to caregivers of older adults with dementia was developed with adequate content and response process validity evidence, enabling adjustments in its construct. Future studies will analyze the remaining evidence of validity and reliability.

Collaboration cognizance: Development of a self-assessment tool to measure intra-professional collaborative practices (IPCP) in postgraduate medical residents at tertiary care hospitals

BackgroundThe ever-evolving healthcare system of today demands physicians who steer their roles as treatment providers, managers and collaborators. Professionals are highly interdependent due to increased complexity of health problems and risk of errors increases with transitions in care. In hospitals, the main workforce is postgraduate residents; therefore, intraprofessional collaboration amongst residents is essential. Several instruments are available to evaluate interprofessional collaboration amongst physicians, nurses and hospital teams but none specifically assessed intra-professional collaborative practices amongst residents working in tertiary care hospitals in multi-disciplinary teams. This study intends to develop and validate an instrument to self-assess intraprofessional collaborative practices in postgraduate residents undergoing residency in multiple specialties at tertiary care hospitals.ApproachThis study on Instrument Development employed mixed method study design executed in two phases. In phase 1, six domains of intraprofessional collaborative practices were identified from literature and 35 items were developed. Fifteen experts participated in qualitative content validation and provided comments. To establish content validity in phase 2, content validity index (CVI) and content clarity average (CCA) were assessed by seventeen experts. Response process validity was established by cognitive interviewing of 5 postgraduate residents. Pilot testing was done on a sample of 407 residents. Cronbach’s alpha was determined, and confirmatory factor analysis established construct validity.ResultsDuring phase 1, items were modified based on qualitative feedback from 15 experts. In round 2, CVI and CCA were determined based on responses of 17 experts. The items having an I-CVI greater than 0.90 were accepted and six items underwent modifications as their I-CVI fell between 0.78 and 0.90. Similarly, four items with a CCA of less than 2.4 were modified to increase clarity. Cognitive interviews of participants on 30 items resulted in the deletion of 1 item and changes in 5 items. The final instrument had 29 items categorized under six constructs. All items had good factor loadings during CFA, so none was deleted. Cronbach’s Alpha α was 0.937.ConclusionIntraprofessional collaborative practices in residents is a valid and reliable self-assessment tool comprising 29 items measuring six constructs. It may be used by residents to assess their collaborative practices and incorporated in curricula to help develop collaborative practices and their assessment during training of postgraduate residents.

Measuring interprofessional collaboration in fluid healthcare teams through the lens of teamness

ABSTRACT Health professionals often need to work together to provide team-based care. With increasing healthcare complexities and manpower shortages, more health professionals are working in multiple, fluid teams instead of one stable team, to provide care to patients. However, there is currently no validated instrument to measure the quality of interprofessional collaboration in fluid teams. One approach is to assess team members’ perceived level of teamness (qualities that make clinical teams effective). This study aimed to examine the validity evidence of using the Assessment for Collaborative Environment (ACE-15) in fluid teams and investigate if teamness varies among health professions and clinical settings. Content and response process validity were gathered through consulting experts and cognitive interviews, resulting in revisions to 11 of 15 items in ACE-15. Through exploratory factor analysis of 194 responses on the revised ACE-15, a 13-item instrument, ACE-13F, with strong validity evidence for use in fluid teams was developed. A two-factor fixed effect ANOVA model revealed that the clinical setting that health professionals work in has a significant impact on the level of teamness (F[3,170] = 6.15, p < .001, η2 = 0.09). ACE-13F can be used as a rapid instrument to measure interprofessional collaboration in fluid healthcare teams.

Development and Validation of a Trauma-Informed Care Communication Skills Assessment Tool

BackgroundTrauma-informed care (TIC) is growing in medical education as health care systems recognize trauma’s impact on health outcomes. TIC acknowledges and responds to the effects of trauma on physical, psychological, and emotional health. As TIC trainings are developed and delivered to health care professionals across the learner continuum, curricula need evaluation beyond learner satisfaction and knowledge to better assess changes in skills. We developed the Gap Kalamazoo Communication Skills Assessment Form for Trauma-Informed Care (GKCSAF-TIC) to evaluate pediatric trainees' communication skills in TIC. We describe the development and validity evidence of the GKCSAF-TIC in assessing pediatric residents' TIC skills during standardized patient encounters. MethodsWe developed and implemented the TIC communication skills assessment tool in a one-year prospective cohort study involving pediatric residents. We conducted simulated patient encounters conducted before and after TIC training, with two pediatric faculty attendings assessing each encounter. We gathered validity evidence using Messick’s framework, focusing on content, response process, internal structure, and relationship with other variables. ResultsWe analyzed 57 standardized patient encounters with 33 pediatric interns, including 23 pre-post matched pairs. The development process and rater training supported content and response process validity. Internal consistency, measured by Cronbach’s alpha, ranged from 0.93 to 0.96, while inter-rater reliability, measured by intraclass correlations, ranged from 0.80 to 0.83. There was a significant improvement in scores from pre-training to post-training (3.7/5 to 4.05/5; P < 0.05). ConclusionThe GKCSAF-TIC demonstrated strong preliminary validity and offers educators a valuable means to assess and provide formative feedback to pediatric trainees about TIC.

Health vulnerability: evidence of validity of an item bank

Objective: To analyze the evidence of content validity and response process of a health vulnerability item bank. Methods: Psychometric study with support of Pasquali’s theoretical pole in which 62 specialists with different sociodemographic and academic characteristics were selected and 15 of them responded to the content validity questionnaire. After adjustments, the bank moved to the response process validity phase and was evaluated by 12 health service users. A content validity coefficient was used to evaluate the items and the bank, which had the reliability assessed by the binomial test, considering a percentage of 80% and p-value &gt;0.05. Results: The bank presented good evidence of content validity by experts. The coefficients were 0.87 for the subject element, 0.86 for the social element and 0.865 for the general element. The target audience made suggestions in the response process validity phase to better understand the items (total coefficient: 0.89), which were considered clear, accurate and understandable. Conclusion: The final version of the item bank has 535 items, 238 for the subject and 297 for the social element. It presents evidence of validity of content and response process and is suitable for testing the validity of internal structure.

Investigation into the intersection between response process validity and answer-until-correct validity: development of the repeated attempt processing issue detection (RAPID) method

With the already widespread nature of multiple-choice assessments and the increasing popularity of answer-until-correct, it is important to have methods available for exploring the validity of these types of assessments as they are developed. This work analyzes a 20-question multiple choice assessment covering introductory undergraduate chemistry topics which was given to students in an answer-until-correct manner. Response process validity was investigated through one-on-one think-aloud interviews with undergraduate chemistry students. Answer-until-correct validity was also explored using an analysis of partial credit assignments. Results indicated the convenience of the quantitative partial credit method came at great cost to the precision of validity issue detection and is therefore not a valid shortcut to more rich qualitative approaches. The repeated attempt processing issue detection (RAPID) method is a novel method developed as a combination of response process and answer-until-correct validity. Results from this new method revealed validity issues that were undetected from the use of either approach individually or in concert.

Evidence of response process validity of a spectrographic voice analysis protocol

RESUMO Objetivo desenvolver a etapa de validade baseada nos processos de resposta do Protocolo de Análise Espectrográfica da Voz (PAEV). Métodos foram recrutados dez fonoaudiólogos e dez alunos de graduação em Fonoaudiologia, que aplicaram o PAEV em dez espectrogramas, realizaram o julgamento dos itens do PAEV e participaram de uma entrevista cognitiva. A partir das respostas, o PAEV foi reanalisado para reformulação ou para exclusão de itens. Utilizou-se o teste Qui-Quadrado e os valores de acurácia para análise das respostas dos questionários, assim como análise qualitativa dos dados da entrevista cognitiva. Resultados os participantes obtiveram acurácia maior que 70% na maioria dos itens do PAE. Apenas sete itens alcançaram acurácia menor ou igual a 70%. Houve diferença entre as respostas de presença versus ausência de dificuldade na identificação dos itens no espectrograma. A maioria dos participantes não teve dificuldade na identificação dos itens do PAEV. Na entrevista cognitiva, apenas seis itens não obtiveram correta identificação da intenção, conforme verificado na análise qualitativa. Além disso, os participantes sugeriram exclusão de cinco itens. Conclusão após a etapa de validação baseada nos processos de resposta, o PAEV foi reformulado. Sete itens foram excluídos e dois itens foram reformulados. Dessa forma, a versão final do PAEV após essa etapa foi reduzida de 25 para 18 itens, distribuídos nos cinco domínios.

Escala de Coping para Fala em Público (ECOFAP): evidências de validade de conteúdo e processos de resposta

Abstract Purpose To present the content and response process validity evidence of the Speaking in Public Coping of Scale (ECOFAP). Methods A methodological study to develop and validate the instrument. It followed the instrument development method with theoretical, empirical, and analytical procedures, based on the validity criteria of the Standards for Educational and Psychological Testing (SEPT). The process of obtaining content validity evidence had two stages: 1) conceptual definition of the construct, based on theoretical precepts of speaking in public and the Motivational Theory of Coping (MTC); 2) developing items and response keys, structuring the instrument, assessment by a committee with 10 specialists, restructuring scale items, and developing the ECOFAP pilot version. Item representativity was analyzed through the item content validity index. The response process was conducted in a single stage with a convenience sample of 30 people with and without difficulties speaking in public, from the campus of a Brazilian university, belonging to various social and professional strata. In this process, the respondents’ verbal and nonverbal reactions were qualitatively analyzed. Results The initial version of ECOFAP, consisting of 46 items, was evaluated by judges and later reformulated, resulting in a second version with 60 items. This second version was again submitted for expert analysis, and the content validity index per item was calculated. 18 items were excluded, resulting in a third version of 42 items. The validity evidence based on the response processes of the 42-item version was applied to a sample of 30 individuals, resulting in the rewriting of one item and the inclusion of six more items, resulting in the pilot version of ECOFAP with 48 items. Conclusion ECOFAP pilot version has items with well-structured semantics and syntactic, representing strategies to cope with speaking in public.

Speaking in Public Coping Scale (ECOFAP): content and response process validity evidence.

To present the content and response process validity evidence of the Speaking in Public Coping of Scale (ECOFAP). A methodological study to develop and validate the instrument. It followed the instrument development method with theoretical, empirical, and analytical procedures, based on the validity criteria of the Standards for Educational and Psychological Testing (SEPT). The process of obtaining content validity evidence had two stages: 1) conceptual definition of the construct, based on theoretical precepts of speaking in public and the Motivational Theory of Coping (MTC); 2) developing items and response keys, structuring the instrument, assessment by a committee with 10 specialists, restructuring scale items, and developing the ECOFAP pilot version. Item representativity was analyzed through the item content validity index. The response process was conducted in a single stage with a convenience sample of 30 people with and without difficulties speaking in public, from the campus of a Brazilian university, belonging to various social and professional strata. In this process, the respondents' verbal and nonverbal reactions were qualitatively analyzed. The initial version of ECOFAP, consisting of 46 items, was evaluated by judges and later reformulated, resulting in a second version with 60 items. This second version was again submitted for expert analysis, and the content validity index per item was calculated. 18 items were excluded, resulting in a third version of 42 items. The validity evidence based on the response processes of the 42-item version was applied to a sample of 30 individuals, resulting in the rewriting of one item and the inclusion of six more items, resulting in the pilot version of ECOFAP with 48 items. ECOFAP pilot version has items with well-structured semantics and syntactic, representing strategies to cope with speaking in public.

"It's More Of A Me-Thing Than An Evolution Thing": Exploring The Validity Of Evolution Acceptance Measures Using Student Interviews.

Researchers who study student acceptance of evolution rely on surveys that are designed to measure evolution acceptance. It is important for these surveys to measure evolution acceptance accurately and in isolation from other constructs, so that researchers can accurately determine what leads to low acceptance. The Inventory of Student Evolution Acceptance (I-SEA) and the Generalized Acceptance of EvolutioN Evaluation (GAENE) are two surveys that were developed to improve upon the limitations of earlier surveys. Yet neither survey has been extensively tested for response process validity, which can assess the extent to which students use constructs other than their acceptance of evolution to answer survey items. In this study, we examined the response-process validity of the I-SEA and GAENE by conducting cognitive interviews with 60 undergraduate students. Interviews revealed that both surveys retain certain response-process issues. The I-SEA conflated knowledge about and acceptance of evolution for a subset of students. The GAENE measured evolution acceptance inconsistently because students interpreted "evolution" in different ways; it also measured willingness to advocate for evolution in addition to acceptance. Researchers can use these findings to better inform their survey choice when designing future studies, and to further improve the measurement of evolution acceptance.

Validation of a culturally adapted Swedish-language version of the Death Literacy Index.

The death literacy index (DLI) was developed in Australia to measure death literacy, a set of experience-based knowledge needed to understand and act on end-of-life (EOL) care options but has not yet been validated outside its original context. The aim of this study was to develop a culturally adapted Swedish-language version of the DLI, the DLI-S, and assess sources of evidence for its validity in a Swedish context. The study involved a multi-step process of translation and cultural adaptation and two validation phases: examining first content and response process validity through expert review (n = 10) and cognitive interviews (n = 10); and second, internal structure validity of DLI-S data collected from an online cross-sectional survey (n = 503). The psychometric evaluation involved analysis of descriptive statistics on item and scale-level, internal consistency and test-retest reliability, and confirmatory factor analysis. During translation and adaptation, changes were made to adjust items to the Swedish context. Additional adjustments were made following findings from the expert review and cognitive interviews. The content validity index exceeded recommended thresholds (S-CVIAve = 0.926). The psychometric evaluation provided support for DLI-S' validity. The hypothesized six-factor model showed good fit (χ2 = 1107.631 p<0.001, CFI = 0.993, TLI = 0.993, RMSEA = 0.064, SRMR = 0.054). High internal consistency reliability was demonstrated for the overall scale (Cronbach's α = 0.94) and each sub-scale (α 0.81-0.92). Test-retest reliability was acceptable, ICC ranging between 0.66-0.85. Through a comprehensive assessment of several sources of evidence, we show that the DLI-S demonstrates satisfactory validity and acceptability to measure death literacy in the Swedish context. There are, however, indications that the sub-scales measuring community capacity perform worse in comparison to other sca and may function differently in Sweden than in the original context. The DLI-S has potential to contribute to research on community-based EOL interventions.

The development, content and response process validation of a caregiver-reported severity measure for CDKL5 deficiency disorder

BackgroundCDKL5 Deficiency Disorder (CDD) is a severe X-linked developmental and epileptic encephalopathy. Existing developmental outcome measures have floor effects and cannot capture incremental changes in symptoms. We modified the caregiver portion of a CDD clinical severity assessment (CCSA) and assessed content and response-process validity. MethodsWe conducted cognitive interviews with 15 parent caregivers of 1–39-year-old children with CDD. Caregivers discussed their understanding and concerns regarding appropriateness of both questions and answer options. Item wording and questionnaire structure were adjusted iteratively to ensure questions were understood as intended. ResultsThe CCSA was refined during three rounds of cognitive interviews into two measures: (1) the CDD Developmental Questionnaire – Caregiver (CDQ-Caregiver) focused on developmental skills, and (2) the CDD Clinical Severity Assessment – Caregiver (CCSA-Caregiver) focused on symptom severity. Branching logic was used to ensure questions were age and skill appropriate. Initial pilot data (n = 11) suggested no floor effects. ConclusionsThis study modified the caregiver portion of the initial CCSA and provided evidence for its content and response process validity.

Development and initial validation of a video-based peroral endoscopic myotomy assessment tool

Video analysis has emerged as a potential strategy for performance assessment and improvement. We aimed to develop a video-based skill assessment tool for peroral endoscopic myotomy (POEM). POEM was deconstructed into basic procedural components through video analysis by an expert panel. A modified Delphi approach and 2 validation exercises were conducted to refine the POEM assessment tool (POEMAT). Twelve assessors used the final POEMAT version to grade 10 videos. Fully crossed generalizability (G) studies investigated the contributions of assessors, endoscopists' performance, and technical elements to reliability. G coefficients below .5 were considered unreliable, between .5 and .7 as modestly reliable, and above .7 as indicative of satisfactory reliability. After task deconstruction, discussions, and the modified Delphi process, the final POEMAT comprised 9 technical elements. G analysis showed low variance for endoscopist performance (.8%-24.9%) and high interrater variability (range, 63.2%-90.1%). The G score was moderately reliable (≥.60) for "submucosal tunneling" and "myotomy" and satisfactorily reliable (≥.70) for "active hemostasis" and "mucosal closure." We developed and established initial content and response process validity evidence for the POEMAT. Future steps include appraisal of the tool using a wider range of POEM videos to establish and improve the discriminative validity of this tool.

Development and validation of an instrument to measure the Micro-Learning Environment of Students (MLEM)

BackgroundThere are multiple instruments to measure different learning environments but no valid and reliable instrument is present for measuring the micro-learning environment. This study aimed to develop and validate an instrument to measure the micro-learning environment of students. Measuring the micro-learning environment can give insight into the real-life experiences of students and enlighten us about the disparity between taught, delivered, and learned curricula.MethodsMulti-institutional Mixed methods study design with consecutive qualitative and, quantitative components was used based on information processing theory. Literature review, results of semi-structured interviews, and focus group discussion were combined to develop a questionnaire. Content and response process validity were established followed by pilot testing, reliability calculation, and exploratory and confirmatory factor analysis.ResultsA forty-nine-item preliminary draft instrument was reduced to a total of twenty-four items final instrument having five themes regarding teaching practices, learners support, competence in teaching, progressive faculty, and teaching environment. The values of SCVI/Ave and S-CVI/UA were calculated to be 0.92 and 0.62 respectively. Reliability was calculated to be 0.94. Fit indices values were within the normal range.ConclusionThe instrument for measuring the micro-learning environment has excellent content, construct, response process validity, and reliability.

Intentional enterotomies: validation of a novel robotic surgery training exercise.

While laparoscopic simulation-based training is a well-established component of general surgery training, no such requirement or standardized curriculum exists for robotic surgery. Furthermore, there is a lack of high-fidelity electrocautery simulation training exercises in the literature. Using Messick's validity framework, we sought to determine the content, response process, internal content and construct validity of a novel inanimate tissue model that utilizes electrocautery for potential incorporation in such curricula. A multi-institutional, prospective study involving medical students (MS) and general surgery residents (PGY1-3) was conducted. Participants performed an exercise using a biotissue bowel model on the da Vinci Xi robotic console during which they created an enterotomy using electrocautery, followed by approximation with interrupted sutures. Participant performance was recorded and then scored by crowd-sourced assessors of technical skill, along with three of the authors. Construct validity was determined via difference in Global Evaluative Assessment of Robotic Skills (GEARS) score, time to completion, and total number of errors between the two cohorts. Upon completion of the exercise, participants were surveyed on their perception of the exercise and its impact on their robotic training to determine content validity. 31 participants were enrolled and separated into two cohorts: MS + PGY1 vs. PGY2-3. Time spent on the robotic trainer (0.8 vs. 8.13h, p = 0.002), number of bedside robotic assists (5.7 vs. 14.8, p < 0.001), and number of robotic cases as primary surgeon (0.3 vs. 13.1, p < 0.001) were statistically significant between the two groups. Differences in GEARS scores (18.5 vs. 19.9, p = 0.001), time to completion (26.1 vs. 14.4min, p < 0.001), and total errors (21.5 vs. 11.9, p = 0.018) between the groups were statistically significant as well. Of the 23 participants that completed the post-exercise survey, 87% and 91.3% reported improvement in robotic surgical ability and confidence, respectively. On a 10-point Likert scale, respondents rated the realism of the exercise 7.5, educational benefit 9.1, and effectiveness in teaching robotic skills 8.7. Controlling for the upfront investment of certain training materials, each exercise iteration cost ~ $30. This study confirmed the content, response process, internal structure and construct validity of a novel, high-fidelity and cost-effective inanimate tissue exercise which successfully incorporates electrocautery. Consideration should be given to its addition to robotic surgery training programs.

Gathering validity evidence in the development of a new version of the Attitude toward the Subject of Chemistry Inventory (ASCI-UE)

Organic chemistry is one of the most feared and failed courses due to its complex and fast-paced nature. Investigating affective metrics that relate to achievement in these courses can be worthwhile, particularly when these metrics show predictive relationships to achievement. Attitude toward chemistry has been investigated utilizing a variety of instruments. We present herein an instrument related to the well-established Attitude toward the Subject of Chemistry Inventory (ASCIv2). This new instrument was developed utilizing The Standards of Psychological and Educational Measurement, which describe five aspects of validity evidence that should be gathered when using instruments in research. Content validity was gathered through consultation with experts. Response process validity was gathered through student interviews. Internal structural validity was collected through confirmatory factor analysis. Relations to other variables were investigated through correlation analysis and structural equation modeling. Consequential validity was studied through measurement invariance testing before comparing scores for subgroups (high- and low-achieving students). Additionally, reliability evidence was tested with Omega coefficients for each of the factors. We showed that all tests and analyses were done with high rigor, and that this new instrument, ASCI-UE, measuring utility and e motional satisfaction, can provide interesting results and implications for research and practice.

Identifying the response process validity of clinical vignette-type multiple choice questions: An eye-tracking study

Introduction Clinical vignette-type multiple choice questions (CV-MCQs) are widely used in assessment and identifying the response process validity (RPV) of questions with low and high integration of knowledge is essential. Answering CV-MCQs of different levels of knowledge application and integration can be understood from a cognitive workload perspective and this can be identified by using eye-tracking. The aim of the pilot study was to identify the cognitive workload and RPV of CV-MCQs of different levels of knowledge application and integration by the use eye-tracking. Methods Fourteen fourth-year medical students answered a test with 40 CV-MCQs, which were equally divided into low-level and high-level complexity (knowledge application and integration). Cognitive workload was measured using screen-based eye tracking, with the number of fixations and revisitations for each area of interest. Results We found a higher cognitive workload for high-level complexity (M = 121.74) compared with lower-level complexity questions (M = 51.94) and also for participants who answered questions incorrectly (M = 94.31) compared with correctly (M = 79.36). Conclusion Eye-tracking has the potential to become a useful and practical approach for helping to identify the RPV of CV-MCQs. This approach can be used for improving the design and development of CV-MCQs, and to provide feedback to inform teaching and learning. Practice points Clinical vignette- type multiple choice questions (CV-MCQs) are widely used in assessment. Identifying the response process validity (RPV) of CV-MCQs with low and high complexity is essential. Low and high complexity CV-MCQs have different cognitive workloads that can be practically identified using eye-tracking. Eye-tracking has the potential for identifying the RPV of CV-MCQs.

Validation of the Child Observation Record Advantage 1.5 Assessment Tool for Preschool Children: A Multilevel Bifactor Modeling Approach

This paper reports an investigation into the psychometric properties of the COR-Advantage1.5 (COR-Adv1.5) assessment tool, a criterion-referenced observation-based instrument designed to assess the developmental abilities of children from birth through kindergarten. Using data from 8534 children participating in a state-funded preschool program for disadvantaged 4-year-olds, we employed a multilevel bifactor model to show that CORAdv1.5 has some potential as an unbiased assessment tool for tracking developmental progress in pre-kindergarten children, under a bifactor model parameterization, with fewer dimensions than were proposed by the test developer, HighScope. Examination of factor adequacy indices revealed that, despite the multidimensional structure, CORAdv1.5 items overwhelmingly reflect variance due to the general factors at the child and classroom levels. Evidence for multilevel reliability, response process validity, predictive validity, longitudinal measurement invariance, and generalizability of the items are also reported. The findings of this study have important implications for the use of COR-Adv1.5.