Evidence Of Predictive Validity Research Articles

Construct validity of a questionnaire for measuring student engagement in problem-based learning tutorials

BackgroundStudent engagement is student investment of time and energy in academic and non-academic experiences that include learning, teaching, research, governance, and community activities. Although previous studies provided some evidence of measuring student engagement in PBL tutorials, there are no existing quantitative studies in which cognitive, behavioral, and emotional engagement of students in PBL tutorials is measured. Therefore, this study aims to develop and examine the construct validity of a questionnaire for measuring cognitive, behavioral, and emotional engagement of students in PBL tutorials.MethodsA 15-item questionnaire was developed guided by a previously published conceptual framework of student engagement. Focus group discussion (n = 12) with medical education experts was then conducted and the questionnaire was piloted with medical students. The questionnaire was then distributed to year 2 and 3 medical students (n = 176) in problem-based tutorial groups at the end of an integrated course, where PBL is the main strategy of learning. The validity of the internal structure of the questionnaire was tested by confirmatory factor analysis using structural equation modeling assuming five different models. Predictive validity evidence of the questionnaire was studied by examining the correlations between students’ engagement and academic achievement.ResultsConfirmatory factor analysis indicates a good fit between the measurement and structural model of an 11-item questionnaire composed of a three-factor structure: behavioral engagement (3 items), emotional engagement (4 items), and cognitive engagement (4 items). Models in which the three latent factors were considered semi-independent provided the best fit. The construct reliabilities of behavioral, cognitive, and emotional factors were 0.82, 0.82, and 0.76, respectively. We failed however to find significant relationships between academic achievement and engagement.ConclusionsWe found a strong evidence to support the construct validity of a three-factor structure of student engagement in PBL tutorial questionnaire. Further studies are required to test the validity of this instrument in other educational settings. The predictive validity is another area needing further scrutiny.

Children’s perceptions of teacher feedback: Portuguese validation of a scale

ABSTRACT This research aims to validate the Portuguese version of the Teacher Feedback Scale (TFS) in a sample of 628 children from third and fourth grade. Teacher feedback is a crucial factor in child development, with an impact in students’ motivation and learning. It provides daily cognitive and emotional information about children’s academic performance and their abilities as learners. The results of exploratory factor analysis (EFA) and subsequent confirmatory factor analysis (CFA) indicated that the TFS consists of five subscales: praise, ability-effort in reading, ability-effort in math, general negative feedback and specific negative feedback. The measure shows that good reliability estimates and factor loadings were invariant across gender. Evidence of predictive validity is presented through gender differences and associations of feedback with children's competence perceptions. The potential of the measure in educational settings and research is discussed.

Psychometric Testing of a Scale Measuring Preceptor Preparedness: The Cap-ExPresSTM Study.

Background and Purpose: Preceptors educate nursing students to practice as novice, generalist nurses. There are no instruments measuring preceptor preparedness. The purpose of this study was to psychometrically test the Capstone-Experience Preceptor Preparedness Scale (Cap-ExPresS™). Methods: A cross-sectional multi-site survey design was used. Results: A convenience sample of 118 preceptors was recruited from four Midwest hospitals. Exploratory factor analysis supported a 22-item scale representing four factors demonstrating internal consistency reliability using Cronbach's alpha: total scale, 0.95, student-centeredness, 0.90, pedagogic competence, 0.90, clinical competence, 0.81, and nurse professionalism, 0.87. Test-retest reliability was adequate for clinical competence at 0.71, 0.50-0.61 for other scores. Some evidence of predictive validity was observed via regression. Conclusions: Psychometric testing supported the validity and reliability of inferences made about preceptor preparedness with Cap-ExPresS™ scores. Scale scores indicate preceptor learning needs, evaluate interventions influencing preceptor preparedness, and assist clinicians to develop best practices for preceptor preparedness.

University Teacher Credibility as a Strategy to Motivate Students

University students’ motivation can be affected by several factors, one being their perceptions of teacher behaviour in the classroom. This study aimed to predict university students’ state motivation from their perceptions of teacher credibility. The participants were 344 students from the University of Seville. A structural equation model was used with the partial least squares method (PLS-SEM), a technique based on variance, employed to test, and validate the proposed hypotheses. The results reveal, on the one hand, a positive effect of teacher credibility on state motivation and, on the other hand, that teacher credibility has predictive power and predictive relevance for state motivation. Likewise, there is evidence of predictive validity in that teacher credibility can predict values for new cases of state motivation. Strategies are provided for a university faculty to manage their behaviour in the classroom to increase their students’ state motivation, highlighting the use of PLS-SEM as a data analysis tool suitable for application in higher education.

Psychometric examination of the Malay version of the Pain Catastrophising Scale in a Malaysian chronic pain sample

Introduction: The purpose of this study was to examine the psychometric properties of the Malay version of the Pain Catastrophizing Scale (PCS-M). Methods: The original PCS was translated into Malay using the back-translation method and was administered to 132 outpatients with non-malignant chronic pain. Results: Confirmatory factor analysis revealed that a 10-item single-factor model had a better fit profile compared to 8 competing models as documented in previous studies. In terms of internal consistency, Cronbach’s α value for the PCS-M was 0.93. In terms of predictive validity, the PCS-M explained 47%, 24%, and 22% of the variance in anxiety, depression, and kinesiophobia, respectively. Conclusion: Best presented as a unidimensional construct, the 10-item PCS-M demonstrated excellent reliability estimate and initial evidence for predictive validity in a Malaysian chronic pain sample. The 10-item PCS-M is a reliable and valid tool to be used in chronic pain management in the Malaysian context.

Ecological momentary assessment of mood regulation eating expectancies in eating disorders: Convergent and predictive validity.

Eating expectancies (EE) are the anticipation of various benefits or detriments from eating, with mood regulation being a salient type of EE associated with eating disorders. This study examined the convergent and predictive validity of ecological momentary assessment (EMA) mood regulation EE items, including mood improvement and mood worsening EE. Thirty women with binge-eating pathology completed a 14-day EMA protocol, which included measures of mood regulation EE, affect, appetite, appearance- and body-related factors and disordered-eating behaviours. Greater within-subjects hunger and lower within-subjects positive affect and fullness were related to elevated mood improvement EE. Higher within-subjects appearance concerns, fullness, body social comparisons and thinness pressure were associated with higher mood worsening EE. Greater within-subjects mood worsening EE predicted greater likelihood of vomiting at the subsequent time point, but there were no within-subjects associations between mood improvement EE and behaviours. Yet, greater between-subjects mood worsening EE were associated with more restraint/restriction and binge eating, and greater between-subjects mood improvement EE were associated with more binge eating. Findings support the convergent validity of EMA mood regulation EE items. There was limited predictive validity evidence, suggesting complexities in how mood regulation EE predict behaviour in daily life.

Brazilian version of Northwestern university famous faces test (NUFFACE‐Br): Evidence of validity between control and clinical groups

AbstractBackgroundNUFFACE was created to identify the locus of name‐face failure common in healthy aging and dementia. Psychometric studies are necessary for the development of the Brazilian version. Therefore, we intend to verify the evidence of predictive (or diagnostic) validity of this instrument.Methodsa total 24 participants were included in this study and divided into 12 cognitively healthy elderly (NC) and 12 with dementia due to Alzheimer’s Disease (AD; CDR‐1). All responded to the NUFFACE‐Br, which has four tasks: 1) facial naming; 2) verbal description of the facts related to the face; 3) verbal description in response to the printed name; and 4) face‐name matching (multiple‐choice format). Different types of errors, such as Semantically Related (SR), Semantically Unrelated (SU), Phonological Similarity (Ph), “I don’t know” (DK), and Others (O), were developed as part of the correction criteria. The comparison of sociodemographic variables and NUFFACE‐Br total tasks were performed by Mann‐Whitney U test, adopting a significance level of 5%.ResultsThe groups did not differ by age, education, and sex (p<0.05). The NC performed better than the AD in almost all tasks of this instrument (p<0.01), except for task 4 (p<0.266). The AD presented more difficulty in the naming faces tasks (average = 14±6.6) than the others and similar performance to the NC in the face‐name correspondence (average score = 30.7±2.1). A higher frequency of all types of errors was verified in the AD group for the first three tasks of this instrument, except for the “phonological similarity,” in which the frequency of errors was the same between the groups. Task 4 presented a higher frequency in SR, SU, and O errors.ConclusionSignificant differences were verified between the NC and AD groups, indicating the predictive (or diagnostic) evidence of NUFFACE‐Br. The face‐name matching task was the only one that did not show differences between the groups. The AD presented a higher frequency of errors when compared to the NC, especially in the task of naming faces. Further psychometric studies are being conducted to elucidate the clinical evidence of this instrument to old Brazilian adults.

Blinding of Admissions Committee to MCAT Scores: Is There a Difference in Subsequent Medical School Performance Between Cohorts?

The Medical College Admission Test (MCAT) is designed to inform members of the admissions committee about applicants' academic readiness for medical school. Although previous work has shown that MCAT scores have some predictive validity evidence for a variety of medical student outcomes, there is also a concern that the MCAT is overly emphasized by admissions committees, which may, for example, affect matriculant diversity. The purpose of this study was to understand whether deemphasizing the MCAT by blinding committee members to applicants' specific scores has resulted in matriculants with different pre-clerkship and clerkship performance. The Admissions Committee from the Uniformed Services University of the Health Sciences (USU) created a policy to blind admissions committee members to MCAT scores. The MCAT-blinded policy was implemented for classes of 2022 to 2024. This MCAT-blinded cohort's performance was compared with a previous cohort, classes of 2018 to 2020. Two analyses of covariance were performed to test for differences in the pre-clerkship and clerkship module scores. Undergraduate grade point average (uGPA) and MCAT percentile for matriculants were included as covariates. There were no statistically significant differences in either pre-clerkship or clerkship performance between the MCAT-revealed and MCAT-blinded cohorts. This study found similar medical school performance between the MCAT-blinded and MCAT-revealed cohorts. The research team plans to continuously follow these two cohorts to understand their performance further down their education path, including step 1 and step 2 examinations.

Surveying the Middle Ground: A Thematic Analysis of the Medical School Applications of Standard Performers

As gatekeepers to the medical profession, admissions committees make important decisions through the analysis of quantitative (e.g. test scores and grade point averages) and qualitative (e.g. letters of recommendation and personal statements) data. One area of the application that deserves additional study is the Work and Activities section, where students describe their extracurricular experiences. Previous research has found different themes that sometimes overlap in the applications of exceptional performing and low performing medical students, but it is unclear if these themes are present in the applications of standard performers.An exceptional performing medical student is defined as one who was inducted into both the Alpha Omega Alpha Honor Medical Society and the Gold Humanism Honor Society. A low performing medical student is one who was referred to the Student Promotions Committee (SPC) where an administrative action was made. A standard performing medical student is defined as someone who was neither a member of an honor society nor referred to the SPC during medical school. Using a constant comparative method, the applications of standard performers who graduated from the Uniformed Services University between 2017 and 2019 was assessed using exceptional performer themes (success in a practiced activity, altruism, teamwork, entrepreneurship, wisdom, passion, and perseverance) and low performer themes (witnessing teamwork, embellishment of achievement, and description of a future event). The presence of novel themes was also assessed. The absolute number of themes and the diversity of themes were determined. Demographic data were collected (age, gender, number of Medical School Admissions Test [MCAT] attempts, highest MCAT score, and cumulative undergraduate grade point average), and descriptive statistics was performed.A total of 327 standard performers were identified between 2017 and 2019. No novel themes were found after coding 20 applications. All exceptional performer themes were identified in the population of standard performers. The low performing theme of embellishment of achievement was not found. Standard performers had a lower number of total exceptional performer themes and a lower diversity of exceptional performer themes compared to low and exceptional performers, but standard performers also had a lower number and less diversity of low performing themes compared to low performers.This study suggests that the diversity and frequency of exceptional performing themes in a medical school application may be useful in distinguishing between exceptional performers and other performers, though the small sample size limits quantitative conclusions. Low performing themes may be specific to low performers and thus could be useful to admissions committees. Future studies should include a larger sample size and could assess for predictive validity evidence of these exceptional performing and low performing themes through a blinded protocol.

A Suicide-Specific Diagnosis – The Case For

A Suicide-Specific Diagnosis – The Case For

Measuring the dosage of brief and skill-targeted social-emotional learning (SEL) activities in humanitarian settings.

In humanitarian settings, social-emotional learning (SEL) programs for children are often delivered using a field-feasible approach where the programs are more easily deployable and adaptable in the field, require minimal training, and depend less on the strict sequence and structure of the program components to elicit the intended treatment effect. However, evidence is lacking on what aspects of this implementation approach enable the SEL programming to be more beneficial to children's SEL development. In this study, we propose and evaluate measures for three dimensions of dosage (quantity, duration, and temporal pattern) of two sets of brief and skill-targeted SEL activities (Mindfulness and Brain Games) implemented in 20 primary schools in two low-income chiefdoms of Sierra Leone. We find preliminary evidence of predictive validity that these dosage measures could predict children's attendance and classroom adaptive behavior. This study is the first to develop procedures to measure the dimensions of dosage of brief SEL activities in humanitarian settings. Our findings illuminate the need for future research on optimizing the dosage and implementation design of SEL programming using brief SEL activities.

Phonological Awareness Assessment Test (PACOF) for Pre-school Children: Evidence of Validity and Reliability

Phonological awareness problems have been identified as predictors of learning difficulties with reading and writing in alphabetic languages. The objective of this study is to present an analysis of the psychometric properties of the Phonological Awareness Assessment Test (PACOF), a computer-based test for assessing syllable, phoneme, and intra-syllable awareness in pre-schoolers. Two studies were conducted to collect: (1) evidence of concurrent validity and reliability (N = 30) and (2) evidence of predictive validity (N = 52). Significant correlations between the scores in PACOF and a different test of phonological awareness were found. Regarding reliability, test-retest results indicated a high stability in the scores over time. Concerning predictive validity, the results revealed that scores on the PACOF and on each of its three sub-scales obtained at the end of preschool predict future reading and writing performance. These findings suggest that the test is a reliable and valid measure to assess phonological awareness in pre-schoolers.

The Autism Symptom Dimensions Questionnaire: Development and psychometric evaluation of a new, open-source measure of autism symptomatology.

To describe the development and initial psychometric evaluation of a new, freely available measure, the Autism Symptom Dimensions Questionnaire (ASDQ). After development and revision of an initial 33-item version, informants completed a revised 39-item version of the ASDQ on 1467 children and adolescents (aged 2-17 years), including 104 with autism spectrum disorder (ASD). The initial 33-item version of the ASDQ had good reliability and construct validity. However, only four specific symptom factors were identified, potentially due to an insufficient number of items. Factor analyses of the expanded instrument identified a general ASD factor and nine specific symptom factors with good measurement invariance across demographic groups. Scales showed good-to-excellent overall and conditional reliability. Exploratory analyses of predictive validity for ASD versus neurotypical and other developmental disability diagnoses indicated good accuracy for population and at-risk contexts. The ASDQ is a free and psychometrically sound informant report instrument with good reliability of measurement across a continuous range of scores and preliminary evidence of predictive validity. The measure may be a useful alternative to existing autism symptom measures but further studies with comparison of clinical diagnoses using criterion-standard instruments are needed. The Autism Symptom Dimensions Questionnaire (ASDQ) is a new, freely available measure of autism symptoms. The ASDQ showed reliable and accurate measurement of autism symptoms. The measure had good screening efficiency for autism spectrum disorder relative to other developmental conditions.

Webcam-based eye tracking to detect mind wandering and comprehension errors.

Recent advances in computer vision have opened the door for scalable eye tracking using only a webcam. Such solutions are particularly useful for online educational technologies, in which a goal is to respond adaptively to students' ongoing experiences. We used WebGazer, a webcam-based eye-tracker, to automatically detect covert cognitive states during an online reading-comprehension task related to task-unrelated thought and comprehension. We present data from two studies using different populations: (1) a relatively homogenous sample of university students (N = 105), and (2) a more diverse sample from Prolific (N = 173, with < 20% White participants). Across both studies, the webcam-based eye-tracker provided sufficiently accurate and precise gaze measurements to predict both task-unrelated thought and reading comprehension from a single calibration. We also present initial evidence of predictive validity, including a positive correlation between predicted rates of task-unrelated thought and comprehension scores. Finally, we present slicing analyses to determine how performance changed under certain conditions (lighting, glasses, etc.) and generalizability of the results across the two datasets (e.g., training on the data Study 1 and testing on data from Study 2, and vice versa). We conclude by discussing results in the context of remote research and learning technologies.

Interrogating the predictive validity of aptitude testing for interpreting: a systematic methodological review

ABSTRACT Aptitude testing is used to select candidates with the greatest potential for professional interpreter training. Implicit in this practice is the expectation that aptitude test scores predict future performance. As such, the predictive validity of score-based inferences and decisions constitutes an important rationale for aptitude testing. Although researchers have provided predictive validity evidence for different aptitudinal variables, very little research has examined the substantive meaning and robustness of such evidence. We therefore conducted this systematic review to investigate or interrogate the methodological rigour of quantitatively-based prospective cohort studies of aptitude for interpreting, focusing on the substantive meaning, psychometric soundness, and statistical analytic rigour underpinning their predictive validity evidence. Our meta-evaluation of 18 eligible studies, identified through a rigorous search and screening process, shows a diverse array of practices in the operationalisation, analysis, and reporting of aptitude tests, interpreting performance assessments, and related validity evidence. Main patterns include the collection of mostly single-site data (i.e., from a single institution), use of self-designed instruments for testing aptitude, and under-reporting of key information on measurement and statistical procedures. These findings could help researchers better interpret existing validity evidence and design future research on aptitude testing.

Профессиональная направленность абитуриентов вузов как предиктор успешности профессионального становления

Background. Numerous methods have been developed to guide schoolchildren toward career choice and professional education. However, empirical evidence for predictive validity of such methods is quite weak. This study specifies the case of university applicants (recent schoolchildren) profession choices. The author’s approach to identification of a professional orientation, developed in previous studies and based on the integrative-­typological classification of the world of work, is used. Objective. The aim is to check the relation between applicants’ professional orientation, requirements of the chosen professional activity with the success of professional development in the initial period of studying at university. Methods. We used the following methods: “Integrative-­typological professional orientation of the personality questionnaire”, “The scale of satisfaction with life”, “The scale of satisfaction with studies”, “The studies engagement questionnaire”, “Organizational and suborganizational identity questionnaire”. Sample. The sample involved 435 applicants who then entered the university and studied in the chosen field. Results. According to the subject matter of the chosen profession, four types of university education areas have been distinguished. These include integral, object-­related, human-­related and information-­related types. The criteria for the compliance of the professional orientation of university applicants with the requirements of these areas are formulated. While studying at university the applicants who met these requirements and became students showed higher indicators of professional development such as academic performance, organizational identity, involvement in studying, and satisfaction with it. Study results qualitatively correspond to the results of other studies revealing various indicators of professional orientation variability and their deterministic influence on some indicators of professional development. Conclusion. The study deduces four types of university education areas based on subjective work area (integral, object-related, human-­related and information-­related). Each type specifies requirements for applicant’s professional orientation. The correspondence between these requirements and applicant orientation turns out to be a predictor of professional development success, involvement in studying, satisfaction with it, organizational identity, and academic performance in the beginning of university education. More than 30% of the applicants who have become university students do not meet the requirements of the chosen professional activity and the university education direction.

The Process Model of Emotion Regulation Questionnaire: Assessing Individual Differences in Strategy Stage and Orientation.

The study of individual differences in emotion regulation has typically focused exclusively either on the stage of the emotion generation process at which regulation occurs or on the engagement versus disengagement orientation of the regulation efforts. We introduce a new measure that samples equally across each stage of the process model of emotion regulation and from both engagement and disengagement orientations. Through five studies (ntotal = 2,543), we present the development and convergent, discriminant, and predictive validity evidence for the Process Model of Emotion Regulation Questionnaire (PMERQ). We show the final 10-scale 45-item questionnaire has acceptable internal consistency, is invariant between genders and across the age range, and has sufficient convergent and discriminant validity. The PMERQ also predicts affective and relationship functioning with strategies occurring earlier in the process model generally showing the strongest effects. We propose the PMERQ is a valuable measure to assess individual differences in the use of emotion regulation strategies.

“Nature is mine/ours”: Measuring individual and collective psychological ownership of nature

Psychological ownership of nature (i.e., the feeling that nature is “mine” or “ours”) has been examined to encourage pro-environmental behaviors. However, a psychometrically validated measure that assesses the core of the construct is currently lacking, which undercuts its value and growing research efforts. Following recommended procedures, this paper developed and validated 6-item scales of individual and collective psychological ownership of nature, using a representative sample of Australian adults (total N = 836). The two scales exhibited high reliability and evidence of convergent, discriminant, predictive, and incremental validity, with measurement invariance across gender, age, and political subgroups. The validated individual and collective psychological ownership of nature scales were statistically distinct from proximate constructs (e.g., connection to nature and place attachment). The two scales had unique power in explaining civic pro-environmental behaviors, over and above connection to nature. Collective (but not individual) psychological ownership of nature contributed additional variance over place attachment in civic pro-environmental behaviors. This paper advances research on psychological ownership of nature by offering validated scales measuring the core of this increasingly important construct.

Treatment Fidelity and Outcome in CBT for Youth with Autism: The MEYA Fidelity Scale

ABSTRACT Objective Assessing treatment fidelity in effectiveness research is critical to interpreting study findings. This paper details the development and initial psychometric evaluation of the Modular Evidence-Based Practices for Youth with Autism Fidelity Scale (MEYA-FS) designed to support the assessment of cognitive-behavioral treatments for youth with autism in effectiveness research. Method Recorded treatment sessions (N = 338) were randomly selected from 77 youth (M age = 9.65 years, SD = 1.87; 50.67% White; 85.33% male) who received the Schema, Emotion, and Behavior-Focused Therapy for Children (SEBASTIEN) (n = 51) or Coping Cat (n = 24) program. Results The MEYA-FS Adherence items displayed acceptable interrater reliability, but more than half of the MEYA-FS Competence items did not. The magnitude and pattern of correlations supported the score validity of the MEYA-FS Adherence and Competence items and subscales. However, some corresponding Adherence and Competence items displayed significant overlap. Scores on each Adherence subscale distinguished between the SEBASTIEN and Coping Cat programs, providing support for discriminant validity. Finally, higher Adherence and Competence subscales predicted significant improvements in youth clinical outcomes (adjustment problems in the school setting, social-communication difficulties, restrictive/repetitive behaviors, and externalizing problems), providing initial evidence for predictive validity. Conclusions The psychometric properties of the MEYA-FS make it appropriate for supporting efforts to evaluate cognitive-behavioral interventions for youth with autism in effectiveness and implementation research.

Analysis of a brief biodata scale as a predictor of job performance and its incremental validity over the Big Five and Dark Tetrad personality traits.

The collection of biographical information (biodata) through CVs and application forms has many advantages, namely easiness of collection, acceptable validity, less prone to faking, and the fulfilment of legal requirements. However, its systematic use among practitioners is scarce. Two of the mains reasons is the overlap with other constructs like personality and the lack of validated biodata scales in articles and public repositories. Aimed to fill this gap, García-Izquierdo and colleagues developed an 8-item scale able to generate positive applicant reactions, but they did not provide empirical evidence that their scale is able to predict job performance. The present paper was developed for this purpose, investigating the scale’s relationship with four different dimensions of job performance (i.e., task performance, contextual performance, counterproductive behaviors, and adaptive performance) and its incremental validity with respect to Big Five and Dark Tetrad personality traits. The study comprises 528 employees from different organizations (Mage = 39.51, SD = 14.25; 52.8% women, Mexperience = 17.06, SD = 13.27) which voluntarily agreed to participate filling a questionnaire with the variables of interest. Results provide evidence of the predictive validity of the biodata scale in a multi-occupational sample; identify that these biodata contribute to predicting two specific types of job performance: contextual performance and adaptive performance; shows that a brief job-related biodata scale achieves results comparable to those of most personality traits in predictive models of job performance dimensions; and provide evidence of the incremental predictive validity of biodata over the Big Five and the Dark Tetrad. As a whole, these results provide support for the use of the scale in researcher and applied settings, and contributes to the advance the knowledge of biodata for personnel selection.