Evidence Of Criterion-related Validity Research Articles

Developing and validating a Domain-specific Grit Scale for College Athletic Students

The aim of this study was to create and validate a ten-item Domain-specific Grit Scale for College Athletic Students (DGSCAS) to assess the level of grit among college athletic students. College athletic students from a single independent college located in a northern city in China (526 participants at time 1 and 589 participants at time 2) were assessed according to the scale. Various analyses were conducted in this study, including exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and measurement invariance analysis across different sex and birthplaces. The results of the EFA revealed two factors: consistency of interests and perseverance of effort. The CFA results demonstrated acceptable fit indices (x2 = 160.048, df = 34, x2/df = 4.707, CFI = 0.983, TLI = 0.978, SRMR = 0.021, and RMSEA = 0.079). The scale exhibited satisfactory convergent validity and discriminant validity. The significant correlation of these factors with the Grit scale provided strong evidence of criterion-related validity. Measurement invariance analysis indicated that the scale performed consistently across different sex and birthplaces. Three limitations and corresponding recommendations were discussed, including sample heterogeneity, the lack of a unified test result as a criterion for predictive validity, and the cross-sectional design of the study. In conclusion, the DGSCAS is a practical and validated instrument that can be used to assess the level of grit among college athletic students in an educational context.

Measuring Linguistic Growth in Sentence-Level Writing Curriculum-Based Measures: Exploring Complementary Scoring Methods.

Picture-word writing curriculum-based measures (PW CBM-Ws) are technically sound, formative measures of descriptive, sentence-level writing but cannot estimate underlying linguistic skills. The purpose of this exploratory alternative scoring investigation was to apply metrics from language sample analysis (LSA) to PW CBM-Ws as a complementary measure of underlying language skills in beginning writers' sentence-level writing. LSA metrics were applied to 104 typically developing first through third graders' PW CBM-W samples across fall and spring semesters. Factorial analyses of variance with post hoc Bonferroni pairwise comparisons were applied after obtaining alternate-form reliability and criterion-related validity estimates. Analyses revealed reliable discrimination between grades and significant growth between fall and spring semesters for three LSA metrics: mean length of T-unit in words, mean length of T-unit in morphemes, and number of different words. While mean length of T-unit in words and morphemes demonstrated evidence of discrimination and growth in first grade only, number of different words showed evidence of reliable discrimination and growth in first and third grades. Mean length of T-unit in words, mean length of T-unit in morphemes, and number of different words showed evidence of adequate criterion-related validity, discrimination among grades, and sensitivity to growth when calculated using PW CBM-W samples to gauge underlying linguistic skills in first- and third-grade students. Implications and future directions for research are discussed. https://doi.org/10.23641/asha.25050290.

A labeled magnitude scale for measuring the anticipated impact of immediate experiences

Lundberg et al. (2022) recently developed a rubric for classifying experience impacts as ordinary, meaningful, memorable, or transformative. This article extends that contribution by developing a measure of anticipated impacts suitable for evaluating the anticipated impacts of immediate, structured experiences, such as heritage interpretation experiences, museum visits, and events. We developed a labeled magnitude scale (LMS) by calibrating six adverbial modifiers of impact intensity (negligible, slight, significant, profound, substantial, transformative) to judges’ ratings of ratio-scaled distances. We examined criterion-related evidence of validity using 2,867 experience observations from 594 university students who visited venues selected to advance their emotional connection to their university. Students completed the LMS immediately after their visits, along with measures of hypothesized determinants and results. Findings supported all hypotheses; the LMS yielded scores from which valid inferences may be made about anticipated impacts of immediate, structured experiences.

The Short Version of the Italian Maastricht Vital Exhaustion Questionnaire (MVEQ): Psychometric Properties and Relationships with Anxiety, Depression, and Stress in a Community Sample of Older Adults.

Vital Exhaustion (VE) refers to a physical and mental state of excessive fatigue, feelings of demoralization, hopelessness, and increased irritability. The short form of the Maastricht Vital Exhaustion Questionnaire (MVEQ) is a widely used measure to assess VE. Despite its utility is broadly recognized, the validity and reliability of the scale have yet to be examined in the Italian context. The present study aimed to evaluate the psychometric properties of the shortened MVEQ in a community sample of Italian older adults. A total of 722 older adults (Mage = 72.97, SD = 7.71; 60.4% females) completed the MVEQ, as well as other self-report questionnaires assessing anxiety, depression and stress in order to evaluate the criterion-related validity of the scale. A confirmatory factor analysis (CFA) was conducted to examine the original MVEQ latent structure. Internal consistency was assessed through model-based omega coefficient. Test-retest reliability was examined by re-administering the MVEQ after three months to a subsample of 568 participants. Factorial invariance tests across gender were conducted by means of multi-group CFAs. The one-factor model showed an acceptable fit to the data. The MVEQ yielded a reliable total score (ω = 0.822)and showed moderate-to-large correlations with measures of anxiety, depression, and stress (r range 0.30 to 0.75, ps < 0.001). Test-retest reliability was supported by an Intraclass Correlation Coefficient (ICC) of 0.661. Lastly, the scale was factorially invariant across gender. Overall,the MVEQ provided evidence of reliability and criterion-related validity in a sample of Italian older adults and may be useful for both clinical and research practices.

Cross-cultural validation and measurement invariance of anxiety and depression symptoms: A study of the Brief Symptom Inventory (BSI) in 42 countries

BackgroundDepression and anxiety are among the most prevalent mental health issues experienced worldwide. However, whereas cross-cultural studies utilize psychometrically valid and reliable scales, fewer can meaningfully compare these conditions across different groups. To address this gap, the current study aimed to psychometrically assess the Brief Symptomatology Index (BSI) in 42 countries. MethodsUsing data from the International Sex Survey (N = 82,243; Mage = 32.39; SDage = 12.52; women: n = 46,874; 57 %), we examined the reliability of depression and anxiety symptom scores of the BSI-18, as well as evaluated evidence of construct, invariance, and criterion-related validity in predicting clinically relevant variables across countries, languages, genders, and sexual orientations. ResultsResults corroborated an invariant, two-factor structure across all groups tested, exhibiting excellent reliability estimates for both subscales. The ‘caseness’ criterion effectively discriminated among those at low and high risk of depression and anxiety, yielding differential effects on the clinical criteria examined. LimitationsThe predictive validation was not made against a clinical diagnosis, and the full BSI-18 scale was not examined (excluding the somatization sub-dimension), limiting the validation scope of the BSI-18. Finally, the study was conducted online, mainly by advertisements through social media, ultimately skewing our sample towards women, younger, and highly educated populations. ConclusionsThe results support that the BSI-12 is a valid and reliable assessment tool for assessing depression and anxiety symptoms across countries, languages, genders, and sexual orientations. Further, its caseness criterion can discriminate well between participants at high and low risk of depression and anxiety.

Measuring SETPOINT vocational interest dimensions: The development and validation of three short scales

Vocational interest research has seen a resurgence in the applied psychology literature, given evidence showing its predictive validity for key work outcomes. There is a need for integrative, reliable, and valid measures to advance research in this space. While the RIASEC model of vocational interests (Holland, 1997) has been the most widely used and studied typology for the assessment of six broad interest types, more recent work with the SETPOINT model (Su et al., 2019) suggests that eight interest dimensions provide better fit to interest data and demonstrates stronger criterion-related validity evidence. However, to date, no short scales are available for measuring the broad SETPOINT dimensions. We developed three short scales with 8, 24, and 41 items, respectively, that capture the eight dimensions of the SETPOINT model in an integrative manner. Using a sample of 972 full-time working adults assessed across two-time points over five weeks, we validated these three short scales following robust psychometric analyses. These scales are shown to have good psychometric properties. The development and validation of the three short scales help close the operational gap for the SETPOINT model and further facilitate the study of interests and use of interest measures in academic and applied settings.

Validating a critical thinking ability questionnaire for EFL learners

The present study validated a self-developed Critical Thinking Ability Questionnaire by analyzing data from 1169 valid questionnaires completed by students, as well as analyzing interview data from with four students and two experts. Through exploratory factor analysis, a seven-factor model was identified, including analysis, inference, argument evaluation, language construction, self-reflection and correction, cultural comparison, and credibility evaluation of language sources. Confirmatory factor analysis further supported the construct of the questionnaire, confirming that the construct of the questionnaire supports the adequacy and appropriateness of inferences made from the interpretation of questionnaire scores. The interviews provided additional evidence of the consistency between the critical thinking ability assessed by the questionnaire and the theoretical model. Experts interviewed also confirmed the relevance and coverage of the questionnaire, so the content of the questionnaire can support appropriate score interpretation. The evidence of criterion-related validity was supported by significant correlation coefficients. Additionally, the interpretation of questionnaire scores demonstrated satisfactory internal reliability and stability over time. With psychometrical and qualitative results, this research indicated that the CTA questionnaire is a sound instrument that demonstrated evidence of construct validity, content validity and criterion-related validity for Chinese high school students. Limitations and future directions are discussed.

The relationship between English language proficiency and academic performance among English learners: A subgroup analysis by home language background

In U.S. K-12 public education, states' English language proficiency (ELP) standards and assessments have undergone changes in their content to correspond to the language demands of new academic content-area standards (e.g., Common Core State Standards). However, empirical investigations about how English learner (EL) students perform on the new assessments are limited despite the continued use of ELP assessments for making various decisions about students' academic paths. Addressing this gap, this study focused on criterion-related validity evidence by examining the relationship between ELP and content assessment performances of fifth-grade EL students in one state. We examined all EL students and EL home language subgroups. The results suggest that the ELP Reading subdomain was a significant predictor of content assessment scores in English language arts and mathematics; however, the patterns and magnitude of ELP and content relationship differed across different home language groups. Implications for practice, policy, and research are discussed. Educational relevance statementStates' standards-based ELP assessments play a vital role in making decisions about individual EL students' academic paths as well as in school planning for resources and instruction for EL students. The findings of the present study provide important validity evidence to support the intended use of ELP assessments. The study also underscores the importance of investigating within group variations, as the EL group is tremendously heterogenous in terms of their cultural and linguistic backgrounds, socioeconomic status, length of U.S. residence, and formal schooling experience. The results of the study shed light on a more nuanced understanding of the relationship between ELP and the academic performance of EL students from different home language backgrounds.

Evidence for Criterion-Related Validity of Pressure-Mediated Reflection Spectroscopy for Approximating Fruit and Vegetable Consumption among Preschool Children

BackgroundSkin carotenoids are a valid biomarker for approximating fruit and vegetable consumption (FVC). Veggie Meter® (VM®, Longevity Link Corp.) is a pressure-mediated reflection spectroscopy (RS)-based device that allows for noninvasive and rapid assessment of skin-carotenoid score (SCS) in adults and children. Although VM® is established as a valid tool to measure FVC in adults, there is limited research supporting the validity evidence of the VM® to approximate FVC among preschool children. ObjectiveThe current study aims to assess evidence supporting the criterion-related validity of RS-based SCS for approximating FVC among preschool children (3–5-y old). MethodsWe collected cross-sectional data from typically developing preschool children (n = 136) attending the Child and Adult Care Food Program-participating family child care home settings (FCCHs) (n = 46) in Nebraska. Research team members collected children’s height and weight to calculate body mass index; and measured children’s SCS using the VM®. Children’s FVC in FCCHs were collected using dietary observation. In addition, parents (n = 89) completed a shortened food frequency questionnaire to report children’s FVC at home. Kendall’s Tau (τ) correlation tests were conducted to measure the association between children’s SCS with FVC in FCCHs and with parent-reported total fruit and vegetable (FV) frequency scores. ResultsChildren’s SCS were significantly correlated with their mean FVC in FCCHs, τ = 0.14 (P = 0.02), total provitamin A carotenoids intake in FCCHs, τ = 0.19 (P < 0.001), and with parent-reported total FV frequency score, τ = 0.16 (P = 0.04). ConclusionThe correlation coefficients in this study are comparable with results from a meta-analysis examining associations between FVC and SCS in 7–10-y-old children (r = 0.20). Thus, evidence suggests that RS offers a potentially valid, objective, and feasible method to assess preschool children’s total FVC in multiple settings, especially in conjunction with other dietary assessment tools.

Students’ feedback literacy in higher education: an initial scale validation study

ABSTRACT Given the crucial role of feedback in supporting learning in higher education, understanding the factors influencing feedback effectiveness is imperative. Student feedback literacy, that is, the set of attitudes and abilities to make sense of and utilize feedback is therefore considered a key concept. Rigorous investigations of feedback literacy require psychometrically sound measurement. To this end, the present paper reports on the development and initial validation (N = 221) of a self-report instrument. Grounded in the conceptual literature and building on previous scale validation efforts, an initial overinclusive item pool is generated. Exploratory factor analysis and the Rasch measurement model yield adequate psychometric properties of an initial scale measuring two dimensions: feedback attitudes and feedback practices with a total of 21 items. We further provide evidence for criterion-related validity. Findings are discussed in light of the emerging feedback literacy literature and avenues for further improvement of the scale are reported.

Development Of a Jealousy Scale in Romantic Relationship: A Validity and Reliability Scale

The article presents the development and validation of a new measurement tool, the scale of Jealousy in romantic relationship (JRRS), designed to assess romantic jealousy in individuals aged 18 and above with or without a romantic relationship. The scale was developed through a literature review and tailored to the Turkish culture. The exploratory factor analysis revealed a three-dimensional structure, with the first dimension explaining 32.94% of the total variance, the second dimension explaining 11.34%, and the third dimension explaining 9.21%. The total variance explained by all three dimensions was 53.50%. The moderate correlation between the Anxious Attachment subdimension of the Experiences in Close Relationships Scale-Short Form (ECRS-SF) and the JRRS provided evidence of criterion-related validity, consistent with previous findings in the literature. As a result of the reliability analysis, the reliability value of the measurement tool was found to be .85 according to the Cronbach alpha. According to the sub-dimensions, reliability results were obtained as .83 for the social environment and past relationships factor, .83 for the jealousy perception factor and .71 for the perception of the waist factor. The JRRS was compared to other established scales measuring jealousy in romantic relationships, and the results further supported its reliability and validity. Overall, the study showed that the JRRS is a valid and reliable tool for assessing romantic jealousy in individuals.

Further Validation of the Religious Surrender and Attendance Scale-3 (RSAS-3)

Background: The Religious Surrender and Attendance Scale −3 (RSAS-3) is a very brief measure used to quantify religious commitment as a protective health factor. Methods: To provide evidence of criterion-related validity of the RSAS-3, 440 community members and undergraduate students completed a survey containing three religiosity measures: the RSAS-3, the Intrinsic/Extrinsic Orientation scale, and the Belief into Action scale (BIAC), and a measure of problematic substance use, Texas Christian University Drug Screen-5. It was hypothesized all religiosity measures would be positively interrelated, the measure of problematic use would be negatively related to all religiosity measures, and that the RSAS-3 would be strongly predictive of absence of problematic substance use. After data filtering and imputation, bivariate correlations were calculated to establish convergent validity. Results: All relationships were in the predicted directions. Specifically, BIAC had the strongest relationship with the RSAS-3, r (440) = .906, p < .001, followed by intrinsic religiosity, r (440) =.814, p < .001, and extrinsic religiosity, r (440) = .694, p < .001. The RSAS-3 was the strongest predictor of problematic use among the religiosity measures, r (440) = −0.230, p <.001. Criterion-related validity of the RSAS-3 was supported using logistic regression to explore intrinsic religiosity, extrinsic religiosity, BIAC, and RSAS-3 as predictors of the presence/absence of problematic substance use. The RSAS-3 was the only significant predictor (OR = .858 [95% CI .757 − .973], p = .017). Conclusion: All results provide further evidence for the validity of the RSAS-3 as a very brief measure of religious commitment useful in health settings.

Examining the predictive validity of the Duolingo English Test: Evidence from a major UK university

The COVID-19 pandemic has changed the university admissions and proficiency testing landscape. One change has been the meteoric rise in use of the fully automated Duolingo English Test (DET) for university entrance purposes, offering test-takers a cheaper, shorter, accessible alternative. This rapid response study is the first to investigate the predictive value of DET test scores in relation to university students’ academic attainment, taking into account students’ degree level, academic discipline, and nationality. We also compared DET test-takers’ academic performance with that of students admitted using traditional proficiency tests. Credit-weighted first-year academic grades of 1881 DET test-takers (1389 postgraduate, 492 undergraduate) enrolled at a large, research-intensive London university in Autumn 2020 were positively associated with DET Overall scores for postgraduate students (adj. r = .195) but not undergraduate students (adj. r = −.112). This result was mirrored in correlational patterns for students admitted through IELTS ( n = 2651) and TOEFL iBT ( n = 436), contributing to criterion-related validity evidence. Students admitted with DET enjoyed lower academic success than the IELTS and TOEFL iBT test-takers, although sample characteristics may have shaped this finding. We discuss implications for establishing cut scores and harnessing test-takers’ academic language development through pre-sessional and in-sessional support.

Developing a Single-Item General Self-Efficacy Scale: An Initial Study

General self-efficacy represents the global sense of personal capability across various situations and tasks. The aim of the present study was to develop and validate a single-item general self-efficacy scale which balances practical demands and psychometric concerns. The psychometric properties of the proposed Single-Item General Self-Efficacy Scale (GSE-SI) were examined among 231 Singaporean adults. Results based on three statistical methods demonstrated good reliability (.594 .607 and .726, respectively, M = .642), as compared with the reliability scores from other single-item scales. It also showed satisfactory criterion-related validity evidence (i.e., correlation with a multiple-item general self-efficacy scale, r = .795). Validity evidence based on relationships with other constructs was supported by the correlations between the scores of general self-efficacy and the scores of six relevant constructs (i.e., positive correlations with life satisfaction and positive emotions, negative correlations with negative emotions, task and perceived stress, and illness symptoms). More importantly, the GSE-SI and multiple-item scale scores showed consistent correlation patterns with their relevant constructs. Both GSE-SI and multiple-item scale scores significantly discriminated between the three clusters in a similar pattern. The present results show that the GSE-SI is a reliable and valid measure of general self-efficacy and can be recommended in future research to complement the constraints of multiple-item scales.

Motives to play videogames across seven countries: Measurement invariance of the Videogaming Motives Questionnaire

Introduction. Gaming motives appear to be an important predictor of time spent gaming and disordered gaming. The Videogaming Motives Questionnaire (VMQ) has shown adequate psychometric properties to assess gaming motives among Spanish college students. However, the utility of this measure has not yet been explored in other cultures. This research aimed to examine the structure and measurement invariance of the VMQ across seven countries and gender groups, and to provide criterion-related validity evidence for VMQ scores. Method. College students who reported having played videogames in the last year (n = 5192; 59.07 % women) from the US, Canada, South Africa, Spain, Argentina, England, and Uruguay completed an online survey to measure time spent gaming, disordered gaming, and the VMQ. Results. Findings support a 24-item 8-intercorrelated factor model structure for the VMQ in the total sample. Our results also support configural, metric, and scalar invariance of the VMQ across gender groups and countries. Students from North America (US and Canada) scored higher on most gaming motives (except recreation and cognitive development) than students from the other countries. The correlations between VMQ and non-VMQ variables were similar across gender and countries, except in England where VMQ correlations with time spent gaming were stronger. Discussion. These results suggest that the VMQ is a useful measure for assessing gaming motives across young adults from different countries.

Resumes vs. application forms: Why the stubborn reliance on resumes?

The focus of this Perspective article is on the comparison of two of the most popular initial applicant screening methods: Resumes and application forms. The viewpoint offered is that application forms are superior to resumes during the initial applicant screening stage of selection. This viewpoint is supported in part based on criterion-related validity evidence that favors application forms over resumes. For example, the biographical data (biodata) inventory, which can contain similar questions to those used in application forms, is one of the most valid predictors of job performance (if empirically keyed), whereas job experience and years of education, which are often inferred from resumes and cover letters, are two of the least valid predictors of job performance (among commonly used screening criteria). In addition to validity evidence, making decisions based on application forms as opposed to resumes is likely to help organizations defend against claims of discriminatory hiring while enhancing their ability to hire in a more diverse, equitable, and inclusive manner. For example, applicant names on resumes can lead to screening bias against members of identifiable subgroups, whereas an applicant’s name can be easily and automatically hidden from decision-makers when reviewing application forms (particularly digital application forms). Despite these convincing arguments focused on applicant quality and diversity, a substantial research–practice gap regarding the use of resumes and cover letters remains.

Development of a Three-Dimensional Measure of the Calling Work Orientation: Assessing Craftsmanship, Kinship, and Serving

Although conceptualized as a multidimensional construct, the majority of measures of calling are unidimensional. In order to further expand the operationalization of calling, this study developed and validated a measure of the three dimensions of calling, namely, craftsmanship, kinship, and serving using three separate samples. Using a sample of 85 undergraduate students, the pilot study established content validity for the three dimensions of calling. Study 1 aimed to refine the developed measure while assessing its dimensionality across 379 participants from Amazon’s Mechanical Turk (MTurk). Furthermore, Study 2 provided evidence of construct and criterion-related validity, as well as test-retest reliability over a period of 3 months across 301 MTurk participants. Recommendations for future research utilizing the more focused dimensions of calling are discussed.

Exploring the application of a text-to-personality technique in job interviews

ABSTRACT This research’s purpose was to develop a valid and transparent text-to-personality technique to fit the requirements for personnel selection assessments. In this research we developed an advanced word-counting technique, the HEXACO text-to-personality (HTTP) technique, based on prior lexical personality research to assess personality from job interviews. To evaluate the technique’s construct and criterion-related validity we conducted three studies and analysed the transcripts of asynchronous (n = 102 and 72) and face-to-face (n = 155) interviews. These studies provided four key insights. First, the HTTP technique showed small to medium correlations with self-reported and interviewer-rated personality. Second, the technique showed mixed, but generally favourable, evidence for criterion-related validity. Third, the technique produced a more construct valid personality score when the interview questions activated the predicted personality trait. Fourth, the technique’s additional features (i.e., having weighted keywords and adjusting the keywords’ weight for adjacent quantifiers) did not improve its validity; unit-weighing was approximately equally effective. Altogether, the results show that a word-count text-analysis technique can discover traces of personality in interview transcripts. Still, significant improvements are needed before these types of automatically computed text-to-personality ratings can be used to replace or supplement interviewer ratings.

Existential Meaninglessness Scale: Scale Development and Psychometric Properties

Grounded in a tripartite existential meaninglessness model, the authors developed the 18-item Existential Meaninglessness Scale (EMS) to assess one’s concern and anxiety of existential meaninglessness. Across three samples, the EMS’s factor structure and evidence of convergent, criterion-related, and incremental validity, internal consistency, and test–retest reliability were examined. Exploratory factor analyses demonstrated three dimensions of the EMS: incomprehension, purposelessness, and insignificance. Confirmatory factor analyses revealed that a bifactor model was a better fit to the data than other models. The bifactor model provided evidence for a general factor and measurement invariance. Ancillary bifactor indices indicated EMS’s unidimensionality. Findings of bivariate correlations and hierarchical regression analyses provided evidence for different aspects of construct validity and internal consistency. Both the Concern and Anxiety measures of the EMS positively predicted depressive symptoms and suicide ideation above and beyond the effects of general existential meaninglessness, general feelings of anxiety, and presence of meaning in life. Based on the findings, the authors discuss future research directions on existential meaninglessness using the EMS.

Factor Structure and Criterion-Related Validity of the Air Force Officer Qualifying Test Form T for Pilot Applicants.

The Air Force Officer Qualifying Test (AFOQT) is the primary selection tool for officer applicants in the U.S. Air Force (USAF) for nearly seven decades. The AFOQT is revised and modified periodically, with rigorous equating and linking effort to ensure comparability and connectivity across forms. The most recent version of AFOQT is Form T that includes 10 cognitive ability and knowledge subtests. Despite the continuing validation effort of the AFOQT across forms, it was mostly directed to the general population of officer applicants, but not to any specific subpopulation. The current investigation reported three studies in an attempt to provide evidence for factor structure and criterion-related validity of AFOQT Form T for pilot applicants via four analytical approaches: meta-analysis, exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and structural equation modeling (SEM). The results suggested that AFOQT Form T data are best represented by a bifactor model with a general ability and four specific abilities, and that each latent construct has a distinct predictive utility for pilot performance criteria.