Evidence Of Predictive Validity Research Articles

Hours high as a proxy for marijuana use quantity in intensive longitudinal designs

BackgroundMeasuring marijuana use quantity in survey research is complicated due to wide variation in the types (e.g., flower, edibles) and potency of marijuana products and in the modes (e.g., smoking, dabbing) used to consume products. There is currently no gold standard marijuana use quantity measure for survey research. This study examined whether number of hours high can be used as a proxy for marijuana use quantity in survey research, particularly in intensive longitudinal designs. MethodsParticipants came from a community sample of young adults participating in a longitudinal study on simultaneous alcohol and marijuana use that used a longitudinal measurement-burst design in which participants completed surveys on up to 14 consecutive days in up to five bursts across nearly two calendar years. Those who reported using marijuana on at least one sampled day were included in present analyses (N = 379; Mage = 21.6; 50.7 % female). Hypotheses were tested using Poisson multilevel models and a logistic regression. ResultsWithin persons, mode-specific marijuana use quantity variables predicted same-day number of hours high indicating evidence of initial criterion validity. In turn, hours high predicted same-day negative marijuana-related consequences indicating evidence of proximal predictive validity. Between persons, participants’ average number of hours high was positively associated with their odds of possible cannabis use disorder following the last burst demonstrating distal predictive validity. ConclusionsNumber of hours high may be a parsimonious proxy for measuring marijuana use quantity (regardless of mode of use) in survey research, particularly in intensive longitudinal designs.

Office job sitting demand scale: Evidence of construct and predictive validity.

Drawing on sedentary behavior and Job Demands-Resources theory, we developed a new scale for assessing office job sitting demand, conceptualized as job demand, and validated it using two independent samples. In Study 1 (N = 252), we developed the items and tested the Office Job Sitting Demand scales' factor structure through exploratory factor analysis. In Study 2 (N = 248), we tested the scales' factorial validity through confirmatory factor analysis. Indicators of measurement invariance showed no differences between participants who engage in leisure-time sport and participants who do not. Moreover, we tested convergent, discriminant, and criterion validity. We found significant positive associations with exhaustion, work pressure, and musculoskeletal complaints. The construct was unrelated to proactive vitality management and relaxation. Our results illustrate the relation between this new job demand in the workplace with employee well-being. Based on our findings, this newly developed instrument is valid and reliable and can be used to assess office job sitting demand to prevent burnout and health complaints.

The Patient is Thriving! Current Issues, Recent Advances, and Future Directions in Creativity Assessment

ABSTRACT In 1998, Plucker and Runco provided an overview of creativity assessment, noting current issues (fluency confounds, generality vs. specificity), recent advances (predictive validity, implicit theories), and promising future directions (moving beyond divergent thinking measures, reliance on batteries of assessments, translation into practice). In the ensuing quarter century, the field experienced large growth in the quantity, breadth, and depth of assessment work, suggesting another analysis is timely. The purpose of this paper is to review the 1998 analysis and identify current issues, advances, and future directions for creativity measurement. Recent advances include growth in assessment quantity and quality and use of semantic distance as a scoring technique. Current issues include mismatches between current conceptions of creativity and those on which many measures are based, the need for psychometric quality standards, and a paucity of predictive validity evidence. The paper concludes with analysis of likely future directions, including use of machine learning to administer and score assessments and refinement of our conceptual frameworks for creativity assessment. Although the 1998 paper was written within an academic climate of harsh criticism of creativity measurement, the current climate is more positive, with reason for optimism about the continued growth of this important aspect of the field.

Mach-IV: Evidence of Structural, Discriminant and Predictive Validity with Argentinian Students

Machiavellians are defined as self-interested and manipulative people, who use others as mere means for their ends. The Mach-IV scale consists of 20 items measuring different aspects of Machiavellianism. In this study, we investigate the psychometric properties of the Mach-IV Scale with Argentinian university students. First, we translated and adapted the Mach-IV Scale to the target culture. Second, we evaluated the structure and internal consistency of the scale. Third, we examined the correlation between Mach-IV and Social Value Orientation (SVO), and between the Mach-IV scale and decisions and expectations in a Dictator Game (DG), both in its “give” and “take” versions. None of the structures evaluated showed good indexes of fit. We opted for the original one-factor structure but without items 19 and 20. Assuming such a structure, we observed negative correlations between the Mach-IV scale and SVO, and between the Mach-IV scale and cooperation in the DG. Materials, data, and scripts are available at https://bit.ly/2In6fgI.

Psychometric Properties of the Sibling Attachment Inventory in Mexican Young Adults.

The aim of this work was to adapt and validate the Sibling Attachment Inventory (SAI) in Mexican young adults and analyze its psychometric properties. Using an Internet-based survey, data were collected from 307 (64.5% female) Mexican young adults university students (aged 18–30). Exploratory and confirmatory factor analyses were performed to determine the factor structure of the revised Mexican version of the Sibling Attachment Inventory (SAI-RMx). Convergent and predictive validity were verified by carrying out correlations with the parent form of the Inventory of Parent and Peer Attachment (IPPA), the Lifespan Sibling Relationship Scale (LSRS), the Satisfaction With Life Scale (SWLS), and the Rosenberg Self-Esteem Scale (RSE). Results indicated that the SAI-RMx presents good levels of internal consistency and a monodimensional structure, also providing evidence for convergent, predictive, and construct validity. Furthermore, secure attachments were linked with high levels of self-esteem, as well as with high levels of life satisfaction. The SAI-RMx is expected to be a reliable instrument for measuring the global level and the three components of secure attachment between siblings in the young adult’s Mexican population.

Considering vocational training as selection criterion for medical students: evidence for predictive validity

Prior work experience in a relevant medical profession is an important admission criterion currently used at many German medical schools in addition to cognitive criteria. In other countries, work experience is often considered in later admission stages (e.g., interviews with pre-selected subgroups of applicants). However, evidence for its predictive validity for study success in addition to cognitive admission criteria is currently lacking. We therefore assessed whether completed vocational training in a relevant medical profession can predict study performance in the first two years of study in addition to cognitive admission criteria. Admission and study performance data of all currently enrolled medical students at two German medical schools (Göttingen and Heidelberg) beginning with the 2013/14 cohort were retrospectively analyzed. Cognitive admission criteria in our sample were GPA grades and a cognitive test (“Test für Medizinische Studiengänge”, TMS). We defined the study outcome parameter as the mean percentile rank over all performance data points over the first two years of study for each location, respectively. A multi-level model with varying intercepts by location, GPA, TMS, vocational training, and sex as predictors accounted for 14.5% of the variance in study outcome. A positive predictive association with study outcome was found for vocational training (ß = 0.33, p = .008) beyond GPA (ß = 0.38, p < .001) and TMS (ß = 0.26, p < .001). Our results support the use of prior vocational training as a selection criterion for medical studies potentially adding predictive validity to cognitive criteria.

Validation of the Comprehensive Inventory of Mindfulness Experiences (CHIME) in Portuguese Children.

ObjectivesDespite the increasing interest in mindfulness, there are few multifaceted mindfulness measures for children, mainly in Portuguese. Thus, this study developed the Portuguese version of the Comprehensive Inventory of Mindfulness Experiences (CHIME). Because this scale was initially created for adolescents, we also aimed to adapt it to children.MethodsAfter translating the original CHIME to Portuguese and adapting the wording for children, the instrument was administered to 223 9–10-year-olds, along with self-report measures of affect and quality of life, teacher-rated measures of inhibition, inattention, and emotional lability, and a performance-based attentional measure. We examined CHIME’s factorial validity, inspected facets’ reliability, and tested convergent, discriminant, and predictive validity.ResultsConfirmatory factor analyses (CFA) supported a 7-factor structure of the scale, after removing the accepting and non-judgmental attitude facet. Factor loadings (> .34) and reliability indices (> .54) were acceptable, though the average variance extracted was less than desirable. Correlations with external correlates provided support to CHIME’s convergent and divergent validity (rs range .13–.55, ps < .05). Finally, evidence of predictive validity was found with a regression analysis showing that external awareness (b = − .16) and openness to experience (b = − .17) predicted academic achievement.ConclusionsThis study provided preliminary validity and reliability evidence on the Portuguese CHIME. This instrument can provide relevant insights about the mindfulness facets and offer useful indications for the development of evidence-based interventions in childhood.

Chimpanzee Extraversion scores vary with epigenetic modification of dopamine receptor gene D2 (DRD2) and early rearing conditions

ABSTRACT Chimpanzees have consistent individual differences in behaviour, also referred to as personality. Similar to human personality structure, five dimensions are commonly found in chimpanzee studies that show evidence for convergent and predictive validity (Dominance, Openness, Extraversion, Agreeableness, and Reactivity/Undependability). These dimensions are to some extent heritable, indicating a genetic component that explains part of the variation in personality scores, but are also influenced by environmental factors, such as the early social rearing background of the individuals. In this study, we investigated the role of epigenetic modification of the dopamine receptor D2 gene (DRD2) as a potential mechanism underlying personality variation in 51 captive chimpanzees. We used previously collected personality trait rating data and determined levels of DRD2 CpG methylation in peripheral blood samples for these same individuals. Results showed that DRD2 methylation is most strongly associated with Extraversion, and that varying methylation levels at specific DRD2 sites are associated with changes in Extraversion in nursery-reared, but not mother-reared, individuals. These results highlight the role of dopaminergic signalling in chimpanzee personality, and indicate that environmental factors, such as social experiences early in life, can have long-lasting behavioural effects, potentially through modification of the epigenome. These findings add to the growing evidence demonstrating the importance of the experience-dependent methylome for the development of complex social traits like personality.

The Self-Appraisal of Masking Instrument

We explore mask-wearing behavior during the coronavirus pandemic using the Self-Appraisal of Masking Instrument (SAMI). We situate this survey-based instrument within a theory in which the decision to mask reflects social identity, an associated identity standard, and appraisals that generate feelings about oneself. Analyses of SAMI’s empirical properties reveal that masking-specific emotional reactions are distinct from emotional reports related to current events and politics (discriminant validity). We also uncover evidence of predictive validity: expressed feelings about masking predict future voting more than 6 months later. We recommend SAMI to researchers interested in studying mask resistance in an increasingly polarized political climate, and the intuition behind SAMI could prove useful in other research contexts in which health decisions reflect a conscious comparison to standards held by those who share an identity or will otherwise pass judgment.

Corporate governance performance ratings with machine learning

SummaryWe use machine learning with a cross‐sectional research design to predict governance controversies and to develop a measure of the governance component of the environmental, social, governance (ESG) metrics. Based on comprehensive governance data from 2,517 companies over a period of 10 years and investigating nine machine‐learning algorithms, we find that governance controversies can be predicted with high predictive performance. Our proposed governance rating methodology has two unique advantages compared with traditional ESG ratings: it rates companies' compliance with governance responsibilities and it has predictive validity. Our study demonstrates a solution to what is likely the greatest challenge for the finance industry today: how to assess a company's sustainability with validity and accuracy. Prior to this study, the ESG rating industry and the literature have not provided evidence that widely adopted governance ratings are valid. This study describes the only methodology for developing governance performance ratings based on companies' compliance with governance responsibilities and for which there is evidence of predictive validity.

Evidence for use of a healthy relationships assessment tool in the CHARISMA pilot study.

IntroductionThe CHARISMA intervention, nested within the MTN-025/HOPE vaginal ring trial in Johannesburg, South Africa, seeks to facilitate women’s use of HIV prevention products by promoting partner dialogue and mitigating intimate partner violence (IPV). We developed “HEART”, a lay counselor-administered relationship assessment tool, for the CHARISMA pilot. The five-scale tool assesses participants’ endorsement of Traditional Values (TV), her HIV Prevention Readiness (HPR) and levels of partner support (PS), abuse and control (PAC), and resistance to HIV prevention (PR), guiding decisions about which of three counselling modules to offer (partner communication/A; ring disclosure/B; and IPV prevention/C).MethodsWe correlated baseline scores on HEART subscales with a) independent measures of relationship stability, disclosure and IPV to assess construct validity, and b) with specific modules offered to determine how HEART was used in the pilot. We examined changes in HEART scores at three and six months. Finally, we ran separate growth models for each subscale to examine changes in scores, accounting for partnership changes and counseling module(s) received.ResultsBaseline HEART scores correlated as predicted among subscales and with other measures. Reliabilities for four subscales were 0.75 or higher. Women who disclosed study participation and ring use scored higher on PS and lower on PR. Women experiencing IPV scored lower on PS, and higher on PAC and PR. During the pilot, 82% of women received one and 17% received two or more modules; over half received the IPV module. Women with higher PAC and PR scores were more likely to receive the IPV than the communication or disclosure modules. Over time, the TV, PAC and PR scores decreased, and PS score increased. Receiving the IPV module was associated with a decreased PAC score.ConclusionsThese data offer preliminary evidence for HEART construct and predictive validity and support its further evaluation to guide implementation and monitor the impact of the CHARISMA intervention in a randomized controlled evaluation.

Association of cognitive performance with clinical staging in schizophrenia spectrum disorders: a prospective 6-year follow-up study

BackgroundClinical staging has been developed to capture the large heterogeneity in schizophrenia spectrum disorders. Including cognitive performance in the staging model may improve its clinical validity. Moreover, cognitive functioning could predict transition across stages. However, current evidence of the association between cognition and clinical staging is inconsistent. Therefore, we aim to assess whether cognitive parameters are associated with clinical stages in a large sample of patients with schizophrenia spectrum disorders and to identify cognitive markers at baseline that are associated with stage-transition at three and six-year follow-up. MethodsWe applied the staging model of Fusar-Poli et al. (2017) in 927 patients with non-affective psychotic disorders, assessed at baseline, and after three and six-year follow-up. Cognitive performance was assessed with a standard test battery. Generalized linear mixed models were used to analyze associations of cognitive performance with staging and stage-transition at follow-up. ResultsFindings showed that higher stages of illness were significantly associated with lower processing speed (F = 3.688, p = 0.025) and deficits in working memory (F = 6.365, p = 0.002) across assessments. No associations between cognitive parameters at baseline and stage-transition at three- and six-year follow-up were found. ConclusionWe conclude that processing speed and working memory were modestly associated with higher stages of illness in schizophrenia spectrum disorders, thereby slightly improving its clinical validity. However, associations were small and we found no evidence for predictive validity.

Psychometric Properties of the Behavior Assessment System for Children Student Observation System (BASC-3 SOS) with Young Children in Special Education.

Measuring classroom behavior among young children is important to guide assessment and intervention decisions, yet there is limited literature on appropriate direct observation tools for this purpose. This article describes the psychometric properties of the Behavior Assessment System for Children, Student Observation System (BASC-3 SOS) with 135 children ages 20 to 67 months (M = 35 months, 64% Latinx, 78% with an established developmental disability) and their teachers (N = 36) as part of a larger randomized control trial of a teacher training intervention. Inter-rater reliability on individual BASC-3 SOS behaviors ranged from poor to good. Correlations between BASC-3 SOS scores across time indicated low to moderate developmental test–retest reliability. Significant correlations between BASC-3 SOS scores and teacher ratings provided evidence for convergent, divergent, and predictive validity. Differences between BASC-3 SOS scores for children with versus without disabilities supported the tool’s discriminant validity. There were no significant pre- to post-treatment changes in BASC-3 SOS scores. Overall, results provide mixed evidence for the psychometric properties of the BASC-3 SOS when used with young, diverse children with and without disabilities. Implications for clinical and research purposes are discussed.Supplementary InformationThe online version contains supplementary material available at 10.1007/s10864-021-09458-x.

An investigation of emotional and evaluative implicit associations with police using four versions of the Implicit Association Test

ABSTRACT In two studies (total N = 829), we assessed civilian implicit associations with police using four modified Implicit Association Tests (IAT). Across studies and IATs, individuals harbored stronger negative implicit associations (associating police with fear/bad) than positive implicit associations (associating police with safety/good). The predictive validity of the implicit associations and magnitude of D scores varied across IAT. In Study 1, the IATs involving categorization of police-related (vs. everyday) symbols were most sensitive, but the versions involving categorization of police (vs. civilian) models provided more evidence for predictive validity. In Study 2, the IAT involving categorization of emotional words (safety/fear) was most sensitive, but the version involving categorization of evaluative words (good/bad) provided more evidence for predictive validity. In both studies, we also assessed individual differences (race, political affiliation) in implicit associations. The findings prompt the need to further examine the underlying cognitive components of civilian attitudes toward police and emphasize the importance of developing several IATs when assessing implicit attitudes.

Classroom Engagement Scale: Validation of a Teacher-Report Assessment Used to Scale in the Kindergarten Report Card of a Large Urban School District

ABSTRACT Research Findings: This study provides psychometric validation of the Classroom Engagement Scale (CES), a 14-item, teacher-report, formative assessment of behavioral engagement. It is currently used for all kindergarten children in a large, urban school district where it is embedded in the report card. Multi-level factor analysis revealed two distinct dimensions of the measure – academic engagement and social engagement – that comport with the educational research literature. Measurement invariance investigations provided initial support for the applicability of the CES across student subgroups and time. Finally, the two dimensions of the CES were found to be differentially predictive of important kindergarten and third-grade outcomes, providing evidence of concurrent and predictive validity. Practice or Policy: This study was designed in response to both the national call for routine monitoring of critical engagement competencies and the lack of validated teacher-report measures of these skills that can be used to inform ongoing instruction. This initial validation is an important step in institutionalizing practical, quality measurement within schools. Because of its strategic placement on the kindergarten report card, the CES provides an opportunity for teachers and families to share information about students’ engagement skills and coordinate confluent supports to foster these critical competencies.

Testing the Invariance of the Metacognitive Monitoring Test

Abstract Metacognition is predominantly measured by the self-report and think-aloud methods. This is problematic since they produce considerable both respondent and confirmatory biases, which implies damage to the measurement. The Metacognitive Monitoring Test (MMT) was created to evaluate metacognition through performance and eliminate the aforementioned biases. There is evidence of MMT convergent, divergent, structural, predictive and incremental validity. This article expands the validity studies about the MMT by analyzing the configural, metric and scalar invariance of MMT across sex, nationality, and educational level variables. The sample is composed of Brazilian and Honduran subjects, as well as 6st to 12st grades and higher education students. Results indicate configural, metric and scalar invariance for the sex variable, as well as configural invariance and metric and scalar partial invariance for nationality and educational level. It is concluded that the MMT allows comparing means of the latent variable measured in the analyzed groups.

Addressing the Need for Spanish Literacy Assessments Within the Context of Bilingual MTSS: Investigating the Technical Adequacy of ISIP Español for Grades 3–5

Building on theories of cross linguistic transfer, reading proficiency of Spanish-speaking Emerging Bilinguals may be enhanced by providing tiered instruction through multitiered systems of support. To realize the benefits of this systems-level framework, technically adequate universal screening assessments to support teachers’ decision making are needed to provide educators with information about students’ English and Spanish literacy skills. The purpose of this study is to investigate the technical adequacy of the ISIP Español universal screening assessment with Spanish and English outcome measures for Grades 3–5. Results suggested moderate concurrent and predictive validity evidence for the criterion assessments in both languages while classification accuracy results indicate comparable predictive utility of the Spanish screener for English and Spanish versions of the state achievement test but better predictive utility for a standardized achievement test of Spanish reading. Practical implications and directions for future research are discussed. Impact Statement Monitoring the literacy development of Spanish-speaking Emerging Bilinguals (EBs) in the context of bilingual programs should include assessments of Spanish literacy. However, there are few universal screening assessments of Spanish literacy for use within the context of Multitiered Systems of Support. This study aims to address this gap by exploring the validity and classification accuracy of the ISIP Español universal screening assessment of Spanish literacy for English and Spanish outcome measures. Supplemental data for this article can be accessed online at https://doi.org/10.1080/2372966X.2021.1986357.

Specificity and incremental predictive validity of implicit attitudes: studies of a race-based phenotype

Four studies involving 2552 White American participants were conducted to investigate bias based on the race-based phenotype of hair texture. Specifically, we probed the existence and magnitude of bias in favor of Eurocentric (straight) over Afrocentric (curly) hair and its specificity in predicting responses to a legal decision involving the phenotype. Study 1 revealed an implicit preference, measured by an Implicit Association Test (IAT), favoring Eurocentric over Afrocentric hair texture among White Americans. This effect was not reducible to a Black/White implicit race attitude nor to mere perceptual preference favoring straight over curly hair. In Study 2, the phenotype (hair) IAT significantly and uniquely predicted expressions of support in response to an actual legal case that involved discrimination on the basis of Afrocentric hair texture. Beyond replicating this result, Studies 3 and 4 (the latter preregistered) provided further, and even more stringent, evidence for incremental predictive validity: in both studies, the phenotype IAT was associated with support for a Black plaintiff above and beyond the effects of two parallel explicit scales and, additionally, a race attitude IAT. Overall, these studies support the idea that race bias may be uniquely detected by examining implicit attitudes elicited by group-based phenotypicality, such as hair texture. Moreover, the present results inform theoretical investigations of the correspondence principle in the context of implicit social cognition: they suggest that tailoring IATs to index specific aspects of an attitude object (e.g., by decomposition of phenotypes) can improve prediction of intergroup behavior.

Reassessing Passion and Perseverance as Dimensions of Individual Entrepreneurial Orientation: A Conceptual and Empirical Investigation into Theory and Measurement

Abstract Santos, G., C. S. Marques, and J. J. Ferreira. 2020. “Passion and Perseverance as Two New Dimensions of an Individual Entrepreneurial Orientation Scale.” Journal of Business Research 112: 190–9 proposed a five-dimensional conceptualization of individual entrepreneurial orientation (IEO), but more evidence is needed before authors can reliably apply their conceptualization and operationalization. We provide a theoretical critique of their conceptualization and continue their scale development process. Four of five dimensions converged with other scales of the same constructs, but their passion dimension did not. The factor structure and predictive validity evidence also suggested that passion should not be included within the scope of IEO, and we assert that Santos et al.’s conceptualization of IEO is better represented with a hierarchical structure. Proactiveness and perseverance as well as risk-taking and innovativeness load onto two separate second-order factors, and these two second-order factors load onto the broader factor of IEO. We suggest that the construct measured by Santos et al.’s passion dimension may instead be a mediator between IEO and entrepreneurial outcomes, and we provide further insights into the measurement of IEO.

Validation and psychometric properties of the Indonesian version of the Fear of Missing Out Scale in adolescents

BackgroundThis study’s main purpose was to examine the psychometric properties of FoMOs’ adaptation among the Indonesian adolescents’ population. The second aim was to investigate the concurrent validity of the Indonesian version to provide evidence for the validity. Also, FoMOs’ difference level between demographic variance analyses was performed.MethodThe study involved a cross-sectional online survey design with 638 Indonesian adolescents aged 16–24 (M = 19.08, SD = 14.70). FoMO was measured by a 16-item that has been modified from the original 10-item. Exploratory and confirmatory factor analyses were carried out to study its scores’ evidence of structural validity. Besides, to study its scores’ evidence of convergent, discriminant, and predictive validity concerning other variables such as stress, anxiety, and depression (Depression and Anxiety Stress Scale-21), and general health condition (General Health Questionnaire Scale-12), correlation analyses were conducted. To study the sensitivity, we assessed the effect of sociodemographic and social media use on the scale’s ability to identify the population’s risk to the FoMO by conducting analyses of variance. The Cronbach alpha values (α = .93) indicated that internal consistency of the scale was at an adequate level.ResultsExploratory factorial analyses revealed adequate adjustment for the new version of the scale showing the three factorial structures. Confirmatory factor analyses showed that the 12-item of Indonesian FoMO had a good fit (χ2/df = 289.324/51; goodness-of-fit index (GFI) = 0.928; RMSEA = 0.086; comparative fit index (CFI) = 0.915; normed fit index (NFI) = 0.899; parsimony normed fit index (PNFI) = .695; Tucker–Lewis index (TLI) = 0.890).ConclusionThis study has shown that the modified 12-item Fear of Missing Out Scale is a valid and reliable instrument for Indonesian adolescents. It showed that the Indonesian version of Fear of Missing Out Scale has adequate psychometric properties to measure Indonesian adolescents’ online behavior.