Evidence Of Predictive Validity Research Articles

Association of cognitive performance with clinical staging in schizophrenia spectrum disorders: a prospective 6-year follow-up study

BackgroundClinical staging has been developed to capture the large heterogeneity in schizophrenia spectrum disorders. Including cognitive performance in the staging model may improve its clinical validity. Moreover, cognitive functioning could predict transition across stages. However, current evidence of the association between cognition and clinical staging is inconsistent. Therefore, we aim to assess whether cognitive parameters are associated with clinical stages in a large sample of patients with schizophrenia spectrum disorders and to identify cognitive markers at baseline that are associated with stage-transition at three and six-year follow-up. MethodsWe applied the staging model of Fusar-Poli et al. (2017) in 927 patients with non-affective psychotic disorders, assessed at baseline, and after three and six-year follow-up. Cognitive performance was assessed with a standard test battery. Generalized linear mixed models were used to analyze associations of cognitive performance with staging and stage-transition at follow-up. ResultsFindings showed that higher stages of illness were significantly associated with lower processing speed (F = 3.688, p = 0.025) and deficits in working memory (F = 6.365, p = 0.002) across assessments. No associations between cognitive parameters at baseline and stage-transition at three- and six-year follow-up were found. ConclusionWe conclude that processing speed and working memory were modestly associated with higher stages of illness in schizophrenia spectrum disorders, thereby slightly improving its clinical validity. However, associations were small and we found no evidence for predictive validity.

Psychometric Properties of the Behavior Assessment System for Children Student Observation System (BASC-3 SOS) with Young Children in Special Education.

Measuring classroom behavior among young children is important to guide assessment and intervention decisions, yet there is limited literature on appropriate direct observation tools for this purpose. This article describes the psychometric properties of the Behavior Assessment System for Children, Student Observation System (BASC-3 SOS) with 135 children ages 20 to 67 months (M = 35 months, 64% Latinx, 78% with an established developmental disability) and their teachers (N = 36) as part of a larger randomized control trial of a teacher training intervention. Inter-rater reliability on individual BASC-3 SOS behaviors ranged from poor to good. Correlations between BASC-3 SOS scores across time indicated low to moderate developmental test–retest reliability. Significant correlations between BASC-3 SOS scores and teacher ratings provided evidence for convergent, divergent, and predictive validity. Differences between BASC-3 SOS scores for children with versus without disabilities supported the tool’s discriminant validity. There were no significant pre- to post-treatment changes in BASC-3 SOS scores. Overall, results provide mixed evidence for the psychometric properties of the BASC-3 SOS when used with young, diverse children with and without disabilities. Implications for clinical and research purposes are discussed.Supplementary InformationThe online version contains supplementary material available at 10.1007/s10864-021-09458-x.

An investigation of emotional and evaluative implicit associations with police using four versions of the Implicit Association Test

ABSTRACT In two studies (total N = 829), we assessed civilian implicit associations with police using four modified Implicit Association Tests (IAT). Across studies and IATs, individuals harbored stronger negative implicit associations (associating police with fear/bad) than positive implicit associations (associating police with safety/good). The predictive validity of the implicit associations and magnitude of D scores varied across IAT. In Study 1, the IATs involving categorization of police-related (vs. everyday) symbols were most sensitive, but the versions involving categorization of police (vs. civilian) models provided more evidence for predictive validity. In Study 2, the IAT involving categorization of emotional words (safety/fear) was most sensitive, but the version involving categorization of evaluative words (good/bad) provided more evidence for predictive validity. In both studies, we also assessed individual differences (race, political affiliation) in implicit associations. The findings prompt the need to further examine the underlying cognitive components of civilian attitudes toward police and emphasize the importance of developing several IATs when assessing implicit attitudes.

Classroom Engagement Scale: Validation of a Teacher-Report Assessment Used to Scale in the Kindergarten Report Card of a Large Urban School District

ABSTRACT Research Findings: This study provides psychometric validation of the Classroom Engagement Scale (CES), a 14-item, teacher-report, formative assessment of behavioral engagement. It is currently used for all kindergarten children in a large, urban school district where it is embedded in the report card. Multi-level factor analysis revealed two distinct dimensions of the measure – academic engagement and social engagement – that comport with the educational research literature. Measurement invariance investigations provided initial support for the applicability of the CES across student subgroups and time. Finally, the two dimensions of the CES were found to be differentially predictive of important kindergarten and third-grade outcomes, providing evidence of concurrent and predictive validity. Practice or Policy: This study was designed in response to both the national call for routine monitoring of critical engagement competencies and the lack of validated teacher-report measures of these skills that can be used to inform ongoing instruction. This initial validation is an important step in institutionalizing practical, quality measurement within schools. Because of its strategic placement on the kindergarten report card, the CES provides an opportunity for teachers and families to share information about students’ engagement skills and coordinate confluent supports to foster these critical competencies.

Testing the Invariance of the Metacognitive Monitoring Test

Abstract Metacognition is predominantly measured by the self-report and think-aloud methods. This is problematic since they produce considerable both respondent and confirmatory biases, which implies damage to the measurement. The Metacognitive Monitoring Test (MMT) was created to evaluate metacognition through performance and eliminate the aforementioned biases. There is evidence of MMT convergent, divergent, structural, predictive and incremental validity. This article expands the validity studies about the MMT by analyzing the configural, metric and scalar invariance of MMT across sex, nationality, and educational level variables. The sample is composed of Brazilian and Honduran subjects, as well as 6st to 12st grades and higher education students. Results indicate configural, metric and scalar invariance for the sex variable, as well as configural invariance and metric and scalar partial invariance for nationality and educational level. It is concluded that the MMT allows comparing means of the latent variable measured in the analyzed groups.

Addressing the Need for Spanish Literacy Assessments Within the Context of Bilingual MTSS: Investigating the Technical Adequacy of ISIP Español for Grades 3–5

Building on theories of cross linguistic transfer, reading proficiency of Spanish-speaking Emerging Bilinguals may be enhanced by providing tiered instruction through multitiered systems of support. To realize the benefits of this systems-level framework, technically adequate universal screening assessments to support teachers’ decision making are needed to provide educators with information about students’ English and Spanish literacy skills. The purpose of this study is to investigate the technical adequacy of the ISIP Español universal screening assessment with Spanish and English outcome measures for Grades 3–5. Results suggested moderate concurrent and predictive validity evidence for the criterion assessments in both languages while classification accuracy results indicate comparable predictive utility of the Spanish screener for English and Spanish versions of the state achievement test but better predictive utility for a standardized achievement test of Spanish reading. Practical implications and directions for future research are discussed. Impact Statement Monitoring the literacy development of Spanish-speaking Emerging Bilinguals (EBs) in the context of bilingual programs should include assessments of Spanish literacy. However, there are few universal screening assessments of Spanish literacy for use within the context of Multitiered Systems of Support. This study aims to address this gap by exploring the validity and classification accuracy of the ISIP Español universal screening assessment of Spanish literacy for English and Spanish outcome measures. Supplemental data for this article can be accessed online at https://doi.org/10.1080/2372966X.2021.1986357.

Specificity and incremental predictive validity of implicit attitudes: studies of a race-based phenotype

Four studies involving 2552 White American participants were conducted to investigate bias based on the race-based phenotype of hair texture. Specifically, we probed the existence and magnitude of bias in favor of Eurocentric (straight) over Afrocentric (curly) hair and its specificity in predicting responses to a legal decision involving the phenotype. Study 1 revealed an implicit preference, measured by an Implicit Association Test (IAT), favoring Eurocentric over Afrocentric hair texture among White Americans. This effect was not reducible to a Black/White implicit race attitude nor to mere perceptual preference favoring straight over curly hair. In Study 2, the phenotype (hair) IAT significantly and uniquely predicted expressions of support in response to an actual legal case that involved discrimination on the basis of Afrocentric hair texture. Beyond replicating this result, Studies 3 and 4 (the latter preregistered) provided further, and even more stringent, evidence for incremental predictive validity: in both studies, the phenotype IAT was associated with support for a Black plaintiff above and beyond the effects of two parallel explicit scales and, additionally, a race attitude IAT. Overall, these studies support the idea that race bias may be uniquely detected by examining implicit attitudes elicited by group-based phenotypicality, such as hair texture. Moreover, the present results inform theoretical investigations of the correspondence principle in the context of implicit social cognition: they suggest that tailoring IATs to index specific aspects of an attitude object (e.g., by decomposition of phenotypes) can improve prediction of intergroup behavior.

Reassessing Passion and Perseverance as Dimensions of Individual Entrepreneurial Orientation: A Conceptual and Empirical Investigation into Theory and Measurement

Abstract Santos, G., C. S. Marques, and J. J. Ferreira. 2020. “Passion and Perseverance as Two New Dimensions of an Individual Entrepreneurial Orientation Scale.” Journal of Business Research 112: 190–9 proposed a five-dimensional conceptualization of individual entrepreneurial orientation (IEO), but more evidence is needed before authors can reliably apply their conceptualization and operationalization. We provide a theoretical critique of their conceptualization and continue their scale development process. Four of five dimensions converged with other scales of the same constructs, but their passion dimension did not. The factor structure and predictive validity evidence also suggested that passion should not be included within the scope of IEO, and we assert that Santos et al.’s conceptualization of IEO is better represented with a hierarchical structure. Proactiveness and perseverance as well as risk-taking and innovativeness load onto two separate second-order factors, and these two second-order factors load onto the broader factor of IEO. We suggest that the construct measured by Santos et al.’s passion dimension may instead be a mediator between IEO and entrepreneurial outcomes, and we provide further insights into the measurement of IEO.

Validation and psychometric properties of the Indonesian version of the Fear of Missing Out Scale in adolescents

BackgroundThis study’s main purpose was to examine the psychometric properties of FoMOs’ adaptation among the Indonesian adolescents’ population. The second aim was to investigate the concurrent validity of the Indonesian version to provide evidence for the validity. Also, FoMOs’ difference level between demographic variance analyses was performed.MethodThe study involved a cross-sectional online survey design with 638 Indonesian adolescents aged 16–24 (M = 19.08, SD = 14.70). FoMO was measured by a 16-item that has been modified from the original 10-item. Exploratory and confirmatory factor analyses were carried out to study its scores’ evidence of structural validity. Besides, to study its scores’ evidence of convergent, discriminant, and predictive validity concerning other variables such as stress, anxiety, and depression (Depression and Anxiety Stress Scale-21), and general health condition (General Health Questionnaire Scale-12), correlation analyses were conducted. To study the sensitivity, we assessed the effect of sociodemographic and social media use on the scale’s ability to identify the population’s risk to the FoMO by conducting analyses of variance. The Cronbach alpha values (α = .93) indicated that internal consistency of the scale was at an adequate level.ResultsExploratory factorial analyses revealed adequate adjustment for the new version of the scale showing the three factorial structures. Confirmatory factor analyses showed that the 12-item of Indonesian FoMO had a good fit (χ2/df = 289.324/51; goodness-of-fit index (GFI) = 0.928; RMSEA = 0.086; comparative fit index (CFI) = 0.915; normed fit index (NFI) = 0.899; parsimony normed fit index (PNFI) = .695; Tucker–Lewis index (TLI) = 0.890).ConclusionThis study has shown that the modified 12-item Fear of Missing Out Scale is a valid and reliable instrument for Indonesian adolescents. It showed that the Indonesian version of Fear of Missing Out Scale has adequate psychometric properties to measure Indonesian adolescents’ online behavior.

The timed 25-foot walk in a large cohort of multiple sclerosis patients

Background: The timed 25-foot walk (T25FW) is a key clinical outcome measure in multiple sclerosis patient management and clinical research. Objectives: To evaluate T25FW performance and factors associated with its change in the Multiple Sclerosis Outcome Assessments Consortium (MSOAC) Placebo Database (n = 2465). Methods: We created confirmed disability progression (CDP) variables for T25FW and Expanded Disability Status Scale (EDSS) outcomes. We used intraclass correlation coefficients (ICCs) and Bland Altman plots to evaluate reliability. We evaluated T25FW changes and predictive validity using a mixed-effects model, survival analysis, and nested case–control analysis. Results: The mean baseline score for the T25FW in this study population was 9.2 seconds, median = 6.1 (standard deviation = 11.0, interquartile range (IQR) = 4.8, 9.0). The T25FW measure demonstrated excellent test–retest reliability (ICC = 0.98). Walk times increased with age, disability, disease type, and disease duration; relapses were not associated with an increase. Patients with T25FW progression had a faster time to EDSS-CDP compared to those without (hazards ratio (HR): 2.6; confidence interval (CI): 2.2, 3.1). Changes in the T25FW were more likely to precede changes in EDSS. Conclusion: This research confirms the association of the T25FW with disability and provides some evidence of predictive validity. Our findings support the continued use of the T25FW in clinical practice and clinical trials.

Early childhood teachers' mathematics-specific professional competence and its relation to the quality of learning support

Teacher expertise research uses different approaches, among them competence models. The model of mathematics-specific professional competence in this study differentiates action-related and reflective competence, which pertain to different types of professional demands according to dual processing theories and should positively impact teaching quality. The cross-sectional study with early childhood teachers investigates the structure of teacher competence and its relations to the quality of learning support in line with the notion of adaptive teaching when playing board games. The findings support a differentiation of competences according to the model (structural validity evidence). Expected relations between competences and the quality of the learning support are partially supported (predictive validity evidence). The study hence adds to research on teacher expertise by developing a specific understanding of competence in line with existing theories. Affordances of using competence models, as well as difficulties and open questions with relation to the emerging approach are discussed.

Development and validation of the parenting and child tooth brushing assessment questionnaire.

We report the development and validation of a comprehensive Parenting and Child Tooth brushing Assessment (PACTA) questionnaire. This cross-sectional study was conducted with community-recruited Australian parents (N=450) of children aged 2-8years. Parents completed an online survey including the newly developed PACTA (comprising four scales assessing children's tooth brushing behaviours, parenting strategies, attitudes and knowledge) and established scales assessing parenting behaviours, attitudes and self-efficacy. Exploratory factor analyses revealed two-factor structures for scales assessing child behaviour ('noncompliance' and 'avoidance behaviour'), parenting strategies ('effective strategies' and 'ineffective strategies') and attitudes ('emotional reactions' and 'lack of concern'), whereas the knowledge scale was unidimensional. Internal consistencies were satisfactory (>0.7) for all except the knowledge scale. There was good evidence of convergent and predictive validity. All subscales predicted children's parent-reported tooth brushing frequency; children were more likely to brush at least twice per day when parents reported fewer tooth brushing behavioural problems, using effective parenting strategies, and having better attitudes and knowledge. Poorer scores on the 'lack of concern' subscale of the attitudes scale were the strongest unique predictor of twice-daily brushing (OR, 95% CI: 1.51, 1.36-1.67). PACTA demonstrates satisfactory validity and reliability. Further research assessing sensitivity to change following intervention is warranted.

Invalid Claims About the Validity of Implicit Association Tests by Prisoners of the Implicit Social-Cognition Paradigm

In a prior publication, I used structural equation modeling of multimethod data to examine the construct validity of Implicit Association Tests. The results showed no evidence that IATs measure implicit constructs (e.g., implicit self-esteem, implicit racial bias). This critique of IATs elicited several responses by implicit social-cognition researchers, who tried to defend the validity and usefulness of IATs. I carefully examine these arguments and show that they lack validity. IAT proponents consistently ignore or misrepresent facts that challenge the validity of IATs as measures of individual differences in implicit cognitions. One response suggests that IATs can be useful even if they merely measure the same constructs as self-report measures, but I find no support for the claim that IATs have practically significant incremental predictive validity. In conclusions, IATs are widely used without psychometric evidence of construct or predictive validity.

Measures of organizational culture, organizational climate, and implementation climate in behavioral health: A systematic review.

Organizational culture, organizational climate, and implementation climate are key organizational constructs that influence the implementation of evidence-based practices. However, there has been little systematic investigation of the availability of psychometrically strong measures that can be used to assess these constructs in behavioral health. This systematic review identified and assessed the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs as defined by the Consolidated Framework for Implementation Research (CFIR) and Ehrhart and colleagues. Data collection involved search string generation, title and abstract screening, full-text review, construct assignment, and citation searches for all known empirical uses. Data relevant to nine psychometric criteria from the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) were extracted: internal consistency, convergent validity, discriminant validity, known-groups validity, predictive validity, concurrent validity, structural validity, responsiveness, and norms. Extracted data for each criterion were rated on a scale from -1 ("poor") to 4 ("excellent"), and each measure was assigned a total score (highest possible score = 36) that formed the basis for head-to-head comparisons of measures for each focal construct. We identified full measures or relevant subscales of broader measures for organizational culture (n = 21), organizational climate (n = 36), implementation climate (n = 2), tension for change (n = 2), compatibility (n = 6), relative priority (n = 2), organizational incentives and rewards (n = 3), goals and feedback (n = 3), and learning climate (n = 2). Psychometric evidence was most frequently available for internal consistency and norms. Information about other psychometric properties was less available. Median ratings for psychometric properties across categories of measures ranged from "poor" to "good." There was limited evidence of responsiveness or predictive validity. While several promising measures were identified, the overall state of measurement related to these constructs is poor. To enhance understanding of how these constructs influence implementation research and practice, measures that are sensitive to change and predictive of key implementation and clinical outcomes are required. There is a need for further testing of the most promising measures, and ample opportunity to develop additional psychometrically strong measures of these important constructs. Organizational culture, organizational climate, and implementation climate can play a critical role in facilitating or impeding the successful implementation and sustainment of evidence-based practices. Advancing our understanding of how these contextual factors independently or collectively influence implementation and clinical outcomes requires measures that are reliable and valid. Previous systematic reviews identified measures of organizational factors that influence implementation, but none focused explicitly on behavioral health; focused solely on organizational culture, organizational climate, and implementation climate; or assessed the evidence base of all known uses of a measure within a given area, such as behavioral health-focused implementation efforts. The purpose of this study was to identify and assess the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs that have been used in behavioral health-focused implementation research. We identified 21 measures of organizational culture, 36 measures of organizational climate, 2 measures of implementation climate, 2 measures of tension for change, 6 measures of compatibility, 2 measures of relative priority, 3 measures of organizational incentives and rewards, 3 measures of goals and feedback, and 2 measures of learning climate. Some promising measures were identified; however, the overall state of measurement across these constructs is poor. This review highlights specific areas for improvement and suggests the need to rigorously evaluate existing measures and develop new measures.

Who Knows Best What the Next Year Will Hold for You? The Validity of Direct and Personality–based Predictions of Future Life Experiences across Different Perceivers

This study explored the validity of person judgements by targets and their acquaintances (‘informants’) in longitudinally predicting a broad range of psychologically meaningful life experiences. Judgements were gathered from four sources (targets, N = 189; and three types of informants, N = 1352), and their relative predictive validity was compared for three types of judgement: direct predictions of future life experiences (e.g. number of new friendships), broad (Big Five) domains (e.g. extraversion), and narrower personality nuances (e.g. sociable). Approximately 1 year later, the targets’ actual life experiences were retrospectively assessed by the targets, and by informants nominated by the targets (TNI). Overall, we found evidence for predictive validity across predictor sources and types. Direct predictions by targets were by far the most valid, followed by TNI. Personality–based predictions by targets and TNI had substantial but lower validity. Domain–based predictions were less valid than nuance–based predictions. Overall, informants with lower ‘liking’ and ‘knowing’ towards targets made less valid predictions. Person–centred multilevel analyses showed both considerable validity of direct predictions (which increased with knowing) and positivity bias (which increased with liking). Taken together, given the relatively high methodological rigour of the study, these results provide an especially realistic picture of the rather moderate predictive power of person judgements regarding future life experiences and corroborate the common practice of obtaining such judgements from targets and their close acquaintances. © 2020 The Authors. European Journal of Personality published by John Wiley & Sons Ltd on behalf of European Association of Personality Psychology

A Comprehensive Review of Instruments Measuring Attitudes Toward Science

The development of attitudes toward science instruments has recently emerged in science education research. However, a comprehensive review of their psychometric properties, using currently accepted assessment standards, has not yet been completed. Consequently, this review discusses the validity and reliability of 18 measures published between 2005 and 2019 in leading science education journals. Findings showed that construct validity and internal consistency reliability was reported for all instruments; however, evidence for predictive validity and temporal stability reliability was rather scarce, which could limit their use in intervention and correlational type of studies. Similarly, content validity was found to be underreported. Consequently, the relevance, comprehensiveness, and comprehensibility of the items in some instruments are currently unknown and yet to be established in future studies. Finally, there is a gap in the literature regarding instruments that can be used across different countries and scientific disciplines, which could restrict accumulative and comparative results worldwide. Since the use of valid and reliable measurement instruments is a crucial aspect of educational research, the findings of this study could be useful in assisting researchers and practitioners in selecting the most appropriate measure for different research designs.

Reliability and Validity Evidence of the STarT-Lower Extremity Screening Tool for Patients With Lower Extremity Fracture: A Prospective Study

ObjectivesTo determine whether a modified version of the STarT Back Screening Tool in its current structure has adequate properties for use in patients with lower extremity fracture. DesignSingle-center, prospective study. SettingLevel I trauma center. ParticipantsPatients with lower extremity fracture without a history of chronic pain (N=114), with 93% follow-up. InterventionsNot applicable. Main Outcome MeasuresSix weeks after surgical fixation, individuals completed the Subgroups for Targeted Treatment of Lower Extremity Screening Tool (STarT-LE). A subsample completed the STarT-LE again 1 week later. The following questionnaires were completed at 6 weeks and 6 months: Pain Catastrophizing Scale, Tampa Scale of Kinesiophobia, Brief Pain Inventory pain intensity subscale, and PROMIS Depression and Pain Interference computer adaptive testing modules. Reliability was evaluated using intraclass correlation coefficients (ICC) and Cronbach’s alpha (α). Convergent validity evidence was measured concurrently using the Spearman ρ correlation between the 6-week STarT-LE and established questionnaires. Predictive validity evidence was evaluated by area under the curve analysis (AUC) using the 6-week STarT-LE total and psychosocial scores and 6-month criterion physical and psychosocial reference standards. ResultsThe STarT-LE has good test-retest reliability (ICC, 0.85; 95% confidence interval, 0.78-0.91) and acceptable internal consistency (α=0.74). The convergent validity evidence was fair to moderate (ρ, 0.53-0.68; P<.001) and the predictive validity evidence was acceptable to excellent (AUC, 0.73-0.84). ConclusionsThe STarT-LE has adequate properties for use in patients with lower extremity fracture. Future larger scale studies are needed to validate risk cutoffs.

Resilience Resources Scale: A brief resilience measure validated with undergraduate students

Objective This paper presents a theory-based brief resilience scale, the Resilience Resources Scale (RRS), and evidence for its factor structure, reliability, and validity in two studies of undergraduate students. Participants: Study 1 sampled 295 students and Study 2 sampled 244 students. Methods: Study 1 participants completed the RRS and other measures online at one of two time points eight weeks apart (n = 193), or at both time points (n = 102). Study 2 participants completed the RRS and other measures online on a single occasion. Results: Factor analyses provided evidence for a one-factor model. Results indicated high internal consistency and strong test-retest reliability. Evidence of concurrent and predictive validity is presented. Conclusions: The RRS measures resilience resources known to be protective of physical and mental health. This brief scale has sound psychometric properties in these initial studies of undergraduate students. We offer possible directions for use of the RRS in this and other populations.

Protective Factors for Reducing Juvenile Reoffending: An Examination of Incremental and Differential Predictive Validity

Juvenile court practitioners and researchers have recently gained interest in evaluating internal and external strengths, or protective factors. Some scholars assert that incorporating measures of strengths into the risk assessment process can increase the accuracy of identifying odds of recidivating. Relatively few juvenile risk assessment validation studies have evaluated the predictive validity of strengths. This study employed a diverse sample ( N = 278) of juveniles under supervision in a Midwestern court. The Protective Factors for Reducing Juvenile Reoffending (PFRJR) significantly predicted recidivism for the total sample, males, and White youth. There was no evidence of differential predictive validity across gender; however, strengths predicted differently across race/ethnicity. Strengths did not increase the amount of variance explained in recidivism after accounting for the variance explained by risk factor scores. Findings contribute to the paucity of validation studies that investigated the differential and incremental predictive validity of strengths.

Self-undermining behavior at work: Evidence of construct and predictive validity.

Self-undermining behavior at work: Evidence of construct and predictive validity.