Evidence Of Predictive Validity Research Articles

Validating the New Primary Care Measure in the Medical Expenditure Panel Survey.

The advancement of primary care research requires reliable and validated measures that capture primary care processes embedded within nationally representative datasets. The objective of this study was to assess the validity of a newly developed measure of primary care processes [Medical Expenditure Panel Survey (MEPS)-PC] with preliminary evidence of moderate to excellent reliability. A retrospective cohort study of community-dwelling adults with history of office-based provider visit/s using the MEPS (2013-2014). The 3 MEPS-PC subscales (Relationship, Comprehensiveness, and Health Promotion) were tested for construct validity against known measures of primary care: Usual Source of Care, Known Provider, and Family-Usual Source of Care. Concurrent and predictive logistic regression analyses were calculated and compared with a priori hypotheses for direction and strength of association. For concurrent validity, all odds ratio estimates conformed with hypotheses, with 91% displaying statistical significance. For predictive validity, all estimates were in the direction of hypotheses, with 92% displaying statistically significant results. Although Relationship and Health Promotion subscales conformed uniformly with hypotheses, the Comprehensiveness subscale yielded significant results in 60% of bivariate odds ratio estimates (P<0.05). The MEPS-PC composite measures display modest to strong preliminary evidence of concurrent and predictive validity relative to known indicators of primary care. The MEPS-PC composite measures display preliminary evidence of concurrent and predictive construct validity, and it may be useful to researchers investigating primary care processes and complexities in the health care environment.

Evaluating a short-form Five Facet Mindfulness Questionnaire in adolescents: Evidence for a four-factor structure and invariance by time, age, and gender

Little is known about whether a widely used mindfulness measure in adults—the Five Facet Mindfulness Questionnaire (FFMQ)—is also reliable and valid in adolescents. The current study evaluated the psychometric properties of a 20-item short-form FFMQ in a sample of 599 high school students (Mage = 16.3 years; 49% female) living in the U.S. Students completed the FFMQ and a battery of self-report questionnaires assessing aspects of psychological well-being and social skills 3 times over the course of one academic year. Confirmatory factor analysis indicated that a modified four-factor hierarchical model (excluding the Observe subscale and 1 item from the Describe subscale) best fit the data. This four-factor, hierarchical FFMQ demonstrated evidence of measurement invariance across time, gender, and grade level. Reliabilities for the FFMQ total score and its subscales ranged from .61 to .88. The FFMQ total score, and its subscales (excluding Observe), demonstrated evidence of convergent (e.g., with self-compassion) and discriminant (e.g., with social perspective taking skills) validity. Finally, the FFMQ total score and Act with Awareness, Nonjudgment, and Nonreactivity subscales demonstrated evidence of incremental predictive validity for cross-time changes in psychological well-being outcomes (e.g., perceived stress). Overall, results provide preliminary support for the reliability and validity of a short-form FFMQ for use in high-school-age adolescents.

Conceptualization and Assessment of Adolescents’ Engagement and Disengagement in School

Abstract. Increasing school engagement is critical for improving academic achievement and reducing dropout rates. In order to increase student engagement and identify those students who are most disengaged from school, we need to conceptualize and measure student engagement appropriately. This study used a mixed method sequential exploratory design to develop and validate a student survey measure of school engagement that reflects a multidimensional conceptualization of engagement. Psychometric tests were conducted with a large racially and socioeconomically diverse sample of 5th–12th graders in the United States ( N = 3,632). Findings demonstrated that a bifactor multidimensional model fit the data appropriately and provided evidence of measurement invariance, construct, and predictive validity. Results provided a psychometrically sound foundation for capturing the behavioral, emotional, cognitive, and social aspects of student engagement and disengagement in school.

“Arrogant, ungrateful, anti-American degenerates”: Development and initial validation of the Attitudes Toward Athlete Activism Questionnaire (ATAAQ)

Despite its increasing prevalence in sports at all levels of competition, there is a startling paucity of research on athlete activism and its broader impact on social and political change. Furthermore, athletes who engage in sport-based political protest – many of whom hold one or more traditionally marginalized identities – often face vicious public backlash. In order to support the role of sport psychologists and other mental health professionals in responding to this issue, there is a need to deepen our understanding of the many views and opinions people hold toward athlete activism, whether they be athletes, coaches, sport psychologists, or the broader public. To stimulate further research on this topic, we developed the Attitudes Toward Athlete Activism Questionnaire (ATAAQ) as a measure of reactions toward athlete political behavior in sport across a range of competitive levels (e.g., professional, collegiate, high school). An initial pool of 40 items was administered to a total sample of 713 participants, recruited from social media and online sport-related forums. Exploratory and confirmatory factor analyses yielded a 19-item scale comprising five-factors: affective reactions (five items), perceived conflict with team (four items), athlete role (three items), lack of political qualifications/credibility (three items), and desired consequences or punishments (four items). Internal consistency estimates for the ATAAQ factors were 0.88 and above. We established initial evidence for convergent, criterion-related, and predictive validity, as ATAAQ scores were correlated in expected directions with scores on personal activism orientation, belief in a just world, and attitudes toward social justice. Strengths, limitations, and implications for research and practice are discussed.

Development and Initial Validation of Cultural Competence Inventory–Preservice Teachers

Despite the critical role of emotions in multicultural teacher education, no attempt has been made to develop an instrument including affect as a dimension in measuring cultural competence for preservice teachers. To bridge this gap, the present three-study research used three distinct samples of 456 preservice teachers to develop and estimate the reliability of scores and validity of inferences for Cultural Competence Inventory–Preservice Teachers (CCI-PT), an 18-item instrument including three subscales: emotional cultural competence (ECC), behavioral cultural competence (BCC), and cognitive cultural competence (CCC). The three-factor model was supported using exploratory factor analysis ( N = 203) in Study 1 and confirmatory factor analyses ( N = 199) in Study 2. Correlation results provided preliminary evidence for the convergent and discriminant validity of CCI-PT in Study 1, and regression results in Study 3 ( N = 113) provided initial evidence of predictive validity of inference for the instrument. Further, the scores of cultural competence assessed through three independent studies also provided initial evidence of the reliability of CCI-PT.

Feasibility of using a smartphone app to assess early signs, basic symptoms and psychotic symptoms over six months: A preliminary report

BackgroundPsychosis relapses are common, have profound adverse consequences for patients, and are costly to health services. ‘Early signs’ have been used to predict relapse, in the hope of prevention or mitigation, with moderate sensitivity and specificity. We investigated the feasibility and validity of adding ‘basic symptoms’ to conventional early signs and monitoring these using a smartphone app. MethodsIndividuals (n = 18) experiencing a relapse within the past year were asked to use a smartphone app (‘ExPRESS’) weekly for six months to report early signs, basic symptoms and psychotic symptoms. Above-threshold increases in app-reported psychotic symptoms prompted a telephone interview (PANSS positive items) to assess relapse. ResultsParticipants completed 65% app assessments and 58% telephone interviews. App items showed high concurrent validity with researcher-rated psychotic symptoms and basic symptoms over six months. There was excellent agreement between telephone call and face-to-face assessed psychotic symptoms. The primary relapse definition, based on telephone assessment and casenotes, compared well with a casenote-only definition but had better specificity. Mixed-effects models provided preliminary evidence of concurrent and predictive validity: early signs and basic symptoms were associated with most app-assessed psychotic symptom variables the same week and with a number of psychotic symptoms variables three weeks later; adding basic symptoms to early signs improved model fit in most of these cases. ConclusionsThis is the first study to test a smartphone app for monitoring early signs and basic symptoms as putative relapse predictors. It demonstrates that weekly app-based monitoring is feasible, valid and acceptable over six months.

Examining the Stability, Accuracy, and Predictive Validity of Behavioral–Emotional Screening Scores Across Time to Inform Repeated Screening Procedures

Preschool has been recognized as an optimal time to provide behavioral and emotional supports to children; however, there is a need to examine the stability, accuracy, and predictive validity of universal emotional and behavioral screening scores in early childhood to inform procedures for repeated screenings across time. The present study included a sample of 1,014 students assessed with the Behavioral and Emotional Screening System Teacher Preschool screening form and comprehensive, omnibus teacher rating scales in preschool through first grade. Results indicated that scores from the Behavioral and Emotional Screening System Teacher screener were relatively stable within the school year and showed evidence of predictive validity of clinical problems in early childhood. To help identify who may benefit from a repeated screening later in the academic year, receiver operating characteristic curve analyses were conducted. A cut score of 51 emerged as a useful cut point to minimize expended resources and maximize sensitivity and specificity in finding new cases of risk. Implications for practitioners are discussed, including those related to the efficiency, timing, and frequency of screening in early childhood.

Reliability and Validity of the Preschool Science Observation Measure

ABSTRACTResearch Findings: Although interest in preschool science is growing, there are few observational measures of science teaching capable of advancing research or supporting professional growth among teachers. To address this gap, the current article presents the development and validation of the Preschool Science Observation Measure (PSOM), which assesses science teaching quality and the content of instruction for structured science lessons in preschool. Within the context of a randomized field trial of a science curriculum, 115 teachers submitted videos of science lessons and 1,003 children ages 4 and 5 years old were assessed on their science skills in the fall and spring. Results indicated adequate range and reliability of PSOM scores. PSOM scores showed moderate positive associations with an observation of teacher–child interactions. Limited positive associations with children’s science outcomes indicated some evidence of predictive validity. Descriptive results from control classrooms indicated wide variability in the content and quality of science teaching in typical preschool classrooms. Practice or Policy: Results support further development and use of the PSOM to evaluate science teaching quality and highlight the need for more research on science in preschool classrooms.

Hospital-Associated Complications of Older People: A Proposed Multicomponent Outcome for Acute Care.

To propose a new multicomponent measure of hospital-associated complications of older people (HAC-OP) and evaluate its validity in a large hospital sample. Observational study using baseline (pre-intervention) data from the Collaboration for Hospitalised Elders Reducing the Impact of Stays in Hospital cluster randomized controlled trial. Acute medical and surgical wards in 4 hospitals in Queensland, Australia. Individuals aged 65 and older (mean age 76, 48% female) with a hospital stay of 72 hours or longer (N=434). We developed a multicomponent measure including 5 well-recognized hospital-associated complications of older people: hospital-associated delirium, functional decline, incontinence, falls, and pressure injuries. To evaluate construct validity, we examined associations with common risk factors (aged ≥75, functional impairment, cognitive impairment, history of falls). To evaluate predictive validity, we examined the association between length of stay, facility discharge, and 6-month mortality and any HAC-OP and total number of HAC-OP. Overall, 192 (44%) participants had 1 or more HAC-OP during their admission. Any HAC-OP was strongly associated with the proposed shared risk factors, and there was a strong and graded association between HAC-OP and length of stay (9.1±7.4 days for any HAC-OP vs 6.8 ±4.1 days with none, p < .001), facility discharge (59/192 (31%) vs 27/242 (11%), p < .001) and 6-month mortality (26/192 (14%) vs 17/242 (7%), p = .02). This study provides evidence of construct and predictive validity of the proposed measure of HAC-OP as a potential outcome measure for research investigating and improving hospital care of older people. J Am Geriatr Soc 67:352-356, 2019.

Psychometric properties of gross motor assessment tools for children: a systematic review

ObjectiveGross motor assessment tools have a critical role in identifying, diagnosing and evaluating motor difficulties in childhood. The objective of this review was to systematically evaluate the psychometric properties and...

Dimensionality, reliability, and predictive evidence of validity for three help-seeking intention instruments: ISCI, GHSQ, and MHSIS.

People's intention to seek help from a mental health professional is thought to be the proximal cause of help-seeking behavior and thus is a dependent variable frequently measured by help-seeking researchers. Using a research design that accounted for actual future help-seeking behavior, the present study documented the dimensionality, internal consistency, and predictive evidence of validity of 3 intention instruments: the Intentions to Seek Counseling Inventory (ISCI), General Help Seeking Questionnaire (GHSQ), and Mental Help-Seeking Intention Scale (MHSIS). The sample was composed of 405 community-dwelling adults who self-identified as currently experiencing a mental health concern. Results provided support for the ISCI's 3-factor structure and the internal consistency of its 3 subscale scores. In contrast, the GHSQ did not demonstrate clear evidence of adequate measurement model fit or internal consistency in the present sample. Results also tentatively suggested that the 3-item MHSIS is a unidimensional instrument that produces an internally consistent total score with appropriate construct replicability. The ability of these instruments to predict who would seek help from a mental health professional in the next 3 months was also examined. The MHSIS demonstrated the strongest evidence of predictive validity (about 70% of participants were correctly classified), followed by the GHSQ and ISCI. (PsycINFO Database Record

Cross-Cultural Validation of the Turkish Version of the Decent Work Scale

This study examined the psychometric properties of the Turkish version of the Decent Work Scale (DWS), a recently developed measure that assesses the psychological experience of the quality of one’s work life. The proposed five-factor structure was verified with a sample of 326 Turkish working adults. Consistent with previous research, a five-factor bifactor model showed best fit to the data. The results of multigroup confirmatory factor analysis showed that the structure of the instrument was invariant across gender, income, and social class groups. Convergent and discriminant validity were supported by positive correlations with person–organization fit, job satisfaction, life satisfaction, and work meaning and by negative correlations with withdrawal intentions. Evidence of predictive validity was obtained by regressing the five subscales on four outcome measures. These findings suggest that the Turkish version of the DWS can be used for assessing decent work among Turkish working adults and cross-cultural psychological research.

Predicting patterns of change and stability in student performance across a medical degree.

Evidence of predictive validity is essential for making robust selection decisions in high-stakes contexts such as medical student selection. Currently available evidence is limited to the prediction of academic performance at single points in time with little understanding of the factors that might undermine the predictive validity of tests of academic and non-academic qualities considered important for success. This study addressed these issues by predicting students' changing performance across a medical degree and assessing whether factors outside an institution's control (such as the uptake of commercial coaching) impact validity. Three cohorts of students (n = 301) enrolled in an undergraduate medical degree from 2007-2013 were used to identify trajectories of student academic performance using growth mixture modelling. Multinomial logistic regression assessed whether past academic performance, a test of cognitive ability and a multiple mini-interview could predict a student's likely trajectory and whether this predictive validity was different for those who undertook commercial coaching compared with those who didn't. Among the medical students who successfully graduated (n = 268), four unique trajectories of academic performance were identified. In three trajectories, performance changed at the time when learning became more self-directed and focused on clinical specialties. Scores on all selection tests, with the exception of a test of abstract reasoning, significantly affected the odds of following a trajectory that was consistently below average. However, selection tests could not distinguish those whose performance improved across time from those whose performance declined after an average start. Commercial coaching increased the odds of being among the below-average performers, but did not alter the predictive validity of the selection tests. Identifying distinct groups of students has important implications for selection, but also for educating medical students. Commercial coaching may result in selecting students who are less suited for coping with the rigours of medical studies.

A systematic review of validated sinus surgery simulators.

Simulation provides a safe and effective opportunity to develop surgical skills. A variety of endoscopic sinus surgery (ESS) simulators has been described in the literature. Validation of these simulators allows for effective utilisation in training. To conduct a systematic review of the published literature to analyse the evidence for validated ESS simulation. Pubmed, Embase, Cochrane and Cinahl were searched from inception of the databases to 11 January 2017. Twelve thousand five hundred and sixteen articles were retrieved of which 10 112 were screened following the removal of duplicates. Thirty-eight full-text articles were reviewed after meeting search criteria. Evidence of face, content, construct, discriminant and predictive validity was extracted. Twenty articles were included in the analysis describing 12 ESS simulators. Eleven of these simulators had undergone validation: 3 virtual reality, 7 physical bench models and 1 cadaveric simulator. Seven of the simulators were shown to have face validity, 7 had construct validity and 1 had predictive validity. None of the simulators demonstrated discriminate validity. This systematic review demonstrates that a number of ESS simulators have been comprehensively validated. Many of the validation processes, however, lack standardisation in outcome reporting, thus limiting a meta-analysis comparison between simulators.

Self-perceived Coparenting of Nonresident Fathers: Scale Development and Validation.

This study reports on the development and validation of the Fatherhood Research and Practice Network coparenting perceptions scale for nonresident fathers. Although other measures of coparenting have been developed, this is the first measure developed specifically for low-income, nonresident fathers. Focus groups were conducted to determine various aspects of coparenting. Based on this, a scale was created and administered to 542 nonresident fathers. Participants also responded to items used to examine convergent and predictive validity (i.e., parental responsibility, contact with the mother, father self-efficacy and satisfaction, child behavior problems, and contact and engagement with the child). Factor analyses and reliability tests revealed three distinct and reliable perceived coparenting factors: undermining, alliance, and gatekeeping. Validity tests suggest substantial overlap between the undermining and alliance factors, though undermining was uniquely related to child behavior problems. The alliance and gatekeeping factors showed strong convergent validity and evidence for predictive validity. Taken together, results suggest this relatively short measure (11 items) taps into three coparenting dimensions significantly predictive of aspects of individual and family life.

The Predictive Validity of a Text-Based Situational Judgment Test in Undergraduate Medical and Dental School Admissions.

Situational judgment tests (SJTs) can be used to assess the nonacademic attributes necessary for medical and dental trainees to become successful practitioners. Evidence for SJTs' predictive validity, however, relates predominantly to selection in postgraduate settings or using video-based SJTs at the undergraduate level; it may not be directly transferable to text-based SJTs in undergraduate medical and dental school selection. This preliminary study aimed to address these gaps by assessing the validity of the UK Clinical Aptitude Test (UKCAT) text-based SJT. Study participants were 218 first-year medical and dental students from four UK undergraduate schools who completed the first UKCAT text-based SJT in 2013. Outcome measures were educational supervisor ratings of in-role performance in problem-based learning tutorial sessions-mean rating across the three domains measured by the SJT (integrity, perspective taking, and team involvement) and an overall judgment of performance-collected in 2015. There were significant correlations between SJT scores and both mean supervisor ratings (uncorrected r = 0.24, P < .001; corrected r = 0.34) and overall judgments (uncorrected rs = 0.16, P < .05; corrected rs = 0.20). SJT scores predicted 6% of variance in mean supervisor ratings across the three nonacademic domains. The results provide evidence that a well-designed text-based SJT can be appropriately integrated, and add value to, the selection process for undergraduate medical and dental school. More evidence is needed regarding the longitudinal predictive validity of SJTs throughout medical and dental training pathways, with appropriate outcome criteria.

Methods to Assess Adverse Childhood Experiences of Children and Families: Toward Approaches to Promote Child Well-being in Policy and Practice

BackgroundAdvances in human development sciences point to tremendous possibilities to promote healthy child development and well-being across life by proactively supporting safe, stable and nurturing family relationships (SSNRs), teaching resilience, and intervening early to promote healing the trauma and stress associated with disruptions in SSNRs. Assessing potential disruptions in SSNRs, such as adverse childhood experiences (ACEs), can contribute to assessing risk for trauma and chronic and toxic stress. Asking about ACEs can help with efforts to prevent and attenuate negative impacts on child development and both child and family well-being. Many methods to assess ACEs exist but have not been compared. The National Survey of Children's Health (NSCH) now measures ACEs for children, but requires further assessment and validation. MethodsWe identified and compared methods to assess ACEs among children and families, evaluated the acceptability and validity of the new NSCH-ACEs measure, and identified implications for assessing ACEs in research and practice. ResultsOf 14 ACEs assessment methods identified, 5 have been used in clinical settings (vs public health assessment or research) and all but 1 require self or parent report (3 allow child report). Across methods, 6 to 20 constructs are assessed, 4 of which are common to all: parental incarceration, domestic violence, household mental illness/suicide, household alcohol or substance abuse. Common additional content includes assessing exposure to neighborhood violence, bullying, discrimination, or parental death. All methods use a numeric, cumulative risk scoring methodology. The NSCH-ACEs measure was acceptable to respondents as evidenced by few missing values and no reduction in response rate attributable to asking about children's ACEs. The 9 ACEs assessed in the NSCH co-occur, with most children with 1 ACE having additional ACEs. This measure showed efficiency and confirmatory factor analysis as well as latent class analysis supported a cumulative risk scoring method. Formative as well as reflective measurement models further support cumulative risk scoring and provide evidence of predictive validity of the NSCH-ACEs. Common effects of ACEs across household income groups confirm information distinct from economic status is provided and suggest use of population-wide versus high-risk approaches to assessing ACEs. ConclusionsAlthough important variations exist, available ACEs measurement methods are similar and show consistent associations with poorer health outcomes in absence of protective factors and resilience. All methods reviewed appear to coincide with broader goals to facilitate health education, promote health and, where needed, to mitigate the trauma, chronic stress, and behavioral and emotional sequelae that can arise with exposure to ACEs. Assessing ACEs appears acceptable to individuals and families when conducted in population-based and clinical research contexts. Although research to date and neurobiological findings compel early identification and health education about ACEs in clinical settings, further research to guide use in pediatric practice is required, especially as it relates to distinguishing ACEs assessment from identifying current family psychosocial risks and child abuse. The reflective as well as formative psychometric analyses conducted in this study confirm use of cumulative risk scoring for the NSCH-ACEs measure. Even if children have not been exposed to ACEs, assessing ACEs has value as an educational tool for engaging and educating families and children about the importance of SSNRs and how to recognize and manage stress and learn resilience.

Adaptation of a Self-Regulated Practice Behaviour Scale for Portuguese music students

The aim of this study was to analyse the validity and reliability of the Self-Regulated Practice Behaviour Scale (SRPB) developed by Miksza (2012) for music students in Portugal. Tasks involved in this study included: (1) translating the original scale and creating a viable Portuguese adaptation for Portuguese musical performance students; (2) analysing the psychometric properties of the Portuguese version of the scale. The translation and adaptation of the original scale resulted in a 52-item Portuguese Self-Regulated Practice Behaviour measure (Portuguese SRPB). Students enrolled in music education programmes from eight conservatoires answered the Portuguese SRPB ( n = 1,200). The results of confirmatory factor analysis show that a five-factor model presenting the five correspondent theoretical dimensions of self-regulation (motive, method, behaviour, time management, and social influence) achieved the best fit to the data collected. All Portuguese SRPB dimensions were predictive of students’ reports of average practice efficiency, grades, and practice time, highlighting preliminary evidence of predictive validity. The internal consistency reliability coefficients ranged from good to excellent (α = .71 to .84). The findings suggest that the Portuguese SRPB produced a valid and reliable measure of the self-regulatory practice behaviour of Portuguese conservatoire students.

A model of leadership motivations, error management culture, leadership capacity, and career success

We integrated ideas from literature on error management culture, leadership motivation, and career development to create several contributions for the research literature. First, we examined two situational factors – error management/aversion culture perceptions – that affect employees’ leadership‐relevant motivations. Second, we distinguish between two types of leader motivations, motivation to lead (MTL) and motivation to develop leadership skills (MTDL). We offer evidence of discriminant and predictive validity of the two leadership motivations on key leadership processes and outcomes. Third, we tested a linkage model in which error management/aversion perceptions influenced leadership motivations (MTL and MTDL) and these motivations predicted leadership capacity and leader career success (i.e., promotions, increased leadership responsibility, and pay increases). Based upon multisource data collected from 151 employees and their supervisors from diverse occupations and organizations over a period of 1 year, we found that error management perceptions were positively associated with social‐normative MTL (the motive to lead out of a sense of duty and obligation) and with MTDL whereas perceptions of error aversion were negatively related with affective‐identity and non‐calculative MTL. MTDL was distinguishable from MTL and demonstrated better predictive validity on leadership capacity and career success than MTL. We discuss a number of implications for both theory and practice.Practitioner Points Creating a culture in which errors are constructively managed enhances leadership motivations, leadership capacity (leader behaviour, development, and potential), and career success. In managing leadership development and performance, distinguishing between motivation to be a leader and motivation to develop leadership skills is important, both in terms of how these motivations are influenced by error management and aversion and in terms of the how the motivations influence leadership capacity and success. Motive to lead out of a sense of duty is key.

Predicting Postsecondary Education and Employment Outcomes Using Results From the Transition Assessment and Goal Generator

We conducted an exploratory study to investigate the relation between nonacademic behavior constructs measured by the Transition Assessment and Goal Generator (TAGG) and postsecondary education and employment outcomes for 297 high school leavers who completed the TAGG during their high school years. Four of eight TAGG constructs predicted postsecondary educational outcomes: (a) Interacting With Others, (b) Student Involvement in the Individualized Education Program (IEP), (c) Support Community, and (d) Goal Setting and Attainment. Four constructs predicted postsecondary employment outcomes: (a) Employment, (b) Student Involvement in the IEP, (c) Support Community, and (d) Interacting With Others. The addition of student grade point average (GPA) strengthened some of the models. The findings appear to add predictive validity evidence to support use of TAGG results to assist with transition planning.