Predictive Validity Estimates Research Articles

An out-of-sample perspective on the assessment of incremental predictive validity

In a psychometric analysis of a new psychological test, we often assess the predictive validity of a new target test over and above a baseline test, known as the incremental predictive validity. Usually, the incremental predictive validity is evaluated using within-sample statistics. Recently, it was argued to use out-of-sample assessment to prevent overfitting and non-replicable findings. In this paper, we elaborate on how to assess incremental predictive validity out-of-sample. In such an approach, we estimate prediction rules in one sample, and evaluate incremental predictive validity in another sample. Using a simulation study, we investigate whether an out-of-sample assessment results in different findings than a within-sample evaluation, taking into account the reliability of the baseline and a target test, and other factors (i.e., sample size). Results show that there is a difference between the in-sample and out-of-sample assessment, especially in small samples. However, the reliability of the two tests has no influence on this difference. In addition, we explore the effects of ridge estimation, ordinary least squares, and SIMEX, three different methods for estimating a prediction rule, on incremental predictive validity. The results show that using SIMEX leads to a bad assessment of incremental predictive validity. Ordinary least squares and ridge estimation result in almost the same incremental predictive validity estimates with a little advantage for ridge regression. In an empirical application, we show how to assess incremental predictive validity in practice and we compare that to the usual assessment.

Feasibility of daily physical activity monitoring and associations with treatment toxicity in older adults with metastatic prostate cancer.

12055 Background: Accelerometry research suggests physical activity may be associated with treatment toxicity, but generalizability to geriatric oncology is unclear. As many older adults have low levels of physical activity and technology use, further exploration is warranted. This study explores the feasibility of daily physical activity monitoring and the association between step counts and symptoms. Methods: Older adults aged 65+ starting treatment (chemotherapy, enzalutamide/abiraterone, or Radium-223) for metastatic prostate cancer were enrolled in a prospective cohort study. Participants reported step counts (measured via smartphone and a free commercial app) and symptoms (Edmonton Symptom Assessment Scale (ESAS)) daily for their first treatment cycle (~1 month). The feasibility of daily monitoring was assessed with descriptive statistics and thematic analysis. The association between a 15% decline in steps (compared to pre-treatment baseline) and the emergence of moderate-to-severe symptoms and pain (ESAS ≥4) in the next 24 hours was assessed using logistic regression. The utility of a 15% step count decline in predicting the development of moderate-to-severe symptoms and pain was evaluated using sensitivity and positive predictive value (PPV). Results: Of 90 participants, 47 engaged in step count monitoring (mean age 75y; 52.2% participation rate). Daily physical activity monitoring was found to be feasible (90.5% median response rate; 94% retention rate) with numerous patient-reported benefits including increased self-awareness and motivation to engage in physical activity. During the first treatment cycle, instances of a 15% decline in steps were common (n=37, 78.7%), as was the emergence of moderate-to-severe symptoms overall (n=40, 85.1%) and pain (n=26, 55.3%). Overall, daily physical activity did not appear to be significantly predictive of symptoms (OR=1.8, 95% CI=0.42-7.74) overall or pain (OR=0.95, 95% CI=0.22-3.81). Predictive validity estimates revealed reasonable sensitivity for both moderate to severe symptoms overall (sensitivity=88.8%, 95% CI=68.7-95.0; PPV=73.0%, 95% CI=58.7-87.3) but poor PPV for pain (sensitivity=77.8%, 95% CI=58.6-97.0; PPV=37.8%, 95% CI=22.2-53.5). Conclusions: Despite recent studies framing physical activity changes as predictive of toxicity, PPV was only poor to fair. However, daily activity monitoring in older cancer patients appears feasible and may have other uses. Clinical trial information: NCT04193657 .

Short-term risk assessment in the long term: A scoping review and meta-analysis of the Brøset Violence Checklist.

INTRODUCTION: Existing literature on the Brøset Violence Checklist (BVC) is examined in the context of usability, implementation and validity to provide evidence-based recommendations on its application and identify opportunities for future development. To identify current knowledge on the BVC and guide clinicians and researchers toward the next steps in using this tool in clinical practice to prevent violence in healthcare settings. A scoping review approach with a meta-analysis supplement was adopted to broadly identify and map available evidence on the BVC and provide specific estimates of predictive validity in different contexts. Sixty-two studies conducted in 23 countries addressed the implementation of the BVC across various settings. Many studies adapted the original BVC, and the clinical utility was noted as an important feature. A meta-analysis of the original BVC format estimated a pooled area under the curve at 0.83 (95% CI 0.78-0.87) in a subset of 15 studies. The BVC combines high predictive validity and good clinical utility across a wide range of settings and cultures. It should continue to be incorporated into routine practice in mental health services focused on preventing violence and coercion. Development of collaborative approaches with service users involved in assessing their own risk of future violence.

Pretrial risk assessment validation research: Range restriction and attenuation of predictive validity estimates.

Pretrial risk assessment validation research: Range restriction and attenuation of predictive validity estimates.

Developing and Validating a Model of Leadership Potential for VUCA Environments

A critical succession planning challenge for organizations across industries is the identification of leadership talent in mid-management roles who are ready for executive-level positions. Despite the incredible demand for greater attention to executive succession practices and the challenges associated with unplanned executive turnover, the published research literature currently lacks a model and assessment process designed for organizations competing in highly complex and volatile business contexts. To develop theory and guide succession management practices, this paper presents a multi-phased study describing the development and validation of such a model: Learning, Engaging, Adapting, and Persevering (LEAP). The first phase presents a qualitative study that identified the leadership competencies associated with executive potential in VUCA (volatile, uncertain, chaotic, and ambiguous) work contexts. The second phase presents a quantitative study of 308 mid-level leaders and 213 of their immediate managers across four organizations in a highly complex and turbulent industry. The third phase consists of a quasi-longitudinal study examining the LEAP’s predictive validity across a series of job performance, promotion, and attitudinal outcomes. Based on convergent, concurrent, and predictive validity estimates provided across the three phases, this paper presents evidence for a theory of leadership potential, providing much needed guidance for executive succession planning. The study’s implications for theory, future research, and a range of practical applications for executive teams, board members, and HR professionals are discussed.

Supplemental Material for Pretrial Risk Assessment Validation Research: Range Restriction and Attenuation of Predictive Validity Estimates

Supplemental Material for Pretrial Risk Assessment Validation Research: Range Restriction and Attenuation of Predictive Validity Estimates

Effects of Situational Judgment Test Format on Reliability and Validity

Single-response situational judgment tests (SRSJTs) differ from multiple-response SJTs (MRSJTS) in that they present test takers with edited critical incidents and simply ask test takers to read over the action described and evaluate it according to its effectiveness. Research comparing the reliability and validity of SRSJTs and MRSJTs is thus far extremely limited. The study reported here directly compares forms of a SRSJT and MRSJT and explores the reliability, convergent validity, and predictive validity of each format. Results from this investigation present preliminary evidence to suggest SRSJTs may produce internal consistency reliability, convergent validity, and predictive validity estimates that are comparable to those achieved with many traditional MRSJTs. We conclude by discussing practical implications for personnel selection and assessment, and future research in psychological science more broadly.

Test-retest reliability and predictive validity of the Implicit Association Test in children.

The Implicit Association Test (IAT) is increasingly used in developmental research despite minimal evidence of whether children's IAT scores are reliable across time or predictive of behavior. When test-retest reliability and predictive validity have been assessed, the results have been mixed, and because these studies have differed on many factors simultaneously (lag-time between testing administrations, domain, etc.), it is difficult to discern what factors may explain variability in existing test-retest reliability and predictive validity estimates. Across five studies (total N = 519; ages 6- to 11-years-old), we manipulated two factors that have varied in previous developmental research-lag-time and domain. An internal meta-analysis of these studies revealed that, across three different methods of analyzing the data, mean test-retest (rs of .48, .38, and .34) and predictive validity (rs of .46, .20, and .10) effect sizes were significantly greater than zero. While lag-time did not moderate the magnitude of test-retest coefficients, whether we observed domain differences in test-retest reliability and predictive validity estimates was contingent on other factors, such as how we scored the IAT or whether we included estimates from a unique sample (i.e., a sample containing gender typical and gender diverse children). Recommendations are made for developmental researchers that utilize the IAT in their research. (PsycINFO Database Record

Predictive Validity of a Student Self-Report Screener of Behavioral and Emotional Risk in an Urban High School

.Increasingly, schools are implementing school-based screening for risk of behavioral and emotional problems; hence, foundational evidence supporting the predictive validity of screening instruments is important to assess. This study examined the predictive validity of the Behavior Assessment System for Children-2 Behavioral and Emotional Screening System Student Form (BESS Student) in the authentic context of an urban high school that conducted universal screening over 3 years. Multivariate regression, sensitivity, specificity, and receiver operating characteristic curve analyses were used to examine the BESS Student scores' prediction of internalizing symptoms. BESS Student scores were able to explain a significant proportion of the variance in internalizing symptoms concurrently, but predictive validity estimates decreased over time. Significant gender differences were present; BESS Student scores were better able to predict internalizing symptoms for females. Implications for research and practice involving screening for behavioral and emotional problems are discussed.

Violence risk assessment in psychiatric patients in China: A systematic review.

Objectives:The aim of this study was to undertake a systematic review on violence risk assessment instruments used for psychiatric patients in China.Methods:A systematic search was conducted from 1980 until 2014 to identify studies that used psychometric tools or structured instruments to assess aggression and violence risk. Information from primary studies was extracted, including demographic characteristics of the samples used, study design characteristics, and reliability and validity estimates.Results:A total of 30 primary studies were identified that investigated aggression or violence; 6 reported on tools assessing aggression while an additional 24 studies reported on structured instruments designed to predict violence. Although measures of reliability were typically good, estimates of predictive validity were mostly in the range of poor to moderate, with only 1 study finding good validity. These estimates were typically lower than that found in previous work for Western samples.Conclusion:There is currently little evidence to support the use of current violence risk assessment instruments in psychiatric patients in China. Developing more accurate and scalable approaches are research priorities.

The Validity of Occupational Performance Assessments: A Validity Generalization Meta-Analysis

ABSTRACTThe purpose of this meta-analysis was to determine sample weighted mean validity effect sizes for occupational performance assessments, and their generalizability from research to clinical settings. The bare-bones Validity Generalization (VG) guidelines developed by Hunter and Schmidt () augmented by Maximum Likelihood (ML) procedures were used to complete the meta-analysis. The sample consisted of 27 studies in which convergent, divergent, and predictive validity estimates of occupational performance assessments were investigated. The mean coefficients of assessments validated in the studies constituting the sample for this meta-analysis ranged from medium to large. Further meta-analysis with complete dis-attenuation of observed mean validity coefficients is indicated.

Contextualizing the Predictive Validity of General Mental Ability

General mental ability (GMA) is backed by substantial evidence demonstrating its utility in predicting learning and performance. Yet despite this convincing evidence, researchers have recently called for a renewed focus on GMA, in order to further our understanding of the more nuanced aspects of its relationship with important workplace outcomes. The purpose of this symposium is therefore to share new research that helps to contextualize our understanding of the predictive validity of GMA, by examining GMA in different workplace settings, using a variety of outcomes and analytical approaches. The first paper by Dierdorff and Rubin helps to contextualize our understanding of the GMA-job performance relationship by examining occupational characteristics as moderators. The second paper by Gonzalez-Mulé and Postlethwaite examines the value of different criminology theories in explaining the GMA-CWB relationship, and provides meta- analytic evidence to address inconsistencies in previous research. The Surface, Gissel, and Borneman study presents a meta-analysis examining the predictive validity of GMA on various foreign language-training outcomes, and explores contextual moderators on the predictive validity of GMA. Finally, the fourth study by Ellington, Mead, Landis, and Overfors uses a simulation design to examine design characteristics that are common to GMA validation studies using supervisory performance ratings as criteria, yet might obscure the predictive validity estimates. Taking a More Complex Look at Complexity in the GMA—Job Performance Relationship Presenter: Erich C. Dierdorff; DePaul U. Presenter: Robert S. Rubin; DePaul U. The Relationship Between GMA and Counterproductive Behaviors: A Meta-Analysis Presenter: Erik Gonzalez-Mule; U. of Iowa Presenter: Bennett E. Postlethwaite; Pepperdine U. Investigating GMA-Training Criteria Relations in Foreign Language Learning Contexts Presenter: Eric Alan Surface; SWA Consulting Inc Presenter: Amanda Gissel; SWA Consulting Inc. Presenter: Matthew J. Borneman; SWA Consulting Inc. Are We Underestimating the Predictive Validity of GMA? Presenter: James Kemp Ellington; Illinois Institute of Technology Presenter: Alan D. Mead; Illinois Institute of Technology Presenter: Ronald S. Landis; Illinois Institute of Technology Presenter: Christine M. Overfors; Illinois Institute of Technology

One is Enough

The Dynamic Analog Scale (DAS)—a generic and flexible technique for creating single-item measures of psychological constructs—comprises detailed definitions and a continuous analog scale on which self and others are rated simultaneously. Using five experimental conditions, the authors investigated the psychometric properties of the DAS as well as the impact of different social contexts and modified Big Five trait definitions on participant ratings and their relationships to measures of volunteerism, religiosity, affect, and alcohol consumption. Results revealed few differences across conditions, and reliability coefficients and predictive validity estimates were similar to those typically obtained using standard, multiple-item questionnaires. The DAS is therefore supported as a consistent and efficacious method for measuring personality traits with single items.

Structured Assessment of Violence Risk in Schizophrenia and Other Psychiatric Disorders: A Systematic Review of the Validity, Reliability, and Item Content of 10 Available Instruments

To undertake a systematic review on structured violence risk assessment tools in individuals with schizophrenia. A systematic search was conducted from 1990 to 2011 to identify violence risk assessment tools and studies examining their predictive validity. Item content of the identified instruments was analyzed, and areas under the curve (AUC) from the studies were extracted. In addition, an 11-item checklist was developed to assess the utility and psychometric properties of these tools. Ten risk assessment tools designed to predict community violence in psychiatric patients were identified, but only 2 studies reporting predictive validity estimates in patients with schizophrenia were found (median AUC = 0.69; interquartile range = 0.60-0.77). When inclusion criteria was broadened to include studies measuring accuracy for any diagnostic group, mixed evidence of predictive validity was found, with median AUCs ranging from 0.62 to 0.85 depending on the population. Item content included mostly clinical, sociodemographic, and criminal history factors. As only 1 tool included a neurobiological item, a structured review of brain-based and cognitive risk factors for violence was included, and 3 clusters (neurocognitive ability, neurocognitive awareness, and attitudinal cognition) were identified. While a number of violence risk assessment tools exist that can be used to predict the likelihood of community violence in psychiatric patients, there is currently little direct evidence for their utility in individuals with schizophrenia. In addition, there is large variation in item content between instruments, and further research is necessary to determine whether the inclusion of alternative factors could improve risk assessment.

Sources of Variability in Estimates of Predictive Validity

Level of Service (LS) is one of the most widely used general risk and need assessment tools in criminal justice agencies across North America. However, there is significant interstudy variability in the magnitude of the validity estimates. This study was conducted to examine possible sources of this variability. The predictive validity of LS risk and need increased with length of follow-up period and with investigator allegiance to LS. The combination of these two variables reveals consistent increases in mean predictive validity estimates from modest (in the .20s) through large (in the mid .30s) to very large (in the .40s) in samples of both male and female offenders. We hypothesized that the “allegiance effect” reflects the integrity of LS implementation and support provided by the agency for risk assessment. This is akin to the difference between “demonstration projects” and “practical” rehabilitation programming in the offender treatment research. Moreover, controls for Canadian versus non-Canadian evaluations reduced the effect of allegiance and length of follow-up to nonsignificant levels. Possible explanations for these findings include the degree of integrity in conducting risk and need assessments, the accuracy of recidivism as the criterion measure, and generalizability across international boundaries.

Quantifying calcium intake in school age children: development and validation of the Calcium Counts! food frequency questionnaire.

Quantifying dietary behavior is difficult and can be intrusive. Calcium, an essential mineral for skeletal development during childhood, is difficult to assess. Few studies have examined the use of food frequency questionnaires (FFQs) for assessing calcium intake in school-age children. This study evaluated the validity and reliability of the Calcium Counts! FFQ (CCFFQ) for estimating calcium intake in school children in the US. Healthy children, aged 7-10 years (n = 139) completed the CCFFQ and 7-day weighed food records. A subset of subjects completed a second CCFFQ within 3.6 months. Concurrent validity was determined using Pearson correlations between the CCFFQ and food record estimates of calcium intake, and the relationship between quintiles for the two measures. Predictive validity was determined using generalized linear regression models to explore the effects of age, race, and gender. Inter- and intra-individual variability in calcium intake was high (>300 mg/day). Calcium intake was approximately 300 mg/day higher by CCFFQ compared to food records. Concurrent validity was moderate (r = 0.61) for the entire cohort and higher for selected subgroups. Predictive validity estimates yielded significant relationships between CCFFQ and food record estimates of calcium intake alone and in the presence of such potential effect modifiers as age group, race, and gender. Test-retest reliability was high (r = 0.74). Although calcium intake estimated by the CCFFQ was greater than that measured by food records, the CCFFQ provides valid and reliable estimates of calcium intake in children. The CCFFQ is especially well-suited as a tool to identify children with low calcium intakes.

A Meta-Analysis of Juvenile Justice Risk Assessment Instruments

Juvenile justice systems have widely adopted risk assessment instruments to support judicial and administrative decisions about sanctioning severity and restrictiveness of care. A little explored property of these instruments is the extent to which their predictive validity generalizes across gender. The article reports on a meta-analysis of risk assessment predictive validity with male and female offenders. Nineteen studies encompassing 20 unique samples met inclusion criteria. Findings indicated that predictive validity estimates are equivalent for male and female offenders and are consistent with results of other meta-analyses in the field. The findings also indicate that when gender differences are observed in individual studies, they provide evidence for gender biases in juvenile justice decision-making and case processing rather than for the ineffectiveness of risk assessment with female offenders.

Risk Assessment Stability: A Revalidation Study of the Arizona Risk/Needs Assessment Instrument

The actuarial method is the gold standard for risk assessment in child welfare, juvenile justice, and criminal justice. It produces risk classifications that are highly predictive and that may be robust to sampling error. This article reports a revalidation study of the Arizona Risk/Needs Assessment instrument, an actuarial instrument for juvenile offenders. Results of this study (N = 29,711) show that the actuarial model provided stable predictive validity estimates for recidivism up to 7 years after its initial validation and that predictive validity did not vary appreciably for males, females, White offenders, Latino youth, African American youth, and Native American youth. These findings lend further support to the actuarial approach to risk assessment and clarify the literature about risk assessment across gender and race/ethnicity.

Predicting sex offender recidivism. I. Correcting for item overselection and accuracy overestimation in scale development. II. Sampling error-induced attenuation of predictive validity over base rate information.

The authors demonstrate a statistical bootstrapping method for obtaining unbiased item selection and predictive validity estimates from a scale development sample, using data (N = 256) of Epperson et al. [2003 Minnesota Sex Offender Screening Tool-Revised (MnSOST-R) technical paper: Development, validation, and recommended risk level cut scores. Retrieved November 18, 2006 from Iowa State University Department of Psychology web site: http://www.psychology.iastate.edu/ approximately dle/mnsost_download.htm] from which the Minnesota Sex Offender Screening Tool-Revised (MnSOST-R) was developed. Validity (area under receiver operating characteristic curve) reported by Epperson et al. was .77 with 16 items selected. The present analysis yielded an asymptotically unbiased estimator AUC = .58. The present article also focused on the degree to which sampling error renders estimated cutting scores (appropriate to local [varying] recidivism base rates) nonoptimal, so that the long-run performance (measured by correct fraction, the total proportion of correct classifications) of these estimated cutting scores is poor, when they are applied to their parent populations (having assumed values for AUC and recidivism rate). This was investigated by Monte Carlo simulation over a range of AUC and recidivism rate values. Results indicate that, except for the AUC values higher than have ever been cross-validated, in combination with recidivism base rates severalfold higher than the literature average [Hanson and Morton-Bourgon, 2004, Predictors of sexual recidivism: An updated meta-analysis. (User report 2004-02.). Ottawa: Public Safety and Emergency Preparedness Canada], the user of an instrument similar in performance to the MnSOST-R cannot expect to achieve correct fraction performance notably in excess of what is achievable from knowing the population recidivism rate alone. The authors discuss the legal implications of their findings for procedural and substantive due process in relation to state sexually violent person commitment statutes and the Supreme Court's Kansas v. Hendricks decision regarding the constitutionality of such statutes.

Violence by psychiatric patients: the impact of archival measurement source on violence base rates and risk assessment accuracy.

This study examined the impact of using various archival sources of violence data on the recorded base rate of violence and on the accuracy of violence risk assessments in a sample of civil psychiatric patients. Violence in the community, after release from involuntary hospitalization, was measured with 3 types of archival data: criminal records, psychiatric hospital readmission records, and readmission records from multiple general hospitals. Findings indicated that the different measurement sources each identified unique incidents of violence, which were correlated moderately or not at all with one another, and that the estimated base rate of violence varied substantially as a function of outcome source. Not surprisingly, using a combination of sources identified the highest base rate of violence. Predictive validity estimates also varied widely under different measurement procedures and were more stable when a combination of sources was used. It is important in practical and research terms for studies of psychiatric patient violence to use multiple sources of outcome data.