Differences In Predictive Validity Research Articles

Dynamic and static predictive modelling of psychotherapy outcome-Comparison of two statistical approaches.

Improving prediction abilities in the therapy process can increase therapeutic success for a variety of reasons, such as more personalised treatment or resource optimisation. The increasingly applied methods of dynamic prediction seem to be very promising for this purpose. Prediction models are usually based on static approaches of frequentist statistics. However, the application of this statistical approach has been widely criticised in this research area. Bayesian statistics has been proposed in the literature as an alternative, especially for the task of dynamic modelling. In this study, we compare the performance of predicting therapy outcome over the course of therapy between both statistical approaches. Based on a sample of 341 patients, a logistic regression analysis was performed using both statistical approaches. Therapy success was conceptualised as reliable pre-post improvement in brief symptom inventory (BSI) scores. As predictors, we used the subscales of the Outcome Questionnaire (OQ-30) and the Helping Alliance Questionnaire (HAQ) measured every fifth session, as well as baseline BSI scores. The influence of the predictors during therapy differs between the frequentist and the Bayesian approach. In contrast, predictive validity is comparable with a mean area under the curve (AUC) of 0.76 in both model types. Bayesian statistic provides an innovative and useful alternative to the frequentist approach in predicting therapy outcome. The theoretical foundation is particularly well suited for dynamic prediction. Nevertheless, no differences in predictive validity were found in this study. More complex methodology as well as further research seems necessary to exploit the potential of Bayesian statistics in this area.

Do the offline and social media Big Five have the same dimensional structure, mean levels, and predictive validity of social media outcomes?

AbstractPrior studies found that the Big Five personality traits are significant predictors of social media outcomes, but they did not specify the situational context of the Big Five. The assumption is that people have the same personality on social media as offline. The present research addressed whether the Big Five are the same on social media as offline in terms of dimensional structure, mean levels, and predictive validity of social media outcomes. Across two samples, 943 college students completed measures of social media outcomes and two versions of the Big Five Inventory-2 adapted from Soto and John (2017), specifying offline and social media contexts. Findings revealed that all of the five dimensions emerged in both contexts, except that a few items might not apply well to the social media context. The mean levels of all five traits were significantly different between contexts, suggesting that the dimensional structure of the offline and social media Big Five are similar but peoples’ levels of expression of the Big Five are not the same between these contexts. Conscientiousness and extraversion were the least similar dimensions out of the five between the contexts. There were also differences in predictive validity between the offline and social media Big Five showing that heavier social media users are more open, conscientious, and extraverted on social media, whereas they are lower on these traits offline. Consequently, studying offline and social media contexts separately and jointly is critical for understanding how the Big Five predict social media outcomes.

The Prediction of Youth Recidivism in a Spanish Roma Population by the Youth Level of Service/Case Management Inventory (YLS/CMI).

Despite the increasing interest in the accuracy of youth risk assessment tools, the amount of research with ethnic minorities remains relatively modest. For this reason, the main goal of this study was to assess the predictive validity and disparate impact of the Youth Level of Service/Case Management Inventory (YLS/CMI) in a Spanish ethnic minority. The participants consisted of 88 Roma youth offenders and 135 non-Roma youth offenders, aged between 14 and 17 years old. Their risk of recidivism was assessed by means of the YLS/CMI Inventory and their recidivism rate was obtained from the Juvenile Justice Department. Results showed that the Inventory presented slightly lower predictive validity for the Roma group. Moreover, Roma juveniles presented higher risk scores and lower strength scores than non-Roma juveniles. These results supported the idea that professionals must therefore be aware of these cultural differences in predictive validity and the existent potentiality for disparate impact.

Predictive Validity of Pretrial Risk Assessments: A Systematic Review of the Literature

Bail reform is sweeping the nation and many jurisdictions are looking to pretrial risk assessment as one potential strategy to support these efforts. This article summarizes the findings of a systematic review of research examining the predictive validity of pretrial risk assessments. We reviewed 11 studies (13 publications) examining the predictive validity of six pretrial risk assessment instruments reported in the gray and peer-reviewed literature as of December, 2018. Findings typically show good to excellent predictive validity. Differences in predictive validity for men and women were mixed and small. When it could be examined, predictive validity was generally comparable across racial/ethnic subgroups; however, three comparisons revealed notably lower, albeit still fair to good, predictive validity for defendants of color than White defendants. Findings suggest that pretrial risk assessments predict pretrial outcomes with acceptable accuracy, but also emphasize the need for continued investigation of predictive validity across gender and racial/ethnic subgroups.

Development and Testing of the Danger Assessment for Law Enforcement (DA-LE)

Abstract One-third of women are victimized by intimate partner violence (IPV) in their lifetime; when women are killed, they are often murdered by a previously abusive intimate partner. Risk-informed collaborative interventions, such as domestic violence high risk teams (DVHRTs), use IPV risk assessment to identify and intervene in high-risk IPV cases. This study reports on the development and testing of the Danger Assessment for Law Enforcement (DA-LE), an IPV risk assessment intended for use with DVHRTs. Data were collected through structured telephone interviews from service-seeking survivors of IPV at two time points approximately seven to eight months apart. One sample (n = 570) was used to develop the DA-LE and another (n = 389) was used to test the predictive validity of the instrument using the receiver operating characteristic area under the curve (AUC). The DA-LE predicted near fatal IPV on follow-up with similar or better accuracy than most validated IPV risk assessment instruments (AUC = 0.6864–0.7516). There were no significant differences in predictive validity based on survivor/offender race or ethnicity. The DA-LE has the potential to identify high-risk police-involved IPV cases. Risk-informed collaborative interventions may enhance outcomes for survivors of IPV by holding offenders accountable, increasing help seeking, and reducing future assaults.

Rater experience and the predictive validity of Psychopathy Checklist: Youth Version scores

We compared the predictive validity of Psychopathy Checklist: Youth Version (PCL:YV) scores assigned by a licensed clinician to scores assigned by a graduate student across a sample of 82 juvenile offenders. Although both raters completed in-depth training and practice scoring cases, the graduate student had no prior clinical experience. The raters showed a high level of agreement in their scoring for 11 reliability check cases (intraclass correlation coefficient, ICCA,1 = .90 for PCL:YV Total score), but the scores assigned by the licensed clinician were better predictors of post-release recidivism (area under the curve, AUC = .77) than those assigned by the graduate student (AUC = .45). There was more variability in the scores assigned by the licensed clinician than those assigned by the graduate student, suggesting that more experienced clinicians’ willingness to assign both high and low scores may help explain rater differences in predictive validity.

Gender Differences in Predictive Validity of Kenyan Primary School Examinations in Secondary Schools

The purpose of this study was to determine gender differences in predictive validity of Kenya Certificate of Primary Education (KCPE) among secondary school students in Kisii central Sub-county, Kenya. The study adopted Correlational and Ex-post-facto research designs. The study population was 3897 Kenya Certificate of Secondary Education (KCSE) candidates from 55 public secondary schools. Stratified random and saturated sampling techniques were used to select 16 secondary schools and 1391 students. Data used included 2006 KCPE scores and 2010 KCSE scores of the same students under study. Data was collected using a researcher made pro forma. It was analyzed using correlations and regression analyses. Results showed a strong positive Pearson’s correlation coefficient (r=0.675; n=607; p < 0.05) between KCPE and KCSE scores for male students and (r=0.710; n=784; p < 0.05) between KCPE and KCSE scores for female students. This was however not a statistically significant difference in correlation of KCSE and KCPE scores based on gender. The study concluded that KCPE scores is a good predictor of KCSE scores regardless of learners’ gender. It recommended that both male and female learners should be accorded equal encouragement to improve their academic outcomes.

A prospective validation study of the START:AV

PurposeThe purpose of this paper is to explore the validity and reliability of the Short-Term Assessment of Risk and Treatability: Adolescent Version (START:AV) to determine if it has predictive accuracy in relation to physical aggression, severe verbal aggression, property damage and self-harm, in a medium secure setting. In addition, the authors hoped to provide some of the first descriptive data available for the START:AV among a UK adolescent population in a medium secure adolescent unit.Design/methodology/approachThe sample consisted of 90 female and male adolescents, with and without developmental disabilities. It was important to explore the measure’s predictive accuracy across specific population groups, such as between males and females, as well as those with developmental disabilities, and those without.FindingsSome significant relationships were found between the START:AV and adverse outcomes. For instance, total strength and vulnerability scores were predictive for verbal and physical aggression. Differences in predictive validity were evident when comparisons were made between males and females, with relationships being evident amongst the male population only. When splitting the male sample into developmental disability and non-developmental disability groups, significant relationships were found between strength and vulnerability scores and verbal and physical aggression.Practical implicationsA number of practical implications are considered, such as the START:AV is relevant for use with adolescents in hospital settings and the significant inverse relationship between strength scores and negative outcomes supports the importance of considering protective/strength factors when working with at risk youths.Originality/valueThere is currently limited validation data for the START:AV in the UK or elsewhere.

PCL-R field validity in prison and hospital settings.

Recent field studies have questioned the interrater reliability (IRR) and predictive validity regarding (violent) recidivism of the Psychopathy Checklist-Revised (PCL-R). Using a forensic psychiatric sample, the current study investigated discrepancies in scoring between hospital and prison settings, as well as differences in predictive validity across these two settings. PCL-R information was collected from prison and hospital files, resulting in 224 PCL-R total scores and 74 double scores. When examining repeated measurements, large individual differences were found together with an intraclass correlation coefficient (ICCA,1) of .42 for the total score. Discrepant results were found for Factor 2, with repeated scores within the same setting having an ICCA,1 of .28 versus an ICCA,1 of .57 for repeated scores between settings. However, areas under the curve (AUCs) from receiver operating characteristic (ROC) analyses for total, factor and facet scores did not differ between settings. For the whole sample, Factor 2 scores marginally predicted violent and general recidivism after 2 years (AUC = .62 and .63), whereas Factor 1 did not predict (violent) recidivism. Consistent with recent studies from other countries, these results suggest inadequate field reliability and validity in prison and hospital settings in Flanders (Belgium). (PsycINFO Database Record

The predictive validity of the Two-Tiered Violence Risk Estimates Scale (TTV) in a long-term follow-up of violent offenders.

Over the past few decades many structured risk appraisal measures have been created to respond to this need. The Two-Tiered Violence Risk Estimates Scale (TTV) is a measure designed to integrate both an actuarial estimate of violence risk with critical risk management indicators. The current study examined interrater reliability and the predictive validity of the TTV in a sample of violent offenders (n = 120) over an average follow-up period of 17.75 years. The TTV was retrospectively scored and compared with the Violence Risk Appraisal Guide (VRAG), the Statistical Information of Recidivism Scale-Revised (SIR-R1), and the Psychopathy Checklist-Revised (PCL-R). Approximately 53% of the sample reoffended violently, with an overall recidivism rate of 74%. Although the VRAG was the strongest predictor of violent recidivism in the sample, the Actuarial Risk Estimates (ARE) scale of the TTV produced a small, significant effect. The Risk Management Indicators (RMI) produced nonsignificant area under the curve (AUC) values for all recidivism outcomes. Comparisons between measures using AUC values and Cox regression showed that there were no statistical differences in predictive validity. The results of this research will be used to inform the validation and reliability literature on the TTV, and will contribute to the overall risk assessment literature. (PsycINFO Database Record

EXAMINING POTENTIAL BIAS IN SCREENING MEASURES FOR MIDDLE SCHOOL STUDENTS BY SPECIAL EDUCATION AND LOW SOCIOECONOMIC STATUS SUBGROUPS

To provide timely and effective supports for students reading below grade level, schools require methods for quickly and accurately identifying those students in need. One method for identifying those students is through universal screening. Assessments such as oral reading fluency (ORF) and Maze reading comprehension are commonly used as screening assessments in middle grades. The current study examined ORF and Maze for evidence of bias across two subgroups known to be at increased risk for failure in reading: (a) students with learning disabilities and (b) students from low‐income households. Data from 4,215 students in the sixth (n = 1,126), seventh (n = 1,361), and eighth grades (n = 1,728) were analyzed. Results indicate no significant differences in predictive validity for students from low‐income households compared to students from middle and upper income households. For students with learning disabilities only 8th grade scores showed any evidence of bias compared to students without diagnosed disabilities. Implications and recommendations for future research are discussed.

Comparing the predictive validity of behavioral codings and behavioral ratings in a working-dog breeding program

Most working-dog breeding programs have a substantial interest in using behavioral assessments of their young dogs to predict their subsequent success. Different methods of measuring behavior may capture different aspects of behavior yet working-dog programs typically use only a single measurement method. Thus, the primary aim of this study was to test whether two different measurement methods (ratings or codings) would differ in their predictive validity with respect to working-dog selection outcomes. Rating methods require observers to intuitively aggregate their observations into a single rating and in doing so may reduce error variance in measurement, resulting in improved validity. Coding methods on the other hand do not demand so much judgment on the part of the observer so may be less influenced by observer biases. Here we analyzed the two methods with respect to their ability to predict selection for training in a sample of odor-detection dogs bred at the U.S. Transportation Security Administration Canine Breeding and Development Center. Behaviors observed in two standardized tests (search & retrieve and environment) at four different time points across the first year of life were measured using nine ratings and 23 codings. Data reduction techniques identified two underlying dimensions in ratings (environmental stability and hunt drive) and nine in codings (confidence, anxiety, exploration, excitability, search performance, dominant possession, independent possession, energy management, and search aptitude). There were no differences in predictive validity between the two methods; both ratings and codings correctly classified a high percentage of dogs that were/were not selected for training at 12 months of age (84.6–88.5%). In the search & retrieve test, codings and ratings appeared to be measuring the same construct. In the environment test the only significant coding predictor of training selection (confidence) was strongly related to the single rating predictor (environmental stability). Rating methods tended to capture behavior that was more consistent, while coding methods tended to capture behavior that was more situation-specific. Our mixed-models approach also allowed us to discriminate between average behavior (between-individual variation) and behavioral change through time (within-individual variation); such findings emphasize different aspects of development that may need to be monitored during rearing. Our results suggest that, in some cases, the use of ratings versus codings may be inconsequential from the standpoint of predicting which dogs get selected for training. Virtually all research on animal behavior assesses behavior via coding or rating methods; further work is needed to verify these results.

Are there differences between behavioral measurement methods? A comparison of the predictive validity of two ratings methods in a working dog program

Consistent behavioral variation within and between individuals is ubiquitous in all working dog populations. Most working dog programs have recognized this fact, and have subsequently attempted to quantify behavior through the use of standardized tests. Standardized tests may employ several measurement methods, but two common ones are behavioral ratings and subjective ratings. The former is characterized by a rating for a behavior (e.g., reaction to a noise) usually based on a single observation or test; the latter is characterized by a rating for a trait (e.g., confidence) that is based across multiple observations of behavior. The main difference between the two rating methods is the level of aggregation, or intuition, that is required by the human observer. Measurement theory predicts that ratings based on multiple observations (i.e., subjective ratings) should be more reliable because measurement error is reduced. However, subjective ratings, by definition, may be susceptible to observer bias, in which case ratings based on fewer, but better-defined observations (i.e., behavioral ratings) could result in greater reliability. In either case, the ultimate criterion of most working dog programs is the predictive validity of measured behaviors in standardized tests. To the best of our knowledge, the relative predictive validity of subjective and behavioral ratings has yet to be tested within the same working dog population. Here we analyzed behavioral test results along with training outcomes from a large sample (∼400) of German shepherd dogs, aged 15–18 months, all bred at the Swedish Armed Forces breeding kennel. Behaviors observed in the test were measured using 25 behavioral and 13 subjective ratings. Data reduction and confirmatory techniques identified five underlying dimensions in behavioral ratings (confidence, physical engagement, social engagement, aggression, and environmental sureness) and three in subjective ratings (engagement, confidence, and aggression). Both ratings methods correctly classified a high percentage of dogs that did/did not complete training (70.3–78.3%). However, only minor differences in predictive validity were observed between the two measurement methods (1.7–6.6%). Engagement and confidence, irrespective of measurement method, were the strongest predictors of training completion, but the two ratings methods identified different aspects of engagement and confidence that may be important to training outcomes in this and other working dog programs. Taken together, our results suggest that in some cases, the use of subjective versus behavioral ratings may be inconsequential from the standpoint of prediction to training criterion. Further empirical verification is needed, along with improvements in the explicit definition and measurement of ‘success’ in this and other working dog programs.

Comparing Current Definitions of Return to Work: A Measurement Approach

Return-to-work (RTW) status is an often used outcome in work and health research. In low back pain, work is regarded as a normal activity a worker should return to in order to fully recover. Comparing outcomes across studies and even jurisdictions using different definitions of RTW can be challenging for readers in general and when performing a systematic review in particular. In this study, the measurement properties of previously defined RTW outcomes were examined with data from two studies from two countries. Data on RTW in low back pain (LBP) from the Canadian Early Claimant Cohort (ECC); a workers' compensation based study, and the Dutch Amsterdam Sherbrooke Evaluation (ASE) study were analyzed. Correlations between outcomes, differences in predictive validity when using different outcomes and construct validity when comparing outcomes to a functional status outcome were analyzed. In the ECC all definitions were highly correlated and performed similarly in predictive validity. When compared to functional status, RTW definitions in the ECC study performed fair to good on all time points. In the ASE study all definitions were highly correlated and performed similarly in predictive validity. The RTW definitions, however, failed to compare or compared poorly with functional status. Only one definition compared fairly on one time point. Differently defined outcomes are highly correlated, give similar results in prediction, but seem to differ in construct validity when compared to functional status depending on societal context or possibly birth cohort. Comparison of studies using different RTW definitions appears valid as long as RTW status is not considered as a measure of functional status.

The Predictive Accuracy of Verbal, Quantitative, and Nonverbal Reasoning Tests: Consequences for Talent Identification and Program Diversity

Effective talent-identification procedures minimize the proportion of students whose subsequent performance indicates that they were mistakenly included in or excluded from the program. Classification errors occur when students who were predicted to excel subsequently do not excel or when students who were not predicted to excel do. Using a longitudinal sample, we assessed the accuracy of measures of verbal reasoning, quantitative reasoning, nonverbal reasoning, and current achievement for predicting later achievement. We found that seemingly small differences in predictive validity substantially changed the number of students erroneously included or excluded from the program. Surprisingly, nonverbal tests not only led to more classification errors but also failed to identify more English language learners and minority students. To increase equity and maintain fairness, practitioners should carefully evaluate claims that scores from alternative assessments are as valid as scores from conventional ability tests and verify that the use of these tests result in greater diversity.

Measuring the Retail Food Environment in Rural and Urban North Carolina Counties

Development of accurate and sensitive methods to characterize the food environment is needed. Thus, we examined convergent and criterion validity of 2 retail food environment data sources and then examined differences in predictive validity between 3 ways of measuring the rural and urban food environment. Ten counties were selected in each of 3 North Carolina regions (n = 30). Number of fast-food restaurants and chain supermarkets were calculated using 2 data sources. Convergent validity was percent agreement between the 2 sources. Criterion validity was percent agreement between each source and the most accurate venue count. Predictive validity of food environment measures (Retail Food Environment Index, fast-food restaurants/capita, and supermarkets/capita) was calculated by associations with county-level mean-weighted body mass index (BMI). Percent agreement for fast-food restaurants ranged from 50% to 100% (mean = 87%) and for supermarkets ranged from 58% to 100% (mean = 89%). The 2 data sources had similar percent agreement with the most accurate count. Retail Food Environment Index was positively associated with BMI, while fast-food restaurants per capita were negatively associated with BMI. Our results lend support to studies using both food environment data sources examined.

Long-Term Predictive Validity of the Risk Matrix 2000

The purpose of this study is to examine the predictive accuracy of the Risk Matrix 2000 on an independent sample of 351 sexual offenders, followed in the community for an average duration of 11.4 years (range 0-20 years, SD = 4.4 years). For comparison purposes, this study also examines the predictive accuracy of two other risk assessment instruments, specifically modified versions of the Static-99 and the Sex Offender Risk Appraisal Guide (SORAG). Results indicate that the Risk Matrix 2000 demonstrates convergent validity by correlating with the other risk assessment instruments. Moreover, the Risk Matrix 2000 is predictive of recidivism above chance levels, exhibiting medium to large effect sizes, although in general, the other two instruments, particularly the SORAG, are superior. Results also indicate differences in predictive validity when comparing 2-year, 5-year, and variable follow-up periods. Finally, a cumulative meta-analysis compares and integrates current findings with those obtained from the accumulation of previous studies, and new cumulative estimates are provided.

Generalizability of GMAT® Validity to Programs outside the U.S.

This study explores the predictive validity of GMAT® scores for predicting performance in graduate management programs outside the United States. Results suggest that the validity estimates based on the combination of GMAT® scores were about a third of a standard deviation higher for non-U.S. programs compared with existing data on U.S. programs. There also appeared to be differences for separate GMAT® sections such that verbal and writing scores had higher validity estimates for non-U.S. programs, but the validity estimates for quantitative scores were higher for U.S. programs. By meta-analytically combining data for more than 1,200 cases across six Western European programs, this study was able to explore the predictive validity of GMAT® scores for students with differing backgrounds, including gender, educational background, native language, and citizenship. The analyses show that predictive validity differences among groups have only a small impact on academic performance measured by grades.

The factor structure and concurrent validity of the alcohol use disorder identification test based on a nationally representative UK sample

To assess the psychometric structure and construct validity of the alcohol use disorder identification test (AUDIT) in a Great Britain population sample. A stratified multi-stage random sample of 7849 participants completed the AUDIT as part of a computer assisted interview. Confirmatory factor analyses were conducted testing one to three factor models. The factors in these models were correlated with demographic variables and scores relating to perceived wellbeing, verbal IQ, and neurotic and psychosis symptoms to assess construct validity of the factor solutions. A two factor solution was deemed to appropriately fit the data, measuring alcohol consumption and alcohol related problems. Correlations between the two factors on demographic, wellbeing, neurosis and psychosis symptomology were significantly different. The two factor solution suggests an advantage to investigating factor specific cut off scores for both consumption and alcohol related problems given their difference in predictive validity on both health and demographic variables.

Employee versus supervisor ratings of performance in the retail customer service sector: Differences in predictive validity for customer outcomes

Based on data from Maxham and Netemeyer [Maxham, J. G. and R. G. Netemeyer (2003). “Firms Reap What They Sow: The Effects of Employee Shared Values and Perceived Organizational Justice on Customer Evaluations of Complaint Handling,” Journal of Marketing, 67, 46–62], the authors present two field samples to examine predictive validity differences of service employee ratings of their performance versus supervisor ratings of employee performance with respect to customer satisfaction and customer likelihood of spreading positive word-of-mouth (WOM) after a service recovery attempt. The results generally show that supervisor ratings are more strongly positively related to customer satisfaction and WOM than are employee ratings of their own performances. The results also show that both supervisor ratings and employee ratings are related to customer satisfaction and WOM in a curvilinear fashion (as well as linear fashion). Employee extra-role performances (toward customers and the firm) show increasing returns at the higher levels of performance, and employee in-role customer performance generally shows a decreasing return at the higher level of customer in-role performance. These results suggest two managerial implications. First, supervisor ratings of customer service employee performances may be the preferred form of measurement for predicting customer outcomes. Second, maximizing in-role performance inputs may have decreasing returns for customer evaluations in the service recovery context; but maximizing extra-role performance inputs may actually “delight” customers, i.e., increasing returns for customer evaluations.