Evidence Of Incremental Validity Research Articles

Do self-reported decision styles relate with others’ impressions of decision quality?

Research has demonstrated the usefulness of decision styles for predicting various performance-related criteria. It is still unclear, however, which particular styles are associated with a general tendency to make high-quality decisions. Participants (n=168) completed a common measure of five decision styles, along with a measure of the traits in the five-factor model of personality. Self and peer evaluations of general decision quality were obtained as performance criteria. Results showed that specific decision styles predicted variance in both measures of decision quality. And, there was clear evidence for incremental validity for specific decision styles when self-ratings were predicted. In the context of past research, this study supports a primary focus on rational or analytical styles for understanding and predicting decision success.

Development and validation of the Customer-Centered Behavior measure

The purpose of this study was to develop and validate a new measure of customer service job performance – the Customer-Centered Behavior (CCB) measure. The CCB measure draws on the content of over 15 existing measures, but captures in-role service performance focused on customer interactions in a multidimensional framework. Three studies were conducted to develop and validate the new measure. Study 1 addresses the item generation process and examines the initial factor structure. Results demonstrate that service behaviors are best represented as a second-order construct composed of three first-order factors – assurance, responsiveness, and recommendation behaviors. Study 2 confirms the factor structure and provides evidence for the measure's predictive validity. Finally, Study 3 provides evidence for incremental validity above other service performance measures.

Trait Emotional Intelligence as a Measure of Introspective Limits: Re-Analysis of the Trait Meta Mood Scale and Evidence of Incremental Validity in Prediction of Symptom Underreporting Among Men

Trait emotional intelligence (EI) is related to mental, psychosomatic, and physical health as well as to work performance. However, given the construct’s associations with the Big Five and alexithymia, investigations on its incremental validity appear paramount. Yet, we propose that trait EI may also serve as a measure of introspective limits that may account for well-known sex differences in self-reports of mental health. This hypothesis was tested with the Trait Meta Mood Scale (TMMS) in an Austrian community sample (N=364). Preliminary structural and psychometric analyses suggested that the TMMS is better represented by a factor structure that allows for a method factor for negatively keyed items. Validity analyses confirmed expected links with the Big Five and alexithymia. Alexithymia was also associated with the tendency to score higher on the negatively keyed items of the TMMS. Incremental validity of trait EI over and above the Big Five and alexithymia in the prediction of mental health was found only for men and only for the TMMS Attention scale. Further analyses confirmed that Attention moderated sex differences in self-reports of mental health, suggesting that emotional self-awareness may be conceived as a response bias indicator among men.

Adapting social neuroscience measures for schizophrenia clinical trials, part 3: fathoming external validity.

It is unknown whether measures adapted from social neuroscience linked to specific neural systems will demonstrate relationships to external variables. Four paradigms adapted from social neuroscience were administered to 173 clinically stable outpatients with schizophrenia to determine their relationships to functionally meaningful variables and to investigate their incremental validity beyond standard measures of social and nonsocial cognition. The 4 paradigms included 2 that assess perception of nonverbal social and action cues (basic biological motion and emotion in biological motion) and 2 that involve higher level inferences about self and others' mental states (self-referential memory and empathic accuracy). Overall, social neuroscience paradigms showed significant relationships to functional capacity but weak relationships to community functioning; the paradigms also showed weak correlations to clinical symptoms. Evidence for incremental validity beyond standard measures of social and nonsocial cognition was mixed with additional predictive power shown for functional capacity but not community functioning. Of the newly adapted paradigms, the empathic accuracy task had the broadest external validity. These results underscore the difficulty of translating developments from neuroscience into clinically useful tasks with functional significance.

The development and validation of the Incivility from Customers Scale.

Scant research has examined customers as sources of workplace incivility, despite evidence suggesting that mistreatment is more common from organizational outsiders, including customers, than from organizational members (Grandey, Kern, & Frone, 2007; Schat & Kelloway, 2005). As an important step in extending the literature on customer incivility, we conducted two studies to develop and validate a measure of this construct. Study 1 used focus groups of retail and restaurant employees (n = 30) to elicit a list of uncivil customer behaviors, based on which we wrote initial scale items. Study 2 used a correlational survey design (n = 439) to pare down the number of scale items to 10 and to garner reliability and validity evidence for the scale. Exploratory and confirmatory factor analyses show that the scale is unidimensional and distinguishable from measures of the related, but distinct, constructs of interpersonal justice and psychological aggression from customers. Reliability analyses show that the scale is internally consistent. Significant correlations between the scale and individuals' job satisfaction, turnover intentions, and general and job-specific psychological strain provide evidence of criterion-related validity. Hierarchical regression analyses show that the scale significantly predicts three of four organizational and personal strain outcomes over and above a workplace incivility measure adapted for customer incivility, providing some evidence of incremental validity. Limitations and future research directions are discussed.

Evidence of Incremental Validity between Pfister Test and Human Figure Drawing

The aim of this study was to seek incremental evidence of validity between the Color Pyramid Test (CPT) and the Human Figure Drawing (HFD). The incremental validity of information obtained fro m a g iven test can predict some- thing unique, which other existing measures or constructs cannot predict. The sample consisted of 118 children fro m public schools in the state of Sao Paulo, Brazil, of which 59 (50%) were male, aged between 6 and 12 years (M = 9.22, SD = 1.75) and education ranging from pre-kindergarten to seventh grade. The tests were ad ministered on each child without prede- termined time for their imp lementation. The results showed significant correlations between the two tests, which contrib- uted to the evidence of incremental valid ity between the two tests. It is suggested that further studies on the evidence of validity and accuracy, with controlled samp les and other indicators,not explored in this study, will contribute to the pro jec- tive techniques surveyed.

Assessment of Romantic Perfectionism

The objective of the present study was to provide validity evidence for the scores from the Romantic Relationship Perfectionism Scale. Results indicate a two-factor structure, adequate reliability, and overall good convergent, concurrent, discriminant, and incremental validity evidence. The strengths and limitations of this measure are discussed.

Inappropriate and excessive guilt: instrument validation and developmental differences in relation to depression.

Inappropriate or excessive guilt is listed as a symptom of depression by the American Psychiatric Association (1994). Although many measures of guilt have been developed, definitional and operational problems exist, especially in the application of such measures in childhood and adolescence. To address these problems, the current study introduces the Inappropriate and Excessive Guilt Scale (IEGS), assesses its validity for use with children and adolescents, and tests its relation to depression across development. From a sample of 370 children between 7 and 16 years old, results provided (1) evidence that items designed to assess inappropriate and excessive guilt converged onto a single underlying factor, (2) support for the convergent, discriminant, and construct validity of the IEGS in a general youth population, and (3) evidence of incremental validity of the IEGS over-and-above other measures of guilt. Results also supported the hypothesis that inappropriate and excessive guilt as well as negative cognitive errors become less normative and more depressotypic with age.

Resilience and Successful Community Reintegration among Female Forensic Psychiatric Patients: A Preliminary Investigation

Research on resiliency and recovery in forensic psychiatric patients is still limited. Information pertaining to factors associated with successful community reintegration would contribute to a more comprehensive assessment of functioning and informed treatment planning that fits within a recovery approach of service provision. Using a retrospective design involving file reviews and a 3-year follow-up period, the authors investigated the rate of successful/unsuccessful community reintegration (defined by the presence or absence of an absolute discharge/readmission to hospital) in female forensic psychiatric patients (N = 48). The study evaluated the extent to which the risk and protective factors captured in the Short-Term Assessment of Risk and Treatability (START) predicted a range of positive and negative outcomes in the study sample. Results showed that 47.9% of the women qualified as having successfully reintegrated into the community, with the remaining 52.1% qualifying as still being in recovery. Successful individuals possessed significantly more protective factors and significantly fewer risk factors than individuals still in recovery. Furthermore, both the vulnerability and the strength scale of the START demonstrated good predictive validity, however we did not find evidence of incremental validity of the strength scale.

Mindfulness Skills and Anxiety-Related Cognitive Processes Among Young Adult Daily Smokers: A Pilot Test

We examined specific mindfulness skills (observing, describing, acting with awareness, accepting without judgment, as measured by the Kentucky Inventory of Mindfulness Skills, in terms of anxiety-related cognitive processes among adult daily smokers (n = 90; 43 females; Mage = 26.6 years, SD = 11.8). Partially consistent with hypotheses, describing and accepting without judgment were each shown to significantly predict perceived control over anxiety-related events. The observed significant effects were evident above and beyond the variance accounted for by gender, smoking rate, and negative affectivity. Although observing also was shown to significantly predict agoraphobic cognition, it was in the opposite direction as was theoretically expected. No evidence of incremental validity for mindfulness skills was evident for anxiety sensitivity. These data highlight the potential explanatory relevance of only specific mindfulness skills in terms of only certain anxiety-based cognitive processes among adult daily smokers.

Predictive validity of the Biomedical Admissions Test: An evaluation and case study

There has been an increase in the use of pre-admission selection tests for medicine. Such tests need to show good psychometric properties. Here, we use a paper by Emery and Bell [2009. The predictive validity of the Biomedical Admissions Test for pre-clinical examination performance. Med Educ 43:557–564] as a case study to evaluate and comment on the reporting of psychometric data in the field of medical student selection (and the comments apply to many papers in the field). We highlight pitfalls when reliability data are not presented, how simple zero-order associations can lead to inaccurate conclusions about the predictive validity of a test, and how biases need to be explored and reported. We show with BMAT that it is the knowledge part of the test which does all the predictive work. We show that without evidence of incremental validity it is difficult to assess the value of any selection tests for medicine.

Clinical Usefulness of the Oppositional Defiant Disorder Rating Scale (ODDRS)

The present study examined the reliability, validity, and clinical utility of the Oppositional Defiant Disorder Rating Scale (ODDRS) in a population of children referred for ADHD evaluation. The diagnostic benefit of using a rating scale specific to Oppositional Defiant Disorder (ODD), in addition to a broad range behavior scale, was also investigated. Parents and teachers of 177 clinic-referred children provided ratings on the ODDRS. The average correlation between parent and teacher ratings was minimal (r = .13). Children diagnosed with ODD through structured parent interviews were found to have elevated scores on the parent version of the ODDRS, as compared to children without a research diagnosis of ODD. However teacher ratings did not discriminate between the ODD and no-ODD groups. There was minimal evidence of incremental validity as classification rates for the ODDRS were almost identical to classification rates based on the Aggression subscale of the Behavior Assessment System for Children (BASC-2).

Development and validation of a Coping with Discrimination Scale: Factor structure, reliability, and validity.

Four studies were conducted to develop and validate the Coping With Discrimination Scale (CDS). In Study 1, an exploratory factor analysis (N = 328) identified 5 factors: Education/Advocacy, Internalization, Drug and Alcohol Use, Resistance, and Detachment, with internal consistency reliability estimates ranging from .72 to .90. In Study 2, a confirmatory factor analysis (N = 328) provided cross-validation of the 5-factor model as well as evidence for validity of the scale. The validity evidence was similar across racial groups and for males and females. In Study 3, the estimated 2-week test-retest reliabilities (N = 53) were between .48 and .85 for the 5 factors. Education/Advocacy, Internalization, Drug and Alcohol Use, and Detachment were positively associated with active coping, self-blame, substance use, and behavioral disengagement, respectively, providing further support for validity of the CDS. Finally, incremental validity evidence was obtained in Study 4 (N = 220), where it was shown that the CDS explained variance in outcome variables (i.e., depression, life satisfaction, self-esteem, and ethnic identity) that could not be explained by general coping strategies.

The Examination for Professional Practice in Psychology (EPPP) in the era of evidence-based practice.

Professional psychology has increasingly moved toward evidence-based practice. However, instrumentsused to assess psychologists seeking licensure, such as the Examination for Professional Practice inPsychology (EPPP), have received relatively little empirical scrutiny. Therefore, the authors evaluatedthe available evidence in support of the EPPP’s validity and current use as a core component ofprofessional licensure. Although the EPPP has in many ways been extensively evaluated, there is apaucity of criterion, predictive, and incremental validity evidence available. Further, several aspects ofthe content validation studies were examined, and the authors question whether the EPPP, as currentlyconstructed, can meet its stated goals. Given that the EPPP is a high-stakes examination and given theauthors’ best estimate (based on a sample of 16 states) that 35% of applicants fail the examination, it isrecommended that the EPPP be more extensively evaluated. An outline of major decision points in thisproposed evaluation process is provided, several suggestions for further research are proposed, and thefield is encouraged to discuss these issues further.Keywords: ExaminationforProfessionalPracticeinPsychology,EPPP,licensure,educationandtraining,competence

A MULTIDIMENSIONAL APPROACH FOR EVALUATING VARIABLES IN ORGANIZATIONAL RESEARCH AND PRACTICE

One of the most difficult tasks facing industrial‐organizational psychologists is evaluating the importance of variables, especially new variables, to be included in the prediction of some outcome. When multiple regression is used, common practices suggest evaluating the usefulness of new variables by showing incremental validity beyond the set of existing variables. This approach assures that the new variables are not statistically redundant with this existing set, but this approach attributes any shared criterion‐related validity to the existing set of variables and none to the new variables. More importantly, incremental validity alone fails to answer the question directly about the importance of variables included in a regression model—arguably the more important statistical concern for practitioners. To that end, the current article reviews 2 indices of relative importance, general dominance weights and relative weights, which may be used to complement incremental validity evidence and permit organizational decision makers to make more precise and informed decisions concerning the usefulness of predictor variables. We illustrate our approach by reanalyzing the correlation matrices from 2 published studies.

The Validity of Assessment Center Ratings and 16PF Personality Trait Scores in Police Sergeant Promotions: A Case of Incremental Validity

Assessment centers (AC) have become a mainstay of promotion systems for both the private and public sectors--police agencies included. (1) While costly to develop and implement, their track record of validity and legal defensibility has justified their increased use. (2) Similarly, the use of personality trait measurement in personnel decisions has also shown a substantial increase in use based on its validity in predicting a range of employee and managerial job behaviors. (3) Unlike ACs, police agencies have used personality assessment primarily for entry level officer screening and fitness for duty assessments. (4) The use of personality assessment in police officer promotional decisions has rarely been reported. Personality trait scores have shown incremental validity, however, when combined with cognitive ability tests in predicting a range of employee and managerial behaviors, (5) but not for state police recruits. (6) In addition, incremental validity has been found for personality measures when combined with biodata (7) and ACs. (8) Other than the attempt of Cortina, Coherty, Schmitt, Kaufmann and Smith, (9) who found no evidence of incremental validity, there have been no other reported studies of incremental validity for personality trait scores when used in combination with any other predictor of police officer or supervisor performance. Evidence of Assessment Center Validity There is substantial literature which documents the validity of ACs in predicting on-the-job performance, especially for managers. (10) In addition to its proven validity, based on candidate reactions and court challenges, the AC method has been labeled fair and legally defensible. (11) The validity of ACs used for police officer selection and promotion has been shown to be commensurate with that found in other organizational settings. For example, AC performance ratings have been shown to be significantly related to supervisor ratings of officer performance, (12) valid for high ranking police officials (13) and accurate in prediction over a 19-year time span. (14) Evidence of Validity for Personality Measurement While early reviews of the literature indicated that personality assessment yielded poor validity in predicting job performance, (15) more recent work has established personality as a bona fide predictor of a range of job-related behaviors across a variety of organizational settings and employee groups, including police officers. (16) A number of meta-analyses have provided reasonable support for the use of personality assessment in selection. (17) The decision to use personality assessment by itself within an employee selection context should be made with caution, given the greater validity shown by other predictor measures, such as cognitive ability tests and assessment centers. The resurgence of personality assessment as a valid predictor of job performance is due in large part to the fact that: 1. Personality has been shown to be a valid predictor of work-related outcomes; 2. Personality measures do not generally display adverse impact against demographic subgroups (e.g., racial, gender, ethnic, etc.); 3. The validity of personality measures is not affected by intentional faking; (18) and 4. The construct of personality has become more structured with the acceptance of the personality traits (e.g., conscientiousness and extraversion) and the consistency with which they are measured. (19) Many studies have dealt specifically with the ability of personality trait scores to predict police officer performance, both on the job and while in training. Personality traits have been measured through traditional clinical self report inventories as well as instruments designed to assess the Big Five dimensions (e.g., the Minnesota Multiphasic Personality Inventory; (20) the Inwald Personality Inventory; (21) the California Psychological Inventory; (22) the Eyesenck Personality Inventory, the SCL-90, and the Bender-Gestalt; (23) the 16PF; (24) and the NEO Personality Inventory (25)). …

Responsibility to continue thinking and worrying: Evidence of incremental validity

This two-part study examined the role of the responsibility to continue thinking, beliefs that one needs prolonged thinking about stressful problems, in the prediction of excessive worry. This construct is considered to reflect high levels of motivation to continue inflexible thinking and the use of rigid stop rules. In Study 1, 122 students completed questionnaires. A regression analysis revealed that responsibility accounted for a unique variance beyond negative meta-cognitive beliefs about worry. One hundred and fifty students participated in Study 2, where worry was regressed on emotional instability (Neuroticism), responsibility, and other worry-related cognitive variables (intolerance of uncertainty, positive/negative meta-cognitive beliefs, poor problem-solving orientation, and cognitive avoidance). Again, responsibility was a significant predictor, after controlling for emotional instability (Neuroticism) and other worry-related cognitions. These results indicate the incremental validity of the responsibility to continue thinking.

Predictive validity of four nicotine dependence measures in a college sample

Background The present study compared the predictive and incremental validity of four commonly used dependence measures (Diagnostic and Statistical Manual-IV [DSM-IV] nicotine dependence criteria, Fagerstrom Test for Nicotine Dependence [FTND], Hooked On Nicotine Checklist [HONC], Nicotine Dependence Syndrome Scale [NDSS]) in a first year college sample reporting light smoking patterns. Methods Nicotine dependence measures were administered at the end of the first semester and follow-up smoking behavior (i.e. continued smoking, quantity, frequency, and length of abstinence) was assessed at the end of the first and second academic years. Results Higher levels of dependence as measured by the HONC and DSM-IV predicted smoking behavior at both follow-up assessments. While higher scores on some of the NDSS factors predicted heavier smoking behavior during follow-up assessments, higher scores on other NDSS factors predicted lighter smoking behavior. The DSM-IV, NDSS-priority, and HONC measures provided some evidence for incremental validity. Higher dependence scores on all four measures were related to shorter lengths of smoking abstinence. Conclusions The four dependence measures were differentially related to smoking behavior outcomes in a light smoking sample. These findings suggest that nicotine dependence can predict a variety of smoking behaviors in light smokers.

Is Complicated Grief a Valid Construct?

Important progress has been made in conceptualizing and demonstrating effective interventions for complicated grief (CG). The author argues, however, that the CG construct is still poorly understood, and that there remain several important but unanswered questions about the construct's basic validity. Recent research is reviewed that provides preliminary evidence for the CG's incremental validity over other forms of grief-related psychopathology. However, further evidence of incremental validity is still needed. Several prospective studies are also reviewed that support the discriminant validity of CG in relation to more enduring depressive symptoms. However, most studies of CG have failed to make this distinction and have tended to confound CG with longer-term depressive states. The author argues that future research on CG, including intervention studies, needs to allot greater attention to these validity issues.

Hardiness Scales in Iranian Managers: Evidence of Incremental Validity in Relationships with the Five Factor Model and with Organizational and Psychological Adjustment

This study examined the incremental validity of Hardiness scales in a sample of Iranian managers. Along with measures of the Five Factor Model and of Organizational and Psychological Adjustment, Hardiness scales were administered to 159 male managers (M age = 39.9, SD = 7.5) who had worked in their organizations for 7.9 yr. (SD=5.4). Hardiness predicted greater Job Satisfaction, higher Organization-based Self-esteem, and perceptions of the work environment as being less stressful and constraining. Hardiness also correlated positively with Assertiveness, Emotional Stability, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness and negatively with Depression, Anxiety, Perceived Stress, Chance External Control, and a Powerful Others External Control. Evidence of incremental validity was obtained when the Hardiness scales supplemented the Five Factor Model in predicting organizational and psychological adjustment. These data documented the incremental validity of the Hardiness scales in a non-Western sample and thus confirmed once again that Hardiness has a relevance that extends beyond the culture in which it was developed.