Evidence For Validity Of Scores Research Articles

Internal Consistency Reliability and Internal Structure Validity of the English Versions of the Patient Health Questionnaire-9 and the Generalized Anxiety Disorder-7: A Psychometric Synthesis

We conducted a psychometric synthesis of the internal consistency reliability and internal structure of scores on the English versions of the Patient Health Questionnaire-9 (PHQ-9) and Generalized Anxiety Disorder-7 (GAD-7) in publications between 2012 and 2022. Results supported acceptable-to-strong reliability and validity evidence of scores among a number of samples.

Validity of a cardiology fellow performance assessment: reliability and associations with standardized examinations and awards

BackgroundMost work on the validity of clinical assessments for measuring learner performance in graduate medical education has occurred at the residency level. Minimal research exists on the validity of clinical assessments for measuring learner performance in advanced subspecialties. We sought to determine validity characteristics of cardiology fellows’ assessment scores during subspecialty training, which represents the largest subspecialty of internal medicine. Validity evidence included item content, internal consistency reliability, and associations between faculty-of-fellow clinical assessments and other pertinent variables.MethodsThis was a retrospective validation study exploring the domains of content, internal structure, and relations to other variables validity evidence for scores on faculty-of-fellow clinical assessments that include the 10-item Mayo Cardiology Fellows Assessment (MCFA-10). Participants included 7 cardiology fellowship classes. The MCFA-10 item content included questions previously validated in the assessment of internal medicine residents. Internal structure evidence was assessed through Cronbach’s α. The outcome for relations to other variables evidence was overall mean of faculty-of-fellow assessment score (scale 1–5). Independent variables included common measures of fellow performance.FindingsParticipants included 65 cardiology fellows. The overall mean ± standard deviation faculty-of-fellow assessment score was 4.07 ± 0.18. Content evidence for the MCFA-10 scores was based on published literature and core competencies. Cronbach’s α was 0.98, suggesting high internal consistency reliability and offering evidence for internal structure validity. In multivariable analysis to provide relations to other variables evidence, mean assessment scores were independently associated with in-training examination scores (beta = 0.088 per 10-point increase; p = 0.05) and receiving a departmental or institutional award (beta = 0.152; p = 0.001). Assessment scores were not associated with educational conference attendance, compliance with completion of required evaluations, faculty appointment upon completion of training, or performance on the board certification exam. R2 for the multivariable model was 0.25.ConclusionsThese findings provide sound validity evidence establishing item content, internal consistency reliability, and associations with other variables for faculty-of-fellow clinical assessment scores that include MCFA-10 items during cardiology fellowship. Relations to other variables evidence included associations of assessment scores with performance on the in-training examination and receipt of competitive awards. These data support the utility of the MCFA-10 as a measure of performance during cardiology training and could serve as the foundation for future research on the assessment of subspecialty learners.

Psychometric Synthesis of the Parent‐Child Relationship Inventory

Assessment in counseling is strengthened when research exists to demonstrate the psychometric strength of scores from a particular instrument across samples. This study presents a psychometric synthesis of data across 52 publications that met criteria and reported use of the Parent‐Child Relationship Inventory (PCRI; Gerard, 1994) across disciplines and in numerous contexts. Results revealed low score reliability, with aggregated alphas of .63 to .83 and test‐retest coefficients of .62 to .79 over a 1‐year retest period across subscales and total score. There was little evidence of score validity, including no exploratory or confirmatory factor analytic studies, or evidence of diagnostic validity. Cautions for use of the PCRI and implications for professional counselors and researchers are discussed.

Beliefs and Evaluations About Counseling Services (BEACS): Scale Development

We developed the Beliefs and Evaluations About Counseling Services (BEACS) scale by incorporating a mathematical formula adopted from the Theory of Reasoned Action. We conducted three studies using distinctive samples for item generation, exploratory and confirmatory factor analyses, and obtained the validity evidence for scores from the scale.

Validity Evidence based on Internal Structure of Scores of the Emotional Quotient-Inventory: Youth Version Short (EQ-i: YV-S) in a Spanish Sample.

The purpose of this study was to analyze the reliability and validity evidence of scores on the Spanish version of EQ-i: YV-S in Spanish adolescents. The total sample was comprised of 508 participants from Grades 7 to 12, 241 males (47.4%) and 267 females (52.6%), each of whom completed the questionnaires on two separate occasions. Three [intrapersonal (α = .83, CR = .86, and McDonald Omega = .86), stress management (α = .83, CR = .86, and McDonald Omega = .85) and adaptability (α = .82, CR = .85, and McDonald Omega = .85)] of the four scales had acceptable internal consistency. Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) were used with FACTOR and EQS version 6.1 software to examine validity evidence based on internal structure drawn from the scores on the EQ-i: YV-S, supporting the multidimensionality of the questionnaire. Three models were tested; the best fit to the data was the hierarchical model (S-Bχ2 / df = 2.11, CFI = .93 and RMSEA = .047), which hypothesized that the four specific factors (interpersonal, intrapersonal, stress management, and adaptability) were explained with a second-order factor, Emotional-Social-Intelligence (ESI). Finally, significant positive correlations were found between general self-concept and EQ-i: YV-S [interpersonal (r = .153, p < .001), intrapersonal (r = .235, p < .001), stress management (r = .145, p < .001), adaptability (r = .311, p < .001) and ESI (r = .360, p < .001)]; ESI showed significant direct power prediction of the general self-concept (.52) as demonstrated through structural equation modeling.

Reliability and Validity Evidence of Scores on the French Version of the Questionnaire about Interpersonal Difficulties for Adolescents.

This study examined the reliability and validity evidence drawn from the scores of the French version of the Questionnaire about Interpersonal Difficulties for Adolescents (QIDA) in a sample of 957 adolescents (48.5% boys) ranging in age from 11 to 18 years (M = 14.48, SD = 1.85). A principal axis factoring (PAF) and confirmatory factor analyses (CFA) were performed to determine the fit of the factor structure of scores on the QIDA. PAF and CFA replicated the previously identified correlated five-factor structure of the QIDA: Assertiveness, Heterosexual Relationships, Public Speaking, Family Relationships, and Close Friendships. The QIDA yielded acceptable reliability scores for French adolescents. Validity evidence of QIDA was also established through correlations with scores on the School Anxiety Inventory and the Social Anxiety Scale for Adolescents. Most of the correlations were positive and exceeded the established criteria of statistical significance, but the magnitude of these varied according to the scales of the QIDA. Results supported the reliability and validity evidence drawn from the scores of the French version of the QIDA.

The Questionnaire about Interpersonal Difficulties for Adolescents: Reliability and validity evidence in Colombian adolescents

Este objetivo de este estudio fue analizar la evidencia de fiabilidad y validez de las puntuaciones de la version colombiana del Cuestionario de Dificultades Interpersonales para Adolescentes (QIDA) en una muestra de 1.628 adolescentes (65.7% chicos) de 12 a 18 anos. Los analisis factoriales confirmatorios replicaron la estructura factorial de cinco factores relacionados: Asertividad, Relaciones Heterosexuales, Hablar en Publico, Relaciones Familiares y Amigos Intimos. La consistencia interna del QIDA y sus subescalas fue excelente. Las chicas informaron mayores niveles de ansiedad percibida en relaciones heterosexuales, mientras los chicos mostraron mas ansiedad en relaciones con amigos intimos. La ansiedad interpersonal disminuyo durante la adolescencia. Los resultados apoyan la fiabilidad y validez de las puntuaciones de la version colombiana del QIDA.

Validity of the AM-PAC “6-Clicks” Inpatient Daily Activity and Basic Mobility Short Forms

Standardized assessment of patients' activity limitations in acute care settings can provide valuable information. Existing measures have not been widely implemented. The aim of this study was to provide evidence for validity of scores on Activity Measure for Post-Acute Care (AM-PAC) "6-Clicks" measures of basic mobility and daily activity in acute care. A retrospective measurement study was conducted. The study used a database from one health system containing "6-Clicks" scores from first and last physical therapist and occupational therapist visits for 84,446 patients. Validity was analyzed by examining differences in "6-Clicks" scores across categories of patient characteristics; the ability of "6-Clicks" scores to predict patients' having more than one therapy visit; correlation of "6-Clicks" scores with Functional Independence Measure (FIM) scores; and internal responsiveness over the episode of care. Internal consistency reliability also was determined. The "6-Clicks" scores differed across patients' age, preadmission living situation, and number of therapy visits. The areas under receiver operating characteristic curves derived using "6-Clicks" scores at the first visit to predict patients receiving more than one visit were 0.703 and 0.652 using basic mobility and daily activity scores, respectively. The "6-Clicks" scores at the final visit were correlated with scores on subscales of the FIM completed on admission to inpatient rehabilitation facilities (r=.65 and .69). Standardized response means were 1.06 and 0.95 and minimal detectable changes with 90% confidence level (MDC90) were 4.72 and 5.49 for basic mobility and daily activity scores, respectively. Internal consistency reliability of basic mobility and daily activity scores was .96 and .91, respectively. Using clinical databases for research purposes has limitations, including missing data, misclassifications, and selection bias. Rater reliability is not known. This study provides evidence for the validity of "6-Clicks" scores for assessing patients' activity limitations in acute care settings.

Validity Evidence based on Internal Structure of Scores on the Spanish Version of the Self-Description Questionnaire-II

The aim of this study was to analyze the reliability and validity evidence of scores on the Spanish version of Self-Description Questionnaire II (SDQ-II). The instrument was administered in a sample of 2022 Spanish students (51.1% boys) from grades 7 to 10. Confirmatory factor analysis (CFA) was used to examine validity evidence based on internal structure drawn from the scores on the SDQ-II. CFA replicated the correlated II first-order factor structure. Furthermore, hierarchical confirmatory factor analysis (HCFA) was used to examine the hierarchical ordering of self-concept, as measured by scores on the Spanish version of the SDQ-II. Although a series of HCFA models were tested to assess academic and non-academic components organization, support for those hierarchical models was weaker than for the correlated 11 first-order factor structure. Results also indicated that scores on the Spanish version of the SDQ-II had internal consistency and test-retest reliability estimates within an acceptable range.

Developing abbreviated OHIP versions for use with TMD patients

The aim of this study was to evaluate the psychometric characteristics of three versions of the Dutch Oral Health Impact Profile (OHIP-NL), for clinical use with temporomandibular disorder (TMD) patients. To that end, two abbreviated OHIP versions (OHIP-NL14 and OHIP-NL5) were developed by respectively selecting 14 and five items from the officially translated and culturally adapted original 49-item OHIP-NL questionnaire. A total of 245 consecutive patients, referred by their dentist to the TMD clinic of the Academic Centre for Dentistry Amsterdam (77% women; mean age ± s.d. = 41·0 ± 14·9 years), completed the Research Diagnostic Criteria for TMD (RDC/TMD) axis II questionnaire and the OHIP-NL. Reliability and validity of all three OHIP versions were compared, and their associations with four psychological axis II variables, indicating the level of impairment of patients with TMD, were examined. According to guidelines for clinical application, internal consistency scores were sufficient for OHIP-NL and OHIP-NL14, but insufficient for OHIP-NL5. Test-retest reliability (n = 64) was excellent for OHIP-NL and OHIP-NL14 and fair to good for OHIP-NL5. For all three versions, there was evidence for score validity: associations between OHIP summary scores on the one hand and validation variables and other RDC/TMD axis II variables on the other hand met the expectations and were statistically significant (P < 0·001). In conclusion, the OHIP-NL and OHIP-NL14 both performed comparatively well and better than the OHIP-NL5. When the length of the questionnaire (i.e. the time needed for its completion) is an issue, the OHIP-14 would therefore be the preferred version.

Reliability and Validity Evidence of Scores on the Slovene Version of the Questionnaire About Interpersonal Difficulties for Adolescents

This study analyzed the psychometric properties of scores on the Slovene version of the Questionnaire about Interpersonal Difficulties for Adolescents (QIDA) in a sample of 1,334 adolescents (44% boys), ranging in age from 12 to 18 years (M = 15.61). Confirmatory factor analyses replicated the correlated five-factor structure of the QIDA: Assertiveness, Heterosexual Relationships, Public Speaking, Family Relationships, and Close Friendships. Internal consistency and test-retest reliability were reasonable. Correlations of scores on the QIDA with scores of neuroticism, low extraversion, and low openness, as measured by the Inventory of Child/Adolescent Individual Differences, and scores of fear of negative evaluation, and tension and inhibition in social contacts, as measured by the Social Anxiety Scale for Adolescents were found, revealing differential links with QIDA subscale scores. Girls reported more difficulties than boys. Age differences showed a small but significant decrease in QIDA total score over adolescence.

The Factor Structure of the CIBS-II-Readiness Assessment

The Brigance Comprehensive Inventory of Basic Skills-II (CIBS-II)-Readiness form is a diagnostic battery intended for children aged 5 and 6 years. The CIBS-II-Readiness is a new version of the CIBS-Revised-Readiness and includes updated normative information on a larger representative sample in comparison to the CIBS-Revised-Readiness. Empirical support for the CIBS-Revised-Readiness was lacking, especially score validity evidence. Adequate score validity information needs to be provided. The purpose of this study was to investigate evidence of construct validity of the composite scores on the CIBS-II-Readiness by examining the internal structure of the instrument and correlations with external measures. Confirmatory factor analysis was employed to compare the model implied by the composite score structure of the assessment to other plausible models. A six-factor model was supported. CIBS-II-Readiness scores had significant correlations with the Dynamic Indicators of Basic Early Literacy Skills scores, another widely used measure assessing five major components of early literacy.

What Do Alternate Assessments of Alternate Academic Achievement Standards Measure? A Multitrait—Multimethod Analysis

This study featured validity evidence for scores from states' alternate assessments of alternate academic achievement standards (AA-AASs). It evaluated students from 6 states who were eligible for an AA-AAS concurrently with measures of academic competence and adaptive behavior. The investigators also assessed students with disabilities who were not eligible for an AA-AAS by using the same measures, as well as by using general achievement tests. The main findings included that AA-AAS reading and math scores may reflect a unitary construct, that AA-AAS scores are highly related to adaptive behavior but also relate to academic competence and achievement, and that all these scores represent unique but overlapping constructs. These results have implications for AA-AAS developers and teachers working with students with significant cognitive disabilities.

Reliability and validity evidence of scores on the Achievement Goal Tendencies Questionnaire in a sample of Spanish students of compulsory secondary education

AbstractThis study examined the reliability and validity evidence drawn from the scores of the Spanish version of the Achievement Goal Tendencies Questionnaire (AGTQ) using a sample of 2,022 (51.1% boys) Spanish students from grades 7 to 10. Confirmatory factor analysis replicated the correlated three‐factor structure of the AGTQ in this sample: Learning Goals (LG), Social Reinforcement Goals (SRG), and Performance Goals (PG). The AGTQ yields relatively reliable scores for Spanish students. Logistic regression analyses were used to provide additional validity evidence of AGTQ scores on academic success (general, Spanish, and mathematics). Results indicated that general academic success was 15% and 5% more likely every time that scores on LG and PG scales increased 1 point, respectively, although general academic success was 9% less likely every time that scores on SRG scale increased 1 point. These results were similar for Spanish and mathematics. However, the score on the LG scale did not forecast success in both subjects. © 2009 Wiley Periodicals, Inc.

Development and Psychometric Assessment of the Collaborative Care for Attention-Deficit Disorders Scale

To describe the development and assess the validity and reliability of the Collaborative Care for Attention-Deficit Disorders Scale (CCADDS), a measure of collaborative care processes for children with attention-deficit/hyperactivity disorder who attend primary care practices. Collaborative care was conceptualized as a multidimensional construct. The 41-item CCADDS was developed from an existing instrument, review of the literature, focus groups, and an expert panel. The CCADDS was field tested in a national mail survey of 600 stratified and randomly selected practicing general pediatricians. Psychometric analysis included assessments of factor structure, construct validity, and internal consistency. The overall response rate was 51%. Most respondents were male (56%), 46 years old or older (59%), and white (69%). Common factor analysis identified 3 subscales: beliefs, collaborative activities, and connectedness. Internal consistency reliability (coefficient alpha) for the overall scale was .91, and subscale scores ranged from .80 to .89. The CCADDS correlated with a validated measure of provider psychosocial orientation (r = -.36, P < .001) and with self-reported frequency of mental health referrals or consultations (r = -.24 to -.42, P < .001). CCADDS scores were similar among physicians by race/ethnicity, gender, age group, and practice location. Scores on the CCADDS were reliable for measuring collaborative care processes in this sample of primary care clinicians who provide treatment for children with attention-deficit/hyperactivity disorder. Evidence for validity of scores was limited. Future research is needed to confirm its psychometric properties and factor structure and provide guidance on score interpretation.

Brief Form of the Social Phobia and Anxiety Inventory (SPAI-B) for Adolescents

Brief Form of the Social Phobia and Anxiety Inventory (SPAI-B) for Adolescents

Reliability and Construct Validity of the Pain Distress Inventory

We conducted three studies to evaluate further the reliability and construct validity of a new self-report instrument, the Pain Distress Inventory (PDI; Osman et al., 2003, The Pain Distress Inventory: Development and initial psychometric properties, J. Clin. Psychol. 59: 767-785). In Study I, exploratory and confirmatory factor analytic results confirmed the replicability of the four-factor oblique solution of the PDI in a mixed sample of students and nonstudents. We also found strong evidence for criterion-related validity of scores on this instrument. In Study II, multisample analyses results found further evidence for equivalence of structure of the PDI across African American and Caucasian young adults. Ethnic and gender group differences were obtained on two of the PDI scale scores. Internal consistency reliability estimates on the PDI total and scale scores were good in both Studies I and II. In Study III, additional analyses of internal consistency and known-groups validity established strong support for construct validity of the PDI.

Self-report on the social skills rating system: Analysis of reliability and validity for an elementary sample

The Social Skills Rating System (SSRS; F.M. Gresham & S.N. Elliott, 1990) is a norm-referenced measure of students' social and problem behaviors. Since its release, much of the published reliability and validity evidence for the SSRS has focused primarily on the Teacher Report Form. The purpose of this study was to explore reliability and validity evidence of scores on the SSRS-Student Elementary Form (SSRS-SEF) for children in Grades 3 to 5. Findings provided support for the use of Total scale as a measure of student social behavior for initial screening purposes; however, evidence for the subscales was not as strong as predicted. Directions for future research regarding reliability and validity of scores from the SSRS-SEF are discussed. © 2005 Wiley Periodicals, Inc. Psychol Schs 42: 345–354, 2005.

Construct Validation of a Revised Strain-Free Negative Affectivity Scale

Based on research demonstrating that many extant measures of negative affectivity (NA) are contaminated with strain content, Fortunato and Stone-Romero developed the Strain-Free Negative Affectivity Scale (SFNA). However, because of several psychometric concerns involving scores on the original scale, a revision of the SFNA was undertaken. The SFNA-Revised (R) was administered to two samples of participants, along with instruments purportedly measuring neuroticism, negative emotionality, trait negative affect, affect intensity, affective disposition, extroversion, and positive affectivity. Estimates of reliability and factor analytic findings indicate that scores on the SFNA-R exhibited better psychometric properties than scores on the original scale. In addition, evidence for the construct validity of scores on the SFNA-R is presented. Based on the above, the authors conclude that the SFNA-R is a worthy alternative to extant measures of NA.

Development of the Physical Therapy Outpatient Satisfaction Survey (PTOPS)

The purposes of this 3-phase study were (1) to identify the underlying components of outpatient satisfaction in physical therapy and (2) to develop a test that would yield reliable and valid measurements of these components. Three samples, consisting of 177, 257, and 173 outpatients from 21 facilities, were used in phases 1, 2, and 3, respectively. In phase 1, principal component analyses (PCAs), reliability checks, and correlations with social desirability scales were used to reduce a pool of 98 items to 32 items. These analyses identified a 5-component model of outpatient satisfaction in physical therapy. The phase 2 PCA, with a revised pool of 48 items, indicated that 4 components rather than 5 components represented the best model and resulted in the 34-item Physical Therapy Outpatient Satisfaction Survey (PTOPS). Factor analyses conducted with phase 2 and phase 3 data supported this conclusion and provided evidence for the internal validity of the PTOPS scores. The 4-component scales were labeled "Enhancers," "Detractors," "Location," and "Cost." Responses from subsamples of phase 3 subjects provided evidence for validity of scores in that the PTOPS components of "Enhancers," "Detractors," and "Cost" appeared to differentiate overtly satisfied patients from overtly dissatisfied patients. "Location" and "Enhancer" scores discriminated subjects with excellent attendance at scheduled physical therapy sessions from those with poor attendance. In this study, we identified components of outpatient satisfaction in physical therapy and used them to develop a test that would yield valid and reliable measurements of these components.