Reliability Estimates For Scores Research Articles

Choosing Between Cronbach’s Coefficient Alpha, McDonald’s Coefficient Omega, and Coefficient H: Confidence Intervals and the Advantages and Drawbacks of Interpretive Guidelines

The purpose of this instructional piece was to provide a nontechnical synthesis of common internal consistency reliability estimates used in professional counseling and in related fields. The article begins with an overview of coefficients alpha, omega, omega hierarchical, and H, with guidelines for their selection. Next, I provide recommendations for interpretive cutoff scores for higher and lower stakes testing followed by commentary on the limitations of relying too heavily on such guidelines. I discuss the importance of reporting confidence intervals (CIs) for reliability estimates to enhance reliability generalizations. When evaluating internal consistency reliability estimates of scores based on sample data counselors are advised to (a) determine the intended use of test scores in terms of higher or lower stakes testing, (b) take a multifaceted approach and consider the requirements of each reliability index, (c) include CIs, and (d) refer to interpretive cutoff scores as tentative general guidelines, not absolute standards.

Evaluating a Brief Big Five Personality Test in a Diverse Chinese Sample: The Role of Midpoint Designs and Reversely-Worded Items

Response styles present a threat to the validity of data from Likert scales. This study evaluates a brief personality test administered by interviewers to a diverse Chinese sample, focusing on two factors with direct impacts on response styles, the midpoint designs and the inclusion of reversely-worded items. We randomly assign respondents into three midpoint designs: masked midpoint, explicit midpoint and removed midpoint. Cronbach’s alpha tends to be low and is not drastically affected by midpoint designs. Score reliability estimates from the removed and explicit midpoint designs are above or close to .60 along the whole score range for openness, but not the other subscales. The masked midpoint design presents some advantages in criterion relationship validity evidence based on correlations with demographic, behavioral, subjective and cognitive variables. Dropping reversely-worded items increases the estimates of Cronbach’s alpha, but shows no positive effects on the criterion relationship validity. In conclusion, for interviewer-administered brief personality scales applied to Chinese respondents, masked midpoint design that aims to reduce a possible overuse of midpoint presents some advantages in the overall psychometric properties, but the inclusion of reversely-worded items shows mixed effects.

Development and preliminary validation of the evaluation use scale for evaluation systems (EUS-ES)

ABSTRACT The usefulness of evaluation is critical. Evaluation use occurs when, from its results or process, decisions are made about the program, it changes people's mindsets, or persuasive or legitimation actions happen (instrumental, conceptual, and symbolic uses respectively). Few quantitative evaluation use studies have been conducted in recent years. Most existing evaluation use scales are outdated due to theoretical advances and were project or program specific, not designed for evaluation systems. This study was designed to develop the Evaluation Use Scale for Evaluation Systems (EUS-ES) and gather initial validity evidence. We collected and analyzed data from Brazilian professors and program coordinators who were consultants of the Brazilian Graduate Evaluation System. Validity evidence was collected based on instrument content and internal structure. CFA results supported a three-factor-correlated model with adequate score reliability estimates. The EUS-ES is valuable for diagnostic purposes to identify low/high evaluation uses in an evaluation system setting.

Statistical Methods Used In Educational Technology Research 2012-2013

This article provides a content analysis of the research methodologies used in quantitative and mixed-methods articles in the top five educational technology journals between 2012 and 2013. These articles represented a total of 32,131 sampling procedures and statistical techniques recorded from 1,171 articles – the largest research synthesis of research methodologies in field of educational technology to date. Results indicate quantitative methods continue to dominate the field as a whole, yet specific journals appear to favor certain research methods over others. Most authors did not report the type of sampling procedure used in their investigations (617 articles). Fewer researchers reported score reliability estimates using their own data – with only 420 articles reporting reliability coefficients. Findings also suggest few authors reported informationally-adequate statistics. Recommendations for best statistical practices and implications for the field of educational technology are discussed.

Assessing Online Viewing Practices among College Students

This article focuses on media literacy education for college students. First, we conducted psychometric analyses to verify the properties of the Critical Evaluation and Analysis of Media (CEAM) scale. CEAM measures college students’ self-reported practices for critically evaluating and analyzing the credibility, audience, and technical design elements of online media, such as news, advertisement, and entertainment media. Using CEAM, our second goal was to identify trends in critical viewing practices among first-year students enrolled in college. Results of confirmatory factor analysis (CFA) and item response theory (IRT) supported a three-factor structure for the CEAM scale. Composite score reliability for all items comprising the total scale displayed strong evidence for the internal consistency of the scale with a Coefficient Alpha (α) of .91. Score reliability estimates for each subscale follow: (a) Questioning Credibility (α = .80), (b) Recognizing Audience (α = .78), and (c) Recognizing Design (α = .81). Findings from the study indicate that while first-year college students generally perceive they have adequate practices in recognizing audience in media messages and questioning the credibility of news, there is room for improvement in questioning the credibility of advertisements, suggesting that college instructors should focus more on advertising literacy.

Further evidence for score reliability and validity of the Anxiety Depression Distress Inventory-27 scale

ABSTRACT We conducted several analyses with data from undergraduate students (N = 309) and active duty US Air Force Security Forces personnel (N = 273) to examine the utility of scores on the Anxiety Depression Distress Inventory-27 (ADDI-27). The three specific dimensions of the ADDI-27 include Positive Affect, Somatic Anxiety, and General Distress, each composed of nine relevant and representative items. Internal consistency reliability estimates for scores on the scales were strong across the study groups (coefficient-α values ≥ .80). Results of analyses using differential item functioning showed that the groups interpreted the contents of the ADDI-27 items similarly. At the scale-level analyses, the undergraduate student sample reported higher somatic and general distress symptoms compared to the Air Force sample. Evidence for concurrent validity was adequate. Examples of the concurrent measures were the meaning in life, mental health functioning, and sources of social support self-report instruments. Taken together, results support the use of the ADDI-27 for assessing clusters of somatic anxiety, depression, and general distress in the current study samples.

Psychometric Fundamentals of the Social Skills Improvement System: Social–Emotional Learning Edition Rating Forms

This study described the development of the Social Skills Improvement System Social Emotional Learning Edition Rating Forms (SSIS SEL RF) for teachers, parents, and students. This new multirater assessment is a reconfiguration of the SSIS Rating Scales items inspired by the CASEL Social Emotional Competency framework. The internal structure and score reliability estimates were examined across three raters for a common sample of more than 200 individual children ages 3 to 18 years. Confirmatory factor analyses tested against the CASEL five-dimensional SEL theoretical model demonstrated adequate fit for the SSIS SEL Parent and Student RFs and mediocre fit of the Teacher RF. Internal consistency, test-retest, and interrater reliability estimates for scores on each of the SSIS SEL RFs all met or exceeded acceptable criteria. Thus, researchers and practitioners interested in measuring the social–emotional behavior of children ages 3 to 18 can expect reliable scores and structurally meaningful behavior content within the Collaborative on Academic Social Emotional Learning (CASEL) SEL competency framework. Limitations to the present findings and suggestions for future research conclude the report.

Variability of Coefficient Alpha: An Empirical Investigation of the Scales of Psychological Wellbeing

Using reliability generalization analysis, the purpose of this study was to characterize the average score reliability, the variability of the score reliability estimates, and explore possible characteristics (e.g., sample size) that influence the reliability of scores across studies using the Scales of Psychological Wellbeing (PWB; Ryff, 1989 , 2014 ). Published studies were included in this investigation if they appeared in a peer-reviewed journal, used 1 or more PWB subscales, estimated coefficient alpha value(s) for the PWB subscale(s), and were written in English. Of the 924 articles generated by the search strategy, a total of 264 were included in the final sample for meta-analysis. The average value reported for coefficient alpha referencing the composite PWB Scale was 0.858, with mean coefficient alphas ranging from 0.722 for the autonomy subscale to 0.801 for the self-acceptance subscale. The 95% prediction intervals ranged from [.653, .996] for the composite PWB. The lower bound of the prediction intervals for specific subscales were >.350. Moderator analyses revealed significant differences in score reliability estimates across select sample and test characteristics. Most notably, R2 values linked with test length ranged from 40% to 71%. Concerns were identified with the use of the 3-item per PWB subscale which reinforces claims advanced by Ryff (2014) . Suggestions for researchers using the PWB are advanced which span measurement considerations and standards of reporting. Psychological researchers who calculate score reliability estimates within their own work should recognize the implications of alpha coefficient values on validity, null hypothesis significant testing, and effect sizes.

Psychometric properties of the Short Inventory of Problems (SIP) with adjudicated DUI intervention participants.

We used responses of two large samples of court-ordered participants from a statewide alcohol/driving safety program to investigate factor structure, score reliability, and criterion-related validity of the Short Inventory of Problems (SIP). Exploratory and confirmatory factor analyses, using both item-level and subscore-level data, support a one-factor structure for the SIP. Internal consistency score reliability estimates were consistent across samples and high enough to warrant use for making decisions about individuals. Item response theory model calibration of the scale, using a two-parameter logistic model, yielded consistent estimates of location and discrimination (slope) across samples. Estimated scale scores correlated moderately with an independent indicator of alcohol problems and poorly with an indicator of risky driving behavior, lending evidence of convergent and discriminant validity. We judge the SIP as adequately described by a single factor, that the joint person-item scale is coherent, and scores behave consistently across samples. (PsycINFO Database Record

Decisional balance and self-efficacy for sun protection: measurement among Turkish adolescents.

Sun protection is important for skin cancer prevention, but many adolescents do not protect themselves from the sun. Instrumentation derived from the transtheoretical model (TTM) can be used to study the process of change in health behaviors like sun protection. The purpose of this study was to translate and adapt TTM-based decisional balance and self-efficacy for sun protection scales from English to Turkish and assess psychometric properties of scores when the scales are used among Turkish adolescents. The Decisional Balance Scale (DBS) and the Self-Efficacy Scale (SES) for sun protection were adapted to Turkish culture using translation and back-translation. The scales were administered to a total of 900 adolescents in two Turkish schools. Confirmatory factor analysis was used to assess dimensionality. External validity was evaluated by comparing subscale scores across reported stages of change for sun protection. Reliability estimates for scores on the DBS Pros and DBS Cons and the SES Sunscreen Use scales were high and SES Hat Use and Sun Avoidance were moderate. The two-factor correlated model for the DBS and the three-factor correlated model for the SES reported in other studies were confirmed. Means increased across the stages of change for sun protection and sunscreen use for the DBS Pros and the SES subscales as predicted by the TTM, but the pattern of DBS Cons means did not. Scores from the Turkish version of the DBS and SES for sun protection were valid, reliable, and appropriate for Turkish culture. The pattern of means for the SES and DBS Pros across the stages of change supported propositions of the TTM. Theoretical inconsistencies in the pattern of DBS Cons scores across the stages of change suggest that greater attention to conceptualization and measurement of the DBS Cons for sun protection and sunscreen use is needed.

Does the Length of a Questionnaire Matter?

Short personality questionnaires are increasingly used in research and practice, with some scales including as few as two to five items per personality domain. Despite the frequency of their use, these short scales are often criticized on the basis of their reduced internal consistencies and their purported failure to assess the breadth of broad constructs, such as the Big 5 factors of personality. One reason for this might be the use of principles routed in Classical Test Theory during test construction. In this study, Generalizability Theory is used to compare psychometric properties of different scales based on the NEO-PI-R and BFI, two widely-used personality questionnaire families. Applying both Classical Test Theory (CTT) and Generalizability Theory (GT) allowed to identify the inner workings of test shortening. CTT-based analyses indicated that longer is generally better for reliability, while GT allowed differentiation between reliability for relative and absolute decisions, while revealing how different variance sources affect test score reliability estimates. These variance sources differed with scale length, and only GT allowed clear description of these internal consequences, allowing more effective identification of advantages and disadvantages of shorter and longer scales. Most importantly, the findings highlight the potential error proneness of focusing solely on reliability and scale length in test construction. Practical as well as theoretical consequences are discussed.

Stakeholder Input and Test Design: A Case Study on Changing the Interlocutor Familiarity Facet of the Group Oral Discussion Test

Test takers should have a voice in testing practices (Mathew, 2004). However, when incorporating their input, systematic processes to ensure the validity of testing practices must be followed. Such processes allow for test development to be a more democratic process (Shohamy, 2001), without sacrificing the value of the resulting inferences made from the test scores. This article describes a case study that incorporated the desires of test takers to change the procedures of a group oral discussion test in a university English as a Foreign Language program. A study was designed to determine the extent to which the proposed changes would threaten the validity of the testing process. Specifically, the procedures for the group oral were altered to investigate the effect of interlocutor familiarity. Students were randomly assigned to class-familiar (n = 146) and class-unfamiliar (n = 159) groups to identify to what extent group familiarity affected test takers' scores in the four assessed categories: pronunciation, fluency, lexis and grammar, and communication skills. For the two groups, no statistically significant difference in scores was found, and score reliability estimates were similar. The implications of the findings are addressed in terms of recommendations for using stakeholder input in the assessment design process.

Evaluating fidelity to the wraparound service model for youth: application of item response theory to the Wraparound Fidelity Index.

The wraparound process is a mechanism for multisystem planning and care coordination for youth with serious emotional and behavioral problems. Fidelity monitoring is critical to effective implementation of evidence-based practices in children's mental health, as it helps ensure that complex interventions like wraparound are implemented as intended. The 40-item Wraparound Fidelity Index, Version 4 (WFI-4; Bruns, Burchard, Suter, Leverentz-Brady, & Force, 2004) is the most frequently used measure of fidelity to the wraparound process, but analysis of its psychometric properties is insufficient. An item response theory approach, Rasch partial credit models for ordered polytomous data, was used on ratings from 1,234 facilitators, 1,006 caregivers, and 221 team members, focused on 1,478 youths (55% male). Results indicated the WFI-4 measured a unidimensional construct, with little evidence of item bias and good item and model fit. However, the item information curve was skewed, with most people endorsing high-fidelity responses, and several items had duplicative location estimates. A reduced 20-item measure is proposed. Internal reliability estimates for scores from this reduced measure were approximately equivalent to the longer measure. However, both versions would benefit from additional items located in the highest fidelity area of either version of the scale where scores by greater than half of our sample fall, but only 3 items are located.

The First Twenty Years of the Will and the Ways: An Examination of Score Reliability Distribution on Snyder’s Dispositional Hope Scale

C. R. Snyder has established hope theory as an important contributor to positive psychology. As the empirical evidence continues to grow, hope researchers need to have confidence that their measures will produce reliable scores. This study presents a reliability generalization on both the internal consistency and test–retest reliability estimates from Snyder’s dispositional hope scale. While over 300 published works were found to have cited the target article 74 present internal consistency scores and 17 reported scores for test–retest reliability. The results of the reliability generalization suggest support for the score reliabilities produced by the dispositional hope scale. However, internal consistency was higher for studies using the eight-item response format (α = 0.82) compared to those using the four-item response format (α = 0.77). Additionally, the test–retest score reliability was high 0.80 with no statistically significant differences by response format. Findings also demonstrated that score reliability estimates were not significantly influenced by the coded sample characteristics.

Factor structure of child behavior scale scores in peruvian preschoolers

AbstractBehavior rating scales aid in the identification of problem behaviors, as well as the development of interventions to reduce such behavior. Although scores on many behavior rating scales have been validated in the United States, there have been few such studies in other cultural contexts. In this study, the structural validity of scores on a Spanish translation of the six‐factor Child Behavior Scale (CBS) was assessed in a sample of 265 Peruvian preschool children who ranged from 2 to 6 years in age. Exploratory factor analysis yielded a four‐factor structure, and reliability estimates for scores on the four factors were adequate. The authors suggest replicating the study and examining the utility of CBS scores in predicting future problem behaviors in this population. © 2011 Wiley Periodicals, Inc.

HOW DO RATERS FROM INDIA PERFORM IN SCORING THE TOEFL IBT™ SPEAKING SECTION AND WHAT KIND OF TRAINING HELPS?

ABSTRACTThis study investigated the scoring of the Test of English as a Foreign Language™ Internet‐based Test (TOEFL iBT™) Speaking section by bilingual or multilingual speakers of English and 1 or more Indian languages. We explored the extent to which raters from India, after being trained and certified, were able to score the Speaking section for TOEFL iBT examinees with mixed first language (L1) backgrounds, especially those speaking an Indian language, accurately and consistently. The effectiveness of a special training package designed for scoring Indian examinees was examined as well. A total of 26 trained and certified raters from India were randomly divided into 2 groups and participated in 2 on‐site scoring sessions in Mumbai. In the first session, both groups received regular training for scoring the TOEFL iBT Speaking section, which was largely similar to that received by raters in North America. In the second scoring session, 1 group continued to receive the regular training while the second group was trained using a special training package. Rater feedback surveys were also given to the raters. It was found that with training similar to that which operational U.S.‐based raters receive, the raters from India performed as well as the operational raters in scoring both Indian and non‐Indian examinees. In addition, the special training helped the raters score Indian examinees more consistently, leading to increased score reliability estimates. It also boosted raters' levels of confidence in scoring Indian examinees.

Score Reliability of Adolescent Alcohol Screening Measures: A Meta-Analytic Inquiry

ABSTRACT This study describes the reliability reporting practices in empirical studies using eight adolescent alcohol screening tools and characterizes and explores variability in internal consistency estimates across samples. Of 119 observed administrations of these instruments, 40 (34%) reported usable reliability information. The Personal Experience Screening Questionnaire—Problem Severity scale generated average reliability estimates exceeding 0.90 (95% CI = 0.90–0.96) and the Adolescent Alcohol Involvement Scale generated average score reliability estimates below 0.80 (95% CI = 0.67–0.85). Average reliability estimates of the remaining instruments were distributed between these extremes. Sample characteristics were identified as potentially important predictors of variability in the reliability estimates of all the instruments and all instruments under evaluation generated more reliable scores in clinical settings (M = 0.89) as opposed to nonclinical settings (M = 0.82; r effect size (38) = 0.29, p < .10). Clinicians facing instrument selection decisions can use these data to guide their choices and researchers evaluating the performance of these instruments can use these data to inform their future studies.

Substance Use Scales of the Minnesota Multiphasic Personality Inventory

Three drug and alcohol use screening scales are embedded within the Minnesota Multiphasic Personality Inventory—2: the MacAndrew Alcoholism Scale (MAC) and its revised version (MAC-R), the Addiction Acknowledgement Scale (AAS), and the Addiction Potential Scale (APS). The current study evaluated the reliability reporting practices among 210 studies administering the MAC/MAC-R, APS, and/or AAS. Furthermore, reliability generalization methods were used to characterize the previously reported reliability estimates associated with each instrument. The vast majority of studies (90.6%) did not provide measurement reliability data, suggesting a need for improved psychometric reporting. Data from the remaining studies yielded mean and median score reliability estimates below .70 for each of the identified measures. Although limited in some instances by sample size constraints, results suggest that these instruments tend not to produce scores with acceptable levels of reliability for most research or clinical situations.

A Rasch Perspective

Measurement error is a common theme in classical measurement models used in testing and assessment. In classical measurement models, the definition of measurement error and the subsequent reliability coefficients differ on the basis of the test administration design. Internal consistency reliability specifies error due primarily to poor item sampling. Rater reliability indicates error due to inconsistency among raters. For estimates of test-retest reliability, error is attributed mainly to changes over time. In alternate-forms reliability, error is assumed to be due largely to variation between samples of items on test forms. Rasch models can also compute reliability estimates of scores under different test situations. The authors therefore present the Rasch perspective on calculating reliability (measurement error) and present Rasch measurement model programs to compute the various reliability estimates.

The Balanced Inventory of Desirable Responding (BIDR)

The Balanced Inventory of Desirable Responding (BIDR) is one of the most widely used social desirability scales. The authors conducted a reliability generalization study to examine the typical reliability coefficients of BIDR scores and explored factors that explained the variability of reliability estimates across studies. The results indicated that the overall BIDR scale produced scores that were adequately reliable. The mean score reliability estimates for the two subscales, Self-Deception Enhancement and Impression Management, were not satisfactory. In addition, although a number of study characteristics were statistically significantly related to reliability estimates, they accounted for only a small portion of the overall variability in reliability estimates. The results of these findings and their implications are also discussed.