Observed Score Variance Research Articles

Extending Applications of Generalizability Theory-Based Bifactor Model Designs

In recent years, researchers have described how to analyze generalizability theory (GT) based univariate, multivariate, and bifactor designs using structural equation models. However, within GT studies of bifactor models, variance components have been limited to those reflecting relative differences in scores for norm-referencing purposes, with only limited guidance provided for estimating key indices when making changes to measurement procedures. In this article, we demonstrate how to derive variance components for multi-facet GT-based bifactor model designs that represent both relative and absolute differences in scores for norm- or criterion-referencing purposes using scores from selected scales within the recently expanded form of the Big Five Inventory (BFI-2). We further develop and apply prophecy formulas for determining how changes in numbers of items, numbers of occasions, and universes of generalization affect a wide variety of indices instrumental in determining the best ways to change measurement procedures for specific purposes. These indices include coefficients representing score generalizability and dependability; scale viability and added value; and proportions of observed score variance attributable to general factor effects, group factor effects, and individual sources of measurement error. To enable readers to apply these techniques, we provide detailed formulas, code in R, and sample data for conducting all demonstrated analyses within this article.

Reconceptualization of Coefficient Alpha Reliability for Test Summed and Scaled Scores

AbstractCoefficient alpha reliability persists as the most common reliability coefficient reported in research. The assumptions for its use are, however, not well‐understood. The current paper challenges the commonly used expressions of coefficient alpha and argues that while these expressions are correct when estimating reliability for summed scores, they are not appropriate to extend coefficient alpha to correctly estimate the reliability for nonlinearly transformed scaled scores such as percentile ranks and stanines. The current paper reconceptualizes coefficient alpha as a complement of the ratio of two unbiased estimates of the summed score variance. These include conditional summed score variance assuming uncorrelated item scores (gives the error score variance) and unconditional summed score variance incorporating intercorrelated item scores (gives the observed score variance). Using this reconceptualization, a new equation of coefficient generalized alpha is introduced for scaled scores. Coefficient alpha is a special case of this new equation since the latter reduces to coefficinet alpha if the scaled scores are the summed scores themselves. Two applications (cognitive and psychological assessments) are used to compare the performance (estimation and bootstrap confidence interval) of the reliability coefficients for different scaled scores. Results support the new equation of coefficient generalized alpha and compare it to coefficient generalized beta for parallel test forms. Coefficient generalized alpha produced different reliability values, which were larger than coefficient generalized beta for different scaled scores.

Improve Quality of Pain Measurement

Objectives: To review limitations of various single-item and multi-items scales of pain measurement and suggest remedial actions for better use of them leading to meaningful comparison of patients and group of patients in terms of pain scores across time and space. Method: Stages to obtain proposed score of a single item scale are: (1) Item Raw scores (2) Equidistant scores (3) Normalized equidistant scores (4) Conversion to a desired score range. For multi-item scale, further stages are (5) Summation of normalized equidistant scores with a desired score range. For transition from a stage to the next stage, method described along with empirical verification of transformation for an item to help clinicians to understand the main features of the proposed methods of scoring and to use them effectively. Results: The proposed method resulted in continuous, monotonic scores satisfying equidistant and normality conditions with a desired score range. Normalized equidistant scores help to compare patients’ scores from different distributions and facilitate application of statistical techniques in parametric set up. Conclusions: Proposed scores reflecting intensity of pain by continuous variable satisfying equidistant property. Normality, help meaningful comparison in terms of pain intensity, change in pain intensity and drawing path of progress for better prognostication. It is possible to compute split-half reliability and theoretical reliability as ratio of true score variance and observed score variance. Future studies suggested.

Reliability from α to ω: A tutorial.

Reliability is a fundamental problem for measurement in all of science. Although defined in multiple ways, and estimated in even more ways, the basic concepts seem straightforward and need to be understood by practitioners as well as methodologists. Reliability theory is not just for the psychometrician estimating latent variables, it is for everyone who wants to make inferences from measures of individuals or of groups. For the case of a single test administration, we consider multiple measures of reliability, ranging from the worst (β) to average (α, λ3) to best (λ4) split half reliabilities, and consider why model-based estimates (ωh, ωt) should be reported. We also address the utility of test-retest and alternate form reliabilities. The advantages of immediate versus delayed retests to decompose observed score variance into specific, state, and trait scores are discussed. But reliability is not just for test scores, it is also important when evaluating the use of ratings. Estimates that may be applied to continuous data include a set of intraclass correlations while discrete categorical data needs to take advantage of the family of κ statistics. Examples of these various reliability estimates are given using state and trait measures of anxiety given with different delays and under different conditions. An online supplemental materials is provided with more detail and elaboration. The online supplemental materials is also used to demonstrate applications of open source software to examples of real data, and comparisons are made between the many types of reliability. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

The internal structure of situational judgement tests reflects candidate main effects: Not dimensions or situations

Despite their popularity and capacity to predict performance, there is no clear consensus on the internal measurement characteristics of situational judgement tests (SJTs). Contemporary propositions in the literature focus on treating SJTs as methods, as measures of dimensions, or as measures of situational responses. However, empirical evidence relating to the internal structure of SJT scores is lacking. Using generalizability theory, we decomposed multiple sources of variance for three different SJTs used with different samples of job candidates (N1 = 2,320; N2 = 989; N3 = 7,934). Results consistently indicated that (1) the vast majority of reliable observed score variance reflected SJT‐specific candidate main effects, analogous to a general judgement factor, and that (2) the contribution of dimensions and situations to reliable SJT variance was, in relative terms, negligible. These findings do not align neatly with any of the proposals in the contemporary literature; however, they do suggest an internal structure for SJTs.Practitioner points To help optimize reliable variance, overall‐level aggregation should be used when scoring SJTs. The majority of reliable variance in SJTs reflects a general performance factor, relative to variance pertaining to specific dimensions or situations. SJT‐based developmental feedback should be delivered in terms of general SJT performance rather than on performance relating to specific dimensions or situations. Generalizability theory, although underutilized in organizational multifaceted measurement, offers an approach to informing on the psychometric properties of SJTs that is well suited to the complexities of SJT measurement designs.

Estimating true standard deviations.

Estimating true standard deviations.

An Application of Generalizability Theory to Evaluate the Technical Quality of an Alternate Assessment

Although federal regulations require testing students with severe cognitive disabilities, there is little guidance regarding how technical quality should be established. It is known that challenges exist with documentation of the reliability of scores for alternate assessments. Typical measures of reliability do little in modeling multiple sources of error, which are characteristic of alternate assessments. Instead, Generalizability theory (G-theory) allows researchers to identify sources of error and analyze the relative contribution of each source. This study demonstrates an application of G-theory to examine reliability for an alternate assessment. A G-study with the facets rater type, assessment attempts, and tasks was examined to determine the relative contribution of each to observed score variance. Results were used to determine the reliability of scores. The assessment design was modified to examine how changes might impact reliability. As a final step, designs that were deemed satisfactory were evaluated regarding the feasibility of adapting them into a statewide standardized assessment and accountability program.

Occasions and the Reliability of Classroom Observations: Alternative Conceptualizations and Methods of Analysis

Student–teacher interactions are dynamic relationships that change and evolve over the course of a school year. Measuring classroom quality through observations that focus on these interactions presents challenges when observations are conducted throughout the school year. Variability in observed scores could reflect true changes in the quality of student–teacher interaction or simply reflect measurement error. Classroom observation protocols should be designed to minimize measurement error while allowing measureable changes in the construct of interest. Treating occasions as fixed multivariate outcomes allows true changes to be separated from random measurement error. These outcomes may also be summarized through trend score composites to reflect different types of growth over the school year. We demonstrate the use of multivariate generalizability theory to estimate reliability for trend score composites, and we compare the results to traditional methods of analysis. Reliability estimates computed for average, linear, quadratic, and cubic trend scores from 118 classrooms participating in the MyTeachingPartner study indicate that universe scores account for between 57% and 88% of observed score variance.

How Cloudy a Crystal Ball: A Psychometric Assessment of Concept Testing

Although traditional and conjoint forms of concept testing play an important role in the new product development process, they largely ignore data quality issues, as evidenced by the traditional reliance on the percent Top-2-Box scores heuristic. The purpose of this research is to reconsider the design of concept testing from a measurement theory (generalizability theory) perspective and to use it to suggest some ways to improve the psychometric quality of concept testing. Generalizability theory is employed because it can account for the multiple facets of variation in concept testing, and it enables a concept test to be designed to provide a required level of accuracy for decision making in the most effective way, whether the purpose of measurement is to scale concepts or something else, such as to scale respondents. The paper identifies four types of sources—concept-related factors, response task factors, situational factors, and respondent factors—that can contribute to the observed variation in concept testing and develops six research propositions that summarize what is known or assumed about their contribution to observed score variance. Four secondary data sets from different concept testing contexts are then used to test the propositions. The results provide new insights into the design of concept tests and the psychometric quality of the concept testing data: (1) the concepts facet is not a major contributor to response variation; (2) of the response task factors, concept formulations are a trivial source of variance, but items are not always a trivial source of variance; (3) the situational factors that are investigated are trivial sources of variance; (4) respondents are always a major contributor to the total variation; (5) concepts by respondents are not always a major contributor and the other interactions are often not trivial; and (6) residual error is always a major source of variance. Additionally, the analyses of the secondary data sets enable some useful managerial conclusions to be drawn about the design of concept testing. First, the sample size needed to reliably scale concepts depends on the types of concepts being tested. Second, averaging over items provides considerably more reliable information than relying on a single item. Third, which specific item performs best is inconsistent and very context specific. The popular purchase intention item is never the best single item to use. Fourth, not much is gained by sampling levels of the response task factors. Finally, concept testing should be designed to meet the needs of specific managerial tasks.

Reliability and Attribute‐Based Scoring in Cognitive Diagnostic Assessment

The attribute hierarchy method (AHM) is a psychometric procedure for classifying examinees’ test item responses into a set of structured attribute patterns associated with different components from a cognitive model of task performance. Results from an AHM analysis yield information on examinees’ cognitive strengths and weaknesses. Hence, the AHM can be used for cognitive diagnostic assessment. The purpose of this study is to introduce and evaluate a new concept for assessing attribute reliability using the ratio of true score variance to observed score variance on items that probe specific cognitive attributes. This reliability procedure is evaluated and illustrated using both simulated data and student response data from a sample of algebra items taken from the March 2005 administration of the SAT. The reliability of diagnostic scores and the implications for practice are also discussed.

Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests

The assessment of cognitive abilities, whether it is for purposes of basic research or applied decision making, is potentially susceptible to both facilitating and debilitating influences. However, relatively little research has examined the degree to which these factors might moderate the criterion-related validity of cognitive ability tests. To address this gap, we use Classical Test Theory formulas to articulate how test anxiety and test familiarity can influence observed scores, observed score variance, and most importantly, the criterion-related validity of observed scores. The resulting equations reveal that understanding the influence of test anxiety and test familiarity on criterion-related validity coefficients requires the consideration of a number of additional parameters. To elucidate the implications of the model, we present a Monte Carlo simulation. Results show that anxiety and familiarity can have a significant negative effect on the observed criterion-related validity, but also show that this effect is highly variable. In particular, the effect depends heavily upon the relation between these factors and the criterion variable. Additionally, we note that the equations we develop highlight important gaps in the literature; there are few clear empirical estimates of several of the parameters in our formulas. We call for future research to better examine these additional relations.

Variability in Reliability Coefficients and the Standard Error of Measurement From School District to District

Measurement textbooks frequently suggest that reliability coefficients will be lower in more homogeneous groups. This expected relation rests on the assumption of constancy in the standard error of measurement (SEM) and that the lower observed-score variance is the result of lower true-score variance. However, the assumption of constancy in the SEM can be challenged on both theoretical and empirical grounds. It is possible that a more heterogeneous group might have a relatively large error score variance and hence, a lower reliability coefficient. In this study, the stability of the SEM and the relation between the reliability coefficient and the variance of both true scores and error scores were empirically examined for 170 school districts within a state. As expected, rXX´ was found to vary as a function of group variability. However, the coefficient was not found to be solely a function of the magnitude of true-score variance. Rather, it depended on error variance as well, and particularly on the ratio of true and error variances. Homogeneous subgroups almost always had lower reliability coefficients, but some instances were found in which the expected relation did not hold.

Dependability of Anchoring Labels of Likert-Type Scales

Generalizability theory was used to examine the dependability of anchoring labels of Likert-type scales. Variance components associated with labeling were estimated in two samples using a two-facet, random effects g-study design. In one sample, 173 graduate students in education were administered seven items measuring attitudes toward quantitative methodology. The other sample consisted of 108 graduate students in education who responded to the eight-item Life Orientation Test. From both samples, variance components associated with labeling were found to be minute, contributing little to the observed score variance. One plausible explanation was that respondents could primarily be using the numerical information in rating a Likert-type scale.

Reliability of Measurement and Power of Significance Tests Based on Differences

The power of significance tests based on differ ence scores is indirectly influenced by the reliability of the measures from which differences are obtained. Reliability depends on the relative magnitude of true score and error score variance, but statistical power is a function of the absolute magnitude of these components. Explicit power calculations reaffirm the paradox put forward by Overall & Woodward (1975, 1976)—that significance tests of differences can be powerful even if the reliability of the difference scores is 0. This anomaly arises because power is a function of observed score variance but is not a function of reliability unless either true score variance or error score variance is constant. Provided that sample size, significance level, directionality, and the alternative hypothesis associated with a significance test remain the same, power always increases when population variance decreases, independently of reliability.

Generalizability Theory as Construct-Related Evidence of the Validity of Job Performance Ratings

Investigations of the construct-related evidence of the validity of performance ratings have been rare, perhaps because researchers are dissuaded by the con- siderable amount of evidence needed to show construct validity (Landy, 1986). It is argued that generalizability (G) theory (Cronbach, Gleser, Nanda, & Rajaratnam, 1972) is well-suited to investigations of construct-related evi- dence of validity because a single generalizability investigation may provide multiple inferences of validity. G theory permits the researcher to partition observed score variance into universe (true) score variance and multiple, distinct estimates of error variance. G theory was illustrated through the anal- ysis of proficiency ratings of 256 Air Force jet engine mechanics. Mechanics were rated on three different rating forms by themselves, peers, and supervi- sors. Interpretation of G study variance components revealed suitable evi- dence of construct validity. Ratings within sources were reliable. Proficiency ratings showe...

Methodology Review: Assessing Unidimensionality of Tests and ltenls

Various methods for determining unidimensionality are reviewed and the rationale of these methods is as sessed. Indices based on answer patterns, reliability, components and factor analysis, and latent traits are reviewed. It is shown that many of the indices lack a rationale, and that many are adjustments of a previous index to take into account some criticisms of it. After reviewing many indices, it is suggested that those based on the size of residuals after fitting a two- or three-parameter latent trait model may be the most useful to detect unidimensionality. An attempt is made to clarify the term unidimensional, and it is shown how it differs from other terms often used inter changeably such as reliability, internal consistency, and homogeneity. Reliability is defined as the ratio of true score variance to observed score variance. Inter nal consistency denotes a group of methods that are intended to estimate reliability, are based on the vari ances and the covariances of test items, and depend on only one administration of a test. Homogeneity seems to refer more specifically to the similarity of the item correlations, but the term is often used as a synonym for unidimensionality. The usefulness of the terms in ternal consistency and homogeneity is questioned. Uni dimensionality is defined as the existence of one latent trait underlying the data.

ASSESSING THE RELIABILITY OF TESTS USED TO MAKE PASS/FAIL DECISIONS

This paper is about the reliability of tests having the following characteristics: 1. The test is made up of discrete items. The examinee's response to each item is scored as right or wrong; the examinee's score on the test is the number of responses scored as right. 2. Scores on the test are to be used as the basis for a pass/fail decision or, more generally, a classification of examinees into high-scoring and low-scoring groups. 3. The classification to be made on the basis of the test scores is acknowledged to be an artificial dichotomy, not an attempt to reproduce an underlying true dichotomy. That is, the difference between those examinees who pass the test and those who fail is assumed to be a difference in degree, not a difference in kind.' The first of these characteristics-discrete items, scored right or wrong and summed to produce a total score-is typical of most pencil-and-paper tests and of some performance tests and oral examinations as well. The second characteristic-a pass/fail or high/low classification-is typical of many tests used for making decisions at all educational levels, from the placement of elementary students in special classes to the certification of medical specialists. The third characteristic-the recognition that the pass/fail classification is artificial-applies to most pass/fail testing situations; the exceptions tend to involve very short tests of very specific instructional objectives. Together, the three characteristics describe a wide range of tests, including accreditation tests, placement tests, licensing examinations, and professional certification examinations. The traditional measures of test reliability are the reliability coefficient and the standard error of measurement. Each of these statistics describes a particular aspect of test reliability. The reliability coefficient has been defined in various ways. Some authors have chosen to define it as the correlation between observed scores on parallel test forms (Gulliksen, 1950; Stanley, 1971). Others define it as the ratio of true-score variance to observed-score variance (Cureton, 1958). And others define it as the squared correlation between true scores and observed scores (Lord & Novick, 1968). The standard error of measurement is the average (root mean square) size of the errors of measurement, averaged over the group of examinees from whose scores it is computed. These statistics have been generalized to take into account several independent sources of error (Cronbach, et al., 1972), but even in their newer, more general form their interpretation is basically the same. They do not provide the test user with information that directly

Sleep Recordings in the Laboratory and Home: A Comparative Analysis

ABSTRACTAll‐night sleep recordings were obtained for 8 subjects who slept for 3 consecutive nights in a standard sleep laboratory and for 3 consecutive nights at home. The order of recording locations was counterbalanced across subjects. No first night effects were found in either location. Subjects' sleep in the two locations was highly correlated, with nonsignificant correlations occurring on only four variables (Number of Stage 1 Arousals, Minutes of Stage 1 Sleep, Latency to First REM, and number of REM periods). Significant differences between mean values were found on four variables: Minutes of Stage 3 Sleep, Percent of Stage 2 Sleep, Percent of Stage 3 Sleep, and Number of REM periods. Variances on Total Sleep Time and Latency to Sleep Onset were greater in the laboratory than at home. Generalizability theory was used to generate estimates of the degree to which each experimental factor (subjects, occasions, and raters) influenced observed score variance in each location. There was less between and within‐subject variability at home on Total Sleep Time, Latency to Sleep Onset, and Stages 1 and 2 Sleep. Minutes Awake after Sleep Onset, Number of Arousals, and REM Sleep were less variable in the laboratory than at home. Taken together, these data suggest strong relationships between estimates of sleep parameters obtained in the two locations. But recordings in different locations may yield different values on some variables for groups of persons and for some individuals within those groups.

Equivalencies among Canonical Factor Analysis, Canonical Reliability Analysis, and Principal Components Analysis: Implications for Data Reduction of Fallible Measures

If variables are scaled so that they have equal errors of measurement, then the maximally reliable composites found in a canonical reliability analysis have highly desirable properties. Under this scaling transformation, canonical factor analysis on all nonerror variance, principal components analysis and canonical reliability analysis all yield equivalent results. Thus, the resulting components are successively maximally reliable and simultaneously provide a least squares fit to both the covariance matrix of rescaled observed scores and of rescaled true scores. In addition, the proportion of true to observed score variance retained in an s-dimensional least squares fit is equal to the ratio of the s-dimensional canonical reliability to the canonical reliability of the entire battery. Based on the above, it is argued that data reduction (when there is no a priori reason to keep the original units of measurement) should proceed by doing a components analysis on the covariance matrix rescaled so as to yield equal errors of measurement.

Generalizability of measures: A computer program for two- and three-facet designs

Generalizability Theory Classical test theory (Gulliksen, 1950) has been shown to be limited in handling a wide variety of measurement problems (Cronbach, Gleser, Nanda, & Rajaratnam, 1972). To overcome some of the problems, Cronbach et a1. (1972) developed a theory of the generalizability of behavioral measurements. This theory does not rely upon the restricted assumptions of the classical test theory. Furthermore, it isolates more than one source of error variation affecting the measurement. Generalizability theory distinguishes two types of studies: a G study and a D study. A G study is a pilot study carried out according to an elaborate design which includes all possible sources that may influence the generalizability of measurement. A D study collects data for the purpose of making decisions or drawing conclusions. One can choose the design for the D study from information provided in the G study, considering, of course, the D study's purposes and the cost of collecting information. In contrast to the classical test theory, generalizability theory distinguishes between absolute decisionsdecisions made on an individual's absolute score (e.g., 80% correct constitutes the criterion on a written driver's examination)-and relative decisions-decisions based upon rank ordering of individuals (e.g., correlational studies). This distinction is used to determine which sources of error variability affect the measurement. In generalizability theory the sources of error affecting the measurement (e.g., judges, observation occasions, items) are called facets (cf. factors in the analysis of variance). The conditions (cf. levels of a factor in the analysis of variance) representing these facets either constitute a random sample from the universe of conditions (a random facet) or exhaust the universe of conditions (a fixed facet). The persons on whom the measurements are taken are considered a random sample from a well-defined target population. A variability in the measurement due to a facet or an interaction among facets is defmed as error variance, while the variability among the persons is defmed as the universe score variance. An observed score variance is the sum of the universe score variance and the sampling error variances. The sampling error variance is defined as an average error variance where the average is taken over the number of conditions in which the source of error is sampled. The coefficient of generalizability (the counterpart of reliability coefficient) is defmed as the ratio of universe score variance to observed score variance.