Criterion-referenced Measurement Research Articles

Reliability estimates for criterion-referenced measures in the psychomotor domain.

Abstract The purpose of this study was to examine procedures for estimating the reliability for a criterion-referenced measure in the psychomotor domain. Reliability is defined as the consistency of classification of examinees into mastery and nonmastery categories. Three trial mastery criteria—6, 7, 8—were utilized along with three test mastery criteria—.6n, .7n, .8n. Motor skill was defined as first-ball scores of each frame in a line of bowling. Since the empirical distribution functions for men and women subjects were significantly different at trial criteria of 6 and 7, separate reliability coefficients were estimated for each sex. The single administration estimates of reliability developed by Huynh and Subkoviak were equally good indicators of the Swaminathan-Hambleton-Algina estimate of P when the test was administered on 2 days (P represents the proportion of agreement of classifications). Variations in trial and test mastery criteria yielded different proportions of subjects assigned to mastery ...

Validity and Reliability of Criterion-Referenced Measures: Issues and Procedures for Special Educators

The controversy over norm-referenced versus criterion-referenced measures is of particular interest to special educators. Over the years, special educators have been encouraged to use CRMs in assessment of learning disorders in order to avoid the biasing and other undesirable features of standardized tests. However, as this article points out, the choice between the two types of measures is not as simple as it might seem. While the reliability of CRMs is generally indisputable, the validity of a measure designed from local curricula is still dependent on the intuitive judgments of teachers. In discussing the advantages and disadvantages of both types of tests, Harris and Wolf emphasize that tests should be selected according to the intended use of the scores, rather than being an all-or-nothing decision. Ultimately, the decision depends on the degree of faith one places in global constructs versus local curricula.

Measures for Psychological Assessment: A Guide to 3,000 Original Sources and Their Applications. Ki-Taek Chun, Sidney Cable, and John R. P. French, Jr. / Identification of Specific Learning Disabilities. E. LaMonte Ohlson / Measurement and Evaluation in the Schools (2nd ed). Louis J. Karmel and Marylin O. Karmel / Psychological and Educational Assessment of Minority Children. Thomas Oakland, Editor /

<i>Measures for Psychological Assessment: A Guide to 3,000 Original Sources and Their Applications.</i> Ki-Taek Chun, Sidney Cable, and John R. P. French, Jr. / <i>Identification of Specific Learning Disabilities.</i> E. LaMonte Ohlson / <i>Measurement and Evaluation in the Schools</i> (2nd ed). Louis J. Karmel and Marylin O. Karmel / <i>Psychological and Educational Assessment of Minority Children.</i> Thomas Oakland, Editor /

Direct Instruction Research in Australia

A series of studies in metropolitan and rural areas involving Direct Instruction methods (Becker, Engelmann, and Thomas, 1971, 1975) have been conducted in Australian schools over the past eight years. These studies involved the use of DISTAR®, Corrective Reading, and Morphographic Spelling instructional programs with children in regular classes, special classes, withdrawal classes (e.g., pull out groups), and special schools at the elementary and junior high levels. Retarded, migrant, aboriginal, learning disabled, and socially disadvantaged children have participated in these investigations. The results demonstrate that the Direct Instruction programs have been effective across settings and populations when independently evaluated against norm-referenced and criterion-referenced dependent measures. In addition, the results exceeded the usual expectations held for these populations in relation to academic and intellectual achievement The positive learning outcomes achieved in all these studies, many of which were longitudinal, highlight the fact that there is now a body of data-based evaluations across settings and populations that show Direct Instruction programs to be effective. The implications for educational administrators, principals, teachers, teacher educators, and parents is clear. Effective instructional programs are now available which can reduce learning gaps, and in many instances completely close the gap between “normal” academic and intellectual development and learning deficits. The body of empirical data on Direct Instruction which now exists offers new options for the administrators, principals, teachers, and teacher educators who seek the best in teaching strategies for children.

A Reply to Marx

It might be profitable if I attempt to answer briefly what I judge to be Marx' three main criticisms of my paper: (i) misconstrues the role of the 'criterion' in criterion-referenced measurement, (2) accepts too readily the nature of the cognitive processes presumed to underlie different types of test items, (3) his tacit assumption instructional feedback plays a role as a reinforcer. In answer to the first criticism it should be recalled the initial discussion took place within a framework of public school and extension of the curricula, and there is little question those responsible for a core curriculum, at least in British Columbia, view it in terms of mastery requirements for most students. Consequently, the use of criterion referenced which suggests that judgements concerning an individual's achievement are dichotomous would not be inconsistent with the context of the discussion. However, I believe a rereading of the paper will show the focus in this case was not on standards of achievement but on a well-defined behavioral domain. Whether or not several performance standards are established with regard to the learning outcomes specified in a core curriculum is irrelevant to the thrust of the argument. I am simply suggesting because a core curriculum usually defines rather carefully the knowledge and skills expected of the students, this logically requires a criterion-referenced interpretation of performance on test items which reflect as closely as possible those learning outcomes described in core curriculum. This brings me to Marx' second criticism I accept too readily the nature of the cognitive processes presumed to underlie different types of test items. While this is not completely true, I will state, without hesitation, different types of test item, for example, completion versus multiple choice, frequently do require different cognitive processes. There are three ways to support this statement: by findings on item difficulty, by introspection, and by studies designed to measure this cognitive difference. Consider first the simple case of the recall of a specific fact in a sentence-completion item versus the recognition of this same fact in a

On Test Purpose and Item Type: A Comment on Mason

The concept of criterion-referenced measurement is certainly not new, as pointed out by Airasian and Madaus (1977), but since almost all educational measurement up to the sixties, and even most current usage, has been based on models developed in differential psychology, the criterion-referenced model is seen by most to be a new and rival conception of measurement. The conflict over these different models has been highlighted recently by the Popham-Ebel debate at the 1978 AERA meeting (Ebel, 1978; Popham, 1978), which Mason uses as the point of departure for his paper. The second development upon which Mason bases much of his case is the now well-accepted distinction between formative and summative evaluation (Scriven, 1967). As Mason clearly points out, these developments are not independent, and for those of us who teach educational measurement and the many more who construct measures of learning outcomes, an integration of these two developments is welcome. The inclusion of a discussion of item type in this integration is doubly welcome, for, if successful, the synthesis of these three areas of measurement could provide a sort of conceptual roadmap for the construction and rationalization of measurement devices. Unfortunately, I think Mason's synthesis breaks down at a number of points.

Criterion-Referenced Measurement: Advantages and Misconceptions

Criterion-Referenced Measurement: Advantages and Misconceptions Get access Leon J. Gross, Ph.D. Leon J. Gross, Ph.D. Assistant Director for Research and Measurement on the Board of Registry staff, Chicago, Illinois. Search for other works by this author on: Oxford Academic PubMed Google Scholar Laboratory Medicine, Volume 9, Issue 11, 1 November 1978, Pages 21–22, https://doi.org/10.1093/labmed/9.11.21 Published: 01 November 1978

A Three-Year Auditory Perception Program for Low-Ses Minority Children

This study evaluates the effectiveness of three instructional methods in promoting speech sound discrimination and reading achievement over a 3-year period. Subjects were black and Mexican-American lower-SES children identified as having auditory discrimination disabilities. Norm-referenced and criterion-referenced measures were administered at the ends of grades one, two, and three to identify changes in auditory perception and reading achievement. One instructional group received a visual-linguistic reading method plus auditory perception activities (VLA); a second group received a visual-linguistic reading method (VL); and a third group received a phonic reading method (P). The hypothesis that VLA would score higher than (&gt;) VL = P in auditory perception was not confirmed consistently. The hypothesis that VLA &gt; VL &gt; P in reading also was not confirmed consistently. No consistent differences were noted for black or Mexican-Americans or for males or females on measures of auditory perception or reading achievement.

Competency verification in the health professions via limited focus measurement.

As American health professionals attempt to create assessment mechanisms to verify the competencies of practitioners, they are discovering that traditional norm-refer enced measurement strategtes are inappro priate. However , in their subsequent adoption of criterion-referenced measure ment strategies they have learned that these highly descriptive measures more clearly reveal not only what is being measured, but also what is not being measured. To create reliable assessment procedures for competency verification, a limited focus conception of measurement is proposed wherein manageable numbers of high-import competencies are validly assessed.

A Comparison of Piagetian Theory and Criterion-referenced Measurement

Two methods of assessing human mental functioning that are currently in vogue in education are criterion-referenced measurement (CRM) and the approach of Piaget. Because of their respective origins and orientations, they could be interpreted as being mutually exclusive; yet despite the differences between them, each approach can, in fact, be considered complementary to the other. Beginning with a review of both perspectives, this paper illustrates their highly comparable natures and indicates how they could be used together in the more adequate assessment of the competencies of individuals.

Aspects of Criterion-Referenced Measurement in Performance Evaluation

Four measurement models for use in performance evaluation (norm, criterion, domain, and objectives-referenced measurement) are presented and the contingent relationships among them are discussed. An argument is made for the use of criterion-referenced measurement in performance testing. Literature on two major problem areas in criterion-referenced measurement, reliability and validity, is briefly reviewed; and a recent example of criterion-referenced performance test development in an applied training context is described.

Book Review: Applying Norm-Referenced and Criterion-Referenced Measurement in Education

Book Review: Applying Norm-Referenced and Criterion-Referenced Measurement in Education

Criterion-Referenced Testing and Measurement: A Review of Technical Issues and Developments

Criterion-Referenced Testing and Measurement: A Review of Technical Issues and Developments

The Criterion Referenced Measurement of an Early Reading Behavior

INVESTIGATES SOME SPECIFIC PROBLEMS in constructing, analyzing and interpreting criterion referenced tests in the area of early reading behaviors. These problems arise from the apparent irreconcilability of 1) a measurement technique that demands precise fractionation of reading behaviors with 2) reading behaviors that resist such fractionation. In order to investigate these problems, a criterion referenced test of students' ability to pronounce consonant-vowelconsonant trigrams (CVC's) was constructed and administered to 144 subjects in grades 2, 3, and 4. The results were analyzed to yield conclusions and implications for further research in statistical aspects of criterion referenced test development (that is, in item generation and analysis); and the performances of the subjects were analyzed to yield information on the interpretation and implications from a reading perspective (that is, differences between good and poor readers, dimensions of the CVC universe, etc.).

The Predictive Validity and Reliability of the Basic School Skills Inventory

The present study is an investigation of the predictive validity and test-retest reliability of a new readiness instrument, the Basic School Skills Inventory (BSSI). Scores obtained by 90 children in kindergarten on the BSSI were used to predict later teacher ratings of their performance in first grade. Test-retest reliability was estimated using a method developed for criterion-referenced measures, and sex and ethnic differences in test scores were examined. It was concluded that the BSSI total score provided an effective and highly reliable means of predicting teachers' ratings of first-grade performance.

A critical note on Harvey's ‘Some thoughts on norm-referenced and criterion-referenced measures’

A critical note on Harvey's ‘Some thoughts on norm-referenced and criterion-referenced measures’

Systematic procedure for determining criterion scores

The present study describes a systematic method for determining a criterion score in criterion-referenced measurement. Reliable estimates were obtained by using only three judges who were subject matter experts. This demonstrates that the method can be applied to the test performance levels of individuals who are required to meet the objectives for a subject matter area. The present study showed that reliable judgments could be made even on subject matter of an advanced nature.

Criterion-Referenced Measurement (CRM) in the Initial Acquisition of a Psychomotor Skill with Exceptional Children

Criterion-Referenced Measurement (CRM) in the Initial Acquisition of a Psychomotor Skill with Exceptional Children

Some Thoughts on Norm-Referenced and Criterion-Referenced Measures

Some Thoughts on Norm-Referenced and Criterion-Referenced Measures

Formative and non-formative evaluation

Formative and non-formative evaluation are examined from the viewpoints of adaptation to improve individual performance and maintenance to preserve an existing system. Distinctions are made in terms of the experimental method, criterion-referenced measures, accessible events, and individual behavior. The functions of other distinguishing features are also examined, such as the quantification of weak and indirect variables for active maintenance. A case is made for the advantages and disadvantages of both types of evaluations. An increased role for formative evaluation is advocated as well as a deliberate rather than an “accidental” use of non-formative evaluation.