Conduct Validation Studies Research Articles

Assessing the Validity of Higher Education Indicators

The U.S. Department of Education, the National Science Foundation (NSF), the Council of Chief State School Officer, and nearly all individual states have efforts under way to create educational indicators. Federal agencies view indicators as essential for monitoring the status of the nation's educational system and for tracking how it changes over time. At the state level, policy makers hope that indicators will provide information they can use to hold local districts and schools accountable for their performance. State policy makers also seek indicator data that can inform new improvement efforts. At the heart of each of these national and state efforts is a fundamental question: What indicators will be most valid and useful [22]? Introduction Indicators come in many forms, but all are descriptive data (for example, frequencies, percentages, ranks, means, rates, ratings) serving as proxies for conditions in education that policy makers wish to monitor. While imprecise and not without flaws, they were powerful numbers, capable of triggering important decisions relating to resource allocation, student admissions, staffing, curriculum, and other matters. What began in 1867 as simple record-keeping of enrollment statistics by the federal government [20] has since become big business [24], as evidenced by the multitude of studies and projects begin conducted. Today's activity results in part from twenty-five years of growing concern for student performance and the effectiveness of our schools. This study was prompted by the author's concern for the rigor with which indicators are developed or selected and for the validity of certain indicators that are being used today to evaluate institutions of higher education. This concern stems from the observation that indicators have the potential to be both technically inaccurate (for example, the numbers are calculated incorrectly or contain various sources of error) and conceptually invalid (that is, they are poor proxies for the concepts or conditions in education we wish to measure). The statistics we identify as indicators of quality of excellence require special justification because the inferences we draw from them extend beyond recorded facts (for example, student enrollment, cost-per-student) to constructs which are difficult to define or isolate. Additionally, many of us in the research and policy communities have, at best a fuzzy understanding about what it means to have a valid indicator. How does one go about testing the validity of an indicator or indicator system? Unlike the known standards for conducting validity studies on tests, there are no agreed-upon procedures for assuring that the indicators measure, or represent, that which we assume they measure. This lack of agreement is not limited to policy users, as de Nuefville [13] wrote in her treatise on a social indicators, but extends to developers of indicator systems, as a review of indicator literature revealed. This lack of a systematic method or conceptual framework lead to confusion as to which types of evidence might be gathered to support the interpretation and use of indicators. The same lack explains, perhaps, why virtually all indicator systems proposed or in place in education todat have no self-evaluate component. Although the practical requirements of good indicators (for example, cost effective, understood by lay audiences, feasible to collect) have been amply listed [for example, 21, 23, 26], most current developers fail to explain how their indicators should be tested or assessed other than by a largely subjective process of literature review, review by committee, and affirmation over time by the public [13, 26, 27]. The validity of current indicators might be less in question had they been developed systematically, beginning with careful definition of an underlying construct (for example, curriculum quality, ethic discrimination) and proceeding to the development and testing of standardized measures. …

Undergraduate Admission Policies, Practices, and Procedures for Applicants with Learning Disabilities

This study examined (a) the academic and nonacademic criteria used by admission personnel to determine the eligibility of undergraduate applicants with learning disabilities, (b) agreement of criteria used by institutions of varying competitiveness, and (c) the frequency with which admission personnel conduct validity studies on these criteria. A nonrandom sample consisting of 66 state universities and colleges in the Northeast was surveyed. The results suggest that the academic and nonacademic criteria employed are similar to those employed for applicants without learning disabilities, the criteria used by different types of institutions differ significantly, and admission personnel do not conduct validity studies. Further clarification and validation of admission criteria appears warranted.

Content Validity Studies of Licensing Examinations

Can measurement specialists’current ideas about content validation be implemented with licensure examinations? Does pressure of litigation facilitate or inhibit conducting validity studies?

ANALYSES OF THE PREDICTIVE VALIDITY OF THE SAT® AND HIGH SCHOOL GRADES FROM 1976 TO 1985

ABSTRACTSince 1964, colleges have been sending data to the College Board Validity Study Service in order to determine the degree to which measures used in admissions predict college performance. These studies have allowed for the monitoring of general trends in the relationship of SAT scores and high school grades with freshman grades. Beginning in the early‐1970s and continuing to the mid‐1970s, the observed average correlations of SAT scores with freshman grades increased dramatically. In the decade that followed, the strength of this relationship gradually declined. In order to determine to what extent the observed variations in the average correlations might be associated with the composition of colleges conducting validity studies each year, validity data were analyzed for the enrolling classes of 1976 to 1985. Estimates of change, based on a repeated measures estimation approach utilizing data from colleges conducting more than one predictive validity study within the 10‐year period, and correcting for the restriction of range in SAT scores and high school record, indicated a general linear trend from 1976 to 1985. The correlations of SAT scores with first‐year college grades declined from about .51 to about .47, for a change of approximately .04, There was less change during this period for private colleges, small colleges, and more selective colleges.

Evaluation of an interactive instructional television program in Hansen's disease.

The use of instructional television (ITV) is becoming increasingly common in developing continuing education programs in Hansen's disease (leprosy). This study attempted to determine if it is educationally advantageous to develop ITV material that incorporates an active learner response as compared with a conventional view-only format among a group of forty-seven professional nurses employed at a public hospital. Results indicated a clear gain in the knowledge level of those nurses who actively assessed their own progress toward preestablished instructional goals. The study also reaffirms the need to conduct validation studies of mediabased instructional material in order to determine educational effectiveness.

Fairness in Assessment for Personnel Selection

A review of several recent court cases (Gold and Bruno, 1975) where personnel selection agencies were called on to defend the job relatedness of their examination procedures suggests that psychometricians might address themselves to some of the less technical issues in testing for personnel selection and the judicial-legal system might better understand the technical limitations of the current state of the art. This paper is addressed to that objective, reviewing some of the practical and legal limitations on conducting validity studies for personnel selection procedures and suggesting an ad hoc model that may be useful in raising and resolving questions about job relatedness of tests.