Large-scale Student Assessments Research Articles

Improving Administrative Data at Scale: Experimental Evidence on Digital Testing in Indian Schools

Abstract Large-scale student assessments are a cornerstone of proposed educational reforms to improve student achievement from very low levels in low- and middle-income countries. Yet, this promise relies on their presumed reliability. I use direct audit evidence from a large Indian state (Andhra Pradesh) to show that, as currently administered, official learning assessments substantially overstate proficiency and understate the ‘learning crisis’ of low student achievement. In an experiment covering over 2,400 schools, I evaluate whether digital tablet-based testing could reduce distortion. Although paper-based assessments proctored by teachers severely exaggerate achievement, tablet-based assessments closely match independent test data and are much less likely to be flagged for cheating. Furthermore, I use the direct audit-based retest to directly validate existing (indirect) statistical procedures for detecting cheating at scale and establish that it would be feasible to monitor data integrity cheaply and at scale with such methods. Overall, these results suggest that well-designed technology-aided interventions may improve data integrity at scale, without which these learning assessments are unlikely to serve as a catalyst for policy action.

Reliability of the g factor over time in Italian INVALSI data (2010-2022): What can achievement-g tell us about the Flynn effect?

Generational intelligence test score gains over large parts of the 20th century have been observed to be negatively associated with psychometric g. Recent reports about changes in the cross-temporal IQ trajectory suggest that ability differentiation may be responsible for both changes in g as well as increasingly (sub-)domain-specific and inconsistent trajectories. Schooling is considered to be a main candidate cause for the Flynn effect, which suggests that school achievement might be expected to show similar cross-temporal developments. In the present study, we investigated evidence for cross-temporal changes in achievement-based g in a formal large-scale student assessment in Italy (i.e., the INVALSI assessment; N = 1,900,000+). Based on data of four school grades (i.e., grades 2, 5, 8, and 10) over 13 years (2010–2022), we observed little evidence for changes in achievement g in general. However, cross-temporal trajectories were differentiated according to school grade, indicating cross-temporal g decreases for lower grade students whilst changes for higher grade students were positive. These findings may be interpreted as tentative evidence for age-dependent achievement-g differentiation. The presently observed achievement g trajectory appears to be consistent with recently observed evidence for a potential stagnation or reversal of cognitive test score gains.

Public Opinion About National Large-Scale Student Assessment: A Case of NAPLAN

ABSTRACT This study examines public opinion about the National Assessment Program – Literacy and Numeracy (NAPLAN) in Australia, based on an online survey conducted with a sample of New South Wales (NSW) residents (n = 2,017). Our study participants did recognize the benefits of NAPLAN despite their opinion generally trending toward negative viewpoints of the testing program. The views about the positive aspects of NAPLAN varied across socio-demographic groups, but there were more similar views concerning the negative views of NAPLAN. As predicted by the social-cultural and institutional performance models, those living in metropolitan areas and those from high SES groups tended to possess stronger positive views. Political party preference was a strong predictor of the perceptions about NAPLAN. Overall, this study offers possible explanations for the underlying mechanisms explaining sub-group differences in attitudes toward large-scale standardized national testing.

What do test scores really capture? Evidence from a large-scale student assessment in Mexico

• This paper studies student test scores in a national assessment in Mexico. • Grade 6 test scores predict enrollment and test scores in grades 9 and 12. • Results are robust to use twin fixed-effects to control for family characteristics. • Results support the use of large-scale student assessments in developing countries. This paper studies the relationship between test scores and cognitive skills using two longitudinal data sets that track student performance in a national standardized exam in grades 6, 9, and 12 and post-secondary school outcomes in Mexico. Exploiting a large sample of twins to control for all between-family differences in school, household, and neighborhood inputs, we find that primary school test scores are a strong predictor of secondary education outcomes. Using a data set that links results in the national standardized test to later outcomes, we find that secondary school test scores predict university enrollment and hourly wages. These results indicate that, despite their limitations, large-scale student assessments can capture the skills they are meant to measure and can therefore be used to monitor student learning in developing countries.

Chemoentrepreneurship Based Student Worksheet Development to Improve Enterprise Interest in Chemical Teachers

The purpose of this study was the development of a chemoentrepreneurship based student worksheet (LKM) to increase the entrepreneurial interest in chemistry teacher candidates. The sample of this study was 115 students of the Department of Chemistry at the University of Riau and the State Islamic University (UIN) Sultan Syarif Kasim in the 6th semester of the 2019 academic year. The results obtained at the validation stage of MFI development were 95% with the valid category. The results of the small-scale test on the one-one test obtained an average percentage of 84.80% which was included in the very practical category. Furthermore, the lecturer response obtained an average percentage of assessment of 90.40% with the LKM category being well used in the learning process. The results of the small-scale trial assessment obtained an average percentage of the assessment of 85.16% which falls into the very practical category. The results of large-scale student assessments after using the LKM developed through filling out the student response questionnaires obtained an average score percentage of 80.63% with the practical category. The results of the increase in the entrepreneurial interest of UR students as a whole were 80.61, including the high category, and the results of the UIN students' interest in entrepreneurship as a whole were 81.64 including the very high category.

Externalisation and structural coupling: Applications in comparative policy studies in education

The article presents two key concepts of sociological systems theory – externalisation and structural coupling – and applies them to explain (a) the exponential growth of international large-scale student assessments and (b) the rise of ‘policy-relevant’ educational research. The author concludes with a comparison between key concepts used in systems theory and those used in comparative policy studies. She identifies resemblances with concepts of pathways in historical institutionalism, the multiple-streams approach and the notion of punctuated equilibrium advanced in the advocacy coalition framework.

Assessments and accountability in secondary education: International trends

International Large-scale Student Assessments are the most prominent example of internationalization processes in education. A number of studies have analysed the policy reactions to such studies, particularly to the Organisation for Economic Co-operation and Development (OECD) Programme for International Student Assessment (PISA) study. PISA and comparable projects have also raised concerns of a feared increase of assessments and accountability procedures. So far, systematic empirical evidence that could corroborate such concerns has been scarce. In this paper, we examine the prevalence of assessment and accountability practices at secondary education level as well as changes in these practices over time. We explicitly focus on changes over time by drawing on data from PISA 2000 to PISA 2015. Analyses over time are not straightforward with PISA, as the questionnaires change between survey rounds. This leads to different coverage of specific indicators over time. We present descriptive analyses for 20 OECD countries. The results show an increasing trend for the vast majority of the generated indicators, indicating that assessments and the use of assessments for purposes of accountability increased within the larger part of the OECD during the last 15 years. Likewise, more horizontal, peer-oriented evaluation procedures focusing on organizational learning gained importance. A cluster analysis based on selected indicators of 2015 revealed that there are four distinct groups of countries, which are mainly distinguished by different levels of the prevalence of assessment, accountability and evaluation practices.

Observing local dynamics of ILSA projections in federal systems: a comparison between Germany and the United States

ABSTRACTBy comparing two federal education systems, namely Germany and the U.S., and their reactions to PISA we show how international, large-scale student assessments (ILSA) have been used by national stakeholders to gain leverage for legitimising or de-legitimising policy reforms in education. From a neo-institutionalist perspective we argue that country-specific path-dependencies and policy legacies, such as different systems of power devolution, testing traditions and also non-governmental actor influence, additionally moderate the impact of ILSA.

정의적 성취특성 측정의 오차요인 및 성취수준별 척도의 동등성 분석1)

Before using the results of large-scale student assessments, it is essential to assure that the results are consistent and accurate regardless of various background variables, particularly for the proficiency level. In this context, the purpose of this study was to analyze the measurement equivalence of the non-cognitive ability data across students’ groups by the proficiency level based on the generalizability theory approach. This study analyzed student survey data (N=7443) from the 2015 administration of National Assessment of Educational Achievement for ninth graders and applied the multivariate generalizability theory model to the data. In the G study, effects of each error source were investigated, while the focus of the D study was on identifying optimal measurement conditions by varying the number of items within each sub-domain that maximizes the reliability-like indices, such as a generalizability coefficient and an index of dependability. The results of the analysis indicates that the overall measurement was generally reliable over the entire range of the achievement levels; however, the study showed a somewhat negative result in ensuring the measurement equivalence for certain sub-domains of mathematics. In addition, the effect of increasing the number of items to achieve appropriate level of reliability also varied according to achievement level. Based on the results of this study, it was confirmed that a multivariate generalizability analysis approach can be useful for investigating the measurement invariance issues.

A Brief Scale on Attitude Toward Learning of Scientific Subjects (ATLoSS) for Middle School Students

This paper aims at developing and validating a brief scale specifically assessing Attitude Toward Learning of Scientific Subjects (ATLoSS) and at testing its measurement invariance across genders. The scale was validated on a sample of 18,436 Italian students (51% males) attending middle schools at 7th grade by using both exploratory factor analysis and confirmatory factor analysis. The four item scale had good internal consistency with a Cronbach’s alpha of .864 and a one-factor solution characterized by measurement invariance across genders. Associations with other measures – in particular with math academic self-concept and interest/enjoyment in math – supported convergent and divergent validity of the scale; besides, good criterion validity based on student achievement and prospective choice of higher education was confirmed. The scale represents a brief and easy instrument which can be used in large scale student assessment and can provide teachers and educators with preliminary information on their students’ attitudes which are relevant for academic achievement.

Understanding business interests in international large-scale student assessments: a media analysis of The Economist, Financial Times, and Wall Street Journal

The media analysis is situated in the larger body of studies that explore the varied reasons why different policy actors advocate for international large-scale student assessments (ILSAs) and adds to the research on the fast advance of the global education industry. The analysis of The Economist, Financial Times, and Wall Street Journal covers publications on ‘PISA’, ‘TIMSS’, and related search items over the period 1996–2016. The three media outlets vary in terms of ILSA reporting. The Economist and Financial Times tend to focus on PISA, whereas the Wall Street Journal pays greater attention to TIMSS than PISA. The content analysis of 59 articles yields interesting results about how the business-oriented readership of the three media outlets frames public education and why it sees education as a profitable business opportunity. The three most common narratives, reflecting the business logic, are the following: (i) public education is in crisis; (ii) there is no correlation between spending and education outcome; and (iii) school accountability, teacher performance, and decentralisation represent the most effective policies to improve the quality of education. Drawing on these three common narratives, the financial media outlets present a particular vision of how to improve education; a vision in which the private sector is supposed to play a major role.

League tables in educational evidence-based policy-making: can we stop the horse race, please?

ABSTRACTInternational large-scale student assessments (ILSAs) in education represent a valuable source of information for policy-makers, not only on student achievements, but also on their relationship with different contextual factors. The results are partly described in the official studies’ reports; more can be derived from the publicly released data sets. However, league tables are often the only evidence used in policy debates and decisions on education. Indeed, the comparison of student achievement across the participating educational systems is a legitimate proxy for estimating countries’ development and productivity, but the use of league tables more often turns into ‘horse-ranking’, ignoring the contexts of teaching and learning. This is often supported by the media, turning the use of results into their abuse. The purpose of this paper is to discuss the use and misuse of league tables in reporting ILSA results, vs. the use of data for in-depth analysis in order to make informed decisions.

Students’ skipping behavior on truancy items and (school) subjects and its relation to test performance in PISA 2012

Most large-scale student assessments focus on cognitive competences which are assumed to be acquired at school. This study investigates how skipping lessons on purpose is related to student achievement in the PISA 2012 competence test (mathematics, reading and science). First, a Rasch-residual factor analysis is carried out on the truancy scale in order to discover the dimensionality of truancy as a latent construct. Second, the PISA proficiency score in each of the three competence domains is predicted using a set of control variables plus the subject-specific truancy. The main findings are that subject-specific truancy comprises four dimensions and that skipping certain school subjects goes along with a lower proficiency in the corresponding PISA domains. This is most obvious in mathematics and science, where students who choose the most extreme response category (skipped ‘3 times and more’) score about 40 or, respectively, 38 points less than their non-truant classmates.

Reflections on the Use of Large-Scale Student Assessment for Improving Student Success [Abstract

Reflections on the Use of Large-Scale Student Assessment for Improving Student Success [Abstract

Large-Scale Student Assessment: Guidelines for Policymakers

Targeted for policy-makers, this article provides suggestions for ensuring that the benefits of using large-scale student assessments are achieved in the face of a number of challenges to their effective use. Fundamental concepts and concerns are reviewed in the context of making valid inferences from test scores. A unique feature of this article is that the author draws on his recent experiences in the educational policy arena.

Stability of School-Level Scores From Large-Scale Student Assessments

More attention is being given to evaluating the quality of school-level assessment scores due to their importance for school-based planning and monitoring effectiveness. In this study, cross-year stability is proposed as an indicator of data quality and the degree of stability that is appropriate for large-scale assessments of student performance is explored. Following a search of Internet sites, Year 1 to Year 2 stability coefficients were calculated for assessment data from 21 states and 2 provinces. The median stability coefficient was .78 in mathematics and reading, but coefficients for writing were generally lower. A stability coefficient of .80 is recommended as the standard for large-scale assessments of student performance. A high degree of cross-year stability makes it easier to detect and attribute changes in school-level scores to school improvement efforts. The link between stability and reliability and several factors that may attenuate stability are discussed.

The Use of Holistic versus Analytic Scoring for Large-Scale Assessment of Writing

Abstract: This article discusses a variety of issues associated with the use of holistic and analytic scoring methods in large-scale student assessments. The article reviews the education literature on the subject, describes the results of a recent Saskatchewan study comparing the results of the two scoring approaches, and discusses some of the study’s implications for future large-scale assessments.

Setting Achievement Standards/Expectations for Large-Scale Student Assessments

Abstract: This article discusses a variety of issues associated with establishing formal standards or expectations for student achievement on large-scale assessment programs. The article reviews the educational literature on the subject of standard-setting methods, describes two approaches to standard setting that have been used in province-wide assessments in Saskatchewan, and evaluates the effectiveness of these approaches. Although the two procedures described were generally successful, numerous issues are raised and recommendations are made to support future standard-setting activity and research.