Large-scale Standardized Tests Research Articles

The Effects of Teacher-Centered and Student-Centered Approaches in TOEIC Reading Instruction

Teacher-centered instruction is prevalent in preparation courses for large-scale standardized language tests. Student-centered instruction, while known to be effective in enhancing communicative competence, has been considered ineffective and time-consuming to implement in preparation courses for large-scale standardized language tests. To investigate this assumption, a quasi-experimental study was conducted to investigate the effectiveness of two teaching approaches— teacher-centered and Jigsaw—on students’ Test of English for International Communication (TOEIC) Reading Comprehension Part Seven scores. The results showed that both groups experienced significant improvements from the pre-test to the post-test in their overall performance; however, there was no statistically significant difference between them. Furthermore, students in the Jigsaw group showed statistically significant improvements in all three passage types (single, double, and triple), while students in the teacher-centered group showed insignificant improvement in one of the three passage types (single passages). These findings suggest that both teaching approaches can effectively improve students’ TOEIC Reading Comprehension scores. The study adds to the body of research on teaching approaches in EFL TOEIC Reading Comprehension university classrooms. The findings suggest that teacher-centered and student-centered approaches can effectively improve students’ TOEIC Reading Comprehension scores.

The Paradigm Shift in Education in Big Data Era: Exploring the Intersection of Historical Assessment Frameworks and AI-Powered Assessment Methods in Education

This paper delves into the dynamic intersection between traditional assessment processes and AI-powered technology assessment models within the expansive landscape of the big data era. Tracing the origins of large-scale standardized testing, from Francis Galton's pioneering intelligence testing proposals to the subsequent development of norm-referenced testing (NRT) by Alfred Binet and Theodore Simon, the historical trajectory underscores the evolution towards authentic assessment paradigms such as computerized adaptive testing (CAT) and performance-based assessment (PBA). These shifts reflect a broader aim of equipping educators with tools to measure student learning effectively and foster continuous learning beyond the confines of the classroom. However, despite the proliferation of methods integrating big data and AI technologies in educational settings, a discernible gap persists between technological capabilities and their practical implementation in education, notably in the realm of student assessment and evaluation. This gap highlights challenges in enhancing assessment accuracy and reliability, evaluating non-cognitive skills, and delivering personalized feedback. The introduction of AI-powered technologies like ChatGPT raises ethical considerations regarding fairness, privacy, and bias, necessitating responsible and equitable deployment.This paper addresses fundamental research questions concerning the historical influences on assessment frameworks, the role of AI-powered technology in driving paradigm shifts, and the ethical considerations surrounding its application. Through a comprehensive literature review, it explores the potential of AI technology to enhance assessment accuracy and reliability while advocating for its responsible use. Despite the challenges posed by traditional assessment processes and AI-powered technology assessment models, educators continually strive to improve student learning and readiness for lifelong learning.By celebrating the invaluable contributions of educators, we can shift the narrative surrounding assessment practices towards a more positive and supportive environment, fostering ethical and innovative uses of AI-powered technology. This celebration serves as inspiration for future research and development in assessment, emphasizing the importance of building upon educators’ innovative practices to shape the future of assessment in the big data era.

Cheating Automatic Short Answer Grading with the Adversarial Usage of Adjectives and Adverbs

Automatic grading models are valued for the time and effort saved during the instruction of large student bodies. Especially with the increasing digitization of education and interest in large-scale standardized testing, the popularity of automatic grading has risen to the point where commercial solutions are widely available and used. However, for short answer formats, automatic grading is challenging due to natural language ambiguity and versatility. While automatic short answer grading models are beginning to compare to human performance on some datasets, their robustness, especially to adversarially manipulated data, is questionable. Exploitable vulnerabilities in grading models can have far-reaching consequences ranging from cheating students receiving undeserved credit to undermining automatic grading altogether—even when most predictions are valid. In this paper, we devise a black-box adversarial attack tailored to the educational short answer grading scenario to investigate the grading models’ robustness. In our attack, we insert adjectives and adverbs into natural places of incorrect student answers, fooling the model into predicting them as correct. We observed a loss of prediction accuracy between 10 and 22 percentage points using the state-of-the-art models BERT and T5. While our attack made answers appear less natural to humans in our experiments, it did not significantly increase the graders’ suspicions of cheating. Based on our experiments, we provide recommendations for utilizing automatic grading systems more safely in practice.

Public Opinion About National Large-Scale Student Assessment: A Case of NAPLAN

ABSTRACT This study examines public opinion about the National Assessment Program – Literacy and Numeracy (NAPLAN) in Australia, based on an online survey conducted with a sample of New South Wales (NSW) residents (n = 2,017). Our study participants did recognize the benefits of NAPLAN despite their opinion generally trending toward negative viewpoints of the testing program. The views about the positive aspects of NAPLAN varied across socio-demographic groups, but there were more similar views concerning the negative views of NAPLAN. As predicted by the social-cultural and institutional performance models, those living in metropolitan areas and those from high SES groups tended to possess stronger positive views. Political party preference was a strong predictor of the perceptions about NAPLAN. Overall, this study offers possible explanations for the underlying mechanisms explaining sub-group differences in attitudes toward large-scale standardized national testing.

A formatação da prova afeta o desempenho dos estudantes? Evidências do Enem (2016)

Abstract This paper analyzes the impact of the position of questions on students’ performance on the National Secondary Education Examination (Enem) in 2016. From a sample of 4,427,790 cases, we calculated the hit rate per question for the different workbooks in the Mathematics and its Technologies test. The results indicate presence of the fatigue effect on the 2016 Enem, that is, the order in which the questions are presented affects the proportion of correct answers, which is diminished as an item is presented closer to the end of the test. The exploratory evidence also suggests that the fatigue effect is manifested in students of both low and high performance. For example, the position of an item reduced the hit rate up to 18%, controlling for performance level. This paper conducts the first empirical evaluation of the fatigue effect during the Enem. The results contribute to the literature on the non-cognitive influences in evaluation, being useful to substantiate more systematic studies on the fatigue effect’s impact on large-scale standardized tests, beyond the case analyzed. At the end, we suggest measures that can mitigate this effect during the Enem.

Developmental neuroscience of cognitive control: the futurepath and layout

<p indent=0mm>Cognitive control is the top-down process by which individuals store, plan, and manipulate relevant information through the deployment of resources during information processing. Cognitive control encompasses different research perspectives such as selective attention, conflict resolution, working memory, cognitive flexibility, and inhibitory control, each of which has its own unique lifelong developmental characteristics. It is of great significance to combine the psychophysical and neuroimaging techniques, in order to elucidate the developmental mechanism of cognitive control, investigate the delicate balance and interaction between cognitive control and other psychological functions, and explore the plasticity and training of cognitive control. This may contribute to solving the problems such as addictive behavioral disorders caused by the imbalance of cognitive control development as well as decline of cognitive control in cognitive aging. At present, large-scale standardized testing of cognitive control is still lacking. Considering that cognitive control consists of several sub-dimensions, future research should focus on the development of standardized test battery, the construction of normative databases, and the combination of existing big data analysis techniques to examine the developmental and regulatory factors of cognitive control at different levels. This review article analyzed the opportunities and challenges faced by the current research in China, and also proposed the future development plan and roadmap based on previous research on the lifelong development of cognitive control.

Investigating the Alignment Between the CELPIP-General Reading Test and the Canadian Language Benchmarks: A Content Validation Study

Seeking evidence to support content validity is essential to test validation. This is especially the case in contexts where test scores are interpreted in relation to external proficiency standards and where new test content is constantly being produced to meet test administration and security demands. In this paper, we describe a modified scale- anchoring approach to assessing the alignment between the Canadian English Language Proficiency Index Program (CELPIP) test and the Canadian Language Benchmarks (CLB), the proficiency framework to which the test scores are linked. We discuss how proficiency frameworks such as the CLB can be used to support the content validation of large-scale standardized tests through an evaluation of the alignment between the test content and the performance standards. By sharing both the positive implications and challenges of working with the CLB in high-stakes language test validation, we hope to help raise the profile of this national language framework among scholars and practitioners.

History of Testing in the United States: Higher Education

Since the founding of Harvard College, colleges and universities have used many types of examinations to serve multiple purposes. In the early days of student assessment, the process was straightforward. Each institution developed and administered its own unique examination to its own students to monitor their progress and to prospective students who applied for admission. Large-scale standardized tests emerged in the twentieth century in part to relieve the burden placed upon high schools of having to prepare students to meet the examination requirements of each institution to which a student applied. Up to that point, local communities of tutors and teachers were attempting to prepare students to succeed on each higher education institution’s unique examination. Large-scale standardized tests have enjoyed more than a century of popularity and growth, and they have helped higher education institutions to solve problems in admissions and placement, and to measure learning outcomes. Over time, they have also become controversial, especially pertaining to race and class. This article is a historical view of educational testing in U.S. higher education, linking its development with past and present societal challenges related to civil rights laws, prominent higher education policies, and the long struggle of African American people in the United States.

From mythical ‘standard’ to standard reality: The need for alternatives to standardized English language tests

This position paper builds on three of our earlier publications on the same subject (Jenkins &amp; Leung 2014; Leung, Lewkowicz &amp; Jenkins 2016; Jenkins &amp; Leung 2017), as well as a number of conference papers we have given both jointly and individually. However, what we have not done up to this point is to propose alternatives to the large-scale standardized English tests administered by the major international examination boards, of which we have been so critical, despite the fact that we have been discussing other possibilities among ourselves for several years. The opportunity to publish a position paper on English language assessment therefore provided an ideal opportunity to present our alternatives, and this we do in the final part of the paper.

The impact of national standardized literacy and numeracy testing on children and teaching staff in remote Australian Indigenous communities

All educational testing is intended to have consequences, which are assumed to be beneficial, but tests may also have unintended, negative consequences (Messick, 1989). The issue is particularly important in the case of large-scale standardized tests, such as Australia’s National Assessment Program - Literacy and Numeracy (NAPLAN), the intended benefits of which are increased accountability and improved educational outcomes. The NAPLAN purpose is comparable to that of other state and national ‘core skills’ testing programs, which evaluate cross-sections of populations in order to compare results between population sub-groupings. Such comparisons underpin ‘accountability’ in the era of population-level testing. This study investigates the impact of NAPLAN testing on one population grouping that is prominent in the NAPLAN results’ comparisons and public reporting: children in remote Indigenous communities. A series of interviews with principals and teachers documents informants’ first-hand experiences of the use and effects of NAPLAN in schools. In the views of most participants, the language and content of the test instruments, the nature of the test engagement, and the test washback have negative impacts on students and staff, with little benefit in terms of the usefulness of the test data. The primary issue is the fact that meaningful participation in the tests depends critically on proficiency in Standard Australian English (SAE) as a first language. This study contributes to the broader discussion of how reform-targeted standardized testing for national populations affects sub-groups who are not treated equitably by the test instrument or reporting for accountability purposes. It highlights a conflict between consequential validity and the notion of accountability that drives reform-targeted testing.

The Study of Content Validation on CET Writing Section After Reform in China

The College English Test (CET) hosted by China Ministry of Education in 1987 is a large-scale standardized test, divided into CET-4 and CET-6. There are two times one year. CET-4 is for the basic requirement, while CET-6 is for higher requirement. CET is the largest exam of English in China. Since the 1980s, more and more people have focused on it, for it has a profound influence on English teaching. In order to test students’ English level, improve the quality of English teaching and meet the requirement of social development, CET conducted two reforms in 2006 and 2012 respectively. The present paper based on the college English test from 2012 to 2015 has chosen 42 sets of writing paper as the research object, and studied content validation of the writing section after reform from two aspects, design of test and content of test, aiming at examining the quality of writing part of the thesis. The results show that genre tends to be diversified; the topics are rich; the principles of openness and authenticity have been adopted appropriately; the rubrics of the writing items contain adequate information. The results also show that the subject matter of the CET writing covers a wide range, which has grasped the principle of authenticity well; openness is moderate; prompted information is sufficient; forms of prompted information are no longer model. However, genre of CET writing questions is limited to argumentation, and interaction is less. Based on these results, this paper puts forward some suggestions for future improvement in design of CET writing test items and for English writing teaching.

The Use of Subscores in Higher Education: When Is This Useful?

Assessment in higher education is challenging because teachers face more students, with less contact time as compared to primary and secondary education. Therefore, teachers and management are often interested in efficient ways of giving students diagnostic feedback and providing information on the basis of subscores is one method that is often used in large-scale standardized testing. In this article we discuss some recent psychometric literature that warns against the use of subscores in addition to the use of total scores. We illustrate how the added value of subscores can be evaluated using two college exams: A multiple choice exam and a combined open-ended question and multiple choice exam; these formats are often used in higher education and represent cases in which using subscores may be informative. We discuss the implications of our findings for future classroom evaluation.

Research on the Results of CET4/6 by Using the Fuzzy Forecast Method

As a kind of large-scale standardized test, the College English Test Band Four and Six (CET 4/6) can evaluate the English level of the students through its detailed test content. How to calculate and estimate the relationship between the pass rate of CET 4/6 and the English level of the students has been long concerned by the teaching staff and related teaching administrative personnel. This paper will analyze the grades of CET 4/6 over time and explain the scores, and then utilize the fuzzy theory to do the multiple objective fuzzy assessment and combine the experts' experience, so as to build evaluation mathematical model of pass rate of CET 4/6 and the English level of the students. Through the forecast of the CET 4/6 grades and the analysis of the English skills of the students, this paper can validate the accuracy of the model and analyze the advantages and disadvantages of the CET 4/6.

The Membrane-Computing-Based Automatic Online Marking-Management Model and its Application

The online marking system, which can improve the marking speed and quality, is widely applied in large-scale standardized tests. The P system had great parallelism is applied in the engineering systems just beginning. In this paper, an automatic online marking-management model (AOMMM) based on membrane computing is constructed, which can find marker had quality problems and remind the markers and the leader automatically. According to the basal rules of membrane computing, the management rules of the proposed AOMMM are designed. Through the analysis of the computational time complexity of the proposed model, we find that only three steps (O (1)) are needed to complete the feedback operation. Experimental result shows that the proposed model is appropriate for online marking management. With benefiting from the advantage of membrane system, the proposed OMMAM implements automatic and real time feedback operation.Key-Words: -standardized tests; online marking; error; control; membrane computing; P system; management automation; membrane system

The Common Goals and Elusive Solutions for Providing Access to Students with Visual Impairments to Standardized Testing

Horace Mann, an early American educational reformer, considered public education essential for democracy. In 1975, P.L. 94142, the Education of All Handicapped Children Act, expanded the rights to a free appropriate public education for children with disabilities, and the Individuals with Disabilities Education Act (IDEA) further refined those rights. In 2001, No Child Left Behind, the most recent reauthorization of the Elementary and Secondary Education Act (ESEA), was intended to improve education for all students. Every child, regardless of their circumstances, would achieve to high standards. If achievement was mandated, then how could schools demonstrate that all students were leaving school for college and careers? Assessment took on new significance as outcomes became ever more important for students, their teachers, and their schools. Yet major differences between schools in funding, community support, curriculum, and achievement still existed. Not all schools produced the same outcomes. Not all children with disabilities had access to a rigorous curriculum and highly qualified teachers. U.S. states began working to standardize the content of educational coursework. By doing so, a diploma would represent the same high standards across states and would emphasize necessary skills, particularly in reading and math. The U.S. Department of Education explained the intent was to close achievement gaps, promote rigorous accountability, and ensure that all students are on track to graduate, college and career ready (Department of Education, 2014). Enter the common core curriculum (which has been adopted by 43 states as of this writing) and on its heels a need for large-scale standardized testing. Twenty states have adopted tests from the Smarter Balanced Assessment Consortium (2014); 12 states and the District of Columbia have adopted tests created by the Partnership for Assessment of Readiness for College and Careers (PAARC, 2014). Both companies are developing tests that will be taken and scored using computers. The past school year (2013-2014) was considered a practice year for these evaluations, an opportunity to test the online assessments and work out any issues. The 2014-2015 school year will be the first year in which these new testing systems will be fully implemented. The computerized tests have embedded and non-embedded supports (calculators on the screen, for example, or enlarged print). There are tools, supports, and in the Smarter Balanced platform. These supports are designed to provide universal accessibility at levels needed by students as designated in their Individualized Education Programs or 504 plans. PAARC provides both embedded and external accommodations. Although there were numerous problems during the trial phase, many of the issues that were identified have been reported and are being addressed. The biggest issues affected braille, readers who needed to have access to refreshable braille displays and tactile graphics printers or needed hardcopy braille with tactile graphics in order to take these tests. At this time, neither company appears to be fully prepared to facilitate access to braille readers in a smooth and coordinated fashion, and these accommodations may encounter additional obstacles in individual school settings. …

A Mixture Cure-Rate Model for Responses and Response Times in Time-Limit Tests.

Many large-scale standardized tests are intended to measure skills related to ability rather than the rate at which examinees can work. Time limits imposed on these tests make it difficult to distinguish between the effect of low proficiency and the effect of lack of time. This paper proposes a mixture cure-rate model approach to address this issue. Maximum likelihood estimation is proposed for parameter and variance estimation for three cases: when examinee parameters are to be estimated given precalibrated item parameters, when item parameters are to be calibrated given known examinee parameters, and when item parameters are to be estimated without assuming known examinee parameters. Large-sample properties are established for the cases under suitable regularity conditions. Simulation studies suggest that the proposed approach is appropriate for inferences concerning model parameters. In addition, not distinguishing between the effect of low proficiency and the effect of lack of time is shown to have considerable consequences for parameter estimation. A real data example is presented to demonstrate the new model. Choice of survival models for the latent power times is also discussed.

State Test Results Are Predictable

Out-of-school, community demographic and family-level variables have an important influence on student achievement as measured by large-scale standardized tests. Studies described here demonstrated that about half of the test score is accounted for by variables outside the control of teachers and school administrators. The results from these studies raise serious questions about the validity of state test results to judge student learning and the quality of teaching and leadership.

Using HLM to Explore the Effects of Perceptions of Learning Environments and Assessments on Students’ Test Performance

The study examines the role that perceptions or impressions of learning environments and assessments play in students’ performance on a large-scale standardized test. Hierarchical linear modeling (HLM) was used to test aspects of the Learning Errors and Formative Feedback model to determine how much variation in students’ performance was explained by students’ and school principals’ perceptions of learning environments and assessments. Results from sequential HLM testing indicated that students’ but not principals’ perceptions explained a significant, although modest, amount of the total variation in students’ test performance. These results suggest that when students perceive learning environments to be safe and valuable, and positive assessment activities to be taking place, they tend to perform better on standardized tests than when they perceive learning environments and assessment activities otherwise. These findings provide a rationale for investigating the variables that can help improve students’ perceptions in order to enhance their test performance.

Limited aspects of reality: Frames of reference in language assessment

Language testers operate within two frames of reference: norm-referenced (NRT) and criterion-referenced testing (CRT). The former underpins the world of large-scale standardized testing that prioritizes variability and comparison. The latter supports substantive score meaning in formative and domain specific assessment. Some claim that the criterion-referenced enterprise is dead, save its legacy in score reporting (Davidson, 2012, p. 198). We argue that announcing the demise of CRT is premature. But we do acknowledge that what now passes as CRT is in fact not criterion-referenced, but is based upon a corruption of the original meaning of “criterion” as domain-specific performance. This distortion took place when NRT co-opted the term “standard” to serve as a rationale for the measurement enterprise of establishing cut-scores to retrofit NR tests with meaning derived from external scales. The true heirs of the CRT movement are researchers who base test design in the careful analysis of construct and content in domain specific communication.

Sequential Effects in Essay Ratings

Contrary to previous research on sequential ratings of student performance, this study found that professional essay raters of a large-scale standardized testing program produced ratings that were drawn toward previous ratings, creating an assimilation effect. Longer intervals between the two adjacent ratings and higher degree of agreement with other raters were associated with smaller assimilation effects. The effect was also found to create a small bias in favor of low-achieving groups of examinees.