Large-scale Assessment Studies Research Articles

Evaluating German PISA stratification designs: a simulation study

Stratification is an important design feature of many studies using complex sampling designs and it is often used in large-scale assessment (LSA) studies, such as the Programme for International Student Assessment (PISA), for two main reasons. First, stratification variables that achieve a high between and low within strata variance can improve the efficiency of a survey design. Second, stratification allows one to, explicitly or implicitly, control for sample sizes across subpopulations. It ensures that some parts of a population are in the sample in predetermined proportions. In this study, we determine through simulation which stratification scheme is best for PISA in Germany. For this, we consider the constraints imposed by the international sampling design, the available information about schools, and specific national characteristics of the German educational system. We examine seven different stratification designs selected based on scenarios used in past LSAs in Germany and theoretical considerations for future implementations. The chosen scenarios were compared with two reference scenarios: (1) an unstratified design and (2) a synthetic optimal stratification design. The simulation study reveals that the stratification design currently applied in PISA produces satisfactory results regarding sampling precision. The present stratification design is based on Germany's federal states and school types. However, this approach leads to small strata, which has been problematic for estimating sampling variance in previous cycles. Therefore, alternative stratification scenarios were considered and, in addition to overcoming the small-strata problem, also led to smaller standard errors for estimates of student mean performance in mathematics, science, and reading. As a result of this study, we recommend considering three different stratification designs for Germany in future cycles of PISA. These recommendations aim to: (1) improve the sampling efficiency while keeping the sample size constant, (2) follow a sound methodological approach, and (3) make conservative and cautious changes while maintaining a reflection of the structure of the German federal school system with different school types. These suggestions include a reinvented stratification of grouped German federal states and designs with school types as explicit stratifiers and federal states as implicit stratifiers.

The potential of following-up international large-scale assessment studies: using PISA 2018 to develop a comprehensive model of effective teaching

ABSTRACT The paper investigates the potential of using international large-scale assessment studies for conducting follow-up studies testing models of educational effectiveness. The impact of teacher factors coming from the dynamic model of educational effectiveness and the dialogic education theory on student literacy achievement is examined. Modern Greek teachers from secondary schools that participated in PISA 2018 (Accessed on https://www.oecd.org/pisa/data/pisa2018technicalreport/) in Cyprus were recruited. Two Grade 11 classes per teacher were selected, giving 392 students in total. Observation data from Greek lessons and student literacy achievement data were collected. Two multilevel regression analyses were conducted: one with the whole student sample and one with only students from whom PISA prior achievement measure was identified. When prior achievement (i.e., PISA measure) was considered, additional and larger effects of factors on student final achievement were detected. Both analyses reveal that when factors from both frameworks were considered, larger variance in student achievement was explained.

Game-Based Assessment of Students’ Digital Literacy Using Evidence-Centered Game Design

This study measured secondary students’ digital literacy using a digital game-based assessment system that was designed and developed based on the Evidence-Centered Game Design (ECGD) approach. A total of 188 secondary students constituted the valid cases in this study. Fine-grained behavioral data generated from students’ gameplay processes were collected and recorded with the assessment system. The Delphi method was used to extract feature variables related to digital literacy from the process data, and the Analytic Hierarchy Process (AHP) method was used to construct the measurement model. The assessment results of the ECGD-based assessment had a high correlation with standardized test scores, which have been shown to be reliable and valid in prior large-scale assessment studies.

An Examination of the Linking Error Currently Used in PISA

ABSTRACT Educational large-scale assessment (LSA) studies like the program for international student assessment (PISA) provide important information about trends in the performance of educational indicators in cognitive domains. The change in the country means in a cognitive domain like reading between two successive assessments is an example of a trend estimate. The uncertainty of trend estimates includes sampling and linking errors, which are regularly reported in the PISA study. This article focuses on the linking error that assesses the variability in trend estimation regarding the choice of items. Since PISA 2015, the linking error estimation method has changed. This article compares the statistical behavior and the concept of the new PISA linking error and the PISA linking error utilized until PISA 2012. It turns out that the newly proposed linking error provides no generalization of the old PISA linking error but rather reflects different aspects of uncertainties and model error.

Effects of an early start in learning English as a foreign language on reading and listening comprehension in Year 9

Abstract For years now, all over Europe a debate has been ongoing concerning the effectiveness of an early start in learning a foreign language. This paper presents findings from a representative large-scale assessment study from Germany and seeks to examine the relationship between receptive skills in English as a foreign language (EFL) of students in Year 9 and their EFL learning onset age. We examined the differences in achievement of early starters – i.e., students who started learning English either before Year 3 or in Year 3 – in comparison with late starters defined as students instructed in EFL from Year 4 or 5 using a sample of more than 30,000 students. The results from multilevel regression modelling point to long-term advantages in reading and listening comprehension for students with an early onset of English instruction in primary school compared to students who began later. The paper finishes by discussing the results and suggesting further research in this field.

Comparing different trend estimation approaches in country means and standard deviations in international large-scale assessment studies

One major aim of international large-scale assessments (ILSA) like PISA is to monitor changes in student performance over time. To accomplish this task, a set of common items (i.e., link items) is repeatedly administered in each assessment. Linking methods based on item response theory (IRT) models are used to align the results from the different assessments on a common scale. This work employs the one-parameter logistic (1PL) and the two-parameter logistic (2PL) IRT models as scaling models for dichotomous item response data. The present article discusses different types of trend estimates in country means and standard deviations for countries in ILSA. These types differ in three aspects. First, the trend can be assessed by an indirect or direct linking approach for linking a country’s performance at an international metric. Second, the linking for the trend estimation can rely on either all items or only the link items. Third, item parameters can be assumed to be invariant or noninvariant across countries. It is shown that the most often employed trend estimation methods of original trends and marginal trends can be conceived as particular cases of indirect and direct linking approaches, respectively. Through a simulation study and analytical derivations, it is demonstrated that trend estimates using a direct linking approach and those that rely on only link items outperformed alternatives for the 1PL model with uniform country differential item functioning (DIF) and the 2PL model with uniform and nonuniform country DIF. We also illustrated the performance of the different scaling models for assessing the PISA trend from PISA 2006 to PISA 2009 in the cognitive domains of reading, mathematics, and science. In this empirical application, linking errors based on jackknifing testlets were utilized that adequately quantify DIF effects in the uncertainty of trend estimates.

Regularized Mislevy-Wu Model for Handling Nonignorable Missing Item Responses

Missing item responses are frequently found in educational large-scale assessment studies. In this article, the Mislevy-Wu item response model is applied for handling nonignorable missing item responses. This model allows that the missingness of an item depends on the item itself and a further latent variable. However, with low to moderate amounts of missing item responses, model parameters for the missingness mechanism are difficult to estimate. Hence, regularized estimation using a fused ridge penalty is applied to the Mislevy-Wu model to stabilize estimation. The fused ridge penalty function is separately defined for multiple-choice and constructed response items because previous research indicated that the missingness mechanisms strongly differed for the two item types. In a simulation study, it turned out that regularized estimation improves the stability of item parameter estimation. The method is also illustrated using international data from the progress in international reading literacy study (PIRLS) 2011 data.

Studying the quality of inquiry-based teaching in science classrooms. A systematic video study of inquiry-based science teaching in primary and lower-secondary schools

ABSTRACT A large body of research has studied the role and potential of inquiry to increase the quality of teaching in science education. While much of this existing research is based on international large-scale assessment studies, we still lack a clear understanding of the factors that influence the quality of inquiry-based science teaching in actual classroom practices. In this paper, we operationalise teaching quality through an observation manual, and we drew on this manual to systematically analyse video data of instructional practices in 20 Norwegian science classrooms at the primary and lower-secondary school level (73 observed lessons and about 450 students). We identified varying quality in the individual inquiry phases and differences between primary and lower-secondary schools. We observed that primary-school students collected and documented data more systematically than lower-secondary students and that consolidations were slightly more emphasised and of higher quality at the lower-secondary than at the primary level. Moreover, our findings indicate that inquiry-based teaching gave students more freedom to make their own choices and increased the quality of student participation in the classroom. Based on our findings, we discuss how teachers can improve the quality of inquiry-based instruction and empower students in the classroom.

IQB Trends in Student Achievement 2018: A Large-Scale Educational Assessment Study in Germany

The <em>IQB Trends in Student Achievement 2018</em> is a nation-wide large-scale educational assessment using a trend design. The study includes achievement tests on mathematics, biology, chemistry, and physics as well as background questionnaires for students, their parents, subject teachers, and school principals. The datasets include data of 51.511 grade-9 students from 1.473 German schools assessed in 2018, data of 5.026 teachers, and 1.264 school principals. The 6 datasets and the study documentation are available via the Research Data Centre (FDZ) at IQB and include a wide variety of constructs such as achievement estimates, indicators of general cognitive abilities, motivational and socio-emotional constructs as well as data on social networks.

Four-Parameter Guessing Model and Related Item Response Models

Guessing effects frequently occur in testing data in educational or psychological applications. Different item response models have been proposed to handle guessing effects in dichotomous test items. However, it has been pointed out in the literature that the often employed three-parameter logistic model poses implausible assumptions regarding the guessing process. The four-parameter guessing model has been proposed as an alternative to circumvent these conceptual issues. In this article, the four-parameter guessing model is compared with alternative item response models for handling guessing effects through a simulation study and an empirical example. It turns out that model selection for item response models should be rather based on the AIC than the BIC. However, the RMSD item fit statistic used with typical cutoff values was found to be ineffective in detecting misspecified item response models. Furthermore, sufficiently large sample sizes are required for sufficiently precise item parameter estimation. Moreover, it is argued that the criterion of the statistical model fit should not be the sole criterion of model choice. The item response model used in operational practice should be valid with respect to the meaning of the ability variable and the underlying model assumptions. In this sense, the four-parameter guessing model could be the model of choice in educational large-scale assessment studies.

Exploiting the linked teaching and learning international survey and programme for international student assessment data in examining school effects: A case study of Singapore

This paper attempts to demonstrate the usefulness of the linkage data from two international large-scale assessment studies, Teaching and Learning International Survey 2013 (TALIS) 2013 and Programme for International Student Assessment (PISA) 2012, in examining the effects of schools. Data from seven educational systems are used to link, and four critical issues with five selection criteria are applied to the data selected. The linking dataset facilitates the investigation of mathematics performance while considering individual learner characteristics, mathematics teacher variables in the classroom environment and the school-level variables. We extend the new avenue of research by developing a linked database geared to the specific mathematics teaching and learning domain to reflect the school mathematics educational environment. The case study using Singapore linkage data demonstrated the feasibility and potential of exploring school effectiveness. In Singapore, schools with teachers of a higher level of education and self-efficacy in teaching mathematics related to a higher level of school mathematics performance. The study offers a guideline and inspiration to the research community to exploit the rich information in both TALIS and PISA studies to facilitate school effectiveness studies.

Some thoughts on analytical choices in the scaling model for test scores in international large-scale assessment studies

International large-scale assessments (LSAs), such as the Programme for International Student Assessment (PISA), provide essential information about the distribution of student proficiencies across a wide range of countries. The repeated assessments of the distributions of these cognitive domains offer policymakers important information for evaluating educational reforms and received considerable attention from the media. Furthermore, the analytical strategies employed in LSAs often define methodological standards for applied researchers in the field. Hence, it is vital to critically reflect on the conceptual foundations of analytical choices in LSA studies. This article discusses the methodological challenges in selecting and specifying the scaling model used to obtain proficiency estimates from the individual student responses in LSA studies. We distinguish design-based inference from model-based inference. It is argued that for the official reporting of LSA results, design-based inference should be preferred because it allows for a clear definition of the target of inference (e.g., country mean achievement) and is less sensitive to specific modeling assumptions. More specifically, we discuss five analytical choices in the specification of the scaling model: (1) specification of the functional form of item response functions, (2) the treatment of local dependencies and multidimensionality, (3) the consideration of test-taking behavior for estimating student ability, and the role of country differential items functioning (DIF) for (4) cross-country comparisons and (5) trend estimation. This article’s primary goal is to stimulate discussion about recently implemented changes and suggested refinements of the scaling models in LSA studies.

Stick model for as-built and retrofitted infilled RC frames

The Stick model is a MDOF system composed of masses lumped at the story level, connected in series by nonlinear shear link elements suitably calibrated to simulate the interstory inelastic behavior. The model has been demonstrated to predict with sufficient accuracy the dynamic response of existing masonry-infilled RC structures in terms of damage concentration and evolution under specific ground motion records. Previous works proposed interstory backbones calibrated on the response of refined MDOF finite element models representative of gravity-load designed buildings. In this work, with the aim of extending the applicability of the Stick model to different building typologies, a simplified procedure is proposed to generate interstory backbones for the Stick model simulating the behavior of existing masonry-infilled RC structures in as-built and retrofitted configurations. The procedure allows generating Stick models accounting for different levels of design (gravity load, obsolete seismic codes), age of construction, number of storeys and in-plan shape and dimensions. Relevant features characterizing existing buildings are accounted for, explicitly simulating typical failure modes of non-conforming RC elements, brittle failures induced by local frame-infill interaction phenomena, and introducing typical building-level collapse modes for existing buildings. The proposed procedure is extended to account for the possible implementation of retrofit intervention at the building level to both increase the structural capacity and the stiffness of the building. The proposed procedure can be employed to generate typological Stick models to be adopted in large-scale loss assessment studies, also evidencing the beneficial effect of possible retrofit interventions.

Exploring the Multiverse of Analytical Decisions in Scaling Educational Large-Scale Assessment Data: A Specification Curve Analysis for PISA 2018 Mathematics Data.

In educational large-scale assessment (LSA) studies such as PISA, item response theory (IRT) scaling models summarize students’ performance on cognitive test items across countries. This article investigates the impact of different factors in model specifications for the PISA 2018 mathematics study. The diverse options of the model specification also firm under the labels multiverse analysis or specification curve analysis in the social sciences. In this article, we investigate the following five factors of model specification in the PISA scaling model for obtaining the two country distribution parameters; country means and country standard deviations: (1) the choice of the functional form of the IRT model, (2) the treatment of differential item functioning at the country level, (3) the treatment of missing item responses, (4) the impact of item selection in the PISA test, and (5) the impact of test position effects. In our multiverse analysis, it turned out that model uncertainty had almost the same impact on variability in the country means as sampling errors due to the sampling of students. Model uncertainty had an even larger impact than standard errors for country standard deviations. Overall, each of the five specification factors in the multiverse analysis had at least a moderate effect on either country means or standard deviations. In the discussion section, we critically evaluate the current practice of model specification decisions in LSA studies. It is argued that we would either prefer reporting the variability in model uncertainty or choosing a particular model specification that might provide the strategy that is most valid. It is emphasized that model fit should not play a role in selecting a scaling strategy for LSA applications.

Flood risk assessment and quantification at the community and property level in the State of Iowa

Flood risk assessment contributes to identifying at-risk communities and supports mitigation decisions to maximize benefits from the investments. Large-scale risk assessments generate invaluable inputs for prioritizing regions for the distribution of limited resources. High-resolution flood maps and accurate parcel information are critical for flood risk analysis to generate reliable outcomes for planning, preparedness, and decision-making applications. Large-scale damage assessment studies in the United States often utilize the National Structure Inventory (NSI) or HAZUS default dataset, which results in inaccurate risk estimates due to the low geospatial accuracy of these datasets. On the other hand, some studies utilize higher resolution datasets, however they are limited to focus on small scales, for example, a city or a Hydrological United Code (HUC)-12 watershed. In this study, we collected extensive detailed flood maps and parcel datasets for many communities in Iowa to carry out a large-scale flood risk assessment. High-resolution flood maps and the most recent parcel information are collected to ensure the accuracy of risk products. The results indicate that the Eastern Iowa communities are prone to a higher risk of direct flood losses. Our model estimates nearly $10 million in average annualized losses, particularly in large communities in the study region. The study highlights that existing risk products based on FEMA's flood risk output underestimate the flood loss, specifically in highly populated urban communities such as Bettendorf, Cedar Falls, Davenport, Dubuque, and Waterloo. Additionally, we propose a flood risk score methodology for two spatial scales (e.g., HUC-12 watershed, property) to prioritize regions and properties for mitigation purposes. Lastly, the watershed-scale study results are shared through a web-based platform to inform the decision-makers and the public.

Copying from a model that does not fit? How the school staff survey TALIS impacted the new ECEC staff survey TALIS Starting Strong

International organisations use large-scale assessment studies to systematically monitor the educational sector. In doing so, they often strive for synergies between different studies. This article focuses on the example of TALIS Starting Strong, an international comparative OECD survey of pedagogical staff and leaders in early childhood education and care (ECEC). Developed in the wake of the OECD Teaching and Learning International Survey (TALIS), a study that targets school teachers and principals, it was conducted for the first time in 2018. This article shows which consequences the links to TALIS and the integration into OECD structures had on the genesis of TALIS Starting Strong and also discusses more concrete aspects like the adaptation of the TALIS methodology to the ECEC sector. Overall, the example shows that the alignment with a study for a different educational level has advantages and synergy effects on the one hand, but also poses specific challenges on the other. Thus, it is important to maintain a critical discourse on the structures that shape surveys and the characteristics of international large-scale surveys as a whole in order to ensure that the data collected is useful for the further development of the ECEC and higher education sectors.

On the Choice of the Item Response Model for Scaling PISA Data: Model Selection Based on Information Criteria and Quantifying Model Uncertainty.

In educational large-scale assessment studies such as PISA, item response theory (IRT) models are used to summarize students’ performance on cognitive test items across countries. In this article, the impact of the choice of the IRT model on the distribution parameters of countries (i.e., mean, standard deviation, percentiles) is investigated. Eleven different IRT models are compared using information criteria. Moreover, model uncertainty is quantified by estimating model error, which can be compared with the sampling error associated with the sampling of students. The PISA 2009 dataset for the cognitive domains mathematics, reading, and science is used as an example of the choice of the IRT model. It turned out that the three-parameter logistic IRT model with residual heterogeneity and a three-parameter IRT model with a quadratic effect of the ability provided the best model fit. Furthermore, model uncertainty was relatively small compared to sampling error regarding country means in most cases but was substantial for country standard deviations and percentiles. Consequently, it can be argued that model error should be included in the statistical inference of educational large-scale assessment studies.

The role of ICT for school leadership and management activities: an international comparison

PurposeInformation and communication technologies (ICT) has an increasing impact on schools. School leaders play a key role in this context as drivers of innovation including those related to ICT. Against this background, the study presented in this article focuses on school leadership and management activities with ICT and related challenges. It sought to analyze how frequently German school principals use ICT compared to principals in other countries, what distinct clusters of German principals could be identified in terms of ICT usage and how principals viewed ICT in schools and related challenges.Design/methodology/approachA mixed-methods approach was chosen, using quantitative data from both the international comparative large-scale assessment study ICILS 2018 and the explorative qualitative data from Germany. For the international comparison, the school principal data sets of the 12 international participants of the International Computer and Information Literacy Study (ICILS) 2018 were taken into account: Chile, Denmark, Finland, France, Germany, Italy, Kazakhstan, Republic of Korea, Luxembourg, Portugal, Uruguay and the United States. To look beyond averaged frequencies, a latent class analysis (LCA) was conducted to identify possible clusters of school leaders with distinct usage patterns of ICT for leadership and management activities.FindingsThe results indicate that, in general, German principals use ICT for leadership and management activities on a similar level as their international colleagues. However, they seem to communicate with education authorities significantly more often than their international colleagues, whereas representative activities (presentations, home page) are rather infrequent. The qualitative data point to significant barriers to fully harnessing the potential of using ICT for leadership, management and school improvement such as lack of competencies and lack of adequate support.Originality/valueTo the best of the authors’ knowledge, this is the first study that focuses on school leadership and management activities using ICT with such a data set. The results provide insights into how German principals use ICT to lead and manage their schools compared to their international counterparts. The qualitative data offers additional insights into possible reasons hindering a more effective use of ICT.

Inequalities in Albanian Education: Evidence from Large-Scale Assessment Studies

Inequalities in Albanian Education: Evidence from Large-Scale Assessment Studies

When barriers are not an issue: Tracing the relationship between hindering factors and technology use in secondary schools across Europe

Many researchers have investigated how barriers to technology integration affect the use of digital technologies in teaching and learning. However, the results have varied across educational contexts and countries. Large-scale assessment studies have described barriers only on a descriptive level instead of analyzing the effects of barriers on actual indicators of technology integration, such as technology use. Therefore, this study investigated the effects of barriers on technology use through the lens of the “will, skill, tool” model (WST model) in different European countries while taking the countries’ technological development level into account. A regression analysis showed that barriers had only a minor impact on the frequency of technology use in the classroom in the large majority of countries. In accordance with theoretical expectations, we found country-specific patterns, with a higher negative impact of technological barriers in less technologically developed countries and teacher-belief related barriers prevalent in developed countries. These findings may help policy makers identify needed interventions in different contexts.